False Discovery in A/B Testing

2021 ◽  
Author(s):  
Ron Berman ◽  
Christophe Van den Bulte
Keyword(s):  
Low Power ◽  
False Discovery Rate ◽  
Decision Makers ◽  
Two Stage ◽  
Testing Platform ◽  
False Discovery ◽  
High Fraction ◽  
B Test

We investigate what fraction of all significant results in website A/B testing is actually null effects (i.e., the false discovery rate (FDR)). Our data consist of 4,964 effects from 2,766 experiments conducted on a commercial A/B testing platform. Using three different methods, we find that the FDR ranges between 28% and 37% for tests conducted at 10% significance and between 18% and 25% for tests at 5% significance (two sided). These high FDRs stem mostly from the high fraction of true null effects, about 70%, rather than from low power. Using our estimates, we also assess the potential of various A/B test designs to reduce the FDR. The two main implications are that decision makers should expect one in five interventions achieving significance at 5% confidence to be ineffective when deployed in the field and that analysts should consider using two-stage designs with multiple variations rather than basic A/B tests. This paper was accepted by Eric Anderson, marketing.

Jointly determining significance levels of primary and replication studies by controlling the false discovery rate in two-stage genome-wide association studies

2017 ◽  
Vol 27 (9) ◽  
pp. 2795-2808 ◽  
Author(s):  
Wei Jiang ◽  
Weichuan Yu
Keyword(s):  
False Discovery Rate ◽  
Association Studies ◽  
Genome Wide Association ◽  
Genome Wide Association Studies ◽  
Two Stage ◽  
Traditional Methods ◽  
Replication Studies ◽  
False Discovery ◽  
Genome Wide ◽  
Significance Levels

In genome-wide association studies, we normally discover associations between genetic variants and diseases/traits in primary studies, and validate the findings in replication studies. We consider the associations identified in both primary and replication studies as true findings. An important question under this two-stage setting is how to determine significance levels in both studies. In traditional methods, significance levels of the primary and replication studies are determined separately. We argue that the separate determination strategy reduces the power in the overall two-stage study. Therefore, we propose a novel method to determine significance levels jointly. Our method is a reanalysis method that needs summary statistics from both studies. We find the most powerful significance levels when controlling the false discovery rate in the two-stage study. To enjoy the power improvement from the joint determination method, we need to select single nucleotide polymorphisms for replication at a less stringent significance level. This is a common practice in studies designed for discovery purpose. We suggest this practice is also suitable in studies with validation purpose in order to identify more true findings. Simulation experiments show that our method can provide more power than traditional methods and that the false discovery rate is well-controlled. Empirical experiments on datasets of five diseases/traits demonstrate that our method can help identify more associations. The R-package is available at: http://bioinformatics.ust.hk/RFdr.html .

scholarly journals Sample size reassessment for a two-stage design controlling the false discovery rate

2015 ◽  
Vol 14 (5) ◽  
Author(s):  
Sonja Zehetmayer ◽  
Alexandra C. Graf ◽  
Martin Posch
Keyword(s):  
Sample Size ◽  
False Discovery Rate ◽  
Effect Sizes ◽  
Rna Seq ◽  
Two Stage ◽  
Stage Design ◽  
False Discovery ◽  
Sample Size Calculations ◽  
Two Stage Design ◽  
High Uncertainty

AbstractSample size calculations for gene expression microarray and NGS-RNA-Seq experiments are challenging because the overall power depends on unknown quantities as the proportion of true null hypotheses and the distribution of the effect sizes under the alternative. We propose a two-stage design with an adaptive interim analysis where these quantities are estimated from the interim data. The second stage sample size is chosen based on these estimates to achieve a specific overall power. The proposed procedure controls the power in all considered scenarios except for very low first stage sample sizes. The false discovery rate (FDR) is controlled despite of the data dependent choice of sample size. The two-stage design can be a useful tool to determine the sample size of high-dimensional studies if in the planning phase there is high uncertainty regarding the expected effect sizes and variability.

scholarly journals Comparison of one-stage and two-stage genome-wide association studies

2017 ◽  
Author(s):  
Shang Xue ◽  
Funda Ogut ◽  
Zachary Miller ◽  
Janu Verma ◽  
Peter J. Bradbury ◽  
...  
Keyword(s):  
False Discovery Rate ◽  
Association Studies ◽  
Experimental Designs ◽  
Genome Wide Association ◽  
Genome Wide Association Studies ◽  
Whole Genome ◽  
Two Stage ◽  
False Discovery ◽  
Genome Wide ◽  
Stage Analysis

AbstractLinear mixed models are widely used in humans, animals, and plants to conduct genome-wide association studies (GWAS). A characteristic of experimental designs for plants is that experimental units are typically multiple-plant plots of families or lines that are replicated across environments. This structure can present computational challenges to conducting a genome scan on raw (plot-level) data. Two-stage methods have been proposed to reduce the complexity and increase the computational speed of whole-genome scans. The first stage of the analysis fits raw data to a model including environment and line effects, but no individual marker effects. The second stage involves the whole genome scan of marker tests using summary values for each line as the dependent variable. Missing data and unbalanced experimental designs can result in biased estimates of marker association effects from two-stage analyses. In this study, we developed a weighted two-stage analysis to reduce bias and improve power of GWAS while maintaining the computational efficiency of two-stage analyses. Simulation based on real marker data of a diverse panel of maize inbred lines was used to compare power and false discovery rate of the new weighted two-stage method to single-stage and other two-stage analyses and to compare different two-stage models. In the case of severely unbalanced data, only the weighted two-stage GWAS has power and false discovery rate similar to the one-stage analysis. The weighted GWAS method has been implemented in the open-source software TASSEL.

Sign in / Sign up

Export Citation Format

Share Document