scholarly journals The Importance of Understanding False Discoveries and the Accuracy Paradox When Evaluating Quantitative Studies

2021 ◽  
Vol 2 (2) ◽  
pp. p1
Author(s):  
Kirk Davis ◽  
Rodney Maiden
Keyword(s):  
Statistical Significance ◽  
Error Rates ◽  
Type I ◽  
False Discovery Rates ◽  
Quantitative Studies ◽  
False Discovery ◽  
Lower Accuracy ◽  
Psychology Literature ◽  
False Discoveries ◽  
Published Research

Although the limitations of null hypothesis significance testing (NHST) are well documented in the psychology literature, the accuracy paradox, which concisely states an important limitation of published research, is never mentioned. The accuracy paradox appears when a test with higher accuracy does a poorer job of correctly classifying a particular outcome than a test with lower accuracy, which suggests that a reliance on accuracy as a metric for a test’s usefulness is not always the best metric. Since accuracy is a function of type I and II error rates, it can be misleading to interpret a study’s results as accurate simply because these errors are minimized. Once a decision has been made regarding statistical significance, type I and II error rates are not directly informative to the reader. Instead, false discovery and false omission rates are more informative when evaluating the results of a study. Given the prevalence of publication bias and small effect sizes in the literature, the possibility of a false discovery is especially important to consider. When false discovery rates are estimated, it is easy to understand why many studies in psychology cannot be replicated.

scholarly journals A direct approach to estimating false discovery rates conditional on covariates

2015 ◽  
Author(s):  
Simina M. Boca ◽  
Jeffrey T. Leek
Keyword(s):  
False Discovery Rate ◽  
Meta Analysis ◽  
Error Rates ◽  
Multiple Hypothesis Testing ◽  
False Discovery Rates ◽  
False Discovery ◽  
A Genome ◽  
The Individual ◽  
False Discoveries ◽  
Discovery Rates

AbstractModern scientific studies from many diverse areas of research abound with multiple hypothesis testing concerns. The false discovery rate is one of the most commonly used error rates for measuring and controlling rates of false discoveries when performing multiple tests. Adaptive false discovery rates rely on an estimate of the proportion of null hypotheses among all the hypotheses being tested. This proportion is typically estimated once for each collection of hypotheses. Here we propose a regression framework to estimate the proportion of null hypotheses conditional on observed covariates. This may then be used as a multiplication factor with the Benjamini-Hochberg adjusted p-values, leading to a plug-in false discovery rate estimator. Our case study concerns a genome-wise association meta-analysis which considers associations with body mass index. In our framework, we are able to use the sample sizes for the individual genomic loci and the minor allele frequencies as covariates. We further evaluate our approach via a number of simulation scenarios.

scholarly journals The Functional False Discovery Rate with Applications to Genomics

2017 ◽  
Author(s):  
Xiongzhi Chen ◽  
David G. Robinson ◽  
John D. Storey
Keyword(s):  
Gene Expression ◽  
False Discovery Rate ◽  
Genetic Marker ◽  
Read Depth ◽  
False Discovery Rates ◽  
Additional Information ◽  
False Discovery ◽  
Gene Expression Trait ◽  
Genetics Of Gene Expression ◽  
False Discoveries

AbstractThe false discovery rate measures the proportion of false discoveries among a set of hypothesis tests called significant. This quantity is typically estimated based on p-values or test statistics. In some scenarios, there is additional information available that may be used to more accurately estimate the false discovery rate. We develop a new framework for formulating and estimating false discovery rates and q-values when an additional piece of information, which we call an “informative variable”, is available. For a given test, the informative variable provides information about the prior probability a null hypothesis is true or the power of that particular test. The false discovery rate is then treated as a function of this informative variable. We consider two applications in genomics. Our first is a genetics of gene expression (eQTL) experiment in yeast where every genetic marker and gene expression trait pair are tested for associations. The informative variable in this case is the distance between each genetic marker and gene. Our second application is to detect differentially expressed genes in an RNA-seq study carried out in mice. The informative variable in this study is the per-gene read depth. The framework we develop is quite general, and it should be useful in a broad range of scientific applications.

scholarly journals Considering Both Statistical and Clinical Significance

2016 ◽  
Vol 5 (5) ◽  
pp. 16 ◽  
Author(s):  
Guolong Zhao
Keyword(s):  
Clinical Significance ◽  
Null Hypothesis ◽  
Type I Error ◽  
Statistical Significance ◽  
Error Rates ◽  
Type I ◽  
Data Set ◽  
Type I Error Rates ◽  
Empirical Coverage ◽  
Superiority Trials

To evaluate a drug, statistical significance alone is insufficient and clinical significance is also necessary. This paper explains how to analyze clinical data with considering both statistical and clinical significance. The analysis is practiced by combining a confidence interval under null hypothesis with that under non-null hypothesis. The combination conveys one of the four possible results: (i) both significant, (ii) only significant in the former, (iii) only significant in the latter or (iv) neither significant. The four results constitute a quadripartite procedure. Corresponding tests are mentioned for describing Type I error rates and power. The empirical coverage is exhibited by Monte Carlo simulations. In superiority trials, the four results are interpreted as clinical superiority, statistical superiority, non-superiority and indeterminate respectively. The interpretation is opposite in inferiority trials. The combination poses a deflated Type I error rate, a decreased power and an increased sample size. The four results may helpful for a meticulous evaluation of drugs. Of these, non-superiority is another profile of equivalence and so it can also be used to interpret equivalence. This approach may prepare a convenience for interpreting discordant cases. Nevertheless, a larger data set is usually needed. An example is taken from a real trial in naturally acquired influenza.

scholarly journals A direct approach to estimating false discovery rates conditional on covariates

PeerJ ◽  
2018 ◽  
Vol 6 ◽  
pp. e6035 ◽  
Author(s):  
Simina M. Boca ◽  
Jeffrey T. Leek
Keyword(s):  
Meta Analysis ◽  
Error Rates ◽  
Multiple Hypothesis Testing ◽  
False Discovery Rates ◽  
False Discovery ◽  
Multiple Hypothesis ◽  
A Genome ◽  
Regression Framework ◽  
Novel Method ◽  
The Individual

Modern scientific studies from many diverse areas of research abound with multiple hypothesis testing concerns. The false discovery rate (FDR) is one of the most commonly used approaches for measuring and controlling error rates when performing multiple tests. Adaptive FDRs rely on an estimate of the proportion of null hypotheses among all the hypotheses being tested. This proportion is typically estimated once for each collection of hypotheses. Here, we propose a regression framework to estimate the proportion of null hypotheses conditional on observed covariates. This may then be used as a multiplication factor with the Benjamini–Hochberg adjusted p-values, leading to a plug-in FDR estimator. We apply our method to a genome-wise association meta-analysis for body mass index. In our framework, we are able to use the sample sizes for the individual genomic loci and the minor allele frequencies as covariates. We further evaluate our approach via a number of simulation scenarios. We provide an implementation of this novel method for estimating the proportion of null hypotheses in a regression framework as part of the Bioconductor package swfdr.

Directional-sum test for nonparametric Behrens-Fisher problem with applications to the dietary intervention trial

2021 ◽  
pp. 096228022110028
Author(s):  
Zhen Meng ◽  
Qinglong Yang ◽  
Qizhai Li ◽  
Baoxue Zhang
Keyword(s):  
Dietary Intervention ◽  
Type I Error ◽  
Statistical Significance ◽  
Error Rates ◽  
Type I ◽  
Simulation Studies ◽  
Intervention Trial ◽  
Combination Strategy ◽  
Type I Error Rates ◽  
Dietary Intervention Trial

For a nonparametric Behrens-Fisher problem, a directional-sum test is proposed based on division-combination strategy. A one-layer wild bootstrap procedure is given to calculate its statistical significance. We conduct simulation studies with data generated from lognormal, t and Laplace distributions to show that the proposed test can control the type I error rates properly and is more powerful than the existing rank-sum and maximum-type tests under most of the considered scenarios. Applications to the dietary intervention trial further show the performance of the proposed test.

scholarly journals Analysis of >30,000 abstracts suggests higher false discovery rates for oncology journals, especially those with low impact factors

2018 ◽  
Author(s):  
LM Hall ◽  
AE Hendricks
Keyword(s):  
Open Access ◽  
Impact Factor ◽  
False Discovery Rate ◽  
Journal Impact Factor ◽  
General Medicine ◽  
False Discovery Rates ◽  
False Discovery ◽  
Published Research ◽  
Journal Impact ◽  
Discovery Rates

AbstractBackgroundRecently, there has been increasing concern about the replicability, or lack thereof, of published research. An especially high rate of false discoveries has been reported in some areas motivating the creation of resource-intensive collaborations to estimate the replication rate of published research by repeating a large number of studies. The substantial amount of resources required by these replication projects limits the number of studies that can be repeated and consequently the generalizability of the findings.Methods and findingsIn 2013, Jager and Leek developed a method to estimate the empirical false discovery rate from journal abstracts and applied their method to five high profile journals. Here, we use the relative efficiency of Jager and Leek’s method to gather p-values from over 30,000 abstracts and to subsequently estimate the false discovery rate for 94 journals over a five-year time span. We model the empirical false discovery rate by journal subject area (cancer or general medicine), impact factor, and Open Access status. We find that the empirical false discovery rate is higher for cancer vs. general medicine journals (p = 5.14E-6). Within cancer journals, we find that this relationship is further modified by journal impact factor where a lower journal impact factor is associated with a higher empirical false discovery rates (p = 0.012, 95% CI: -0.010, -0.001). We find no significant differences, on average, in the false discovery rate for Open Access vs closed access journals (p = 0.256, 95% CI: -0.014, 0.051).ConclusionsWe find evidence of a higher false discovery rate in cancer journals compared to general medicine journals, especially those with a lower journal impact factor. For cancer journals, a lower journal impact factor of one point is associated with a 0.006 increase in the empirical false discovery rate, on average. For a false discovery rate of 0.05, this would result in over a 10% increase to 0.056. Conversely, we find no significant evidence of a higher false discovery rate, on average, for Open Access vs. closed access journals from InCites. Our results provide identify areas of research that may need of additional scrutiny and support to facilitate replicable science. Given our publicly available R code and data, others can complete a broad assessment of the empirical false discovery rate across other subject areas and characteristics of published research.

scholarly journals Modified Significance Analysis of Microarrays in Heterogeneous Diseases

2021 ◽  
Vol 11 (2) ◽  
pp. 62
Author(s):  
I-Shiang Tzeng
Keyword(s):  
Type I Error ◽  
Statistical Tests ◽  
Error Rates ◽  
Repeated Measurements ◽  
Type I ◽  
Scoring Method ◽  
Significance Analysis ◽  
False Discovery ◽  
Rate Controlled ◽  
General Statistical

Significance analysis of microarrays (SAM) provides researchers with a non-parametric score for each gene based on repeated measurements. However, it may lose certain power in general statistical tests to correctly detect differentially expressed genes (DEGs) which violate homogeneity. Monte Carlo simulation shows that the “half SAM score” can maintain type I error rates of about 0.05 based on assumptions of normal and non-normal distributions. The author found 265 DEGs using the half SAM scoring, more than the 119 DEGs detected by SAM, with the false discovery rate controlled at 0.05. In conclusion, the author recommends the half SAM scoring method to detect DEGs in data that show heterogeneity.

scholarly journals De novo detection of differentially bound regions for ChIP-seq data using peaks and windows: controlling error rates correctly

2014 ◽  
Vol 42 (11) ◽  
pp. e95-e95 ◽  
Author(s):  
Aaron T.L. Lun ◽  
Gordon K. Smyth
Keyword(s):  
False Discovery Rate ◽  
Error Control ◽  
De Novo ◽  
Hybrid Approach ◽  
Careful Consideration ◽  
Error Rates ◽  
Type I ◽  
Data Set ◽  
False Discovery ◽  
Differential Binding

Abstract A common aim in ChIP-seq experiments is to identify changes in protein binding patterns between conditions, i.e. differential binding. A number of peak- and window-based strategies have been developed to detect differential binding when the regions of interest are not known in advance. However, careful consideration of error control is needed when applying these methods. Peak-based approaches use the same data set to define peaks and to detect differential binding. Done improperly, this can result in loss of type I error control. For window-based methods, controlling the false discovery rate over all detected windows does not guarantee control across all detected regions. Misinterpreting the former as the latter can result in unexpected liberalness. Here, several solutions are presented to maintain error control for these de novo counting strategies. For peak-based methods, peak calling should be performed on pooled libraries prior to the statistical analysis. For window-based methods, a hybrid approach using Simes’ method is proposed to maintain control of the false discovery rate across regions. More generally, the relative advantages of peak- and window-based strategies are explored using a range of simulated and real data sets. Implementations of both strategies also compare favourably to existing programs for differential binding analyses.

Sign in / Sign up

Export Citation Format

Share Document