Supplemental Material for Meta-Analysis of Correlations Revisited: Attempted Replication and Extension of Field's (2001) Simulation Studies

Publication bias and questionable research practices in primary research can lead to badly overestimated effects in meta-analysis. Methodologists have proposed a variety of statistical approaches to correct for such overestimation. However, it is not clear which methods work best for data typically seen in psychology. Here, we present a comprehensive simulation study in which we examined how some of the most promising meta-analytic methods perform on data that might realistically be produced by research in psychology. We simulated several levels of questionable research practices, publication bias, and heterogeneity, and used study sample sizes empirically derived from the literature. Our results clearly indicated that no single meta-analytic method consistently outperformed all the others. Therefore, we recommend that meta-analysts in psychology focus on sensitivity analyses—that is, report on a variety of methods, consider the conditions under which these methods fail (as indicated by simulation studies such as ours), and then report how conclusions might change depending on which conditions are most plausible. Moreover, given the dependence of meta-analytic methods on untestable assumptions, we strongly recommend that researchers in psychology continue their efforts to improve the primary literature and conduct large-scale, preregistered replications. We provide detailed results and simulation code at https://osf.io/rf3ys and interactive figures at http://www.shinyapps.org/apps/metaExplorer/ .

Download Full-text

Controlling bias and inflation in epigenome- and transcriptome-wide association studies using the empirical null distribution

10.1101/055772 ◽

2016 ◽

Cited By ~ 1

Author(s):

Maarten van Iterson ◽

Erik van Zwet ◽

P. Eline Slagboom ◽

Bastiaan T. Heijmans ◽

Keyword(s):

Association Studies ◽

Meta Analysis ◽

False Positive Rate ◽

Null Distribution ◽

Simulation Studies ◽

Test Statistics ◽

Empirical Null Distribution ◽

Novel Approach ◽

Level Data ◽

Positive Rate

ABSTRACTAssociation studies on omic-level data other then genotypes (GWAS) are becoming increasingly common, i.e., epigenome-and transcriptome-wide association studies (EWAS/TWAS). However, a tool box for the analysis of EWAS and TWAS studies is largely lacking and often approaches from GWAS are applied despite the fact that epigenome and transcriptome data have vedifferent characteristics than genotypes. Here, we show that EWASs and TWASs are prone not only to significant inflation but also bias of the test statistics and that these are not properly addressed by GWAS-based methodology (i.e. genomic control) and state-of-the-art approaches to control for unmeasured confounding (i.e. RUV, sva and cate). We developed a novel approach that is based on the estimation of the empirical null distribution using Bayesian statistics. Using simulation studies and empirical data, we demonstrate that our approach maximizes power while properly controlling the false positive rate. Finally, we illustrate the utility of our method in the application of meta-analysis by performing EWASs and TWASs on age and smoking which highlighted an overlap in differential methylation and expression of associated genes. An implementation of our new method is available from http://bioconductor.org/packages/bacon/.

Download Full-text

A Simplification and Implementation of Random-effects Meta-analyses Based on the Exact Distribution of Cochran’s Q

Methods of Information in Medicine ◽

10.3414/me13-01-0073 ◽

2014 ◽

Vol 53 (01) ◽

pp. 54-61 ◽

Cited By ~ 6

Author(s):

M. Preuß ◽

A. Ziegler

Keyword(s):

Distribution Function ◽

Cumulative Distribution Function ◽

Type I Error ◽

Meta Analysis ◽

Real Data ◽

R Package ◽

Cumulative Distribution ◽

Data Sets ◽

Type I ◽

Simulation Studies

SummaryBackground: The random-effects (RE) model is the standard choice for meta-analysis in the presence of heterogeneity, and the stand ard RE method is the DerSimonian and Laird (DSL) approach, where the degree of heterogeneity is estimated using a moment-estimator. The DSL approach does not take into account the variability of the estimated heterogeneity variance in the estimation of Cochran’s Q. Biggerstaff and Jackson derived the exact cumulative distribution function (CDF) of Q to account for the variability of Ť 2.Objectives: The first objective is to show that the explicit numerical computation of the density function of Cochran’s Q is not required. The second objective is to develop an R package with the possibility to easily calculate the classical RE method and the new exact RE method.Methods: The novel approach was validated in extensive simulation studies. The different approaches used in the simulation studies, including the exact weights RE meta-analysis, the I 2 and T 2 estimates together with their confidence intervals were implemented in the R package metaxa.Results: The comparison with the classical DSL method showed that the exact weights RE meta-analysis kept the nominal type I error level better and that it had greater power in case of many small studies and a single large study. The Hedges RE approach had inflated type I error levels. Another advantage of the exact weights RE meta-analysis is that an exact confidence interval for T 2is readily available. The exact weights RE approach had greater power in case of few studies, while the restricted maximum likelihood (REML) approach was superior in case of a large number of studies. Differences between the exact weights RE meta-analysis and the DSL approach were observed in the re-analysis of real data sets. Application of the exact weights RE meta-analysis, REML, and the DSL approach to real data sets showed that conclusions between these methods differed.Conclusions: The simplification does not require the calculation of the density of Cochran’s Q, but only the calculation of the cumulative distribution function, while the previous approach required the computation of both the density and the cumulative distribution function. It thus reduces computation time, improves numerical stability, and reduces the approximation error in meta-analysis. The different approaches, including the exact weights RE meta-analysis, the I 2 and T 2estimates together with their confidence intervals are available in the R package metaxa, which can be used in applications.

Download Full-text

Data-generating models of dichotomous outcomes: Heterogeneity in simulation studies for a random-effects meta-analysis

Statistics in Medicine ◽

10.1002/sim.7569 ◽

2017 ◽

Vol 37 (7) ◽

pp. 1115-1124 ◽

Cited By ~ 13

Author(s):

Konstantinos Pateras ◽

Stavros Nikolakopoulos ◽

Kit Roes

Keyword(s):

Random Effects ◽

Meta Analysis ◽

Simulation Studies ◽

Dichotomous Outcomes

Download Full-text

A meta-analysis of the predicted effects of climate change on wheat yields using simulation studies

Field Crops Research ◽

10.1016/j.fcr.2013.11.008 ◽

2014 ◽

Vol 156 ◽

pp. 180-190 ◽

Cited By ~ 50

Author(s):

Julia Wilcox ◽

David Makowski

Keyword(s):

Climate Change ◽

Meta Analysis ◽

Simulation Studies

Download Full-text

Correcting for bias in psychology: A comparison of meta-analytic methods

10.31234/osf.io/9h3nu ◽

2017 ◽

Cited By ~ 7

Author(s):

Evan C Carter ◽

Felix D. Schönbrodt ◽

Will M Gervais ◽

Joseph Hilgard

Keyword(s):

Publication Bias ◽

Large Scale ◽

Meta Analysis ◽

Sensitivity Analyses ◽

Simulation Studies ◽

Research Practices ◽

Simulation Code ◽

Questionable Research Practices ◽

Analytic Methods ◽

Primary Literature

Publication bias and questionable research practices in primary research can lead to badly overestimated effects in meta-analysis. Methodologists have proposed a variety of statistical approaches to correct for such overestimation. However, much of this work has not been tailored specifically to psychology, so it is not clear which methods work best for data typically seen in our field. Here, we present a comprehensive simulation study to examine how some of the most promising meta-analytic methods perform on data that might realistically be produced by research in psychology. We created such scenarios by simulating several levels of questionable research practices, publication bias, heterogeneity, and using study sample sizes empirically derived from the literature. Our results clearly indicated that no single meta-analytic method consistently outperformed all others. Therefore, we recommend that meta-analysts in psychology focus on sensitivity analyses—that is, report on a variety of methods, consider the conditions under which these methods fail (as indicated by simulation studies such as ours), and then report how conclusions might change based on which conditions are most plausible. Moreover, given the dependence of meta-analytic methods on untestable assumptions, we strongly recommend that researchers in psychology continue their efforts on improving the primary literature and conducting large-scale, pre-registered replications. We provide detailed results and simulation code at https://osf.io/rf3ys and interactive figures at http://www.shinyapps.org/apps/metaExplorer/.

Download Full-text