Association Between Unreported Outcomes and Effect Size Estimates in Cochrane Meta-analyses

Multiple meta-analyses may use similar search criteria and focus on the same topic of interest, but they may yield different or sometimes discordant results. The lack of statistical methods for synthesizing these findings makes it challenging to properly interpret the results from multiple meta-analyses, especially when their results are conflicting. In this paper, we first introduce a method to synthesize the meta-analytic results when multiple meta-analyses use the same type of summary effect estimates. When meta-analyses use different types of effect sizes, the meta-analysis results cannot be directly combined. We propose a two-step frequentist procedure to first convert the effect size estimates to the same metric and then summarize them with a weighted mean estimate. Our proposed method offers several advantages over existing methods by Hemming et al. (2012). First, different types of summary effect sizes are considered. Second, our method provides the same overall effect size as conducting a meta-analysis on all individual studies from multiple meta-analyses. We illustrate the application of the proposed methods in two examples and discuss their implications for the field of meta-analysis.

Download Full-text

Neglect of publication bias compromises meta-analyses of educational research

PLoS ONE ◽

10.1371/journal.pone.0252415 ◽

2021 ◽

Vol 16 (6) ◽

pp. e0252415

Author(s):

Ivan Ropovik ◽

Matus Adamkovic ◽

David Greger

Keyword(s):

Educational Research ◽

Publication Bias ◽

Effect Size ◽

State Of The Art ◽

Research Review ◽

Bias Adjustment ◽

Negative Findings ◽

Meta Analyses ◽

Size Estimates ◽

Practical Recommendations

Because negative findings have less chance of getting published, available studies tend to be a biased sample. This leads to an inflation of effect size estimates to an unknown degree. To see how meta-analyses in education account for publication bias, we surveyed all meta-analyses published in the last five years in the Review of Educational Research and Educational Research Review. The results show that meta-analyses usually neglect publication bias adjustment. In the minority of meta-analyses adjusting for bias, mostly non-principled adjustment methods were used, and only rarely were the conclusions based on corrected estimates, rendering the adjustment inconsequential. It is argued that appropriate state-of-the-art adjustment (e.g., selection models) should be attempted by default, yet one needs to take into account the uncertainty inherent in any meta-analytic inference under bias. We conclude by providing practical recommendations on dealing with publication bias.

Download Full-text

The puzzling relationship between multi-lab replications and meta-analyses of the rest of the literature

10.31234/osf.io/pbrdk ◽

2020 ◽

Author(s):

Molly Lewis ◽

Maya B Mathur ◽

Tyler VanderWeele ◽

Michael C. Frank

Keyword(s):

Empirical Evidence ◽

Effect Size ◽

Meta Analysis ◽

Scientific Research ◽

Psychological Phenomena ◽

Analytic Estimate ◽

Meta Analyses ◽

The Relationship ◽

Size Estimates

What is the best way to estimate the size of important effects? Should we aggregate across disparate findings using statistical meta-analysis, or instead run large, multi-lab replications (MLR)? A recent paper by Kvarven, Strømland, and Johannesson (2020) compared effect size estimates derived from these two different methods for 15 different psychological phenomena. The authors report that, for the same phenomenon, the meta-analytic estimate tends to be about three times larger than the MLR estimate. These results pose an important puzzle: What is the relationship between these two estimates? Kvarven et al. suggest that their results undermine the value of meta-analysis. In contrast, we argue that both meta-analysis and MLR are informative, and that the discrepancy between estimates obtained via the two methods is in fact still unexplained. Informed by re-analyses of Kvarven et al.’s data and by other empirical evidence, we discuss possible sources of this discrepancy and argue that understanding the relationship between estimates obtained from these two methods is an important puzzle for future meta-scientific research.

Download Full-text

How robust are cross-population signatures of polygenic adaptation in humans?

10.1101/2020.07.13.200030 ◽

2020 ◽

Author(s):

Alba Refoyo-Martínez ◽

Siyang Liu ◽

Anja Moltke Jørgensen ◽

Xin Jin ◽

Anders Albrechtsen ◽

...

Keyword(s):

Effect Size ◽

Association Studies ◽

Gwas Data ◽

Genome Wide Association Studies ◽

Summary Statistics ◽

Uk Biobank ◽

Polygenic Adaptation ◽

The Uk ◽

Meta Analyses ◽

Size Estimates

AbstractOver the past decade, summary statistics from genome-wide association studies (GWAS) have been used to detect and quantify polygenic adaptation in humans. Several studies have reported signatures of natural selection at sets of SNPs associated with complex traits, like height and body mass index. However, more recent studies suggest that some of these signals may be caused by biases from uncorrected population stratification in the GWAS data with which these tests are performed. Moreover, past studies have predominantly relied on SNP effect size estimates obtained from GWAS panels of European ancestries, which are known to be poor predictors of phenotypes in non-European populations. Here, we collated GWAS data from multiple anthropometric and metabolic traits that have been measured in more than one cohort around the world, including the UK Biobank, FINRISK, Chinese NIPT, Biobank Japan, APCDR and PAGE. We then evaluated how robust signals of polygenic adaptation are to the choice of GWAS cohort used to identify associated variants and their effect size estimates, while using the same panel to obtain population allele frequencies (The 1000 Genomes Project). We observe many discrepancies across tests performed on the same phenotype and find that association studies performed using multiple different cohorts, like meta-analyses, tend to produce scores with strong overdispersion across populations. This results in apparent signatures of polygenic adaptation which are not observed when using effect size estimates from biobank-based GWAS of homogeneous ancestries. Indeed, we were able to artificially create score overdispersion when taking the UK Biobank cohort and simulating a meta-analysis on multiple subsets of the cohort. This suggests that extreme caution should be taken in the execution and interpretation of future tests of polygenic adaptation based on population differentiation, especially when using summary statistics from GWAS meta-analyses.

Download Full-text

Preprint - Meta-Analyzing the Multiverse: A Peek Under the Hood of Selective Reporting

10.31234/osf.io/43yae ◽

2021 ◽

Author(s):

Anton Olsson-Collentine ◽

Robbie Cornelis Maria van Aert ◽

Marjan Bakker ◽

Jelte M. Wicherts

Keyword(s):

Effect Size ◽

Degrees Of Freedom ◽

Meta Analysis ◽

Extreme Case ◽

Large Degree ◽

Selective Reporting ◽

Primary Research ◽

Standard Deviations ◽

Meta Analyses ◽

Size Estimates

There are arbitrary decisions to be made (i.e., researcher degrees of freedom) in the execution and reporting of most research. These decisions allow for many possible outcomes from a single study. Selective reporting of results from this ‘multiverse’ of outcomes, whether intentional (_p_-hacking) or not, can lead to inflated effect size estimates and false positive results in the literature. In this study, we examine and illustrate the consequences of researcher degrees of freedom in primary research, both for primary outcomes and for subsequent meta-analyses. We used a set of 10 preregistered multi-lab direct replication projects from psychology (Registered Replication Reports) with a total of 14 primary outcome variables, 236 labs and 37,602 participants. By exploiting researcher degrees of freedom in each project, we were able to compute between 3,840 and 2,621,440 outcomes per lab. We show that researcher degrees of freedom in primary research can cause substantial variability in effect size that we denote the Underlying Multiverse Variability (UMV). In our data, the median UMV across labs was 0.1 standard deviations (interquartile range = 0.09 – 0.15). In one extreme case, the effect size estimate could change by _d_ = 1.27, evidence that _p_-hacking in some (rare) cases can provide support for almost any conclusion. We also show that researcher degrees of freedom in primary research provide another source of uncertainty in meta-analysis beyond those usually estimated. This would not be a large concern for meta-analysis if researchers made all arbitrary decisions at random. However, emulating selective reporting of lab results led to inflation of meta-analytic average effect size estimates in our data by as much as 0.1 - 0.48 standard deviations, depending to a large degree on the number of possible outcomes at the lab level (i.e., multiverse size). Our results illustrate the importance of making research decisions transparent (e.g., through preregistration and multiverse analysis), evaluating studies for selective reporting, and whenever feasible making raw data available.

Download Full-text

Examining the Reproducibility of Meta-Analyses in Psychology: A Preliminary Report

10.31222/osf.io/xfbjf ◽

2017 ◽

Cited By ~ 3

Author(s):

Daniel Lakens ◽

Elizabeth Page-Gould ◽

Marcel A. L. M. van Assen ◽

Bobbie Spellman ◽

Felix D. Schönbrodt ◽

...

Keyword(s):

Effect Size ◽

Preliminary Report ◽

Effect Sizes ◽

Individual Effect ◽

Raw Data ◽

Lack Of Information ◽

Meta Analyses ◽

The Impact ◽

Size Estimates ◽

Unpublished Research

Meta-analyses are an important tool to evaluate the literature. It is essential that meta-analyses can easily be reproduced to allow researchers to evaluate the impact of subjective choices on meta-analytic effect sizes, but also to update meta-analyses as new data comes in, or as novel statistical techniques (for example to correct for publication bias) are developed. Research in medicine has revealed meta-analyses often cannot be reproduced. In this project, we examined the reproducibility of meta-analyses in psychology by reproducing twenty published meta-analyses. Reproducing published meta-analyses was surprisingly difficult. 96% of meta-analyses published in 2013-2014 did not adhere to reporting guidelines. A third of these meta-analyses did not contain a table specifying all individual effect sizes. Five of the 20 randomly selected meta-analyses we attempted to reproduce could not be reproduced at all due to lack of access to raw data, no details about the effect sizes extracted from each study, or a lack of information about how effect sizes were coded. In the remaining meta-analyses, differences between the reported and reproduced effect size or sample size were common. We discuss a range of possible improvements, such as more clearly indicating which data were used to calculate an effect size, specifying all individual effect sizes, adding detailed information about equations that are used, and how multiple effect size estimates from the same study are combined, but also sharing raw data retrieved from original authors, or unpublished research reports. This project clearly illustrates there is a lot of room for improvement when it comes to the transparency and reproducibility of published meta-analyses.

Download Full-text

p-Hacking and Publication Bias Interact to Distort Meta-Analytic Effect Size Estimates

10.31234/osf.io/bvm58 ◽

2020 ◽

Author(s):

Malte Friese ◽

Julius Frankenbach

Keyword(s):

Publication Bias ◽

Effect Size ◽

Large Scale ◽

Meta Analysis ◽

Scientific Progress ◽

Reliable Technique ◽

Research Areas ◽

Broad Array ◽

Meta Analyses ◽

Size Estimates

Science depends on trustworthy evidence. Thus, a biased scientific record is of questionable value because it impedes scientific progress, and the public receives advice on the basis of unreliable evidence that has the potential to have far-reaching detrimental consequences. Meta-analysis is a valid and reliable technique that can be used to summarize research evidence. However, meta-analytic effect size estimates may themselves be biased, threatening the validity and usefulness of meta-analyses to promote scientific progress. Here, we offer a large-scale simulation study to elucidate how p-hacking and publication bias distort meta-analytic effect size estimates under a broad array of circumstances that reflect the reality that exists across a variety of research areas. The results revealed that, first, very high levels of publication bias can severely distort the cumulative evidence. Second, p-hacking and publication bias interact: At relatively high and low levels of publication bias, p-hacking does comparatively little harm, but at medium levels of publication bias, p-hacking can considerably contribute to bias, especially when the true effects are very small or are approaching zero. Third, p-hacking can severely increase the rate of false positives. A key implication is that, in addition to preventing p-hacking, policies in research institutions, funding agencies, and scientific journals need to make the prevention of publication bias a top priority to ensure a trustworthy base of evidence.

Download Full-text

The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates

Review of General Psychology ◽

10.1037/gpr0000034 ◽

2015 ◽

Vol 19 (2) ◽

pp. 172-182 ◽

Cited By ~ 26

Author(s):

Michèle B. Nuijten ◽

Marcel A. L. M. van Assen ◽

Coosje L. S. Veldkamp ◽

Jelte M. Wicherts

Keyword(s):

Low Power ◽

Sample Size ◽

Publication Bias ◽

Effect Size ◽

Effect Sizes ◽

Population Effect ◽

Replication Studies ◽

Unpublished Studies ◽

Meta Analyses ◽

Size Estimates

Replication is often viewed as the demarcation between science and nonscience. However, contrary to the commonly held view, we show that in the current (selective) publication system replications may increase bias in effect size estimates. Specifically, we examine the effect of replication on bias in estimated population effect size as a function of publication bias and the studies’ sample size or power. We analytically show that incorporating the results of published replication studies will in general not lead to less bias in the estimated population effect size. We therefore conclude that mere replication will not solve the problem of overestimation of effect sizes. We will discuss the implications of our findings for interpreting results of published and unpublished studies, and for conducting and interpreting results of meta-analyses. We also discuss solutions for the problem of overestimation of effect sizes, such as discarding and not publishing small studies with low power, and implementing practices that completely eliminate publication bias (e.g., study registration).

Download Full-text

Association of study characteristics with estimates of effect size in studies of ecstasy use

Journal of Psychopharmacology ◽

10.1177/0269881111408955 ◽

2011 ◽

Vol 25 (11) ◽

pp. 1573-1577 ◽

Cited By ~ 4

Author(s):

Eleanor M Taylor ◽

Natasha MP Greene ◽

Celia JA Morgan ◽

Marcus R Munafò

Keyword(s):

Effect Size ◽

Meta Analysis ◽

Control Group ◽

Effect Size Estimate ◽

Size Estimate ◽

Study Characteristics ◽

Drug Naïve ◽

Meta Analyses ◽

And Control ◽

Size Estimates

Studies of the chronic effects of MDMA, or ‘ecstasy’, in humans have been largely inconsistent. We explored whether study-level characteristics are associated with the effect size estimate reported. We based our analyses on the recent systematic review by Rogers and colleagues, focusing on those meta-analyses within this report where there was a relatively large number of studies contributing to each individual meta-analysis. Linear regression was used to investigate the association between study level variables and effect size estimate, weighted by the inverse of the SE of the effect size estimate, with cluster correction for studies which contributed multiple estimates. This indicated an association between effect size estimate and both user group, with smaller estimates among studies recruiting former users compared with those recruiting current users, and control group, with smaller estimates among studies recruiting polydrug user controls compared with those recruiting drug-naïve controls. In addition, increasing year of publication was associated with reduced effect size estimate, and there was a trend level association with prevalence of ecstasy use, reflecting smaller estimates among studies conducted in countries with higher prevalence of ecstasy use. Our data suggest a number of study-level characteristics which appear to influence individual study effect size estimates. These should be considered when designing future studies, and also when interpreting the ecstasy literature as a whole.

Download Full-text

How to Detect Publication Bias in Psychological Research

Zeitschrift für Psychologie ◽

10.1027/2151-2604/a000386 ◽

2019 ◽

Vol 227 (4) ◽

pp. 261-279 ◽

Cited By ~ 2

Author(s):

Frank Renkewitz ◽

Melanie Keiner

Keyword(s):

Publication Bias ◽

Effect Size ◽

Statistical Power ◽

Type I Error ◽

Psychological Research ◽

Type I ◽

True Effect Size ◽

Questionable Research Practices ◽

True Effect ◽

Meta Analyses

Abstract. Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates. Both of these problems lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect such bias in meta-analytic results. We present an evaluation of the performance of six of these tools. To assess the Type I error rate and the statistical power of these methods, we simulated a large variety of literatures that differed with regard to true effect size, heterogeneity, number of available primary studies, and sample sizes of these primary studies; furthermore, simulated studies were subjected to different degrees of publication bias. Our results show that across all simulated conditions, no method consistently outperformed the others. Additionally, all methods performed poorly when true effect sizes were heterogeneous or primary studies had a small chance of being published, irrespective of their results. This suggests that in many actual meta-analyses in psychology, bias will remain undiscovered no matter which detection method is used.

Download Full-text