scholarly journals Neglect of publication bias compromises meta-analyses of educational research

PLoS ONE ◽  
2021 ◽  
Vol 16 (6) ◽  
pp. e0252415
Author(s):  
Ivan Ropovik ◽  
Matus Adamkovic ◽  
David Greger
Keyword(s):  
Educational Research ◽  
Publication Bias ◽  
Effect Size ◽  
State Of The Art ◽  
Research Review ◽  
Bias Adjustment ◽  
Negative Findings ◽  
Meta Analyses ◽  
Size Estimates ◽  
Practical Recommendations

Because negative findings have less chance of getting published, available studies tend to be a biased sample. This leads to an inflation of effect size estimates to an unknown degree. To see how meta-analyses in education account for publication bias, we surveyed all meta-analyses published in the last five years in the Review of Educational Research and Educational Research Review. The results show that meta-analyses usually neglect publication bias adjustment. In the minority of meta-analyses adjusting for bias, mostly non-principled adjustment methods were used, and only rarely were the conclusions based on corrected estimates, rendering the adjustment inconsequential. It is argued that appropriate state-of-the-art adjustment (e.g., selection models) should be attempted by default, yet one needs to take into account the uncertainty inherent in any meta-analytic inference under bias. We conclude by providing practical recommendations on dealing with publication bias.

scholarly journals Neglect of publication bias compromises meta-analyses of educational research

2019 ◽  
Author(s):  
Ivan Ropovik ◽  
Matus Adamkovic ◽  
David Greger
Keyword(s):  
Educational Research ◽  
Publication Bias ◽  
Bias Correction ◽  
State Of The Art ◽  
Effect Sizes ◽  
Research Review ◽  
Negative Findings ◽  
Trim And Fill ◽  
Meta Analyses ◽  
Practical Recommendations

Because negative findings have less chance of getting published, available studies tend to be a biased sample. This leads to a higher prevalence of false positives and the inflation of effect sizes to an unknown degree. To see how meta-analyses in education account for publication bias, we surveyed all meta-analyses published in the last three years in the Review of Educational Research and Educational Research Review. The results show that meta-analyses usually neglect publication bias correction. In the minority of studies adjusting for bias, only the outdated trim and fill method was used, and none of the meta-analyses based their conclusions on corrected estimates, rendering the adjustment inconsequential. It is argued that appropriate state-of-the-art adjustment (e.g., selection models) should be carried out by default, yet one needs to take into account the uncertainty inherent in any meta-analytic inference under bias. We conclude by providing practical recommendations on dealing with publication bias.

scholarly journals vanAert_Niemeyer21

2020 ◽  
Author(s):  
Robbie Cornelis Maria van Aert ◽  
Helen Niemeyer
Keyword(s):  
Statistical Method ◽  
Publication Bias ◽  
Effect Size ◽  
Clinical Psychology ◽  
State Of The Art ◽  
Meta Analysis ◽  
Major Threat ◽  
Art Methods ◽  
Meta Analyses

Meta-analysis is the statistical method for synthesizing studies on the same topic and is often used in clinical psychology to quantify the efficacy of treatments. A major threat to the validity of meta-analysis is publication bias, which implies that some studies are less likely to be published and are therefore less often included in a meta-analysis. A consequence of publication bias is the overestimation of the meta-analytic effect size that may give a false impression with respect to the efficacy of a treatment, which might result in (avoidable) suffering of patients and waste of resources. Guidelines recommend to routinely assess publication bias in meta-analyses, but this is currently not common practice. This chapter describes popular and state-of-the-art methods to assess publication bias in a meta-analysis and summarizes recommendations for applying these methods. We also illustrate how these methods can be applied to two meta-analyses that are typical for clinical psychology such that psychologists can readily apply the methods in their own meta-analyses.

scholarly journals p-Hacking and Publication Bias Interact to Distort Meta-Analytic Effect Size Estimates

2020 ◽  
Author(s):  
Malte Friese ◽  
Julius Frankenbach
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Large Scale ◽  
Meta Analysis ◽  
Scientific Progress ◽  
Reliable Technique ◽  
Research Areas ◽  
Broad Array ◽  
Meta Analyses ◽  
Size Estimates

Science depends on trustworthy evidence. Thus, a biased scientific record is of questionable value because it impedes scientific progress, and the public receives advice on the basis of unreliable evidence that has the potential to have far-reaching detrimental consequences. Meta-analysis is a valid and reliable technique that can be used to summarize research evidence. However, meta-analytic effect size estimates may themselves be biased, threatening the validity and usefulness of meta-analyses to promote scientific progress. Here, we offer a large-scale simulation study to elucidate how p-hacking and publication bias distort meta-analytic effect size estimates under a broad array of circumstances that reflect the reality that exists across a variety of research areas. The results revealed that, first, very high levels of publication bias can severely distort the cumulative evidence. Second, p-hacking and publication bias interact: At relatively high and low levels of publication bias, p-hacking does comparatively little harm, but at medium levels of publication bias, p-hacking can considerably contribute to bias, especially when the true effects are very small or are approaching zero. Third, p-hacking can severely increase the rate of false positives. A key implication is that, in addition to preventing p-hacking, policies in research institutions, funding agencies, and scientific journals need to make the prevention of publication bias a top priority to ensure a trustworthy base of evidence.

scholarly journals The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates

2015 ◽  
Vol 19 (2) ◽  
pp. 172-182 ◽  
Author(s):  
Michèle B. Nuijten ◽  
Marcel A. L. M. van Assen ◽  
Coosje L. S. Veldkamp ◽  
Jelte M. Wicherts
Keyword(s):  
Low Power ◽  
Sample Size ◽  
Publication Bias ◽  
Effect Size ◽  
Effect Sizes ◽  
Population Effect ◽  
Replication Studies ◽  
Unpublished Studies ◽  
Meta Analyses ◽  
Size Estimates

Replication is often viewed as the demarcation between science and nonscience. However, contrary to the commonly held view, we show that in the current (selective) publication system replications may increase bias in effect size estimates. Specifically, we examine the effect of replication on bias in estimated population effect size as a function of publication bias and the studies’ sample size or power. We analytically show that incorporating the results of published replication studies will in general not lead to less bias in the estimated population effect size. We therefore conclude that mere replication will not solve the problem of overestimation of effect sizes. We will discuss the implications of our findings for interpreting results of published and unpublished studies, and for conducting and interpreting results of meta-analyses. We also discuss solutions for the problem of overestimation of effect sizes, such as discarding and not publishing small studies with low power, and implementing practices that completely eliminate publication bias (e.g., study registration).

How to Detect Publication Bias in Psychological Research

2019 ◽  
Vol 227 (4) ◽  
pp. 261-279 ◽  
Author(s):  
Frank Renkewitz ◽  
Melanie Keiner
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Statistical Power ◽  
Type I Error ◽  
Psychological Research ◽  
Type I ◽  
True Effect Size ◽  
Questionable Research Practices ◽  
True Effect ◽  
Meta Analyses

Abstract. Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates. Both of these problems lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect such bias in meta-analytic results. We present an evaluation of the performance of six of these tools. To assess the Type I error rate and the statistical power of these methods, we simulated a large variety of literatures that differed with regard to true effect size, heterogeneity, number of available primary studies, and sample sizes of these primary studies; furthermore, simulated studies were subjected to different degrees of publication bias. Our results show that across all simulated conditions, no method consistently outperformed the others. Additionally, all methods performed poorly when true effect sizes were heterogeneous or primary studies had a small chance of being published, irrespective of their results. This suggests that in many actual meta-analyses in psychology, bias will remain undiscovered no matter which detection method is used.

scholarly journals A Statistical Method for Synthesizing Meta-Analyses

2013 ◽  
Vol 2013 ◽  
pp. 1-9 ◽  
Author(s):  
Liansheng Larry Tang ◽  
Michael Caudy ◽  
Faye Taxman
Keyword(s):  
Statistical Method ◽  
Effect Size ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Weighted Mean ◽  
Summary Effect ◽  
Different Types ◽  
Meta Analyses ◽  
Size Estimates ◽  
Similar Search

Multiple meta-analyses may use similar search criteria and focus on the same topic of interest, but they may yield different or sometimes discordant results. The lack of statistical methods for synthesizing these findings makes it challenging to properly interpret the results from multiple meta-analyses, especially when their results are conflicting. In this paper, we first introduce a method to synthesize the meta-analytic results when multiple meta-analyses use the same type of summary effect estimates. When meta-analyses use different types of effect sizes, the meta-analysis results cannot be directly combined. We propose a two-step frequentist procedure to first convert the effect size estimates to the same metric and then summarize them with a weighted mean estimate. Our proposed method offers several advantages over existing methods by Hemming et al. (2012). First, different types of summary effect sizes are considered. Second, our method provides the same overall effect size as conducting a meta-analysis on all individual studies from multiple meta-analyses. We illustrate the application of the proposed methods in two examples and discuss their implications for the field of meta-analysis.

scholarly journals Is Replication Possible without Fidelity?

2021 ◽  
Author(s):  
Michelle Renee Ellefson ◽  
Daniel Oppenheimer
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Experimental Psychology ◽  
Intervention Science ◽  
Experimental Protocols ◽  
Meta Analyses

Failure of replication attempts in experimental psychology might extend beyond p-hacking, publication bias or hidden moderators; reductions in experimental power can be caused by violations of fidelity to a set of experimental protocols. In this paper, we run a series of simulations to systematically explore how manipulating fidelity influences effect size. We find statistical patterns that mimic those found in ManyLabs style replications and meta-analyses, suggesting that fidelity violations are present in many replication attempts in psychology. Scholars in intervention science, medicine, and education have developed methods of improving and measuring fidelity, and as replication becomes more mainstream in psychology, the field would benefit from adopting such approaches as well.

Association Between Unreported Outcomes and Effect Size Estimates in Cochrane Meta-analyses

JAMA ◽  
2007 ◽  
Vol 297 (5) ◽  
pp. 465 ◽  
Author(s):  
Toshi A. Furukawa ◽  
Norio Watanabe ◽  
Ichiro M. Omori ◽  
Victor M. Montori ◽  
Gordon H. Guyatt
Keyword(s):  
Effect Size ◽  
Meta Analyses ◽  
Size Estimates

scholarly journals Has the evidence for moral licensing been inflated by publication bias?

2019 ◽  
Vol 3 ◽  
Author(s):  
Niclas Kuper ◽  
Antonia Bott
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Empirical Work ◽  
True Effect Size ◽  
Replication Studies ◽  
Immoral Behavior ◽  
Moral Licensing ◽  
Self Image ◽  
Meta Analyses ◽  
Licensing Effects

Moral licensing describes the phenomenon that displaying moral behavior can lead to subsequent immoral behavior. This is usually explained by the idea that an initial moral act affirms the moral self-image and hence licenses subsequent immoral acts. Previous meta-analyses on moral licensing indicate significant overall effects of d> .30. However, several large replication studies have either not found the effect or reported a substantially smaller effect size. The present article investigated whether this can be attributed to publication bias. Datasets from two previous meta-analyses on moral licensing were compared and when necessary modified. The larger dataset was used for the present analyses. Using PET-PEESE and a three-parameter-selection-model (3-PSM), we found some evidence for publication bias. The adjusted effect sizes were reduced to d= -0.05, p= .64 and d= 0.18, p= .002, respectively. While the first estimate could be an underestimation, we also found indications that the second estimate might exaggerate the true effect size. It is concluded that both the evidence for and the size of moral licensing effects has likely been inflated by publication bias. Furthermore, our findings indicate that culture moderates the moral licensing effect. Recommendations for future meta-analytic and empirical work are given. Subsequent studies on moral licensing should be adequately powered and ideally pre-registered.  

scholarly journals The role of meta-analysis and preregistration in assessing the evidence for cleansing effects

2021 ◽  
Vol 44 ◽  
Author(s):  
Robert M. Ross ◽  
Robbie C. M. van Aert ◽  
Olmo R. van den Akker ◽  
Michiel van Elk
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Degrees Of Freedom ◽  
Meta Analysis ◽  
Replication Studies ◽  
Size Estimates

Abstract Lee and Schwarz interpret meta-analytic research and replication studies as providing evidence for the robustness of cleansing effects. We argue that the currently available evidence is unconvincing because (a) publication bias and the opportunistic use of researcher degrees of freedom appear to have inflated meta-analytic effect size estimates, and (b) preregistered replications failed to find any evidence of cleansing effects.

Sign in / Sign up

Export Citation Format

Share Document