scholarly journals Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers

Author(s):  
Maya B Mathur ◽  
Tyler VanderWeele

Selective publication and reporting in individual papers compromise the scientific record, but are meta-analyses as compromised as their constituent studies? We systematically sampled 63 meta-analyses (each comprising at least 40 studies) in PLOS One, top medical journals, top psychology journals, and Metalab, an online, open-data database of developmental psychology meta-analyses. We empirically estimated publication bias in each. Across all meta-analyses, “statistically significant” results in the expected direction were only 1.17 times more likely to be published than “nonsignificant” results or those in the unexpected direction (95%CI: [0.94, 1.47]), with a confidence interval substantially overlapping the null. Comparable estimates were 0.83 for meta-analyses in PLOS One, 1.02 for top medical journals, 1.54 for top psychology journals, and 4.70 for Metalab. The severity of publication bias did differ across individual meta-analyses; in a small minority (10%; 95% CI: [2%, 21%]), publication bias appeared to favor "significant" results in the expected direction by more than 3-fold. We estimated that for 89% of meta-analyses, the amount of publication bias that would be required to attenuate the point estimate to the null exceeded the amount of publication estimated to be actually present in the vast majority of meta-analyses from the relevant scientific discipline (exceeding the 95th percentile of publication bias). Study-level measures (“statistical significance” with a point estimate in the expected direction and point estimate size) did not indicate more publication bias in higher-tier versus lower-tier journals, nor in the earliest studies published on a topic versus later studies. Overall, the mere act of performing a meta-analysis with a large number of studies (at least 40) and that includes non-headline results may largely mitigate publication bias in meta-analyses, suggesting optimism about the validity of meta-analytic results.

2019 ◽  
Author(s):  
Sho Tsuji ◽  
Alejandrina Cristia ◽  
Michael C. Frank ◽  
Christina Bergmann

Meta-analyses are an indispensable research synthesis tool for characterizing bodies of literature and advancing theories. One important open question concerns the inclusion of unpublished data into meta-analyses. Finding such studies can be effortful, but their exclusion potentially leads to consequential biases like overestimation of a literature’s mean effect. We address two key questions using MetaLab, a collection of community-augmented meta-analyses focused on developmental psychology. First, we assess to what extent these datasets include grey literature, and by what search strategies they are unearthed. An average of 11% of datapoints are from unpublished literature, and that standard search strategies like database searches, complemented with individualized approaches like including authors’ own data, contribute the majority of this literature. Second, we analyze the effect of including versus excluding unpublished literature on estimates of effect size and publication bias, and find this decision does not affect outcomes. We discuss lessons learned and implications.


2019 ◽  
Author(s):  
Maya B Mathur ◽  
Tyler VanderWeele

We propose sensitivity analyses for publication bias in meta-analyses. We consider a publication process such that "statistically significant" results are more likely to be published than negative or "nonsignificant" results by an unknown ratio, eta. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse-probability weighting and robust estimation that accommodates non-normal population effects, small meta-analyses, and clustering, we develop sensitivity analyses that enable statements such as: "For publication bias to shift the observed point estimate to the null, 'significant' results would need to be at least 30-fold more likely to be published than negative or 'nonsignificant' results." Comparable statements can be made regarding shifting to a chosen non-null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of eta across disciplines. We show that a worst-case meta-analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta-analysis of only the negative and "nonsignificant" studies; this method sometimes indicates that no amount of such publication bias could "explain away" the results. We illustrate the proposed methods using real-life meta-analyses and provide an R package, PublicationBias.


2019 ◽  
Vol 14 (4) ◽  
pp. 705-708 ◽  
Author(s):  
Maya B. Mathur ◽  
Tyler J. VanderWeele

Independent meta-analyses on the same topic can sometimes yield seemingly conflicting results. For example, prominent meta-analyses assessing the effects of violent video games on aggressive behavior have reached apparently different conclusions, provoking ongoing debate. We suggest that such conflicts are sometimes partly an artifact of reporting practices for meta-analyses that focus only on the pooled point estimate and its statistical significance. Considering statistics that focus on the distributions of effect sizes and that adequately characterize effect heterogeneity can sometimes indicate reasonable consensus between “warring” meta-analyses. Using novel analyses, we show that this seems to be the case in the video-game literature. Despite seemingly conflicting results for the statistical significance of the pooled estimates in different meta-analyses of video-game studies, all of the meta-analyses do in fact point to the conclusion that, in the vast majority of settings, violent video games do increase aggressive behavior but that these effects are almost always quite small.


2020 ◽  
Vol 228 (1) ◽  
pp. 50-61 ◽  
Author(s):  
Sho Tsuji ◽  
Alejandrina Cristia ◽  
Michael C. Frank ◽  
Christina Bergmann

Abstract. Meta-analyses are an indispensable research synthesis tool for characterizing bodies of literature and advancing theories. One important open question concerns the inclusion of unpublished data into meta-analyses. Finding such studies can be effortful, but their exclusion potentially leads to consequential biases like overestimation of a literature’s mean effect. We address two questions about unpublished data using MetaLab, a collection of community-augmented meta-analyses focused on developmental psychology. First, we assess to what extent MetaLab datasets include gray literature, and by what search strategies they are unearthed. We find that an average of 11% of datapoints are from unpublished literature; standard search strategies like database searches, complemented with individualized approaches like including authors’ own data, contribute the majority of this literature. Second, we analyze the effect of including versus excluding unpublished literature on estimates of effect size and publication bias, and find this decision does not affect outcomes. We discuss lessons learned and implications.


2019 ◽  
Author(s):  
Maya B Mathur ◽  
Tyler VanderWeele

Independent meta-analyses on the same topic can sometimes yield seemingly conflicting results. For example, prominent meta-analyses assessing the effects of violent video games on aggressive behavior have reached apparently different conclusions, provoking ongoing debate. We suggest that such conflicts are sometimes partly an artifact of reporting practices for meta-analyses that focus only on the pooled point estimate and its statistical significance. Considering statistics that focus on the distributions of effect sizes and that adequately characterize effect heterogeneity can sometimes indicate reasonable consensus between “warring” meta-analyses. Using novel analyses, we show that this seems to be the case in the video-game literature. Despite seemingly conflicting results for the statistical significance of the pooled estimates in different meta-analyses of video-game studies, all of the meta-analyses do in fact point to the conclusion that, in the vast majority of settings, violent video games do increase aggressive behavior but that these effects are almost always quite small.


2019 ◽  
Author(s):  
Amanda Kvarven ◽  
Eirik Strømland ◽  
Magnus Johannesson

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.


2021 ◽  
Author(s):  
Yan Yu ◽  
Jiasu Liu

Objectives: This meta-analysis aimed to identify the therapeutic effect of 0.01% atropine with on ocular axial elongation for myopia children. Methods: We searched PubMed, Cochrane Library, and CBM databases from inception to July 2021. Meta-analysis was conducted using STATA version 14.0 and Review Manager version 5.3 softwares. We calculated the weighted mean differences(WMD) to analyze the change of ocular axial length (AL) between orthokeratology combined with 0.01% atropine (OKA) and orthokeratology (OA) alone. The Cochran's Q-statistic and I2 test were used to evaluate potential heterogeneity between studies. To evaluate the influence of single studies on the overall estimate, a sensitivity analysis was performed. We also performed sub group and meta-regression analyses to investigate potential sources of heterogeneity. We conducted Begger's funnel plots and Egger's linear regression tests to investigate publication bias. Results: Nine studies that met all inclusion criteria were included in this meta-analysis. A total of 191 children in OKA group and 196 children in OK group were assessed. The pooled summary WMD of AL change was -0.90(95%CI=-1.25~-0.55) with statistical significance(t=-5.03, p<0.01), which indicated there was obvious difference between OKA and OK in myopic children. Subgroup analysis also showed that OKA treatment resulted in significantly less axial elongation compared to OK treatment alone according to SER. We found no evidence for publication bias. Conclusions:  Our meta-analysis indicates 0.01% atropine atropine is effective in slowing axial elongation in myopia children with orthokeratology.


2020 ◽  
Author(s):  
Nasrin Amiri Dashatan ◽  
Marzieh Ashrafmansouri ◽  
Mehdi Koushki ◽  
Nayebali Ahmadi

Abstract Background Leishmaniasis is one of the most important health problems worldwide. The evidence has suggested that resveratrol and its derivatives have anti-leishmanial effects; however, the results are inconsistent and inconclusive. The aim of this study was to assess the effect of resveratrol and its derivatives on the Leishmania viability through a systematic review and meta-analysis of available relevant studies. Methods The electronic databases PubMed, ScienceDirect, Embase, Web of Science and Scopus were queried between October 2000 and April 2020 using a comprehensive search strategy. The eligible articles selected and data extraction conducted by two reviewers. Mean differences of IC50 (concentration leading to reduction of 50% of Leishmania) for each outcome was calculated using random-effects models. Sensitivity analyses and prespecified subgroup were conducted to evaluate potential heterogeneity and the stability of the pooled results. Publication bias was evaluated using the Egger’s and Begg’s tests. We also followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines for this review. Results Ten studies were included in the meta-analysis. We observed that RSV and its derivatives had significant reducing effects on Leishmania viability in promastigote [24.02 µg/ml; (95% CI 17.1, 30.8); P < 0.05; I2 = 99.8%; P heterogeneity = 0.00] and amastigote [18.3 µg/ml; (95% CI 13.5, 23.2); P < 0.05; I2 = 99.6%; P heterogeneity = 0.00] stages of Leishmania. A significant publication bias was observed in the meta-analysis. Sensitivity analyses showed a similar effect size while reducing the heterogeneity. Subgroup analysis indicated that the pooled effects of leishmanicidal of resveratrol and its derivatives were affected by type of stilbenes and Leishmania species. Conclusions Our findings clearly suggest that the strategies for the treatment of leishmaniasis should be focused on natural products such as RSV and its derivatives. Further study is needed to identify the mechanisms mediating this protective effects of RSV and its derivatives in leishmaniasis.


2020 ◽  
Author(s):  
Joshua Pritsker

Brand, von der Post, Ounsley, and Morgan (2019) introduced Bayesian posterior passing as an alternative to traditional meta-analyses. In this commentary I relate their procedure to traditional meta-analysis, showing that posterior passing is equivalent to fixed effects meta-analysis. To overcome the limitations of simple posterior passing, I introduce improved posterior passing methods to account for heterogeneity and publication bias. Additionally, practical limitations of posterior passing and the role that it can play in future research are discussed.


2021 ◽  
Vol 2021 ◽  
pp. 1-9
Author(s):  
Mistire Teshome Guta ◽  
Tiwabwork Tekalign ◽  
Nefsu Awoke ◽  
Robera Olana Fite ◽  
Getahun Dendir ◽  
...  

Aims. This systemic review and meta-analysis were aimed at determining the level of anxiety and depression among cystic fibrosis patients in the world. Methods. We conducted a systematic search of published studies from PubMed, EMBASE, MEDLINE, Cochrane, Scopus, Web of Science, CINAHL, and manually on Google Scholar. This meta-analysis follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The quality of studies was assessed by the modified Newcastle-Ottawa Scale (NOS). Meta-analysis was carried out using a random-effects method using the STATA™ Version 14 software. Trim and fill analysis was done to correct the presence of significant publication bias. Result. From 419,820 obtained studies, 26 studies from 2 different parts of the world including 9766. The overall global pooled prevalence of anxiety and depression after correction for publication bias by trim and fill analysis was found to be 24.91(95% CI: 20.8-28.9) for anxiety. The subgroup analyses revealed with the lowest prevalence, 23.59%, (95% CI: 8.08, 39.09)) in North America and the highest, 26.77%, (95% CI: 22.5, 31.04) seen in Europe for anxiety and with the highest prevalence, 18.67%, (95% CI: 9.82, 27.5) in North America and the lowest, 13.27%, (95% CI: -10.05, 16.5) seen in Europe for depression. Conclusion. The global prevalence of anxiety and depression among cystic fibrosis patients is common. Therefore, close monitoring of the patient, regularly screening for anxiety and depression, and appropriate prevention techniques is recommended.


Sign in / Sign up

Export Citation Format

Share Document