scholarly journals Neglect of publication bias compromises meta-analyses of educational research

2019 ◽  
Author(s):  
Ivan Ropovik ◽  
Matus Adamkovic ◽  
David Greger
Keyword(s):  
Educational Research ◽  
Publication Bias ◽  
Bias Correction ◽  
State Of The Art ◽  
Effect Sizes ◽  
Research Review ◽  
Negative Findings ◽  
Trim And Fill ◽  
Meta Analyses ◽  
Practical Recommendations

Because negative findings have less chance of getting published, available studies tend to be a biased sample. This leads to a higher prevalence of false positives and the inflation of effect sizes to an unknown degree. To see how meta-analyses in education account for publication bias, we surveyed all meta-analyses published in the last three years in the Review of Educational Research and Educational Research Review. The results show that meta-analyses usually neglect publication bias correction. In the minority of studies adjusting for bias, only the outdated trim and fill method was used, and none of the meta-analyses based their conclusions on corrected estimates, rendering the adjustment inconsequential. It is argued that appropriate state-of-the-art adjustment (e.g., selection models) should be carried out by default, yet one needs to take into account the uncertainty inherent in any meta-analytic inference under bias. We conclude by providing practical recommendations on dealing with publication bias.

scholarly journals Neglect of publication bias compromises meta-analyses of educational research

PLoS ONE ◽  
2021 ◽  
Vol 16 (6) ◽  
pp. e0252415
Author(s):  
Ivan Ropovik ◽  
Matus Adamkovic ◽  
David Greger
Keyword(s):  
Educational Research ◽  
Publication Bias ◽  
Effect Size ◽  
State Of The Art ◽  
Research Review ◽  
Bias Adjustment ◽  
Negative Findings ◽  
Meta Analyses ◽  
Size Estimates ◽  
Practical Recommendations

Because negative findings have less chance of getting published, available studies tend to be a biased sample. This leads to an inflation of effect size estimates to an unknown degree. To see how meta-analyses in education account for publication bias, we surveyed all meta-analyses published in the last five years in the Review of Educational Research and Educational Research Review. The results show that meta-analyses usually neglect publication bias adjustment. In the minority of meta-analyses adjusting for bias, mostly non-principled adjustment methods were used, and only rarely were the conclusions based on corrected estimates, rendering the adjustment inconsequential. It is argued that appropriate state-of-the-art adjustment (e.g., selection models) should be attempted by default, yet one needs to take into account the uncertainty inherent in any meta-analytic inference under bias. We conclude by providing practical recommendations on dealing with publication bias.

Identification of and Correction for Publication Bias: Comment

2019 ◽  
Author(s):  
Amanda Kvarven ◽  
Eirik Strømland ◽  
Magnus Johannesson
Keyword(s):  
Publication Bias ◽  
False Positive ◽  
Large Scale ◽  
Meta Analysis ◽  
False Positive Rate ◽  
Effect Sizes ◽  
Replication Studies ◽  
Moderate Reduction ◽  
Positive Rate ◽  
Meta Analyses

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

scholarly journals A protocol for the systematic review and meta-analysis of thigmotactic behaviour in the open field test in rodent models associated with persistent pain

2021 ◽  
Vol 5 (1) ◽  
pp. e100135
Author(s):  
Xue Ying Zhang ◽  
Jan Vollert ◽  
Emily S Sena ◽  
Andrew SC Rice ◽  
Nadia Soliman
Keyword(s):  
Systematic Review ◽  
Outcome Measure ◽  
Meta Analysis ◽  
Persistent Pain ◽  
Effect Sizes ◽  
Bias Assessment ◽  
Animal Pain ◽  
Trim And Fill ◽  
Meta Analyses ◽  
The Impact

ObjectiveThigmotaxis is an innate predator avoidance behaviour of rodents and is enhanced when animals are under stress. It is characterised by the preference of a rodent to seek shelter, rather than expose itself to the aversive open area. The behaviour has been proposed to be a measurable construct that can address the impact of pain on rodent behaviour. This systematic review will assess whether thigmotaxis can be influenced by experimental persistent pain and attenuated by pharmacological interventions in rodents.Search strategyWe will conduct search on three electronic databases to identify studies in which thigmotaxis was used as an outcome measure contextualised to a rodent model associated with persistent pain. All studies published until the date of the search will be considered.Screening and annotationTwo independent reviewers will screen studies based on the order of (1) titles and abstracts, and (2) full texts.Data management and reportingFor meta-analysis, we will extract thigmotactic behavioural data and calculate effect sizes. Effect sizes will be combined using a random-effects model. We will assess heterogeneity and identify sources of heterogeneity. A risk-of-bias assessment will be conducted to evaluate study quality. Publication bias will be assessed using funnel plots, Egger’s regression and trim-and-fill analysis. We will also extract stimulus-evoked limb withdrawal data to assess its correlation with thigmotaxis in the same animals. The evidence obtained will provide a comprehensive understanding of the strengths and limitations of using thigmotactic outcome measure in animal pain research so that future experimental designs can be optimised. We will follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guidelines and disseminate the review findings through publication and conference presentation.

scholarly journals Global Burden of Anxiety and Depression among Cystic Fibrosis Patient: Systematic Review and Meta-Analysis

2021 ◽  
Vol 2021 ◽  
pp. 1-9
Author(s):  
Mistire Teshome Guta ◽  
Tiwabwork Tekalign ◽  
Nefsu Awoke ◽  
Robera Olana Fite ◽  
Getahun Dendir ◽  
...  
Keyword(s):  
Cystic Fibrosis ◽  
North America ◽  
Publication Bias ◽  
Meta Analysis ◽  
Systemic Review ◽  
Anxiety And Depression ◽  
Close Monitoring ◽  
The World ◽  
Trim And Fill ◽  
Meta Analyses

Aims. This systemic review and meta-analysis were aimed at determining the level of anxiety and depression among cystic fibrosis patients in the world. Methods. We conducted a systematic search of published studies from PubMed, EMBASE, MEDLINE, Cochrane, Scopus, Web of Science, CINAHL, and manually on Google Scholar. This meta-analysis follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The quality of studies was assessed by the modified Newcastle-Ottawa Scale (NOS). Meta-analysis was carried out using a random-effects method using the STATA™ Version 14 software. Trim and fill analysis was done to correct the presence of significant publication bias. Result. From 419,820 obtained studies, 26 studies from 2 different parts of the world including 9766. The overall global pooled prevalence of anxiety and depression after correction for publication bias by trim and fill analysis was found to be 24.91(95% CI: 20.8-28.9) for anxiety. The subgroup analyses revealed with the lowest prevalence, 23.59%, (95% CI: 8.08, 39.09)) in North America and the highest, 26.77%, (95% CI: 22.5, 31.04) seen in Europe for anxiety and with the highest prevalence, 18.67%, (95% CI: 9.82, 27.5) in North America and the lowest, 13.27%, (95% CI: -10.05, 16.5) seen in Europe for depression. Conclusion. The global prevalence of anxiety and depression among cystic fibrosis patients is common. Therefore, close monitoring of the patient, regularly screening for anxiety and depression, and appropriate prevention techniques is recommended.

Effects of the endocrine disruptor vinclozolin in male reproduction: a systematic review and meta-analysis†

2021 ◽  
Author(s):  
Mariana Feijó ◽  
Roberta V L Martins ◽  
Sílvia Socorro ◽  
Luísa Pereira ◽  
Sara Correia
Keyword(s):  
Systematic Review ◽  
Publication Bias ◽  
Meta Analysis ◽  
Endocrine Disrupting Chemicals ◽  
Reproductive Organs ◽  
Male Reproduction ◽  
Mean Differences ◽  
Trim And Fill ◽  
Meta Analyses ◽  
Q Statistic

Abstract Endocrine-disrupting chemicals have become an issue of scientific and public discussion. Vinclozolin (VNZ) is a fungicide that competitively antagonizes the binding of natural androgens to their receptor, disturbing the function of tissues that are sensitive to these hormones, as is the case of the male reproductive organs. A systematic review with meta-analyses of rodent studies was conducted to answer the following question: Does exposure to VNZ affect sperm parameters and testicular/epididymal weight? The methodology was prespecified according to the Cochrane Handbook for Systematic Reviews and PRISMA recommendations. Sixteen articles met the inclusion criteria, comprising a total of 1189 animals. The risk of publication bias was assessed using the Trim and Fill adjustment, funnel plot, and Egger regression test. Heterogeneity and inconsistency across the findings were tested using the Q-statistic and I2 of Higgins, respectively. Sensitivity was also analyzed. Statistical analysis was performed on Comprehensive Meta-Analysis software (Version 2.0), using random models and weighted mean differences along with a 95% confidence interval. Sperm motility, counts, daily sperm production (evidence of publication bias), and epididymis weight were decreased in VNZ-treated animals. Exposure length and dose, as well as the time point of exposure, influenced the obtained results. Despite the moderate/high heterogeneity observed, the sensitivity analysis overall demonstrated the robustness of the findings. The quality scores of the included studies were superior to 4 in a total of 9, then classified as good. The obtained data corroborate the capability of VNZ exposure to disrupt spermatogenic output and compromise male fertility.

scholarly journals Publication Bias in Meta-Analyses from Psychology and Medicine: A Meta-Meta-Analysis

2017 ◽  
Author(s):  
Robbie Cornelis Maria van Aert ◽  
Jelte M. Wicherts ◽  
Marcel A. L. M. van Assen
Keyword(s):  
Publication Bias ◽  
Systematic Reviews ◽  
Large Scale ◽  
Effect Sizes ◽  
Psychological Bulletin ◽  
Data Set ◽  
Monte Carlo Simulation Study ◽  
True Effect Size ◽  
Meta Analyses ◽  
Psychology And Medicine

Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. The Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). No evidence of bias was obtained using the publication bias tests. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.

scholarly journals A Series of Meta-Analytic Tests of the Efficacy of Long-Term Psychoanalytic Psychotherapy

2020 ◽  
Vol 25 (1) ◽  
pp. 51-72 ◽  
Author(s):  
Christian Franz Josef Woll ◽  
Felix D. Schönbrodt
Keyword(s):  
Mental Disorders ◽  
Publication Bias ◽  
Psychiatric Symptoms ◽  
Meta Analysis ◽  
Research Process ◽  
Open Science ◽  
Effect Sizes ◽  
Psychoanalytic Psychotherapy ◽  
Meta Analyses

Abstract. Recent meta-analyses come to conflicting conclusions about the efficacy of long-term psychoanalytic psychotherapy (LTPP). Our first goal was to reproduce the most recent meta-analysis by Leichsenring, Abbass, Luyten, Hilsenroth, and Rabung (2013) who found evidence for the efficacy of LTPP in the treatment of complex mental disorders. Our replicated effect sizes were in general slightly smaller. Second, we conducted an updated meta-analysis of randomized controlled trials comparing LTPP (lasting for at least 1 year and 40 sessions) to other forms of psychotherapy in the treatment of complex mental disorders. We focused on a transparent research process according to open science standards and applied a series of elaborated meta-analytic procedures to test and control for publication bias. Our updated meta-analysis comprising 191 effect sizes from 14 eligible studies revealed small, statistically significant effect sizes at post-treatment for the outcome domains psychiatric symptoms, target problems, social functioning, and overall effectiveness (Hedges’ g ranging between 0.24 and 0.35). The effect size for the domain personality functioning (0.24) was not significant ( p = .08). No signs for publication bias could be detected. In light of a heterogeneous study set and some methodological shortcomings in the primary studies, these results should be interpreted cautiously. In conclusion, LTPP might be superior to other forms of psychotherapy in the treatment of complex mental disorders. Notably, our effect sizes represent the additional gain of LTPP versus other forms of primarily long-term psychotherapy. In this case, large differences in effect sizes are not to be expected.

scholarly journals Childhood maltreatment and suicide attempts in prisoners: a systematic meta-analytic review

2019 ◽  
Vol 50 (1) ◽  
pp. 1-10 ◽  
Author(s):  
Ioannis Angelakis ◽  
Jennifer L. Austin ◽  
Patricia Gooding
Keyword(s):  
Publication Bias ◽  
Childhood Maltreatment ◽  
Emotional Abuse ◽  
Suicide Attempts ◽  
Random Effect ◽  
Traumatic Experiences ◽  
Analytic Review ◽  
Physical Neglect ◽  
Trim And Fill ◽  
Meta Analyses

AbstractIn the past decade, the links between core types of childhood maltreatment and suicidal acts have become an increasingly important area of investigation. However, no meta-analytic review has examined this relationship in prisoners. We undertook the first systematic meta-analytic review examining the link between childhood maltreatment and suicide attempts in prisoners to redress this important gap. We searched Medline, PsychINFO, Embase, Web of Science and CINAHL from inception until August 2019. Meta-analyses using random effect models were applied, and heterogeneity was quantified using the I2 statistic. Publication bias and risk of bias across studies were assessed. We identified 24 studies comprising 16 586 prisoners. The rates of different types of childhood maltreatment ranged between 29% and 68% [95% confidence interval (CI) 18–81%]. The rate of suicide attempts in prisoners was 23% (95% CI 18–27%). Main results demonstrated that sexual abuse [odds ratio (OR) 2.68, 95% CI 1.86–3.86], physical abuse (OR 2.16, 95% CI 1.60–2.91), emotional abuse (OR 2.70, 95% CI 1.92–3.79), emotional neglect (OR 2.29, 95% CI 1.69–3.10), physical neglect (OR 1.57, 95% CI 1.27–1.94) and combined abuse (OR 3.09, 95% CI 2.14–4.45) were strongly associated with suicide attempts in prisoners. There was an indication of publication bias. Duval and Tweedie's trim-and-fill method was applied, which increased the odds for suicide attempts. Given the high rates of prison suicide deaths and suicide attempts, our findings suggest an urgent need for targeted suicide prevention priorities for prisoners, with a particular focus on ameliorating the effects of childhood traumatic experiences on suicidal prisoners.

scholarly journals Using Selection Models to Assess Sensitivity to Publication Bias: A Tutorial and Call for More Routine Use

2021 ◽  
Author(s):  
Maximilian Maier ◽  
Tyler VanderWeele ◽  
Maya B Mathur
Keyword(s):  
Publication Bias ◽  
Selection Models ◽  
Bias Assessment ◽  
Funnel Plot ◽  
Reporting Practices ◽  
Point Estimates ◽  
Trim And Fill ◽  
Meta Analyses ◽  
Effect Heterogeneity ◽  
User Friendly

In meta-analyses, it is critical to assess the extent to which publication bias might have compromised the results. Classical methods based on the funnel plot, including Egger’s test and Trim-and-Fill, have become the de facto default methods to do so, with a large majority of recent meta-analyses in top medical journals (85%) assessing for publication bias exclusively using these methods. However, these classical funnel plot methods have important limitations when used as the sole means of assessing publication bias: they essentially assume that the publication process favors large point estimates for small studies and does not affect the largest studies, and they can perform poorly when effects are heterogeneous. In light of these limitations, we recommend that meta-analyses routinely apply other publication bias methods in addition to or instead of classical funnel plot methods. To this end, we describe how to use and interpret selection models. These methods make the often more realistic assumption that publication bias favors ``statistically significant'' results and that also directly accommodate effect heterogeneity. Selection models are well-established in the statistics literature and are supported by user-friendly software, yet remain rarely reported in many disciplines. We use previously published meta-analyses to demonstrate that selection models can yield insights that extend beyond those provided by funnel plot methods, suggesting the importance of establishing more comprehensive reporting practices for publication bias assessment.

scholarly journals A Trim and Fill Examination of the Extent of Publication Bias in Communication Research

2019 ◽  
Author(s):  
Christopher Carpenter
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Communication Research ◽  
Academic Journal ◽  
Worked Example ◽  
Trim And Fill ◽  
Meta Analyses ◽  
The Impact

Publication bias can occur when a study is not published in an academic journal because the study did not find a statistically significant result. It can bias meta-analytic estimates upwards because the meta-analyses are missing studies that provide small estimates of the effect size under study. This paper describes the trim and fill technique of estimating the impact of publication bias on meta-analyses (Duval and Tweedie, 2000a, 2000b) and provides a worked example. A sample of 68 meta-analyses of communication research were tested for publication bias using the trim and fill technique. The results suggest that most communication meta-analyses are not substantially affected by publication bias.

Sign in / Sign up

Export Citation Format

Share Document