scholarly journals Publication Bias in Meta-Analyses from Psychology and Medicine: A Meta-Meta-Analysis

2017 ◽  
Author(s):  
Robbie Cornelis Maria van Aert ◽  
Jelte M. Wicherts ◽  
Marcel A. L. M. van Assen
Keyword(s):  
Publication Bias ◽  
Systematic Reviews ◽  
Large Scale ◽  
Effect Sizes ◽  
Psychological Bulletin ◽  
Data Set ◽  
Monte Carlo Simulation Study ◽  
True Effect Size ◽  
Meta Analyses ◽  
Psychology And Medicine

Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. The Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). No evidence of bias was obtained using the publication bias tests. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.

Identification of and Correction for Publication Bias: Comment

2019 ◽  
Author(s):  
Amanda Kvarven ◽  
Eirik Strømland ◽  
Magnus Johannesson
Keyword(s):  
Publication Bias ◽  
False Positive ◽  
Large Scale ◽  
Meta Analysis ◽  
False Positive Rate ◽  
Effect Sizes ◽  
Replication Studies ◽  
Moderate Reduction ◽  
Positive Rate ◽  
Meta Analyses

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

scholarly journals Dissertation R.C.M. van Aert

2018 ◽  
Author(s):  
Robbie Cornelis Maria van Aert
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Meta Analysis ◽  
Original Study ◽  
The Other ◽  
Effect Sizes ◽  
True Effect Size ◽  
True Effect ◽  
Meta Analyses ◽  
Q Statistic

More and more scientific research gets published nowadays, asking for statistical methods that enable researchers to get an overview of the literature in a particular research field. For that purpose, meta-analysis methods were developed that can be used for statistically combining the effect sizes from independent primary studies on the same topic. My dissertation focuses on two issues that are crucial when conducting a meta-analysis: publication bias and heterogeneity in primary studies’ true effect sizes. Accurate estimation of both the meta-analytic effect size as well as the between-study variance in true effect size is crucial since the results of meta-analyses are often used for policy making. Publication bias distorts the results of a meta-analysis since it refers to situations where publication of a primary study depends on its results. We developed new meta-analysis methods, p-uniform and p-uniform*, which estimate effect sizes corrected for publication bias and also test for publication bias. Although the methods perform well in many conditions, these and the other existing methods are shown not to perform well when researchers use questionable research practices. Additionally, when publication bias is absent or limited, traditional methods that do not correct for publication bias outperform p¬-uniform and p-uniform*. Surprisingly, we found no strong evidence for the presence of publication bias in our pre-registered study on the presence of publication bias in a large-scale data set consisting of 83 meta-analyses and 499 systematic reviews published in the fields of psychology and medicine. We also developed two methods for meta-analyzing a statistically significant published original study and a replication of that study, which reflects a situation often encountered by researchers. One method is a frequentist whereas the other method is a Bayesian statistical method. Both methods are shown to perform better than traditional meta-analytic methods that do not take the statistical significance of the original study into account. Analytical studies of both methods also show that sometimes the original study is better discarded for optimal estimation of the true effect size. Finally, we developed a program for determining the required sample size in a replication analogous to power analysis in null hypothesis testing. Computing the required sample size with the method revealed that large sample sizes (approximately 650 participants) are required to be able to distinguish a zero from a small true effect.Finally, in the last two chapters we derived a new multi-step estimator for the between-study variance in primary studies’ true effect sizes, and examined the statistical properties of two methods (Q-profile and generalized Q-statistic method) to compute the confidence interval of the between-study variance in true effect size. We proved that the multi-step estimator converges to the Paule-Mandel estimator which is nowadays one of the recommended methods to estimate the between-study variance in true effect sizes. Two Monte-Carlo simulation studies showed that the coverage probabilities of Q-profile and generalized Q-statistic method can be substantially below the nominal coverage rate if the assumptions underlying the random-effects meta-analysis model were violated.

scholarly journals Estimates of Between-Study Heterogeneity for 705 Meta-Analyses Reported in Psychological Bulletin From 1990-2013

2017 ◽  
Author(s):  
Sara van Erp ◽  
Josine Verhagen ◽  
Raoul P P P Grasman ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Monte Carlo ◽  
Prior Distribution ◽  
Open Science ◽  
Effect Sizes ◽  
Psychological Bulletin ◽  
Data Set ◽  
Study Heterogeneity ◽  
Science Framework ◽  
Meta Analyses ◽  
Q Statistic

We present a data set containing 705 between-study heterogeneity estimates as reported in 61 articles published in Psychological Bulletin from 1990-2013. The data set also includes information about the number and type of effect sizes, the Q-statistic, and publication bias. The data set is stored in the Open Science Framework repository and can be used for several purposes: (1) to compare a specific heterogeneity estimate to the distribution of between-study heterogeneity estimates in psychology; (2) to construct an informed prior distribution for the between-study heterogeneity in psychology; (3) to obtain realistic population values for Monte Carlo simulations investigating the performance of meta-analytic methods.

scholarly journals Comparing Meta-Analyses and Pre-Registered Multiple Labs Replication Projects

2019 ◽  
Author(s):  
Amanda Kvarven ◽  
Eirik Strømland ◽  
Magnus Johannesson
Keyword(s):  
Publication Bias ◽  
Large Scale ◽  
Research Evidence ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Selective Reporting ◽  
Behavioral Sciences ◽  
Scientific Results ◽  
Meta Analyses

Many researchers rely on meta-analysis to summarize research evidence. However, recent replication projects in the behavioral sciences suggest that effect sizes of original studies are overestimated, and this overestimation is typically attributed to publication bias and selective reporting of scientific results. As the validity of meta-analyses depends on the primary studies, there is a concern that systematic overestimation of effect sizes may translate into biased meta-analytic effect sizes. We compare the results of meta-analyses to large-scale pre-registered replications in psychology carried out at multiple labs. The multiple labs replications provide relatively precisely estimated effect sizes, which do not suffer from publication bias or selective reporting. Searching the literature, 17 meta-analyses – spanning more than 1,200 effect sizes and more than 370,000 participants - on the same topics as multiple labs replications are identified. We find that the meta-analytic effect sizes are significantly different from the replication effect sizes for 12 out of the 17 meta-replication pairs. These differences are systematic and on average meta-analytic effect sizes are about three times as large as the replication effect sizes.

How to Detect Publication Bias in Psychological Research

2019 ◽  
Vol 227 (4) ◽  
pp. 261-279 ◽  
Author(s):  
Frank Renkewitz ◽  
Melanie Keiner
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Statistical Power ◽  
Type I Error ◽  
Psychological Research ◽  
Type I ◽  
True Effect Size ◽  
Questionable Research Practices ◽  
True Effect ◽  
Meta Analyses

Abstract. Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates. Both of these problems lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect such bias in meta-analytic results. We present an evaluation of the performance of six of these tools. To assess the Type I error rate and the statistical power of these methods, we simulated a large variety of literatures that differed with regard to true effect size, heterogeneity, number of available primary studies, and sample sizes of these primary studies; furthermore, simulated studies were subjected to different degrees of publication bias. Our results show that across all simulated conditions, no method consistently outperformed the others. Additionally, all methods performed poorly when true effect sizes were heterogeneous or primary studies had a small chance of being published, irrespective of their results. This suggests that in many actual meta-analyses in psychology, bias will remain undiscovered no matter which detection method is used.

scholarly journals Application of the ESMO-Magnitude of Clinical Benefit Scale (V.1.1) to the field of early breast cancer therapies

ESMO Open ◽  
2020 ◽  
Vol 5 (5) ◽  
pp. e000743 ◽  
Author(s):  
Shani Paluch-Shimon ◽  
Nathan I Cherny ◽  
Elisabeth G E de Vries ◽  
Urania Dafni ◽  
Martine J Piccart ◽  
...  
Keyword(s):  
Breast Cancer ◽  
Early Breast Cancer ◽  
Clinical Benefit ◽  
Large Scale ◽  
Disease Free Survival ◽  
Cancer Therapies ◽  
Data Set ◽  
Clinical Scenarios ◽  
Meta Analyses

Click here to listen to the PodcastBackgroundThe European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS) is a validated value scale for solid tumour anticancer treatments. Form 1 of the ESMO-MCBS, used to grade therapies with curative intent including adjuvant therapies, has only been evaluated for a limited number of studies. This is the first large-scale field testing in early breast cancer to assess the applicability of the scale to this data set and the reasonableness of derived scores and to identify any shortcomings to be addressed in future modifications of the scale.MethodRepresentative key studies and meta-analyses of the major modalities of adjuvant systemic therapy of breast cancer were identified for each of the major clinical scenarios (HER2-positive, HER2-negative, endocrine-responsive) and were graded with form 1 of the ESMO-MCBS. These generated scores were reviewed by a panel of experts for reasonableness. Shortcomings and issues related to the application of the scale and interpretation of results were identified and critically evaluated.ResultsSixty-five studies were eligible for evaluation: 59 individual studies and 6 meta-analyses. These studies incorporated 101 therapeutic comparisons, 61 of which were scorable. Review of the generated scores indicated that, with few exceptions, they generally reflected contemporary standards of practice. Six shortcomings were identified related to grading based on disease-free survival (DFS), lack of information regarding acute and long-term toxicity and an inability to grade single-arm de-escalation scales.ConclusionsForm 1 of the ESMO-MCBS is a robust tool for the evaluation of the magnitude of benefit studies in early breast cancer. The scale can be further improved by addressing issues related to grading based on DFS, annotating grades with information regarding acute and long-term toxicity and developing an approach to grade single-arm de-escalation studies.

An Analysis of “Publication Bias” in the Travel, Tourism, and Hospitality Research

2017 ◽  
Vol 6 (4) ◽  
pp. 19-37
Author(s):  
Atila Yüksel ◽  
Ekrem Tufan
Keyword(s):  
Publication Bias ◽  
Systematic Reviews ◽  
Research Articles ◽  
Null Results ◽  
Tourism And Hospitality ◽  
Meta Analyses

This article examines whether studies with favorable or statistically significant outcomes are more likely to be published than studies with null results. Should such a publication tendency be in the form of favoring significant findings exist, then the integrity of science, suggestions and conclusions becomes controversial. This also includes those particularly drawn from meta-analyses and systematic reviews. Drawing on a sample of research articles, an examination was undertaken to determine whether studies reporting significant findings were published more. Additional analyses were conducted to examine the validity of reject/support decisions in relation to null hypotheses tested in these studies. The share of the published articles, in which null hypotheses were rejected, was found to be much larger (81%). Interestingly however, calculated power levels and actual samples sizes of these studies were too small to confidently reject/support null hypotheses. Implications for research are discussed in the concluding section of the article.

scholarly journals Is psychotherapy effective? A re-analysis of treatments for depression

2018 ◽  
Vol 28 (03) ◽  
pp. 268-274 ◽  
Author(s):  
T. Munder ◽  
C. Flückiger ◽  
F. Leichsenring ◽  
A. A. Abbass ◽  
M. J. Hilsenroth ◽  
...  
Keyword(s):  
Natural History ◽  
Publication Bias ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Wait List ◽  
Control Groups ◽  
Data Set ◽  
A Value ◽  
History Of ◽  
History Of Depression

AbstractAimsThe aim of this study was to reanalyse the data from Cuijpers et al.'s (2018) meta-analysis, to examine Eysenck's claim that psychotherapy is not effective. Cuijpers et al., after correcting for bias, concluded that the effect of psychotherapy for depression was small (standardised mean difference, SMD, between 0.20 and 0.30), providing evidence that psychotherapy is not as effective as generally accepted.MethodsThe data for this study were the effect sizes included in Cuijpers et al. (2018). We removed outliers from the data set of effects, corrected for publication bias and segregated psychotherapy from other interventions. In our study, we considered wait-list (WL) controls as the most appropriate estimate of the natural history of depression without intervention.ResultsThe SMD for all interventions and for psychotherapy compared to WL controls was approximately 0.70, a value consistent with past estimates of the effectiveness of psychotherapy. Psychotherapy was also more effective than care-as-usual (SMD = 0.31) and other control groups (SMD = 0.43).ConclusionsThe re-analysis reveals that psychotherapy for adult patients diagnosed with depression is effective.

Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia

2005 ◽  
Vol 20 (8) ◽  
pp. 550-553 ◽  
Author(s):  
José Luis R. Martin ◽  
Víctor Pérez ◽  
Montse Sacristán ◽  
Enric Álvarez
Keyword(s):  
Publication Bias ◽  
Systematic Reviews ◽  
Meta Analysis ◽  
Grey Literature ◽  
Risk Of Bias ◽  
Medical Databases ◽  
Study Results ◽  
Heterogeneous Nature ◽  
Meta Analyses ◽  
Literature Material

AbstractSystematic reviews in mental health have become useful tools for health professionals in view of the massive amount and heterogeneous nature of biomedical information available today. In order to determine the risk of bias in the studies evaluated and to avoid bias in generalizing conclusions from the reviews it is therefore important to use a very strict methodology in systematic reviews. One bias which may affect the generalization of results is publication bias, which is determined by the nature and direction of the study results. To control or minimize this type of bias, the authors of systematic reviews undertake comprehensive searches of medical databases and expand on the findings, often undertaking searches of grey literature (material which is not formally published). This paper attempts to show the consequences (and risk) of generalizing the implications of grey literature in the control of publication bias, as was proposed in a recent systematic work. By repeating the analyses for the same outcome from three different systematic reviews that included both published and grey literature our results showed that confusion between grey literature and publication bias may affect the results of a concrete meta-analysis.

Sign in / Sign up

Export Citation Format

Share Document