A Series of Meta-Analytic Tests of the Efficacy of Long-Term Psychoanalytic Psychotherapy

Author(s):

Christian Franz Josef Woll ◽

Felix D. Schönbrodt

Keyword(s):

Mental Disorders ◽

Publication Bias ◽

Psychiatric Symptoms ◽

Meta Analysis ◽

Research Process ◽

Open Science ◽

Effect Sizes ◽

Psychoanalytic Psychotherapy ◽

Recent meta-analyses come to conflicting conclusions about the efficacy of long-term psychoanalytic psychotherapy (LTPP). Our first goal was to reproduce the most recent meta-analysis by Leichsenring, Abbass, Luyten, Hilsenroth, and Rabung (2013) who found evidence for the efficacy of LTPP in the treatment of complex mental disorders. Our replicated effect sizes were in general slightly smaller. Second, we conducted an updated meta-analysis of randomized controlled trials comparing LTPP (lasting for at least one year and 40 sessions) to other forms of psychotherapy in the treatment of complex mental disorders. We focused on a transparent research process according to open science standards and applied a series of elaborated meta-analytic procedures to test and control for publication bias. Our updated meta-analysis comprising 191 effect sizes from 14 eligible studies revealed small, statistically significant effect sizes at post-treatment for the outcome domains psychiatric symptoms, target problems, social functioning, and overall effectiveness (Hedges’ g ranging between 0.24 and 0.35). The effect size for the domain personality functioning (0.24) was not significant (p = .08). No signs for publication bias could be detected. In light of a heterogeneous study set and some methodological shortcomings in the primary studies, these results should be interpreted cautiously. In conclusion, LTPP might be superior to other forms of psychotherapy in the treatment of complex mental disorders. Notably, our effect sizes represent the additional gain of LTPP vs. other forms of primarily long-term psychotherapy. In this case, large differences in effect sizes are not to be expected.

A Series of Meta-Analytic Tests of the Efficacy of Long-Term Psychoanalytic Psychotherapy

European Psychologist ◽

10.1027/1016-9040/a000385 ◽

2020 ◽

Vol 25 (1) ◽

pp. 51-72 ◽

Cited By ~ 4

Author(s):

Christian Franz Josef Woll ◽

Felix D. Schönbrodt

Keyword(s):

Mental Disorders ◽

Publication Bias ◽

Psychiatric Symptoms ◽

Meta Analysis ◽

Research Process ◽

Open Science ◽

Effect Sizes ◽

Psychoanalytic Psychotherapy ◽

Abstract. Recent meta-analyses come to conflicting conclusions about the efficacy of long-term psychoanalytic psychotherapy (LTPP). Our first goal was to reproduce the most recent meta-analysis by Leichsenring, Abbass, Luyten, Hilsenroth, and Rabung (2013) who found evidence for the efficacy of LTPP in the treatment of complex mental disorders. Our replicated effect sizes were in general slightly smaller. Second, we conducted an updated meta-analysis of randomized controlled trials comparing LTPP (lasting for at least 1 year and 40 sessions) to other forms of psychotherapy in the treatment of complex mental disorders. We focused on a transparent research process according to open science standards and applied a series of elaborated meta-analytic procedures to test and control for publication bias. Our updated meta-analysis comprising 191 effect sizes from 14 eligible studies revealed small, statistically significant effect sizes at post-treatment for the outcome domains psychiatric symptoms, target problems, social functioning, and overall effectiveness (Hedges’ g ranging between 0.24 and 0.35). The effect size for the domain personality functioning (0.24) was not significant ( p = .08). No signs for publication bias could be detected. In light of a heterogeneous study set and some methodological shortcomings in the primary studies, these results should be interpreted cautiously. In conclusion, LTPP might be superior to other forms of psychotherapy in the treatment of complex mental disorders. Notably, our effect sizes represent the additional gain of LTPP versus other forms of primarily long-term psychotherapy. In this case, large differences in effect sizes are not to be expected.

Identification of and Correction for Publication Bias: Comment

10.31222/osf.io/dh87m ◽

2019 ◽

Author(s):

Amanda Kvarven ◽

Eirik Strømland ◽

Magnus Johannesson

Keyword(s):

Publication Bias ◽

False Positive ◽

Large Scale ◽

Meta Analysis ◽

False Positive Rate ◽

Effect Sizes ◽

Replication Studies ◽

Moderate Reduction ◽

Positive Rate ◽

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

Long-term psychodynamic psychotherapy in complex mental disorders: update of a meta-analysis

The British Journal of Psychiatry ◽

10.1192/bjp.bp.110.082776 ◽

2011 ◽

Vol 199 (1) ◽

pp. 15-22 ◽

Cited By ~ 174

Author(s):

Falk Leichsenring ◽

Sven Rabung

Keyword(s):

Mental Disorders ◽

Psychodynamic Psychotherapy ◽

Meta Analysis ◽

Dose Effect ◽

Effect Sizes ◽

Inclusion Criteria ◽

Comparative Efficacy ◽

Short Term ◽

Prospective Design

BackgroundDose–effect relationship data suggest that short-term psychotherapy is insufficient for many patients with chronic distress or personality disorders (complex mental disorders).AimsTo examine the comparative efficacy of long-term psychodynamic psychotherapy (LTPP) in complex mental disorders.MethodWe conducted a meta-analysis of controlled trials of LTPP fulfilling the following inclusion criteria: therapy lasting for at least a year or 50 sessions; active comparison conditions; prospective design; reliable and valid outcome measures; treatments terminated. Ten studies with 971 patients were included.ResultsBetween-group effect sizes in favour of LTPP compared with less intensive (lower dose) forms of psychotherapy ranged between 0.44 and 0.68.ConclusionsResults suggest that LTPP is superior to less intensive forms of psychotherapy in complex mental disorders. Further research on long-term psychotherapy is needed, not only for psychodynamic psychotherapy, but also for other therapies.

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

10.31234/osf.io/ytsvw ◽

2018 ◽

Author(s):

Michele B. Nuijten ◽

Marcel A. L. M. van Assen ◽

Hilde Augusteijn ◽

Elise Anne Victoire Crompvoets ◽

Jelte M. Wicherts

Keyword(s):

Publication Bias ◽

Effect Size ◽

Meta Analysis ◽

Convincing Evidence ◽

Small Study ◽

Effect Sizes ◽

Medium Effect ◽

Average Effect ◽

Average Effect Size ◽

In this meta-study, we analyzed 2,442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson’s correlation of .26, and the median sample size was 60. Furthermore, across primary studies, we found a median power of 11.9% to detect a small effect, 54.5% to detect a medium effect, and 93.9% to detect a large effect. We documented differences in average effect size and median estimated power between different types of in intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). On average, across all meta-analyses (but not in every meta-analysis), we found evidence for small study effects, potentially indicating publication bias and overestimated effects. We found no differences in small study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We conclude that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

Unresolved Heterogeneity in Meta-Analysis: Combined Construct Invalidity, Confounding, and Other Challenges to Understanding Mean Effect Sizes

Human Communication Research ◽

10.1093/hcr/hqz019 ◽

2020 ◽

Vol 46 (2-3) ◽

pp. 343-354 ◽

Author(s):

Timothy R Levine ◽

René Weber

Keyword(s):

Content Analysis ◽

Publication Bias ◽

Effect Size ◽

Meta Analysis ◽

Effect Sizes ◽

Size Distributions ◽

Questionable Research Practices ◽

Graphical Displays ◽

Meta Analyses ◽

Effect Heterogeneity

Abstract We examined the interplay between how communication researchers use meta-analyses to make claims and the prevalence, causes, and implications of unresolved heterogeneous findings. Heterogeneous findings can result from substantive moderators, methodological artifacts, and combined construct invalidity. An informal content analysis of meta-analyses published in four elite communication journals revealed that unresolved between-study effect heterogeneity was ubiquitous. Communication researchers mainly focus on computing mean effect sizes, to the exclusion of how effect sizes in primary studies are distributed and of what might be driving effect size distributions. We offer four recommendations for future meta-analyses. Researchers are advised to be more diligent and sophisticated in testing for heterogeneity. We encourage greater description of how effects are distributed, coupled with greater reliance on graphical displays. We council greater recognition of combined construct invalidity and advocate for content expertise. Finally, we endorse greater awareness and improved tests for publication bias and questionable research practices.

Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

Journal of Intelligence ◽

10.3390/jintelligence8040036 ◽

2020 ◽

Vol 8 (4) ◽

pp. 36

Author(s):

Michèle B. Nuijten ◽

Marcel A. L. M. van Assen ◽

Hilde E. M. Augusteijn ◽

Elise A. V. Crompvoets ◽

Jelte M. Wicherts

Keyword(s):

Publication Bias ◽

Effect Size ◽

Meta Analysis ◽

Convincing Evidence ◽

Small Study ◽

Effect Sizes ◽

Medium Effect ◽

Average Effect ◽

Average Effect Size ◽

In this meta-study, we analyzed 2442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson’s correlation of 0.26, and the median sample size was 60. Furthermore, across primary studies, we found a median power of 11.9% to detect a small effect, 54.5% to detect a medium effect, and 93.9% to detect a large effect. We documented differences in average effect size and median estimated power between different types of intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). On average, across all meta-analyses (but not in every meta-analysis), we found evidence for small-study effects, potentially indicating publication bias and overestimated effects. We found no differences in small-study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We concluded that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

Comparing the Ability of MMPI-2 and MMPI-2-RF Validity Scales to Detect Feigning: A Meta-Analysis

Assessment ◽

10.1177/10731911211067535 ◽

2022 ◽

pp. 107319112110675

Author(s):

Maria Aparcero ◽

Emilie H. Picard ◽

Alicia Nijdam-Jones ◽

Barry Rosenfeld

Keyword(s):

Study Design ◽

Minnesota Multiphasic Personality Inventory ◽

Psychiatric Symptoms ◽

Meta Analysis ◽

Personality Inventory ◽

Effect Sizes ◽

Validity Scales ◽

Symptom Exaggeration ◽

Meta Analyses ◽

Medical Symptoms

Several meta-analyses of the Minnesota Multiphasic Personality Inventory-2 (MMPI-2) and Minnesota Multiphasic Personality Inventory-2 Restructured Form (MMPI-2-RF) have examined these instruments’ ability to detect symptom exaggeration or feigning. However, limited research has directly compared whether the scales across these two instruments are equally effective. This study used a moderated meta-analysis to compare 109 MMPI-2 and 41 MMPI-2-RF feigning studies, 83 (56.46%) of which were not included in previous meta-analyses. Although there were differences between the two test versions, with most MMPI-2 validity scales generating larger effect sizes than the corresponding MMPI-2-RF scales, these differences were not significant after controlling for study design and type of symptoms being feigned. Additional analyses showed that the F and Fp-r scales generated the largest effect sizes in identifying feigned psychiatric symptoms, while the FBS and RBS were better at detecting exaggerated medical symptoms. The findings indicate that the MMPI-2 validity scales and their MMPI-2-RF counterparts were similarly effective in differentiating genuine responders from those exaggerating or feigning psychiatric and medical symptoms. These results provide reassurance for the use of both the MMPI-2 and MMPI-2-RF in settings where symptom exaggeration or feigning is likely. Findings are discussed in the context of the recently released MMPI-3.

Dissertation R.C.M. van Aert

10.31222/osf.io/eqhjd ◽

2018 ◽

Cited By ~ 2

Author(s):

Robbie Cornelis Maria van Aert

Keyword(s):

Publication Bias ◽

Effect Size ◽

Meta Analysis ◽

Original Study ◽

The Other ◽

Effect Sizes ◽

True Effect Size ◽

True Effect ◽

Meta Analyses ◽

Q Statistic

More and more scientific research gets published nowadays, asking for statistical methods that enable researchers to get an overview of the literature in a particular research field. For that purpose, meta-analysis methods were developed that can be used for statistically combining the effect sizes from independent primary studies on the same topic. My dissertation focuses on two issues that are crucial when conducting a meta-analysis: publication bias and heterogeneity in primary studies’ true effect sizes. Accurate estimation of both the meta-analytic effect size as well as the between-study variance in true effect size is crucial since the results of meta-analyses are often used for policy making. Publication bias distorts the results of a meta-analysis since it refers to situations where publication of a primary study depends on its results. We developed new meta-analysis methods, p-uniform and p-uniform*, which estimate effect sizes corrected for publication bias and also test for publication bias. Although the methods perform well in many conditions, these and the other existing methods are shown not to perform well when researchers use questionable research practices. Additionally, when publication bias is absent or limited, traditional methods that do not correct for publication bias outperform p¬-uniform and p-uniform*. Surprisingly, we found no strong evidence for the presence of publication bias in our pre-registered study on the presence of publication bias in a large-scale data set consisting of 83 meta-analyses and 499 systematic reviews published in the fields of psychology and medicine. We also developed two methods for meta-analyzing a statistically significant published original study and a replication of that study, which reflects a situation often encountered by researchers. One method is a frequentist whereas the other method is a Bayesian statistical method. Both methods are shown to perform better than traditional meta-analytic methods that do not take the statistical significance of the original study into account. Analytical studies of both methods also show that sometimes the original study is better discarded for optimal estimation of the true effect size. Finally, we developed a program for determining the required sample size in a replication analogous to power analysis in null hypothesis testing. Computing the required sample size with the method revealed that large sample sizes (approximately 650 participants) are required to be able to distinguish a zero from a small true effect.Finally, in the last two chapters we derived a new multi-step estimator for the between-study variance in primary studies’ true effect sizes, and examined the statistical properties of two methods (Q-profile and generalized Q-statistic method) to compute the confidence interval of the between-study variance in true effect size. We proved that the multi-step estimator converges to the Paule-Mandel estimator which is nowadays one of the recommended methods to estimate the between-study variance in true effect sizes. Two Monte-Carlo simulation studies showed that the coverage probabilities of Q-profile and generalized Q-statistic method can be substantially below the nominal coverage rate if the assumptions underlying the random-effects meta-analysis model were violated.

Times are Changing, Bias isn’t: A Meta-Meta-Analysis on Publication Bias Detection Practices, Prevalence Rates, and Predictors in Industrial/Organizational Psychology

10.31234/osf.io/mtv2h ◽

2020 ◽

Author(s):

Magdalena Siegel ◽

Junia Eder ◽

Jelte M. Wicherts ◽

Jakob Pietschnig

Keyword(s):

Publication Bias ◽

Meta Analysis ◽

Open Science ◽

Detection Methods ◽

Applied Psychology ◽

Psychological Science ◽

Industrial Organizational Psychology ◽

Prevalence Estimates ◽

Bias Detection ◽

Inflated or outright false effects plague Psychological Science, but advances in the identification of dissemination biases in general and publication bias in particular have helped in dealing with biased effects in the literature. However, the application of publication bias detection methods appears to be not equally prevalent across subdisciplines. It has been suggested that particularly in I/O Psychology, appropriate publication bias detection methods are underused. In this meta-meta-analysis, we present prevalence estimates, predictors, and time trends of publication bias in 128 meta-analyses that were published in the Journal of Applied Psychology (7,263 effect sizes, 3,000,000+ participants). Moreover, we reanalyzed data of 87 meta-analyses and applied nine standard and more modern publication bias detection methods. We show that (i) the bias detection method applications are underused (only 41% of meta-analyses use at least one method) but have increased in recent years, (ii) those meta-analyses that apply such methods now use more, but mostly inappropriate methods, and (iii) the prevalence of publication bias is disconcertingly high (15% to 20% show severe, 33% to 48% some bias indication) but mostly remains undetected. Although our results indicate somewhat of a trend towards higher bias awareness, they also indicate that concerns about publication bias in I/O Psychology are justified and researcher awareness about appropriate and state-of-the-art bias detection needs to be further increased. Embracing open science practices such as data sharing or study preregistration is needed to raise reproducibility and ultimately strengthen Psychological Science in general and I/O Psychology in particular.

Comparing Meta-Analyses and Pre-Registered Multiple Labs Replication Projects

10.31219/osf.io/brzwt ◽

2019 ◽

Author(s):

Amanda Kvarven ◽

Eirik Strømland ◽

Magnus Johannesson

Keyword(s):

Publication Bias ◽

Large Scale ◽

Research Evidence ◽

Meta Analysis ◽

Effect Sizes ◽

Selective Reporting ◽

Behavioral Sciences ◽

Scientific Results ◽

Many researchers rely on meta-analysis to summarize research evidence. However, recent replication projects in the behavioral sciences suggest that effect sizes of original studies are overestimated, and this overestimation is typically attributed to publication bias and selective reporting of scientific results. As the validity of meta-analyses depends on the primary studies, there is a concern that systematic overestimation of effect sizes may translate into biased meta-analytic effect sizes. We compare the results of meta-analyses to large-scale pre-registered replications in psychology carried out at multiple labs. The multiple labs replications provide relatively precisely estimated effect sizes, which do not suffer from publication bias or selective reporting. Searching the literature, 17 meta-analyses – spanning more than 1,200 effect sizes and more than 370,000 participants - on the same topics as multiple labs replications are identified. We find that the meta-analytic effect sizes are significantly different from the replication effect sizes for 12 out of the 17 meta-replication pairs. These differences are systematic and on average meta-analytic effect sizes are about three times as large as the replication effect sizes.