scholarly journals The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else is Known

2018 ◽  
Author(s):  
Robert Calin-Jageman ◽  
Geoff Cumming
Keyword(s):  
Confidence Intervals ◽  
Scientific Practice ◽  
Meta Analysis ◽  
Confirmation Bias ◽  
Open Science ◽  
Effect Sizes ◽  
Small Samples ◽  
Science Practices ◽  
New Statistics

&&& Now published in the American Statistician: https://amstat.tandfonline.com/doi/full/10.1080/00031305.2018.1518266 *** The "New Statistics" emphasizes effect sizes, confidence intervals, meta-analysis and the use of Open Science practices. We present 3 specific ways in which a New Statistics approach can help improve scientific practice: by reducing over-confidence in small samples, by reducing confirmation bias, and by fostering more cautious judgements of consistency.

scholarly journals Accelerating open science: The collaborative replications and education project (CREP)

2018 ◽  
Author(s):  
Gerit Pfuhl ◽  
Jon Grahe
Keyword(s):  
Behavioral Science ◽  
Research Training ◽  
Meta Analysis ◽  
Academic Research ◽  
Open Data ◽  
Open Science ◽  
Open Access Publishing ◽  
Quality Data ◽  
Education Project ◽  
Science Practices

Watch the VIDEO.Recent years have seen a revolution in publishing, and large support for open access publishing. There has been a slower acceptance and transition to other open science principles such as open data, open materials, and preregistration. To accelerate the transition and make open science the new standard, the collaborative replications and education project (CREP; http://osf.io/wfc6u/)) was launched in 2013, hosted on the Open Science Framework (osf.io). OSF is like a preprint, collecting partial data with each individual contributors project. CREP introduces open science at the start of academic research, facilitating student research training in open science and solidifying behavioral science results. The CREP team attempts to achieve this by inviting contributors to replicate one of several replication studies selected for scientific impact and suitability for undergraduates to complete during one academic term. Contributors follow clear protocols with students interacting with a CREP team that reviews the materials and video of the procedure to ensure quality data collection while students are learning science practices and methods. By combining multiple replications from undergraduates across the globe, the findings can be pooled to conduct meta-analysis and so contribute to generalizable and replicable research findings. CREP is careful to not interpret any single result. CREP has recently joined forces with the psychological science accelerator (PsySciAcc), a globally distributed network of psychological laboratories accelerating the accumulation of reliable and generalizable results in the behavioral sciences. The Department of Psychology at UiT is part of the network and has two ongoing CREP studies, maintaining open science practices early on. In this talk, we will present our experiences of conducting transparent replicable research, and experience with preprints from a supervisor and researcher perspective.

scholarly journals A Series of Meta-Analytic Tests of the Efficacy of Long-Term Psychoanalytic Psychotherapy

2020 ◽  
Vol 25 (1) ◽  
pp. 51-72 ◽  
Author(s):  
Christian Franz Josef Woll ◽  
Felix D. Schönbrodt
Keyword(s):  
Mental Disorders ◽  
Publication Bias ◽  
Psychiatric Symptoms ◽  
Meta Analysis ◽  
Research Process ◽  
Open Science ◽  
Effect Sizes ◽  
Psychoanalytic Psychotherapy ◽  
Meta Analyses

Abstract. Recent meta-analyses come to conflicting conclusions about the efficacy of long-term psychoanalytic psychotherapy (LTPP). Our first goal was to reproduce the most recent meta-analysis by Leichsenring, Abbass, Luyten, Hilsenroth, and Rabung (2013) who found evidence for the efficacy of LTPP in the treatment of complex mental disorders. Our replicated effect sizes were in general slightly smaller. Second, we conducted an updated meta-analysis of randomized controlled trials comparing LTPP (lasting for at least 1 year and 40 sessions) to other forms of psychotherapy in the treatment of complex mental disorders. We focused on a transparent research process according to open science standards and applied a series of elaborated meta-analytic procedures to test and control for publication bias. Our updated meta-analysis comprising 191 effect sizes from 14 eligible studies revealed small, statistically significant effect sizes at post-treatment for the outcome domains psychiatric symptoms, target problems, social functioning, and overall effectiveness (Hedges’ g ranging between 0.24 and 0.35). The effect size for the domain personality functioning (0.24) was not significant ( p = .08). No signs for publication bias could be detected. In light of a heterogeneous study set and some methodological shortcomings in the primary studies, these results should be interpreted cautiously. In conclusion, LTPP might be superior to other forms of psychotherapy in the treatment of complex mental disorders. Notably, our effect sizes represent the additional gain of LTPP versus other forms of primarily long-term psychotherapy. In this case, large differences in effect sizes are not to be expected.

An analysis of psychotherapy versus placebo studies

1983 ◽  
Vol 6 (2) ◽  
pp. 275-285 ◽  
Author(s):  
Leslie Prioleau ◽  
Martha Murdock ◽  
Nathan Brody
Keyword(s):  
Meta Analysis ◽  
Placebo Treatment ◽  
Convincing Evidence ◽  
Psychological Therapy ◽  
Self Report ◽  
Effect Sizes ◽  
Small Samples ◽  
Control Treatment ◽  
Zero Effect ◽  
Real Patients

AbstractSmith, Glass, and Miller (1980) have reported a meta-analysis of over 500 studies comparing some form of psychological therapy with a control condition. They report that when averaged over all dependent measures of outcome, psychological therapy is. 85 standard deviations better than the control treatment. We examined the subset of studies included in the Smith et al. metaanalysis that contained a psychotherapy and a placebo treatment. The median of the mean effect sizes for these 32 studies was. 15. There was a nonsignificant inverse relationship between mean outcome and the following: sample size, duration of therapy, use of measures of outcome other than undisguised self-report, measurement of outcome at follow-up, and use of real patients rather than subjects solicited for the purposes of participation in a research study. A qualitative analysis of the studies in terms of the type of patient involved indicates that those using psychiatric outpatients had essentially zero effect sizes and that none using psychiatric inpaticnts provide convincing evidence for psychotherapeutic effectiveness. The onty studies clearly demonstrating significant effects of psychotherapy were the ones that did not use real patients. For the most part, these studies involved small samples of subjects and brief treatments, occasionally described in quasibeliavioristic language. It was concluded that for real patients there is no evidence that the benefits of psychotherapy are greater than those of placebo treatment.

scholarly journals Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility

2021 ◽  
Author(s):  
Sophia C. Weissgerber ◽  
Matthias Brunmair ◽  
Ralf Rummer
Keyword(s):  
Recall Performance ◽  
Meta Analysis ◽  
Methodological Approach ◽  
Open Science ◽  
Point Of View ◽  
Outcome Variable ◽  
Science Practices ◽  
Level 3 ◽  
Analytical Requirements ◽  
Level 2

AbstractIn the 2018 meta-analysis of Educational Psychology Review entitled “Null effects of perceptual disfluency on learning outcomes in a text-based educational context” by Xie, Zhou, and Liu, we identify some errors and inconsistencies in both the methodological approach and the reported results regarding coding and effect sizes. While from a technical point of view the meta-analysis aligns with current meta-analytical guidelines (e.g., PRISMA) and conforms to general meta-analytical requirements (e.g., considering publication bias), it exemplifies certain insufficient practices in the creation and review of meta-analysis. We criticize the lack of transparency and negligence of open-science practices in the generation and reporting of results, which complicate evaluation of the meta-analytical reproducibility, especially given the flexibility in subjective choices regarding the analytical approach and the flexibility in creating the database. Here we present a framework applicable to pre- and post-publication review on improving the Methods Reproducibility of meta-analysis. Based on considerations of the transparency and openness (TOP)-guidlines (Nosek et al. Science 348: 1422–1425, 2015), the Reproducibility Enhancement Principles (REP; Stodden et al. Science 354:1240–1241, 2016), and recommendations by Lakens et al. (BMC Psychology 4: Article 24, 2016), we outline Computational Reproducibility (Level 1), Computational Verification (Level 2), Analysis Reproducibility (Level 3), and Outcome Reproducibility (Level 4). Applying reproducibility checks to TRANSFER performance as the chosen outcome variable, we found Xie’s and colleagues’ results to be (rather) robust. Yet, regarding RECALL performance and the moderator analysis, the identified problems raise doubts about the credibility of the reported results.

Sign in / Sign up

Export Citation Format

Share Document