Replication. Part I

2021 ◽  
pp. 116-129
Author(s):  
Ronald Laymon ◽  
◽  
Allan Franklin ◽  
Keyword(s):  
Case Studies ◽  
Systematic Uncertainty ◽  
Medical Sciences ◽  
Single Experiment ◽  
Considerable Debate ◽  
Replication Crisis ◽  
Crisis In Psychology ◽  
Considerable Detail

There has been considerable debate and analysis about the nature and extent of what many believe to be a replication crisis in psychology and other social sci­ences. And perhaps to a lesser degree in the medical sciences. Our approach to the nature and value of replication has been based on the idea that the overriding purpose of replication is to ferret out and identify confounding causes. In other words, to reduce systematic uncertainty. This has led us to understand replication in a broader sense than ordinarily understood. We have developed this approach in considerable detail in three recent books which include both abstract analysis and many case studies drawn primarily, but not exclusively, from physics. We have, for example, considered the difficulty of deciding whether a replication has been successful or has failed, the roles of null experiments, and episodes in which a single experiment has been sufficient to decide, or to further investigate, an issue. In this two-part essay we will review and summarize our approach and results.

Replication Part II

2021 ◽  
pp. 118-131
Author(s):  
Ronald Laymon ◽  
◽  
Allan Franklin ◽  
Keyword(s):  
Case Studies ◽  
Systematic Uncertainty ◽  
Medical Sciences ◽  
Single Experiment ◽  
Considerable Debate ◽  
Replication Crisis ◽  
Crisis In Psychology ◽  
Considerable Detail

There has been considerable debate and analysis about the nature and extent of what many believe to be a replication crisis in psychology and other social sci­ences. And perhaps to a lesser degree in the medical sciences. Our approach to the nature and value of replication has been based on the idea that the overriding purpose of replication is to ferret out and identify confounding causes. In other words, to reduce systematic uncertainty. This has led us to understand replication in a broader sense than ordinarily understood. We have developed this approach in considerable detail in three recent books which include both abstract analysis and many case studies drawn primarily, but not exclusively, from physics. We have, for example, considered the difficulty of deciding whether a replication has been successful or has failed, the roles of null experiments, and episodes in which a single experiment has been sufficient to decide, or to further investigate, an issue. In this two-part essay we will review and summarize our approach and results.

Psychology Exceptionalism and the Multiple Discovery of the Replication Crisis

2021 ◽  
pp. 108926802110465
Author(s):  
Nicole C. Nelson ◽  
Julie Chung ◽  
Kelsey Ichikawa ◽  
Momin M. Malik
Keyword(s):  
Subject Matter ◽  
Common Core ◽  
Early Years ◽  
Institutional Factors ◽  
Scientific Practices ◽  
Incentive Structure ◽  
Biomedical Sciences ◽  
Current Crisis ◽  
Replication Crisis ◽  
Crisis In Psychology

This article outlines what we call the “narrative of psychology exceptionalism” in commentaries on the replication crisis: many thoughtful commentaries link the current crisis to the specificity of psychology’s history, methods, and subject matter, but explorations of the similarities between psychology and other fields are comparatively thin. Historical analyses of the replication crisis in psychology further contribute to this exceptionalism by creating a genealogy of events and personalities that shares little in common with other fields. We aim to rebalance this narrative by examining the emergence and evolution of replication discussions in psychology alongside their emergence and evolution in biomedicine. Through a mixed-methods analysis of commentaries on replication in psychology and the biomedical sciences, we find that these conversations have, from the early years of the crisis, shared a common core that centers on concerns about the effectiveness of traditional peer review, the need for greater transparency in methods and data, and the perverse incentive structure of academia. Drawing on Robert Merton’s framework for analyzing multiple discovery in science, we argue that the nearly simultaneous emergence of this narrative across fields suggests that there are shared historical, cultural, or institutional factors driving disillusionment with established scientific practices.

scholarly journals Registered reports: an early example and analysis

PeerJ ◽  
2019 ◽  
Vol 7 ◽  
pp. e6232 ◽  
Author(s):  
Richard Wiseman ◽  
Caroline Watt ◽  
Diana Kornbrot
Keyword(s):  
Data Collection ◽  
Peer Review ◽  
Circumstantial Evidence ◽  
Focused Attention ◽  
Behavioral Sciences ◽  
Positive Role ◽  
Questionable Research Practices ◽  
Methodological Rigor ◽  
Replication Crisis ◽  
Crisis In Psychology

The recent ‘replication crisis’ in psychology has focused attention on ways of increasing methodological rigor within the behavioral sciences. Part of this work has involved promoting ‘Registered Reports’, wherein journals peer review papers prior to data collection and publication. Although this approach is usually seen as a relatively recent development, we note that a prototype of this publishing model was initiated in the mid-1970s by parapsychologist Martin Johnson in the European Journal of Parapsychology (EJP). A retrospective and observational comparison of Registered and non-Registered Reports published in the EJP during a seventeen-year period provides circumstantial evidence to suggest that the approach helped to reduce questionable research practices. This paper aims both to bring Johnson’s pioneering work to a wider audience, and to investigate the positive role that Registered Reports may play in helping to promote higher methodological and statistical standards.

scholarly journals Does health informatics have a replication crisis?

2018 ◽  
Vol 25 (8) ◽  
pp. 963-968 ◽  
Author(s):  
Enrico Coiera ◽  
Elske Ammenwerth ◽  
Andrew Georgiou ◽  
Farah Magrabi
Keyword(s):  
Health Informatics ◽  
Context Effects ◽  
Evidence Based ◽  
Medical Sciences ◽  
Replication Studies ◽  
Research Fields ◽  
Replication Crisis ◽  
The Cost ◽  
The Impact

Abstract Objective Many research fields, including psychology and basic medical sciences, struggle with poor reproducibility of reported studies. Biomedical and health informatics is unlikely to be immune to these challenges. This paper explores replication in informatics and the unique challenges the discipline faces. Methods Narrative review of recent literature on research replication challenges. Results While there is growing interest in re-analysis of existing data, experimental replication studies appear uncommon in informatics. Context effects are a particular challenge as they make ensuring replication fidelity difficult, and the same intervention will never quite reproduce the same result in different settings. Replication studies take many forms, trading-off testing validity of past findings against testing generalizability. Exact and partial replication designs emphasize testing validity while quasi and conceptual studies test generalizability of an underlying model or hypothesis with different methods or in a different setting. Conclusions The cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics. The benefits of replication include increased rigor in research, and the development of evaluation methods that distinguish the impact of context and the nonreproducibility of research. Taking replication seriously is essential if biomedical and health informatics is to be an evidence-based discipline.

scholarly journals Dealing with the replication crisis in psychological science: The contribution of Type M and Type S errors

2019 ◽  
Author(s):  
Giulia Bertoldo
Keyword(s):  
Statistical Inference ◽  
Design Analysis ◽  
Original Study ◽  
Type I ◽  
Retrospective Design ◽  
Error Magnitude ◽  
Replication Crisis ◽  
Crisis In Psychology ◽  
Type Ii Errors

The present work aims to analyze the replicability crisis in psychology with a focus on statistical inference. The main objective is to highlight the risks to beware when performing hypotheses tests in a Frequentist framework. In addition to the classic Type I and Type II errors, two other errors that are not commonly considered are Type M error (magnitude) and Type S error (sign), concerning the size and direction of the effects. The first chapter introduces Null Hypothesis Significance Testing (NHST), the prevalent approach to statistical inference in the social sciences, following a historical perspective and presenting also the approaches of the statisticians Ronald Fisher, Jerzy Neyman and Egon Pearson. The second chapter discusses the replicability crisis in psychology with an analysis of the origins, the factors that contributed to the crisis and solutions proposed for a change of direction. The third chapter analyzes the role of Type M and Type S errors in the replicability crisis. Studies with a high probability of committing these two types of errors could provide estimates of effects that are exaggerated and/or in the wrong direction. Two types of analysis are presented to examine these errors before conducting a study (prospective design analysis) or once the study has already been conducted (retrospective design analysis). The fourth chapter aims to link Type M and Type S errors with the decline effect, which is the observation that the magnitude of effects decreases with repeated replications. Although there may be multiple reasons behind the decline effect, a possible explanation is that the original study overestimated the effect. A case study illustrates how a retrospective design analysis of the original study can provide information on the probability of Type M and Type S errors and give support to the hypothesis of overestimation. The final chapter summarizes the contributions of Type M and Type S errors to the replication crisis and the role of a design analysis in planning studies and analyzing results.

scholarly journals Can hypnotic suggestibility be measured online?

2018 ◽  
Author(s):  
Bence Palfi ◽  
Peter Lush ◽  
Ryan Bradley Scott ◽  
Zoltan Dienes
Keyword(s):  
Psychometric Properties ◽  
Large Scale ◽  
Experimental Manipulation ◽  
Online Assessment ◽  
Heterogeneous Samples ◽  
Replication Crisis ◽  
Crisis In Psychology ◽  
Hypnotic Suggestions

Hypnosis and hypnotic suggestions are gradually gaining popularity within the consciousness community as established tools for the experimental manipulation of illusions of involuntariness, hallucinations and delusions. However, hypnosis is still far from being a widespread instrument; a crucial hindrance to taking it up is the amount of time needed to invest in identifying people high and low in responsiveness to suggestion. In this study, we introduced an online assessment of hypnotic response and estimated the extent to which the scores and psychometric properties of an online screening differ from an offline one. We propose that the online screening of hypnotic response is viable as it reduces the level of responsiveness only by a slight extent. The application of online screening may prompt researchers to run large-scale studies with more heterogeneous samples, which would help researchers to overcome some of the issues underlying the current replication crisis in psychology.

The Replication Crisis in Psychology: An Overview for Theoretical and Philosophical Psychology

2021 ◽  
Author(s):  
Bradford Jay Wiggins ◽  
Cody D Christopherson
Keyword(s):  
Reform Movement ◽  
Philosophical Psychology ◽  
Philosophical Critique ◽  
Self Reflection ◽  
Replication Crisis ◽  
History Of ◽  
Crisis In Psychology

Psychology is in a replication crisis that has brought about a period of self-reflection and reform. Yet this reform appears in many ways to focus primarily on methodological and statistical practices, with little consideration for the foundational issues that concern many theoretical and philosophical psychologists and that may provide a richer account of the crisis. In this paper we offer an overview of the history of the replication crisis, the critiques and reforms at the heart of the crisis, and several points of intersection between the reform movement and broader theoretical and philosophical issues. We argue that the problems of the replication crisis and the concerns of the reform movement in fact provide various points of entry for theoretical and philosophical psychologists to collaborate with reformers in providing a more deeply philosophical critique and reform.

Sign in / Sign up

Export Citation Format

Share Document