A (Very) Few Concluding Thoughts

In this chapter, educational recommendations for future scientists are suggested followed by possible scenarios that may characterize the future of the reproducibility initiatives discussed in previous chapters. One such scenario, while quite pessimistic, is not without historical precedent. Namely, that the entire movement may turn out to be little more than a publishing opportunity for methodologically oriented scientists—soon replaced by something else and forgotten by most—thereby allowing it to be reprised a few decades later under a different name by different academics. Alternately, and more optimistically, the procedural and statistical behaviors discussed here will receive an increased emphasis in the scientific curricula accompanied by a sea change in actual scientific practice and its culture—thereby producing a substantial reduction in the prevalence of avoidable false-positive scientific results. And indeed recent evidence does appear to suggest that the reproducibility initiatives instituted by the dedicated cadre of methodologically oriented scientists chronicled in this book have indeed begun the process of making substantive improvements in the quality and veracity of scientific inquiry itself.

Multiple-Study Replication Initiatives

The “mass” replications of multiple studies, some employing dozens of investigators distributed among myriad sites, is unique to the reproducibility movement. The most impressive of these initiatives was employed by the Open Science Collaboration directed by Brian Nosek, who recruited 270 investigators to participate in the replication of 100 psychological experiments via a very carefully structured, prespecified protocol that avoided questionable research practices. Just before this Herculean effort, two huge biotech firms (Amegen and Bayer Health Care) respectively conducted 53 and 67 preclinical replications of promising published studies to ascertain which results were worth pursuing for commercial applications. Amazingly, in less than a 10-year period, a number of other diverse multistudy replications were also conducted involving hundreds of effects. Among these were the three “many lab” multistudy replications based on the Open Science Model (but also designed to ascertain if potential confounders of the approach itself existed, such as differences in participant types, settings, and timing), replications of social science studies published in Science and Nature, experimental economics studies, and even self-reported replications ascertained from a survey. Somewhat surprisingly, the overall successful replication percentage for this diverse collection of 811 studies was 46%, mirroring the modeling results discussed in Chapter 3 and supportive of John Ioannidis’s pejorative and often quoted conclusion that most scientific results are incorrect.

Damage Control upon Learning That One’s Study Failed to Replicate

But what happens to investigators whose studies fails to replicate? The answer is complicated by the growing use of social media by scientists and the tenor of the original investigators’ responses to the replicators. Alternative case studies are presented including John Bargh’s vitriolic outburst following a failure of his classic word priming study to replicate, Amy Cuddy’s unfortunate experience with power posing, and Matthew Vees’s low-keyed response in which he declined to aggressively disparage his replicators, complemented the replicators’ interpretation of their replication, and neither defended his original study or even suggested that its findings might be wrong. In addition to such case studies, surveys on the subject suggest that there are normally no long-term deleterious career or reputational effects on investigators for a failure of a study to replicate and that a reasoned (or no) response to a failed replication is the superior professional and affective solution.

A Few Case Studies of QRP-Driven Irreproducible Results

Examples of a dispiriting number of jaw-dropping, irreproducible scientific results are described here, emanating from disparate fields of inquiry. These examples include epidemiological studies that regularly report diametrically conflicting relationships regarding risk factors and health, completely incompetent studies conducted involving the relationship between functional magnetic resonance imaging results and psychosocial variables, and positive genetic associations with cognitive and affective constructs that do not replicate. Studies (and perhaps even some entire disciplines) such as these may have some amusement value when their results are reported in the press, but their untoward effects on science should not be underestimated. Methodological reasons for the appearance of these almost universally positive studies include the previously discussed questionable research practices, several of which are famously blended into a metaphoric scientific journey through the garden of forking paths by Andrew Gelman and Eric Loken, whose article is discussed in this chapter.

The Replication Process

While replication of research is the ultimate arbitrator of reproducibility, the process is a bit more complex than it appears. And, like any empirical study, a replication can itself be wrong. However, replications are the best tool available for determining reproducibility if (a) they employ sufficient statistical power; (b) they follow the original study procedures as closely as possible (sans any questionable research practices present therein); (c) their investigators are able to obtain the necessary information, advice, and materials from the original authors; and (d) the replication protocol is preregistered. The chapter describes different types of replications, such as exact (seldom possible for experimental research), direct (the recommended approach, which involves employing the same methodological procedures, outcome variables, and statistical approaches as the original study), conceptual (not recommended since they customarily presume the original results to be correct and are conducted to determine the extent to which said results can be extended), self (primarily useful for the original investigators to convince themselves of the validity of a finding via a replication of an original study to ensure that its results are reproducible), and partial (seldom necessary but useful when there is no alternative, such as when all of the procedures cannot be duplicated for ethical reasons).

Publication Bias

Publication bias, defined as a “tendency for positive results to be overrepresented in the published literature,” was recognized and bemoaned as early as the 17th century by the chemist Robert Boyle. In the latter half of the 20th century, it began to be recognized as an increasingly serious scientific problem characterized by a deluge of positive published results (actually exceeded 95% in some areas of psychology). And, by the second decade of the 21st century, data mining techniques indicated that the phenomenon had reached epic proportions, not only in psychology and the other social sciences, but in many of the life and physical sciences as well: a finding that might have been viewed as an amusing idiosyncratic scientific fact of life if not for a concomitant realization that most of these positive scientific findings were wrong. And that publication bias, if not a cause of this debacle, was at least a major facilitator. This chapter provides documentation for the high prevalence of this odd phenomenon in a wide swath of myriad empirical scientific literatures along with the accompanying compulsion it fosters for producing positive rather than reproducible results.

Questionable Research Practices (QRPs) and Their Devastating Scientific Effects

The linchpin of both publication bias and irreproducibility involves an exhaustive list of more than a score of individually avoidable questionable research practices (QRPs) supplemented by 10 inane institutional research practices. While these untoward effects on the production of false-positive results are unsettling, a far more entertaining (in a masochistic sort of way) pair of now famous iconoclastic experiments conducted by Simmons, Nelson, and Simonsohn are presented in which, with the help of only a few well-chosen QRPs, research participants can actually become older after simply listening to a Beatle’s song. In addition, surveys designed to estimate the prevalence of these and other QRPs in the published literatures are also described.

False-Positive Results and a Nontechnical Overview of Their Modeling

This chapter explores three empirical concepts (the p-value, the effect size, and statistical power) integral to the avoidance of false positive scientific. Their relationship to reproducibility is explained in a nontechnical manner without formulas or statistical jargon, with p-values and statistical power presented in terms of probabilities from zero to 1.0 with the values of most interest to scientists being 0.05 (synonymous with a positive, hence, publishable result) and 0.80 (the most commonly recommended probability that a positive result will be obtained if the hypothesis that generated it was correct and the study will be properly designed and conducted). Unfortunately many scientists circumvent both by artifactually inflating the 0.05 criterion, overstating the available statistical power, and engaging in a number of other questionable research practices. These issues are discussed via statistical models from the genetic and psychological fields and then extended to a number of different p-values, statistical power levels, effect sizes, and the prevalence of “true,” effects expected to exist in the research literature. Among the basic conclusions of these modeling efforts are that employing more stringent p-values and larger sample sizes constitute the most effective statistical approaches for increasing the reproducibility of published results in all empirically based scientific literatures. This chapter thus lays the necessary foundation for understanding and appreciating the effects of appropriate p-values, sufficient statistical power, reaslistic effect sizes, and the avoidance of questionable research practices upon the production of reproducible results.

The Return of Pathological Science Accompanied by a Pinch of Replication

No discussion of irreproducible science would be complete without at least a brief consideration of what happens when scientists go a step or two beyond questionable research practice (QRP)-driven research. So, continuing the metaphor of scientific journeys, Robert Park’s iconic book title, Voodoo Science: The Road from Foolishness to Fraud, encapsulates the interdisciplinary examples of what Irving Langmuir (a Nobel Prize recipient in chemistry) termed pathological science more than 65 years ago. The chapter discusses more recent examples of this phenomenon in some detail from both the physical sciences (cold fusion) and their sociobehavioral counterparts (the Daryl Bem psi episode). The latter (undoubtedly a virtual mentee of Joseph Banks Rhine whose exploits were exposed by Professor Langmuir) is given more prominence here because of its influence on the genesis of reproducibility crisis itself.

Introduction

The scientific reproducibility crisis is introduced as a paradigmatic shift in the culture and behaviors of the members of one of the most crucial societal institutions. Technically it involves a change from the 20th century methodological emphasis upon the internal validity and generalizability of scientific results to a greater emphasis upon the extent to which results are reproducible and not simply “wrong.” The need for this sea change has been recognized by some methodologically oriented scientists for decades, but it wasn’t until this century that the virtual blizzard of positive but false results began to worry enough scientists to approach the status of anything approaching the status of a movement. A movement if you will that received a steroidal boost by John Ioannidis’ iconic and reasoned 2005 article entitled “Why Most Published Research Findings Are False.” The book is therefore designed to tell the story of this dedicated cadre of 21st century researchers who recognized what had become a genuine scientific crisis and consequently dedicated their considerable talents and expertise to ameliorate its effects. It is a story involving a plethora of correctable questionable behaviors that allowed scientists to get themselves into this extraordinary situation. The book leans heavily on the well documented work of hundreds of other scientists who have taken it upon themselves to show their colleagues how to conduct science that can be reproduced by others and thereby contribute to the greater good. So in a sense this a story of modern science at its best and worst, but ultimately it is an optimistic telling of a corrective change in the culture and the credibility of an absolutely crucial societal institution.