The Reproducibility Crisis

This chapter reviews and evaluates reports that scientists often cannot repeat, or “reproduce” published work. It begins by defining what “reproducibility” means and how reproducibility applies to various kinds of science. The focus then shifts to the Reproducibility Project: Psychology, which was a systematic effort to repeat published findings in psychology, and which gave rise to many of the present concerns about reproducibility. The chapter critically examines the Reproducibility Project and points out how the nature of science and the complexity of nature can stymie the best attempts at reproducibility. The chapter also reviews the statistical criticisms of science that John Ioannidis and Katherine Button and their colleagues have raised. The hypothesis is a central issue because it is inconsistently defined across various branches of science. The statisticians’ strongest attacks are directed against work that differs from most laboratory experimental science. A weak point in the reasoning behind the Reproducibility Project and the statistical arguments is the assumption that a multi-pronged scientific investigation can be legitimately criticized by close examination of one of its components. Experimental science relies on multiple tests and multiple hypotheses to arrive at its conclusions. Reproducibility is a valid concern for science; it is not a “crisis.”