A general commentary by Walleczek and von Stillfried (2020) was recently published in Frontiers in Psychology. The present work provides an account of (i) the detailed research record and (ii) the main arguments behind the commentary for the purpose of full transparency and disclosure. For historical overview, Walleczek and von Stillfried (2019) had previously reported (i) the absence of any true-positive effects and (ii) the presence of one false-positive effect in a commissioned replication study of the Radin double-slit (DS) experiment on observer consciousness. In their subsequent misrepresentations, Radin et al. (2019, 2020) regrettably used the malpractice of undisclosed HARKing, i.e., undisclosed hypothesizing after the results are known. HARKing can increase greatly the risk of false-negative or false-positive conclusions. Specifically, Radin et al. (2019, 2020) deviated in two major ways from the pre-specified protocol for this commissioned study, which (i) was agreed to by Radin before data collection was started (Radin, 2011) and (ii) included data encryption to prevent the use of p-hacking and HARKing. First, Radin et al. (2019) violate the original research design by reporting a so-called “true-positive outcome of a secondary planned hypothesis”. Contrary to the claim by Radin et al. (2019, 2020), that hypothesis was not, however, part of the planned test strategy, but, instead, the associated statistical analysis – a chi-square test – was chosen by Radin sometime after the planned statistical analysis had been completed and the data unblinded. Second, Radin et al. (2019, 2020) violate the funder-approved research design in an additional way by falsely claiming that the newly developed protocol, i.e., the advanced meta-experimental protocol (AMP), implements a non-predictive test strategy when – in fact – the AMP-based test strategy is strictly predictive. Put simply, Radin et al. (2019, 2020) are mistaken that the funder-approved hypotheses posited the random occurrence of effects for the test categories in this replication experiment; instead, a different specific prediction was tested in each of the eight planned test categories, and true-positive effects were predicted to occur for only two (12.5%) of the 16 possible measurement outcomes of the eight planned single-test categories. Therefore, in the predictive single-testing regime, a statistical correction for non-predictive, i.e., random, multiple testing would not be appropriate and would thus violate the AMP-based strategy, which was implemented in the commissioned study based upon the planned outcome predictions as pre-specified in Radin (2011). Neither of these post-hoc changes by Radin et al. (on the basis of HARKing) were disclosed in Radin et al. (2019, 2020) and both these changes violate the funder-approved, original methodology agreed upon in Radin (2011) and pre-specified in the research contract. In summary, the present work reconfirms that – exactly as reported in Walleczek and von Stillfried (2019) – “the false-positive effect, which would be indistinguishable from the predicted true-positive effect, was significant at p = 0.021 (σ = −2.02; N = 1,250 test trials)” and “no statistically significant effects could be identified” in those two groups for which true-positives were predicted to occur. These observations are consistent also with an independent statistical reanalysis of the Radin DS-experiment by Tremblay (2019) and a replication attempt by Guerrer (2019). Tremblay reported significant false-positives in control groups and Guerrer found significant effects with post-hoc analyses only, but null results only when using the planned confirmatory analysis. As a general recommendation, the authors call for the implementation of advanced control-test strategies, including novel approaches from the metascience reform movement, for empirically detecting and preventing uncontrolled false-positive effects in parapsychological research.
Is the Replicability Crisis Overblown? Three Arguments Examined
