We studied how academics assess the results of a set of four experiments that all test a given theory. We found that participants’ belief in the theory increases with the number of significant results, and that direct replications were considered to be more important than conceptual replications. We found no difference between authors and reviewers in their propensity to submit or recommend to publish sets of results, but we did find that authors are generally more likely to desire an additional experiment. In a preregistered secondary analysis of individual participant data, we examined the heuristics academics use to assess the results of four experiments. Only 6 out of 312 (1.9%) participants we analyzed used the normative method of Bayesian inference, whereas the majority of participants used vote counting approaches that tend to undervalue the evidence for the underlying theory if two or more results are statistically significant.
Uncertainty Estimation for Pseudo‐Bayesian Inference Under Complex Sampling
