The Role of Measurement Error in Replicability of Psychological Findings
The role of measurement error in replicability of psychological findings has become of increasing interest, with some researchers suggesting it is critical to replicability, and others arguing that it is likely secondary to other effects on generalizability of findings. This work examined the relationship between reliability, as reflected in internal consistency indices, and effect size in published many-labs projects (313 samples from 44 studies). Among multiple-item designs, at lower reliabilities effect size was near zero regardless of reliability; at greater reliabilities (above approximately 0.80), effect size appeared to increase with reliability for some effects but not others. However, among the broader set of studies, including single-item designs, number of items was not associated with greater effect size, and in fact decreased with measure length. Results point to the importance of measurement precision in replicability of psychological findings, but also to the importance of precision per se and not proxies such as measure length.