Statistical Power in Content Analysis Designs: How Effect Size, Sample Size and Coding Accuracy Jointly Affect Hypothesis Testing ‐ A Monte Carlo Simulation Approach.

This study uses Monte Carlo simulation techniques to estimate the minimum required levels of intercoder reliability in content analysis data for testing correlational hypotheses, depending on sample size, effect size and coder behavior under uncertainty. The ensuing procedure is analogous to power calculations for experimental designs. In most widespread sample size/effect size settings, the rule-of-thumb that chance-adjusted agreement should be ≥.80 or ≥.667 corresponds to the simulation results, resulting in acceptable α and β error rates. However, this simulation allows making precise power calculations that can consider the specifics of each study’s context, moving beyond one-size-fits-all recommendations. Studies with low sample sizes and/or low expected effect sizes may need coder agreement above .800 to test a hypothesis with sufficient statistical power. In studies with high sample sizes and/or high expected effect sizes, coder agreement below .667 may suffice. Such calculations can help in both evaluating and in designing studies. Particularly in pre-registered research, higher sample sizes may be used to compensate for low expected effect sizes and/or borderline coding reliability (e.g. when constructs are hard to measure). I supply equations, easy-to-use tables and R functions to facilitate use of this framework, along with example code as online appendix.

On the Statistical Properties of the Dankel-Loenneke Method

Dankel & Loenneke (2019) recently presented a new approach to identifying subgroups in parallel group study designs. Here, we briefly discuss our statistical concerns with proposed approach. We reveal that the error rates of the Danke-Loenneke approach are much higher than the claimed 5%, and that these error rates are dependent on numerous factors, including sample size, effect variance, and random error. The Dankel-Loenneke method has poor statistical properties; as such, we suggest that the method not be used and the manuscript constitutes an "honest error" per the Committee on Publication Ethics (COPE) guidelines.