Choosing Between the Two-sample T Test and its Alternatives: A Practical Guideline

The traditional guideline for choosing between the two-sample $t$ test and its alternatives is primarily based on assessing assumptions. Many flaws of this approach have been documented. In this paper, I address those flaws briefly and propose a new guideline for choosing between the two-sample $t$ test and its alternatives. I propose to select the hypothesis that operationalizes the research question best. This selection is carried out before data collection and entails identifying the hypotheses that in principle produce meaningful results, and among those, the most appropriate one. I advise to not only report on the most appropriate hypothesis but also on the remaining meaningful hypotheses, as they provide valuable complementary information. For testing the selected hypotheses, I recommend bootstrap and permutation tests instead of the traditionally used parametric tests. The role of assessing assumptions is downgraded to deciding whether the results of a test are reliable. An important implication of the proposed guideline is that in most cases, a nonparametric permutation test should be used instead of the $t$ test.