scholarly journals Psychologists Should Use Brunner-Munzel’s Instead of Mann-Whitney’s U Test as the Default Nonparametric Procedure

2021 ◽  
Vol 4 (2) ◽  
pp. 251524592199960
Author(s):  
Julian D. Karch
Keyword(s):  
Rate Control ◽  
Type I Error ◽  
Standard Procedure ◽  
Nonparametric Test ◽  
T Test ◽  
Type I ◽  
Large Samples ◽  
Statistical Inferences ◽  
Error Rate Control ◽  
Nonparametric Procedure

To investigate whether a variable tends to be larger in one population than in another, the t test is the standard procedure. In some situations, the parametric t test is inappropriate, and a nonparametric procedure should be used instead. The default nonparametric procedure is Mann-Whitney’s U test. Despite being a nonparametric test, Mann-Whitney’s test is associated with a strong assumption, known as exchangeability. I demonstrate that if exchangeability is violated, Mann-Whitney’s test can lead to wrong statistical inferences even for large samples. In addition, I argue that in psychology, exchangeability is typically not met. As a remedy, I introduce Brunner-Munzel’s test and demonstrate that it provides good Type I error rate control even if exchangeability is not met and that it has similar power as Mann-Whitney’s test. Consequently, I recommend using Brunner-Munzel’s test by default. To facilitate this, I provide advice on how to perform and report on Brunner-Munzel’s test.

scholarly journals Psychologists Should Use Brunner-Munzel's Instead of Mann-Whitney's U test as the Default Nonparametric Procedure

2020 ◽  
Author(s):  
Julian Karch
Keyword(s):  
Type I Error ◽  
Standard Procedure ◽  
Nonparametric Test ◽  
T Test ◽  
Type I ◽  
Large Samples ◽  
Statistical Inferences ◽  
Error Rate Control ◽  
Nonparametric Procedure ◽  
Equal Power

For comparing two groups with regard to their central tendencies, the t-test is the standard procedure. In some situations, the parametric t-test is inappropriate, and a nonparametric procedure should be used instead. The default nonparametric procedure is Mann-Whitney's U test. Despite being a nonparametric test, Mann-Whitney's U test is associated with a strong assumption, known as exchangeability. I demonstrate that if exchangeability is violated, Mann-Whitney's U test can lead to wrong statistical inferences even for large samples. Additionally, I argue that in psychology, exchangeability is often not met. As a remedy, I introduce Brunner-Munzel's test and demonstrate that it provides good type I error rate control even if exchangeability is not met, and has almost equal power as Mann-Whitney's U test. Consequently, I recommend using Brunner-Munzel's test by default. To facilitate this, I provide advice on how to perform and report on Brunner-Munzel's test.

How Does Polytomous Item Bias Affect Total-group Survey Score Comparisons?

2015 ◽  
Vol 46 (3) ◽  
pp. 586-603 ◽  
Author(s):  
Ma Dolores Hidalgo ◽  
Isabel Benítez ◽  
Jose-Luis Padilla ◽  
Juana Gómez-Benito
Keyword(s):  
Effect Size ◽  
Type I Error ◽  
Error Rates ◽  
T Test ◽  
Type I ◽  
Polytomous Items ◽  
Item Functioning ◽  
Scale Scores ◽  
Comparative Group ◽  
The Impact

The growing use of scales in survey questionnaires warrants the need to address how does polytomous differential item functioning (DIF) affect observed scale score comparisons. The aim of this study is to investigate the impact of DIF on the type I error and effect size of the independent samples t-test on the observed total scale scores. A simulation study was conducted, focusing on potential variables related to DIF in polytomous items, such as DIF pattern, sample size, magnitude, and percentage of DIF items. The results showed that DIF patterns and the number of DIF items affected the type I error rates and effect size of t-test values. The results highlighted the need to analyze DIF before making comparative group interpretations.

Type I Error Rates for Welch’s Test and James’s Second-Order Test Under Nonnormality and Inequality of Variance When There Are Two Groups

1994 ◽  
Vol 19 (3) ◽  
pp. 275-291 ◽  
Author(s):  
James Algina ◽  
T. C. Oshima ◽  
Wen-Ying Lin
Keyword(s):  
Degrees Of Freedom ◽  
Type I Error ◽  
Total Sample ◽  
Error Rates ◽  
Second Order ◽  
T Test ◽  
Type I ◽  
Sample Sizes ◽  
Unequal Variances ◽  
Type I Error Rates

Type I error rates were estimated for three tests that compare means by using data from two independent samples: the independent samples t test, Welch’s approximate degrees of freedom test, and James’s second-order test. Type I error rates were estimated for skewed distributions, equal and unequal variances, equal and unequal sample sizes, and a range of total sample sizes. Welch’s test and James’s test have very similar Type I error rates and tend to control the Type I error rate as well or better than the independent samples t test does. The results provide guidance about the total sample sizes required for controlling Type I error rates.

scholarly journals A Fully-Adjusted Two-Stage Procedure for Rank Normalization in Genetic Association Studies

2018 ◽  
Author(s):  
Tamar Sofer ◽  
Xiuwen Zheng ◽  
Stephanie M. Gogarten ◽  
Cecelia A. Laurie ◽  
Kelsey Grinde ◽  
...  
Keyword(s):  
Statistical Power ◽  
Type I Error ◽  
Association Studies ◽  
Genetic Association Studies ◽  
Statistical Properties ◽  
Type I ◽  
Residual Distribution ◽  
Two Stage ◽  
Trait Distribution ◽  
Error Rate Control

AbstractWhen testing genotype-phenotype associations using linear regression, departure of the trait distribution from normality can impact both Type I error rate control and statistical power, with worse consequences for rarer variants. While it has been shown that applying a rank-normalization transformation to trait values before testing may improve these statistical properties, the factor driving them is not the trait distribution itself, but its residual distribution after regression on both covariates and genotype. Because genotype is expected to have a small effect (if any) investigators now routinely use a two-stage method, in which they first regress the trait on covariates, obtain residuals, rank-normalize them, and then secondly use the rank-normalized residuals in association analysis with the genotypes. Potential confounding signals are assumed to be removed at the first stage, so in practice no further adjustment is done in the second stage. Here, we show that this widely-used approach can lead to tests with undesirable statistical properties, due to both a combination of a mis-specified mean-variance relationship, and remaining covariate associations between the rank-normalized residuals and genotypes. We demonstrate these properties theoretically, and also in applications to genome-wide and whole-genome sequencing association studies. We further propose and evaluate an alternative fully-adjusted two-stage approach that adjusts for covariates both when residuals are obtained, and in the subsequent association test. This method can reduce excess Type I errors and improve statistical power.

Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms

1998 ◽  
Vol 10 (7) ◽  
pp. 1895-1923 ◽  
Author(s):  
Thomas G. Dietterich
Keyword(s):  
Cross Validation ◽  
Type I Error ◽  
Learning Algorithm ◽  
Statistical Tests ◽  
Computational Cost ◽  
Learning Task ◽  
T Test ◽  
Type I ◽  
Mcnemar’S Test ◽  
Mcnemar's Test

This article reviews five approximate statistical tests for determining whether one learning algorithm outperforms another on a particular learning task. These test sare compared experimentally to determine their probability of incorrectly detecting a difference when no difference exists (type I error). Two widely used statistical tests are shown to have high probability of type I error in certain situations and should never be used: a test for the difference of two proportions and a paired-differences t test based on taking several random train-test splits. A third test, a paired-differences t test based on 10-fold cross-validation, exhibits somewhat elevated probability of type I error. A fourth test, McNemar's test, is shown to have low type I error. The fifth test is a new test, 5 × 2 cv, based on five iterations of twofold cross-validation. Experiments show that this test also has acceptable type I error. The article also measures the power (ability to detect algorithm differences when they do exist) of these tests. The cross-validated t test is the most powerful. The 5×2 cv test is shown to be slightly more powerful than McNemar's test. The choice of the best test is determined by the computational cost of running the learning algorithm. For algorithms that can be executed only once, Mc-Nemar's test is the only test with acceptable type I error. For algorithms that can be executed 10 times, the 5 × 2 cv test is recommended, because it is slightly more powerful and because it directly measures variation due to the choice of training set.

scholarly journals Assessment of Type I Error Rates and Power of Common Analysis Methods in Murine Obesity-Related Study: ‘Plasmode-Based’ Simulation (P13-011-19)

2019 ◽  
Vol 3 (Supplement_1) ◽  
Author(s):  
Keisuke Ejima ◽  
Andrew Brown ◽  
Daniel Smith ◽  
Ufuk Beyaztas ◽  
David Allison
Keyword(s):  
Sample Size ◽  
Error Rate ◽  
Type I Error ◽  
Error Rates ◽  
T Test ◽  
Small Samples ◽  
Type I ◽  
Type I Error Rates ◽  
Type I Error Rate ◽  
Weight Distributions

Abstract Objectives Rigor, reproducibility and transparency (RRT) awareness has expanded over the last decade. Although RRT can be improved from various aspects, we focused on type I error rates and power of commonly used statistical analyses testing mean differences of two groups, using small (n ≤ 5) to moderate sample sizes. Methods We compared data from five distinct, homozygous, monogenic, murine models of obesity with non-mutant controls of both sexes. Baseline weight (7–11 weeks old) was the outcome. To examine whether type I error rate could be affected by choice of statistical tests, we adjusted the empirical distributions of weights to ensure the null hypothesis (i.e., no mean difference) in two ways: Case 1) center both weight distributions on the same mean weight; Case 2) combine data from control and mutant groups into one distribution. From these cases, 3 to 20 mice were resampled to create a ‘plasmode’ dataset. We performed five common tests (Student's t-test, Welch's t-test, Wilcoxon test, permutation test and bootstrap test) on the plasmodes and computed type I error rates. Power was assessed using plasmodes, where the distribution of the control group was shifted by adding a constant value as in Case 1, but to realize nominal effect sizes. Results Type I error rates were unreasonably higher than the nominal significance level (type I error rate inflation) for Student's t-test, Welch's t-test and permutation especially when sample size was small for Case 1, whereas inflation was observed only for permutation for Case 2. Deflation was noted for bootstrap with small sample. Increasing sample size mitigated inflation and deflation, except for Wilcoxon in Case 1 because heterogeneity of weight distributions between groups violated assumptions for the purposes of testing mean differences. For power, a departure from the reference value was observed with small samples. Compared with the other tests, bootstrap was underpowered with small samples as a tradeoff for maintaining type I error rates. Conclusions With small samples (n ≤ 5), bootstrap avoided type I error rate inflation, but often at the cost of lower power. To avoid type I error rate inflation for other tests, sample size should be increased. Wilcoxon should be avoided because of heterogeneity of weight distributions between mutant and control mice. Funding Sources This study was supported in part by NIH and Japan Society for Promotion of Science (JSPS) KAKENHI grant.

A Nonparametric Test Statistic for the General Linear Model

1989 ◽  
Vol 14 (4) ◽  
pp. 351-371 ◽  
Author(s):  
Michael R. Harwell ◽  
Ronald C. Serlin
Keyword(s):  
Linear Model ◽  
Type I Error ◽  
Nonparametric Test ◽  
General Linear Model ◽  
Error Rates ◽  
General Linear ◽  
Type I ◽  
Test Statistic ◽  
Rank Statistics ◽  
Nonparametric Test Statistic

Puri and Sen (1969Puri and Sen (1985) presented a nonparametric test statistic based on a general linear model approach that is appropriate for testing a wide class of hypotheses. The two forms of this statistic, pure- and mixed-rank, differ according to whether the original predictor values or their ranks are used. Both forms permit the use of standard statistical packages to perform the analyses. The applicability of these statistics in testing a number of hypotheses is highlighted, and an example of their use is given. A simulation study for the multivariate-multiple-regression case is used to examine the distributional behavior of the pure- and mixed-rank statistics and an important competitor, the rank transformation of Conover and Iman (1981). The results suggest that the pure- and mixed-rank statistics are superior with respect to minimizing liberal Type I error rates, whereas the Conover and Iman statistic produces larger power values.

scholarly journals On the Performance of the Marginal Homogeneity Test to Detect Rater Drift

2017 ◽  
Vol 42 (4) ◽  
pp. 307-320 ◽  
Author(s):  
Adrienne Sgammato ◽  
John R. Donoghue
Keyword(s):  
Error Control ◽  
Type I Error ◽  
Study Data ◽  
T Test ◽  
Homogeneity Test ◽  
Type I ◽  
Marginal Homogeneity ◽  
Size Number ◽  
Almost All ◽  
Rater Drift

When constructed response items are administered repeatedly, “trend scoring” can be used to test for rater drift. In trend scoring, raters rescore responses from the previous administration. Two simulation studies evaluated the utility of Stuart’s Q measure of marginal homogeneity as a way of evaluating rater drift when monitoring trend scoring. In the first study, data were generated based on trend scoring tables obtained from an operational assessment. The second study tightly controlled table margins to disentangle certain features present in the empirical data. In addition to Q, the paired t test was included as a comparison, because of its widespread use in monitoring trend scoring. Sample size, number of score categories, interrater agreement, and symmetry/asymmetry of the margins were manipulated. For identical margins, both statistics had good Type I error control. For a unidirectional shift in margins, both statistics had good power. As expected, when shifts in the margins were balanced across categories, the t test had little power. Q demonstrated good power for all conditions and identified almost all items identified by the t test. Q shows substantial promise for monitoring of trend scoring.

Sign in / Sign up

Export Citation Format

Share Document