Directional Error Rates of Closed Testing Procedures

2013 ◽  
Vol 5 (4) ◽  
pp. 345-355 ◽  
Author(s):  
Peter H. Westfall ◽  
Frank Bretz ◽  
Randall D. Tobias
Keyword(s):  
Error Rates ◽  
Testing Procedures ◽  
Directional Error ◽  
Closed Testing Procedures

Comparison of methods to account for autocorrelation in correlation analyses of fish data

1998 ◽  
Vol 55 (9) ◽  
pp. 2127-2140 ◽  
Author(s):  
Brian J Pyper ◽  
Randall M Peterman
Keyword(s):  
Monte Carlo ◽  
Hypothesis Testing ◽  
Type I Error ◽  
Low Frequency ◽  
Error Rates ◽  
Type I ◽  
Testing Procedures ◽  
Type I Error Rates ◽  
Fish Recruitment ◽  
Correlation Analyses

Autocorrelation in fish recruitment and environmental data can complicate statistical inference in correlation analyses. To address this problem, researchers often either adjust hypothesis testing procedures (e.g., adjust degrees of freedom) to account for autocorrelation or remove the autocorrelation using prewhitening or first-differencing before analysis. However, the effectiveness of methods that adjust hypothesis testing procedures has not yet been fully explored quantitatively. We therefore compared several adjustment methods via Monte Carlo simulation and found that a modified version of these methods kept Type I error rates near . In contrast, methods that remove autocorrelation control Type I error rates well but may in some circumstances increase Type II error rates (probability of failing to detect some environmental effect) and hence reduce statistical power, in comparison with adjusting the test procedure. Specifically, our Monte Carlo simulations show that prewhitening and especially first-differencing decrease power in the common situations where low-frequency (slowly changing) processes are important sources of covariation in fish recruitment or in environmental variables. Conversely, removing autocorrelation can increase power when low-frequency processes account for only some of the covariation. We therefore recommend that researchers carefully consider the importance of different time scales of variability when analyzing autocorrelated data.

Comparison of SPRT and Sequential Bayes Procedures for Classifying Examinees Into Two Categories Using a Computerized Test

1996 ◽  
Vol 21 (4) ◽  
pp. 405-414 ◽  
Author(s):  
Judith A. Spray ◽  
Mark D. Reckase
Keyword(s):  
Accurate Estimate ◽  
Error Rates ◽  
Computerized Testing ◽  
Classification Error ◽  
Ratio Test ◽  
College Selection ◽  
Entry Level ◽  
Testing Procedures ◽  
Test Items ◽  
Sequential Probability

Many testing applications focus on classifying examinees into one of two categories (e.g., pass/fail) rather than on obtaining an accurate estimate of level of ability. Examples of such applications include licensure and certification, college selection, and placement into entry-level or developmental college courses. With the increased availability of computers for the administration and scoring of tests, computerized testing procedures have been developed for efficiently making these classification decisions. The purpose of the research reported in this article was to compare two such procedures, one based on the sequential probability ratio test and the other on sequential Bayes methodology, to determine which required fewer items for classification when the procedures were matched on classification error rates. The results showed that under the conditions studied, the SPRT procedure required fewer test items than the sequential Bayes procedure to achieve the same level of classification accuracy.

scholarly journals Magnitude Based Inference in Relation to One-sided Hypotheses Testing Procedures

2020 ◽  
Author(s):  
Janet Aisbett ◽  
Daniel Lakens ◽  
Kristin Sainani
Keyword(s):  
Error Control ◽  
Hypothesis Test ◽  
A Priori ◽  
Error Rates ◽  
Equivalence Testing ◽  
High Type ◽  
Testing Procedures ◽  
Type 1 Error ◽  
Sample Size Calculations

Magnitude based inference (MBI) was widely adopted by sport science researchers as an alternative to null hypothesis significance tests. It has been criticized for lacking a theoretical framework, mixing Bayesian and frequentist thinking, and encouraging researchers to run small studies with high Type 1 error rates. MBI terminology describes the position of confidence intervals in relation to smallest meaningful effect sizes. We show these positions correspond to combinations of one-sided tests of hypotheses about the presence or absence of meaningful effects, and formally describe MBI as a multiple decision procedure. MBI terminology operates as if tests are conducted at multiple alpha levels. We illustrate how error rates can be controlled by limiting each one-sided hypothesis test to a single alpha level. To provide transparent error control in a Neyman-Pearson framework and encourage the use of standard statistical software, we recommend replacing MBI with one-sided tests against smallest meaningful effects, or pairs of such tests as in equivalence testing. Researchers should pre-specify their hypotheses and alpha levels, perform a priori sample size calculations, and justify all assumptions. Our recommendations show researchers what tests to use and how to design and report their statistical analyses to accord with standard frequentist practice.

scholarly journals Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates

2004 ◽  
Vol 3 (1) ◽  
pp. 1-69 ◽  
Author(s):  
Sandrine Dudoit ◽  
Mark J. van der Laan ◽  
Katherine S. Pollard
Keyword(s):  
Error Rate ◽  
Multiple Testing ◽  
Type I Error ◽  
Null Distribution ◽  
Error Rates ◽  
Single Step ◽  
Type I ◽  
Test Statistics ◽  
Testing Procedures ◽  
Multiple Testing Procedures

The present article proposes general single-step multiple testing procedures for controlling Type I error rates defined as arbitrary parameters of the distribution of the number of Type I errors, such as the generalized family-wise error rate. A key feature of our approach is the test statistics null distribution (rather than data generating null distribution) used to derive cut-offs (i.e., rejection regions) for these test statistics and the resulting adjusted p-values. For general null hypotheses, corresponding to submodels for the data generating distribution, we identify an asymptotic domination condition for a null distribution under which single-step common-quantile and common-cut-off procedures asymptotically control the Type I error rate, for arbitrary data generating distributions, without the need for conditions such as subset pivotality. Inspired by this general characterization of a null distribution, we then propose as an explicit null distribution the asymptotic distribution of the vector of null value shifted and scaled test statistics. In the special case of family-wise error rate (FWER) control, our method yields the single-step minP and maxT procedures, based on minima of unadjusted p-values and maxima of test statistics, respectively, with the important distinction in the choice of null distribution. Single-step procedures based on consistent estimators of the null distribution are shown to also provide asymptotic control of the Type I error rate. A general bootstrap algorithm is supplied to conveniently obtain consistent estimators of the null distribution. The special cases of t- and F-statistics are discussed in detail. The companion articles focus on step-down multiple testing procedures for control of the FWER (van der Laan et al., 2004b) and on augmentations of FWER-controlling methods to control error rates such as tail probabilities for the number of false positives and for the proportion of false positives among the rejected hypotheses (van der Laan et al., 2004a). The proposed bootstrap multiple testing procedures are evaluated by a simulation study and applied to genomic data in the fourth article of the series (Pollard et al., 2004).

Sign in / Sign up

Export Citation Format

Share Document