Correction: “Influence of Selection Bias on the Test Decision – A Simulation Study”

2014 ◽  
Vol 53 (05) ◽  
pp. 343-343
Keyword(s):  
Selection Bias ◽  
Simulation Study ◽  
Error Rate ◽  
Type I Error ◽  
Block Size ◽  
Error Rates ◽  
Type I ◽  
Type I Error Rate ◽  
Representation Error ◽  
Numeric Representation

We have to report marginal changes in the empirical type I error rates for the cut-offs 2/3 and 4/7 of Table 4, Table 5 and Table 6 of the paper “Influence of Selection Bias on the Test Decision – A Simulation Study” by M. Tamm, E. Cramer, L. N. Kennes, N. Heussen (Methods Inf Med 2012; 51: 138 –143). In a small number of cases the kind of representation of numeric values in SAS has resulted in wrong categorization due to a numeric representation error of differences. We corrected the simulation by using the round function of SAS in the calculation process with the same seeds as before. For Table 4 the value for the cut-off 2/3 changes from 0.180323 to 0.153494. For Table 5 the value for the cut-off 4/7 changes from 0.144729 to 0.139626 and the value for the cut-off 2/3 changes from 0.114885 to 0.101773. For Table 6 the value for the cut-off 4/7 changes from 0.125528 to 0.122144 and the value for the cut-off 2/3 changes from 0.099488 to 0.090828. The sentence on p. 141 “E.g. for block size 4 and q = 2/3 the type I error rate is 18% (Table 4).” has to be replaced by “E.g. for block size 4 and q = 2/3 the type I error rate is 15.3% (Table 4).”. There were only minor changes smaller than 0.03. These changes do not affect the interpretation of the results or our recommendations.

scholarly journals Assessment of Type I Error Rates and Power of Common Analysis Methods in Murine Obesity-Related Study: ‘Plasmode-Based’ Simulation (P13-011-19)

2019 ◽  
Vol 3 (Supplement_1) ◽  
Author(s):  
Keisuke Ejima ◽  
Andrew Brown ◽  
Daniel Smith ◽  
Ufuk Beyaztas ◽  
David Allison
Keyword(s):  
Sample Size ◽  
Error Rate ◽  
Type I Error ◽  
Error Rates ◽  
T Test ◽  
Small Samples ◽  
Type I ◽  
Type I Error Rates ◽  
Type I Error Rate ◽  
Weight Distributions

Abstract Objectives Rigor, reproducibility and transparency (RRT) awareness has expanded over the last decade. Although RRT can be improved from various aspects, we focused on type I error rates and power of commonly used statistical analyses testing mean differences of two groups, using small (n ≤ 5) to moderate sample sizes. Methods We compared data from five distinct, homozygous, monogenic, murine models of obesity with non-mutant controls of both sexes. Baseline weight (7–11 weeks old) was the outcome. To examine whether type I error rate could be affected by choice of statistical tests, we adjusted the empirical distributions of weights to ensure the null hypothesis (i.e., no mean difference) in two ways: Case 1) center both weight distributions on the same mean weight; Case 2) combine data from control and mutant groups into one distribution. From these cases, 3 to 20 mice were resampled to create a ‘plasmode’ dataset. We performed five common tests (Student's t-test, Welch's t-test, Wilcoxon test, permutation test and bootstrap test) on the plasmodes and computed type I error rates. Power was assessed using plasmodes, where the distribution of the control group was shifted by adding a constant value as in Case 1, but to realize nominal effect sizes. Results Type I error rates were unreasonably higher than the nominal significance level (type I error rate inflation) for Student's t-test, Welch's t-test and permutation especially when sample size was small for Case 1, whereas inflation was observed only for permutation for Case 2. Deflation was noted for bootstrap with small sample. Increasing sample size mitigated inflation and deflation, except for Wilcoxon in Case 1 because heterogeneity of weight distributions between groups violated assumptions for the purposes of testing mean differences. For power, a departure from the reference value was observed with small samples. Compared with the other tests, bootstrap was underpowered with small samples as a tradeoff for maintaining type I error rates. Conclusions With small samples (n ≤ 5), bootstrap avoided type I error rate inflation, but often at the cost of lower power. To avoid type I error rate inflation for other tests, sample size should be increased. Wilcoxon should be avoided because of heterogeneity of weight distributions between mutant and control mice. Funding Sources This study was supported in part by NIH and Japan Society for Promotion of Science (JSPS) KAKENHI grant.

scholarly journals Analysis of Variance Models with Stochastic Group Weights

2019 ◽  
Author(s):  
Axel Mayer ◽  
Felix Thoemmes
Keyword(s):  
Analysis Of Variance ◽  
Error Rate ◽  
Latent Variables ◽  
Structural Equation Models ◽  
Type I Error ◽  
Error Rates ◽  
Type I ◽  
Type I Error Rate ◽  
Inflated Type ◽  
Group Weights

The analysis of variance (ANOVA) is still one of the most widely used statistical methods in the social sciences. This paper is about stochastic group weights in ANOVA models – a neglected aspect in the literature. Stochastic group weights are present whenever the experimenter does not determine the exact group sizes before conducting the experiment. We show that classic ANOVA tests based on estimated marginal means can have an inflated type I error rate when stochastic group weights are not taken into account, even in randomized experiments. We propose two new ways to incorporate stochastic group weights in the tests of average effects - one based on the general linear model and one based on multigroup structural equation models (SEMs). We show in simulation studies that our methods have nominal type I error rates in experiments with stochastic group weights while classic approaches show an inflated type I error rate. The SEM approach can additionally deal with heteroscedastic residual variances and latent variables. An easy-to-use software package with graphical user interface is provided.

scholarly journals Adding new experimental arms to randomised clinical trials: Impact on error rates

2020 ◽  
Vol 17 (3) ◽  
pp. 273-284 ◽  
Author(s):  
Babak Choodari-Oskooei ◽  
Daniel J Bratton ◽  
Melissa R Gannon ◽  
Angela M Meade ◽  
Matthew R Sydes ◽  
...  
Keyword(s):  
Error Rate ◽  
Type I Error ◽  
Controlled Trial ◽  
Late Phase ◽  
Pairwise Comparison ◽  
Error Rates ◽  
Pairwise Comparisons ◽  
Type I ◽  
Test Statistics ◽  
Type I Error Rate

Background: Experimental treatments pass through various stages of development. If a treatment passes through early-phase experiments, the investigators may want to assess it in a late-phase randomised controlled trial. An efficient way to do this is adding it as a new research arm to an ongoing trial while the existing research arms continue, a so-called multi-arm platform trial. The familywise type I error rate is often a key quantity of interest in any multi-arm platform trial. We set out to clarify how it should be calculated when new arms are added to a trial some time after it has started. Methods: We show how the familywise type I error rate, any-pair and all-pairs powers can be calculated when a new arm is added to a platform trial. We extend the Dunnett probability and derive analytical formulae for the correlation between the test statistics of the existing pairwise comparison and that of the newly added arm. We also verify our analytical derivation via simulations. Results: Our results indicate that the familywise type I error rate depends on the shared control arm information (i.e. individuals in continuous and binary outcomes and primary outcome events in time-to-event outcomes) from the common control arm patients and the allocation ratio. The familywise type I error rate is driven more by the number of pairwise comparisons and the corresponding (pairwise) type I error rates than by the timing of the addition of the new arms. The familywise type I error rate can be estimated using Šidák’s correction if the correlation between the test statistics of pairwise comparisons is less than 0.30. Conclusions: The findings we present in this article can be used to design trials with pre-planned deferred arms or to add new pairwise comparisons within an ongoing platform trial where control of the pairwise error rate or familywise type I error rate (for a subset of pairwise comparisons) is required.

scholarly journals Correcting the Bias Correction for the Bootstrap Confidence Interval in Mediation Analysis

2021 ◽  
Author(s):  
Tristan Tibbe ◽  
Amanda Kay Montoya
Keyword(s):  
Confidence Interval ◽  
Error Rate ◽  
Indirect Effect ◽  
Mediation Analysis ◽  
Type I Error ◽  
Error Rates ◽  
Type I ◽  
Bootstrap Confidence Interval ◽  
Type I Error Rates ◽  
Type I Error Rate

The bias-corrected bootstrap confidence interval (BCBCI) was once the method of choice for conducting inference on the indirect effect in mediation analysis due to its high power in small samples, but now it is criticized by methodologists for its inflated type I error rates. In its place, the percentile bootstrap confidence interval (PBCI), which does not adjust for bias, is currently the recommended inferential method for indirect effects. This study proposes two alternative bias-corrected bootstrap methods for creating confidence intervals around the indirect effect. Using a Monte Carlo simulation, these methods were compared to the BCBCI, PBCI, and a bias-corrected method introduced by Chen and Fritz (2021). The results showed that the methods perform on a continuum, where the BCBCI has the best balance (i.e., having closest to an equal proportion of CIs falling above and below the true effect), highest power, and highest type I error rate; the PBCI has the worst balance, lowest power, and lowest type I error rate; and the alternative bias-corrected methods fall between these two methods on all three performance criteria. An extension of the original simulation that compared the bias-corrected methods to the PBCI after controlling for type I error rate inflation suggests that the increased power of these methods might only be due to their higher type I error rates. Thus, if control over the type I error rate is desired, the PBCI is still the recommended method for use with the indirect effect. Future research should examine the performance of these methods in the presence of missing data, confounding variables, and other real-world complications to enhance the generalizability of these results.

scholarly journals Extending the CWM approach to intraspecific trait variation: how to deal with overly optimistic standard tests?

2021 ◽  
Author(s):  
David Zelený ◽  
Kenny Helsen ◽  
Yi-Nuo Lee
Keyword(s):  
Error Rate ◽  
Real World ◽  
Type I Error ◽  
Cloud Forest ◽  
Error Rates ◽  
Type I ◽  
Trait Variation ◽  
Site Specific ◽  
Intraspecific Trait Variation ◽  
Type I Error Rate

AbstractCommunity weighted means (CWMs) are widely used to study the relationship between community-level functional traits and environment variation. When relationships between CWM traits and environmental variables are directly assessed using linear regression or ANOVA and tested by standard parametric tests, results are prone to inflated Type I error rates, thus producing overly optimistic results. Previous research has found that this problem can be solved by permutation tests (i.e. the max test). A recent extension of this CWM approach, that allows the inclusion of intraspecific trait variation (ITV) by partitioning information in fixed, site-specific and intraspecific CWMs, has proven popular. However, this raises the question whether the same kind of Type I error rate inflation also exists for site-specific CWM or intraspecific CWM-environment relationships. Using simulated community datasets and a real-world dataset from a subtropical montane cloud forest in Taiwan, we show that site-specific CWM-environment relationships also suffer from Type I error rate inflation, and that the severity of this inflation is negatively related to the relative ITV magnitude. In contrast, for intraspecific CWM-environment relationships, standard parametric tests have the correct Type I error rate, while being somewhat conservative, with reduced statistical power. We introduce an ITV-extended version of the max test for the ITV-extended CWM approach, which can solve the inflation problem for site-specific CWM-environment relationships, and which, without considering ITV, becomes equivalent to the “original” max test used for the CWM approach. On both simulated and real-world data, we show that this new ITV-extended max test works well across the full possible magnitude of ITV. We also provide guidelines and R codes of max test solutions for each CWM type and situation. Finally, we suggest recommendations on how to handle the results of previously published studies using the CWM approach without controlling for Type I error rate inflation.

Influence of Selection Bias on the Test Decision

2012 ◽  
Vol 51 (02) ◽  
pp. 138-143 ◽  
Author(s):  
E. Cramer ◽  
L. N. Kennes ◽  
N. Heussen ◽  
M. Tamm
Keyword(s):  
Clinical Trials ◽  
Selection Bias ◽  
Error Rate ◽  
Randomized Clinical Trials ◽  
Type I Error ◽  
Patient Characteristics ◽  
Allocation Concealment ◽  
Type I ◽  
Type I Error Rate ◽  
Block Randomization

SummaryBackground: Selection bias arises in clinical trials by reason of selective assignment of patients to treatment groups. Even in randomized clinical trials with allocation concealment this phenomenon can occur if future assignments can be predicted due to knowledge of former allocations.Objectives: Considering unmasked randomized clinical trials with allocation concealment the impact of selection bias on type I error rate under permuted block randomization is investigated. We aimed to extend the existing research into this topic by including practical assumptions concerning misclassification of patient characteristics to get an estimate of type I error close to clinical routine. To establish an upper bound for the type I error rate different biasing strategies of the investigator are compared first. In addition, the aspect of patient availability is considered.Methods: To evaluate the influence of selection bias on type I error rate under several practical situations, different block sizes, selection effects, biasing strategies and success rates of patient classification were simulated using SAS.Results: Type I error rate exceeds 5 percent significance level; it reaches values up to 21 percent. More cautious biasing strategies and misclassification of patient characteristics may diminish but cannot eliminate selection bias. The number of screened patients is about three times larger than the needed number for the trial.Conclusions: Even in unmasked randomized clinical trials using permuted block randomization with allocation concealment the influence of selection bias must not be disregarded evaluating the test decision. It should be incorporated when designing and reporting a clinical trial.

scholarly journals The Importance of Type I Error Rates When Studying Bias in Monte Carlo Studies in Statistics

2020 ◽  
Vol 18 (1) ◽  
pp. 2-11
Author(s):  
Michael Harwell
Keyword(s):  
Monte Carlo ◽  
Error Rate ◽  
Type I Error ◽  
Error Rates ◽  
Type I ◽  
Type I Error Rates ◽  
Type I Error Rate ◽  
Monte Carlo Studies

Two common outcomes of Monte Carlo studies in statistics are bias and Type I error rate. Several versions of bias statistics exist but all employ arbitrary cutoffs for deciding when bias is ignorable or non-ignorable. This article argues Type I error rates should be used when assessing bias.

Controlling type I error rate for fast track drug development programmes

2003 ◽  
Vol 22 (5) ◽  
pp. 665-675 ◽  
Author(s):  
Weichung J. Shih ◽  
Peter Ouyang ◽  
Hui Quan ◽  
Yong Lin ◽  
Bart Michiels ◽  
...  
Keyword(s):  
Drug Development ◽  
Error Rate ◽  
Fast Track ◽  
Type I Error ◽  
Type I ◽  
Type I Error Rate

Alternative models and randomization techniques for Bayesian response-adaptive randomization with binary outcomes

2021 ◽  
pp. 174077452110101
Author(s):  
Jennifer Proper ◽  
John Connett ◽  
Thomas Murray
Keyword(s):  
Logistic Regression ◽  
Sample Size ◽  
Error Rate ◽  
Adaptive Design ◽  
Type I Error ◽  
Probability Model ◽  
Binary Outcomes ◽  
Type I ◽  
Operating Characteristics ◽  
Type I Error Rate

Background: Bayesian response-adaptive designs, which data adaptively alter the allocation ratio in favor of the better performing treatment, are often criticized for engendering a non-trivial probability of a subject imbalance in favor of the inferior treatment, inflating type I error rate, and increasing sample size requirements. The implementation of these designs using the Thompson sampling methods has generally assumed a simple beta-binomial probability model in the literature; however, the effect of these choices on the resulting design operating characteristics relative to other reasonable alternatives has not been fully examined. Motivated by the Advanced R2 Eperfusion STrategies for Refractory Cardiac Arrest trial, we posit that a logistic probability model coupled with an urn or permuted block randomization method will alleviate some of the practical limitations engendered by the conventional implementation of a two-arm Bayesian response-adaptive design with binary outcomes. In this article, we discuss up to what extent this solution works and when it does not. Methods: A computer simulation study was performed to evaluate the relative merits of a Bayesian response-adaptive design for the Advanced R2 Eperfusion STrategies for Refractory Cardiac Arrest trial using the Thompson sampling methods based on a logistic regression probability model coupled with either an urn or permuted block randomization method that limits deviations from the evolving target allocation ratio. The different implementations of the response-adaptive design were evaluated for type I error rate control across various null response rates and power, among other performance metrics. Results: The logistic regression probability model engenders smaller average sample sizes with similar power, better control over type I error rate, and more favorable treatment arm sample size distributions than the conventional beta-binomial probability model, and designs using the alternative randomization methods have a negligible chance of a sample size imbalance in the wrong direction. Conclusion: Pairing the logistic regression probability model with either of the alternative randomization methods results in a much improved response-adaptive design in regard to important operating characteristics, including type I error rate control and the risk of a sample size imbalance in favor of the inferior treatment.

Sign in / Sign up

Export Citation Format

Share Document