scholarly journals Type I error control for cluster randomized trials under varying small sample structures

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Joshua R. Nugent ◽  
Ken P. Kleinman
Keyword(s):  
Cluster Size ◽  
Error Control ◽  
Type I Error ◽  
Error Rates ◽  
Small Sample ◽  
Type I ◽  
Number Of Clusters ◽  
Type I Error Rates ◽  
Wald Tests ◽  
Cluster Randomized

Abstract Background Linear mixed models (LMM) are a common approach to analyzing data from cluster randomized trials (CRTs). Inference on parameters can be performed via Wald tests or likelihood ratio tests (LRT), but both approaches may give incorrect Type I error rates in common finite sample settings. The impact of different combinations of cluster size, number of clusters, intraclass correlation coefficient (ICC), and analysis approach on Type I error rates has not been well studied. Reviews of published CRTs find that small sample sizes are not uncommon, so the performance of different inferential approaches in these settings can guide data analysts to the best choices. Methods Using a random-intercept LMM stucture, we use simulations to study Type I error rates with the LRT and Wald test with different degrees of freedom (DF) choices across different combinations of cluster size, number of clusters, and ICC. Results Our simulations show that the LRT can be anti-conservative when the ICC is large and the number of clusters is small, with the effect most pronouced when the cluster size is relatively large. Wald tests with the between-within DF method or the Satterthwaite DF approximation maintain Type I error control at the stated level, though they are conservative when the number of clusters, the cluster size, and the ICC are small. Conclusions Depending on the structure of the CRT, analysts should choose a hypothesis testing approach that will maintain the appropriate Type I error rate for their data. Wald tests with the Satterthwaite DF approximation work well in many circumstances, but in other cases the LRT may have Type I error rates closer to the nominal level.

scholarly journals Type I Error Control for Cluster Randomized Trials Under Varying Small Sample Structures

2019 ◽  
Author(s):  
Joshua Nugent ◽  
Ken Kleinman
Keyword(s):  
Cluster Size ◽  
Error Control ◽  
Type I Error ◽  
Error Rates ◽  
Small Sample ◽  
Type I ◽  
Number Of Clusters ◽  
Type I Error Rates ◽  
Wald Tests ◽  
Cluster Randomized

Abstract Background: Linear mixed models (LMM) are a common approach to analyzing data from cluster randomized trials (CRTs). Inference on parameters can be performed via Wald tests or likelihood ratio tests (LRT), but both approaches may give incorrect Type I error rates in common finite sample settings. The impact of interactions of cluster size, number of clusters, intraclass correlation coefficient (ICC), and analysis approach on Type I error rates have not been well studied. Reviews of published CRTs find that small sample sizes are not uncommon, so the performance of different inferential approaches in these settings can guide data analysts to the best choices. Methods: Using a random-intercept LMM stucture, we use simulations to study Type I error rates with the LRT and Wald test with different degrees of freedom (DF) choices across different combinations of cluster size, number of clusters, and ICC.Results: Our simulations show that the LRT can be anti-conservative when the ICC is large and the number of clusters is small, with the effect most pronouced when the cluster size is relatively large. Wald tests with the Between-Within DF method or the Satterthwaite DF approximation maintain Type I error control at the stated level, though they are conservative when the number of clusters, the cluster size, and the ICC are small. Conclusions: Depending on the structure of the CRT, analysts should choose a hypothesis testing approach that will maintain the appropriate Type I error rate for their data. Wald tests with the Satterthwaite DF approximation work well in many circumstances, but in other cases the LRT may have Type I error rates closer to the nominal level.

scholarly journals Type I Error Control for Cluster Randomized Trials Under Varying Small Sample Structures

2020 ◽  
Author(s):  
Joshua Nugent ◽  
Ken Kleinman
Keyword(s):  
Cluster Size ◽  
Error Control ◽  
Type I Error ◽  
Error Rates ◽  
Small Sample ◽  
Type I ◽  
Number Of Clusters ◽  
Type I Error Rates ◽  
Wald Tests ◽  
Cluster Randomized

Abstract Background: Linear mixed models (LMM) are a common approach to analyzing data from cluster randomized trials (CRTs). Inference on parameters can be performed via Wald tests or likelihood ratio tests (LRT), but both approaches may give incorrect Type I error rates in common finite sample settings. The impact of different combinations of cluster size, number of clusters, intraclass correlation coefficient (ICC), and analysis approach on Type I error rates has not been well studied. Reviews of published CRTs nd that small sample sizes are not uncommon, so the performance of different inferential approaches in these settings can guide data analysts to the best choices.Methods: Using a random-intercept LMM stucture, we use simulations to study Type I error rates with the LRT and Wald test with different degrees of freedom (DF) choices across different combinations of cluster size, number of clusters, and ICC.Results: Our simulations show that the LRT can be anti-conservative when the ICC is large and the number of clusters is small, with the effect most pronounced when the cluster size is relatively large. Wald tests with the between-within DF method or the Satterthwaite DF approximation maintain Type I error control at the stated level, though they are conservative when the number of clusters, the cluster size, and the ICC are small.Conclusions: Depending on the structure of the CRT, analysts should choose a hypothesis testing approach that will maintain the appropriate Type I error rate for their data. Wald tests with the Satterthwaite DF approximation work well in many circumstances, but in other cases the LRT may have Type I error rates closer to the nominal level.

scholarly journals Cluster Wild Bootstrapping to Handle Dependent Effect Sizes in Meta-Analysis with a Small Number of Studies

2021 ◽  
Author(s):  
Megha Joshi ◽  
James E Pustejovsky ◽  
S. Natasha Beretvas
Keyword(s):  
Effect Size ◽  
Type I Error ◽  
Meta Analysis ◽  
Error Rates ◽  
Small Sample ◽  
Type I ◽  
Hypothesis Tests ◽  
Type I Error Rates ◽  
Meta Analyses ◽  
Small Sample Correction

The most common and well-known meta-regression models work under the assumption that there is only one effect size estimate per study and that the estimates are independent. However, meta-analytic reviews of social science research often include multiple effect size estimates per primary study, leading to dependence in the estimates. Some meta-analyses also include multiple studies conducted by the same lab or investigator, creating another potential source of dependence. An increasingly popular method to handle dependence is robust variance estimation (RVE), but this method can result in inflated Type I error rates when the number of studies is small. Small-sample correction methods for RVE have been shown to control Type I error rates adequately but may be overly conservative, especially for tests of multiple-contrast hypotheses. We evaluated an alternative method for handling dependence, cluster wild bootstrapping, which has been examined in the econometrics literature but not in the context of meta-analysis. Results from two simulation studies indicate that cluster wild bootstrapping maintains adequate Type I error rates and provides more power than extant small sample correction methods, particularly for multiple-contrast hypothesis tests. We recommend using cluster wild bootstrapping to conduct hypothesis tests for meta-analyses with a small number of studies. We have also created an R package that implements such tests.

scholarly journals On Small-Sample Inference in Multiphase Stepped Wedge Cluster Randomized Trial (MSW-CRT) With Binary Outcomes

2021 ◽  
Author(s):  
Jing Peng ◽  
Abigail Shoben ◽  
Pengyue Zhang ◽  
Philip M. Westgate ◽  
Soledad Fernandez
Keyword(s):  
Randomized Trial ◽  
Type I Error ◽  
Event Rate ◽  
Cluster Randomized Trial ◽  
Small Sample ◽  
Binary Outcomes ◽  
Type I ◽  
Stepped Wedge ◽  
Number Of Clusters ◽  
Cluster Randomized

Abstract BackgroundThe stepped wedge cluster randomized trial (SW-CRT) design is now preferred for many health- related trials because of its flexibility on resource allocation and clinical ethics concerns. However, as a necessary extension of studying multiple interventions, multiphase stepped wedge designs (MSW-CRT) have not been studied adequately. Since estimated intervention effect from Generalized estimating equations (GEE) has a population-average interpretation, valid inference methods for binary outcomes based on GEE are preferred by public health policy makers.MethodsWe form hypothesis testing of add-on effect of a second treatment based on GEE analysis in an MSW-CRT design with limited number of clusters. Four variance-correction estimators are used to adjust the bias of the sandwich estimator. Simulation studies have been used to compare the statistical power and type I error rate of these methods under different correlation matrices.Results We demonstrate that an average estimator with t(I-3) can stably maintain type I error close to the nominal level with limited sample sizes in our settings. We show that power of testing the add-on effect depends on the baseline event rate, effect sizes of two interventions and the number of clusters. Moreover, by changing the design with including more sequences, power benefit can be achieved. ConclusionsFor designing the MSW-CRT, we suggest using more sequences and checking event rate after initiating the first intervention via interim analysis. When the number of clusters is not very large in MSW-CRTs, inference can be conduct using GEE analysis with an average estimator with t(I-3) sampling distribution.

Performance of Monte Carlo Permutation and Approximate Tests for Multivariate Means Comparisons With Small Sample Sizes When Parametric Assumptions are Violated

Methodology ◽  
2009 ◽  
Vol 5 (2) ◽  
pp. 60-70 ◽  
Author(s):  
W. Holmes Finch ◽  
Teresa Davenport
Keyword(s):  
Monte Carlo ◽  
Type I Error ◽  
Permutation Tests ◽  
Error Rates ◽  
Covariance Matrices ◽  
Small Sample ◽  
Type I ◽  
Permutation Testing ◽  
Sample Sizes ◽  
Type I Error Rates

Permutation testing has been suggested as an alternative to the standard F approximate tests used in multivariate analysis of variance (MANOVA). These approximate tests, such as Wilks’ Lambda and Pillai’s Trace, have been shown to perform poorly when assumptions of normally distributed dependent variables and homogeneity of group covariance matrices were violated. Because Monte Carlo permutation tests do not rely on distributional assumptions, they may be expected to work better than their approximate cousins when the data do not conform to the assumptions described above. The current simulation study compared the performance of four standard MANOVA test statistics with their Monte Carlo permutation-based counterparts under a variety of conditions with small samples, including conditions when the assumptions were met and when they were not. Results suggest that for sample sizes of 50 subjects, power is very low for all the statistics. In addition, Type I error rates for both the approximate F and Monte Carlo tests were inflated under the condition of nonnormal data and unequal covariance matrices. In general, the performance of the Monte Carlo permutation tests was slightly better in terms of Type I error rates and power when both assumptions of normality and homogeneous covariance matrices were not met. It should be noted that these simulations were based upon the case with three groups only, and as such results presented in this study can only be generalized to similar situations.

scholarly journals Analysis of type I and II error rates of Bayesian and frequentist parametric and nonparametric two-sample hypothesis tests under preliminary assessment of normality

2020 ◽  
Author(s):  
Riko Kelter
Keyword(s):  
Null Hypothesis ◽  
Error Control ◽  
Type I Error ◽  
Error Rates ◽  
Type I ◽  
Type Ii ◽  
Type Ii Error ◽  
Preliminary Assessment ◽  
Type I Error Rates ◽  
Practical Research

Abstract Testing for differences between two groups is among the most frequently carried out statistical methods in empirical research. The traditional frequentist approach is to make use of null hypothesis significance tests which use p values to reject a null hypothesis. Recently, a lot of research has emerged which proposes Bayesian versions of the most common parametric and nonparametric frequentist two-sample tests. These proposals include Student’s two-sample t-test and its nonparametric counterpart, the Mann–Whitney U test. In this paper, the underlying assumptions, models and their implications for practical research of recently proposed Bayesian two-sample tests are explored and contrasted with the frequentist solutions. An extensive simulation study is provided, the results of which demonstrate that the proposed Bayesian tests achieve better type I error control at slightly increased type II error rates. These results are important, because balancing the type I and II errors is a crucial goal in a variety of research, and shifting towards the Bayesian two-sample tests while simultaneously increasing the sample size yields smaller type I error rates. What is more, the results highlight that the differences in type II error rates between frequentist and Bayesian two-sample tests depend on the magnitude of the underlying effect.

scholarly journals Differences of Type I error rates for ANOVA and Multilevel-Linear-Models using SAS and SPSS for repeated measures designs

2019 ◽  
Vol 3 ◽  
Author(s):  
Nicolas Haverkamp ◽  
André Beauducel
Keyword(s):  
Repeated Measures ◽  
Linear Models ◽  
Type I Error ◽  
Error Rates ◽  
Small Sample ◽  
Small Samples ◽  
Type I ◽  
Sample Sizes ◽  
Type I Error Rates ◽  
Multilevel Linear Models

  To derive recommendations on how to analyze longitudinal data, we examined Type I error rates of Multilevel Linear Models (MLM) and repeated measures Analysis of Variance (rANOVA) using SAS and SPSS. We performed a simulation with the following specifications: To explore the effects of high numbers of measurement occasions and small sample sizes on Type I error, measurement occasions of m = 9 and 12 were investigated as well as sample sizes of n = 15, 20, 25 and 30. Effects of non-sphericity in the population on Type I error were also inspected: 5,000 random samples were drawn from two populations containing neither a within-subject nor a between-group effect. They were analyzed including the most common options to correct rANOVA and MLM-results: The Huynh-Feldt-correction for rANOVA (rANOVA-HF) and the Kenward-Roger-correction for MLM (MLM-KR), which could help to correct progressive bias of MLM with an unstructured covariance matrix (MLM-UN). Moreover, uncorrected rANOVA and MLM assuming a compound symmetry covariance structure (MLM-CS) were also taken into account. The results showed a progressive bias for MLM-UN for small samples which was stronger in SPSS than in SAS. Moreover, an appropriate bias correction for Type I error via rANOVA-HF and an insufficient correction by MLM-UN-KR for n < 30 were found. These findings suggest MLM-CS or rANOVA if sphericity holds and a correction of a violation via rANOVA-HF. If an analysis requires MLM, SPSS yields more accurate Type I error rates for MLM-CS and SAS yields more accurate Type I error rates for MLM-UN.

A Monte Carlo Comparison of Seven ε-Adjustment Procedures in Repeated Measures Designs With Small Sample Sizes

1994 ◽  
Vol 19 (1) ◽  
pp. 57-71 ◽  
Author(s):  
Stephen M. Quintana ◽  
Scott E. Maxwell
Keyword(s):  
Repeated Measures ◽  
Type I Error ◽  
Error Rates ◽  
Small Sample ◽  
Small Samples ◽  
Type I ◽  
Sample Sizes ◽  
Type I Error Rates ◽  
Repeated Measures Designs ◽  
Small Sample Sizes

The purpose of this study was to evaluate seven univariate procedures for testing omnibus null hypotheses for data gathered from repeated measures designs. Five alternate approaches are compared to the two more traditional adjustment procedures (Geisser and Greenhouse’s ε̂ and Huynh and Feldt’s ε̃), neither of which may be entirely adequate when sample sizes are small and the number of levels of the repeated factors is large. Empirical Type I error rates and power levels were obtained by simulation for conditions where small samples occur in combination with many levels of the repeated factor. Results suggested that alternate univariate approaches were improvements to the traditional approaches. One alternate approach in particular was found to be most effective in controlling Type I error rates without unduly sacrificing power.

Sign in / Sign up

Export Citation Format

Share Document