scholarly journals Waiting for Baseline Stability in Single-Case Designs: Is It Worth the Time and Effort?

2021 ◽  
Author(s):  
Marc J Lanovaz ◽  
Rachel Primiani
Keyword(s):  
Machine Learning ◽  
Decision Making ◽  
Error Rate ◽  
Type I Error ◽  
Single Case ◽  
Support Vector ◽  
Type I ◽  
Type I Error Rate ◽  
Support Vector Classifier ◽  
Single Case Designs

Researchers and practitioners often use single-case designs (SCDs), or n-of-1 trials, to develop and validate novel treatments. Standards and guidelines have been published to provide guidance as to how to implement SCDs, but many of their recommendations are not derived from the research literature. For example, one of these recommendations suggests that researchers and practitioners should wait for baseline stability prior to introducing an independent variable. However, this recommendation is not strongly supported by empirical evidence. To address this issue, we used a Monte Carlo simulation to generate a total of 480,000 AB graphs with fixed, response-guided, and random baseline lengths. Then, our analyses compared the Type I error rate and power produced by two methods of analysis: the conservative dual-criteria method (a structured visual aid) and a support vector classifier (a model derived from machine learning). The conservative dual-criteria method produced more power when using response-guided decision-making (i.e., waiting for stability) with negligeable effects on Type I error rate. In contrast, waiting for stability did not reduce decision-making errors with the support vector classifier. Our findings question the necessity of waiting for baseline stability when using SCDs with machine learning, but the study must be replicated with other designs to support our results.

scholarly journals Machine Learning to Analyze Single-Case Graphs: A Comparison to Visual Inspection

2021 ◽  
Author(s):  
Marc J Lanovaz ◽  
Kieva Hranchuk
Keyword(s):  
Machine Learning ◽  
Error Rate ◽  
Effect Size ◽  
Visual Inspection ◽  
Type I Error ◽  
Single Case ◽  
Type I ◽  
Type I Error Rate ◽  
Novel Approach ◽  
Behavior Analysts

Behavior analysts commonly use visual inspection to analyze single-case graphs, but studies on its reliability have produced mixed results. To examine this issue, we compared the Type I error rate and power of visual inspection with a novel approach, machine learning. Five expert visual raters analyzed 1,024 simulated AB graphs, which differed on number of points per phase, autocorrelation, trend, variability, and effect size. The ratings were compared to those obtained by the conservative dual-criteria method and two models derived from machine learning. On average, visual raters agreed with each other on only 73% of graphs. In contrast, both models derived from machine learning showed the best balance between Type I error rate and power while producing more consistent results across different graph characteristics. The results suggest that machine learning may support researchers and practitioners in making less error when analyzing single-case graphs, but further replications remain necessary.

scholarly journals Using AB Designs With Nonoverlap Effect Size Measures to Support Clinical Decision-Making: A Monte Carlo Validation

2019 ◽  
pp. 014544551986021 ◽  
Author(s):  
Antonia R. Giannakakos ◽  
Marc J. Lanovaz
Keyword(s):  
Error Rate ◽  
Effect Size ◽  
Clinical Decision Making ◽  
Type I Error ◽  
Single Case ◽  
Simulated Data ◽  
Type I ◽  
Type I Error Rate ◽  
Sufficient Power ◽  
Quasi Experimental

Single-case experimental designs often require extended baselines or the withdrawal of treatment, which may not be feasible or ethical in some practical settings. The quasi-experimental AB design is a potential alternative, but more research is needed on its validity. The purpose of our study was to examine the validity of using nonoverlap measures of effect size to detect changes in AB designs using simulated data. In our analyses, we determined thresholds for three effect size measures beyond which the type I error rate would remain below 0.05 and then examined whether using these thresholds would provide sufficient power. Overall, our analyses show that some effect size measures may provide adequate control over type I error rate and sufficient power when analyzing data from AB designs. In sum, our results suggest that practitioners may use quasi-experimental AB designs in combination with effect size to rigorously assess progress in practice.

Correction: “Influence of Selection Bias on the Test Decision – A Simulation Study”

2014 ◽  
Vol 53 (05) ◽  
pp. 343-343
Keyword(s):  
Selection Bias ◽  
Simulation Study ◽  
Error Rate ◽  
Type I Error ◽  
Block Size ◽  
Error Rates ◽  
Type I ◽  
Type I Error Rate ◽  
Representation Error ◽  
Numeric Representation

We have to report marginal changes in the empirical type I error rates for the cut-offs 2/3 and 4/7 of Table 4, Table 5 and Table 6 of the paper “Influence of Selection Bias on the Test Decision – A Simulation Study” by M. Tamm, E. Cramer, L. N. Kennes, N. Heussen (Methods Inf Med 2012; 51: 138 –143). In a small number of cases the kind of representation of numeric values in SAS has resulted in wrong categorization due to a numeric representation error of differences. We corrected the simulation by using the round function of SAS in the calculation process with the same seeds as before. For Table 4 the value for the cut-off 2/3 changes from 0.180323 to 0.153494. For Table 5 the value for the cut-off 4/7 changes from 0.144729 to 0.139626 and the value for the cut-off 2/3 changes from 0.114885 to 0.101773. For Table 6 the value for the cut-off 4/7 changes from 0.125528 to 0.122144 and the value for the cut-off 2/3 changes from 0.099488 to 0.090828. The sentence on p. 141 “E.g. for block size 4 and q = 2/3 the type I error rate is 18% (Table 4).” has to be replaced by “E.g. for block size 4 and q = 2/3 the type I error rate is 15.3% (Table 4).”. There were only minor changes smaller than 0.03. These changes do not affect the interpretation of the results or our recommendations.

Controlling type I error rate for fast track drug development programmes

2003 ◽  
Vol 22 (5) ◽  
pp. 665-675 ◽  
Author(s):  
Weichung J. Shih ◽  
Peter Ouyang ◽  
Hui Quan ◽  
Yong Lin ◽  
Bart Michiels ◽  
...  
Keyword(s):  
Drug Development ◽  
Error Rate ◽  
Fast Track ◽  
Type I Error ◽  
Type I ◽  
Type I Error Rate

Alternative models and randomization techniques for Bayesian response-adaptive randomization with binary outcomes

2021 ◽  
pp. 174077452110101
Author(s):  
Jennifer Proper ◽  
John Connett ◽  
Thomas Murray
Keyword(s):  
Logistic Regression ◽  
Sample Size ◽  
Error Rate ◽  
Adaptive Design ◽  
Type I Error ◽  
Probability Model ◽  
Binary Outcomes ◽  
Type I ◽  
Operating Characteristics ◽  
Type I Error Rate

Background: Bayesian response-adaptive designs, which data adaptively alter the allocation ratio in favor of the better performing treatment, are often criticized for engendering a non-trivial probability of a subject imbalance in favor of the inferior treatment, inflating type I error rate, and increasing sample size requirements. The implementation of these designs using the Thompson sampling methods has generally assumed a simple beta-binomial probability model in the literature; however, the effect of these choices on the resulting design operating characteristics relative to other reasonable alternatives has not been fully examined. Motivated by the Advanced R2 Eperfusion STrategies for Refractory Cardiac Arrest trial, we posit that a logistic probability model coupled with an urn or permuted block randomization method will alleviate some of the practical limitations engendered by the conventional implementation of a two-arm Bayesian response-adaptive design with binary outcomes. In this article, we discuss up to what extent this solution works and when it does not. Methods: A computer simulation study was performed to evaluate the relative merits of a Bayesian response-adaptive design for the Advanced R2 Eperfusion STrategies for Refractory Cardiac Arrest trial using the Thompson sampling methods based on a logistic regression probability model coupled with either an urn or permuted block randomization method that limits deviations from the evolving target allocation ratio. The different implementations of the response-adaptive design were evaluated for type I error rate control across various null response rates and power, among other performance metrics. Results: The logistic regression probability model engenders smaller average sample sizes with similar power, better control over type I error rate, and more favorable treatment arm sample size distributions than the conventional beta-binomial probability model, and designs using the alternative randomization methods have a negligible chance of a sample size imbalance in the wrong direction. Conclusion: Pairing the logistic regression probability model with either of the alternative randomization methods results in a much improved response-adaptive design in regard to important operating characteristics, including type I error rate control and the risk of a sample size imbalance in favor of the inferior treatment.

Anova Tests for Homogeneity of Variance: Nonnormality and Unequal Samples

1977 ◽  
Vol 2 (3) ◽  
pp. 187-206 ◽  
Author(s):  
Charles G. Martin ◽  
Paul A. Games
Keyword(s):  
Error Rate ◽  
Type I Error ◽  
Type I ◽  
Empirical Comparison ◽  
Jackknife Test ◽  
Type I Error Rate ◽  
Power And Control ◽  
Homogeneity Of Variance ◽  
Test Use ◽  
And Control

This paper presents an exposition and an empirical comparison of two potentially useful tests for homogeneity of variance. Control of Type I error rate, P(EI), and power are investigated for three forms of the Box test and for two forms of the jackknife test with equal and unequal n's under conditions of normality and nonnormality. The Box test is shown to be robust to violations of the assumption of normality. The jackknife test is shown not to be robust. When n's are unequal, the problem of heterogeneous within-cell variances of the transformed values and unequal n's affects the jackknife and Box tests. Previously reported suggestions for selecting subsample sizes for the Box test are shown to be inappropriate, producing an inflated P(EI). Two procedures which alleviate this problem are presented for the Box test. Use of the jack-knife test with a reduced alpha is shown to provide power and control of P(EI) at approximately the same level as the Box test. Recommendations for the use of these techniques and computational examples of each are provided.

Group sequential designs with robust semiparametric recurrent event models

2018 ◽  
Vol 28 (8) ◽  
pp. 2385-2403 ◽  
Author(s):  
Tobias Mütze ◽  
Ekkehard Glimm ◽  
Heinz Schmidli ◽  
Tim Friede
Keyword(s):  
Error Rate ◽  
Joint Distribution ◽  
Recurrent Events ◽  
Type I Error ◽  
Type I ◽  
Sequential Designs ◽  
Group Sequential ◽  
Type I Error Rate ◽  
Sequential Procedures ◽  
Group Sequential Designs

Robust semiparametric models for recurrent events have received increasing attention in the analysis of clinical trials in a variety of diseases including chronic heart failure. In comparison to parametric recurrent event models, robust semiparametric models are more flexible in that neither the baseline event rate nor the process inducing between-patient heterogeneity needs to be specified in terms of a specific parametric statistical model. However, implementing group sequential designs in the robust semiparametric model is complicated by the fact that the sequence of Wald statistics does not follow asymptotically the canonical joint distribution. In this manuscript, we propose two types of group sequential procedures for a robust semiparametric analysis of recurrent events. The first group sequential procedure is based on the asymptotic covariance of the sequence of Wald statistics and it guarantees asymptotic control of the type I error rate. The second procedure is based on the canonical joint distribution and does not guarantee asymptotic type I error rate control but is easy to implement and corresponds to the well-known standard approach for group sequential designs. Moreover, we describe how to determine the maximum information when planning a clinical trial with a group sequential design and a robust semiparametric analysis of recurrent events. We contrast the operating characteristics of the proposed group sequential procedures in a simulation study motivated by the ongoing phase 3 PARAGON-HF trial (ClinicalTrials.gov identifier: NCT01920711) in more than 4600 patients with chronic heart failure and a preserved ejection fraction. We found that both group sequential procedures have similar operating characteristics and that for some practically relevant scenarios, the group sequential procedure based on the canonical joint distribution has advantages with respect to the control of the type I error rate. The proposed method for calculating the maximum information results in appropriately powered trials for both procedures.

scholarly journals Data-driven region-of-interest selection without inflating Type I error rate

2016 ◽  
Vol 54 (1) ◽  
pp. 100-113 ◽  
Author(s):  
Joseph L. Brooks ◽  
Alexia Zoumpoulaki ◽  
Howard Bowman
Keyword(s):  
Error Rate ◽  
Type I Error ◽  
Region Of Interest ◽  
Data Driven ◽  
Type I ◽  
Type I Error Rate
Sign in / Sign up

Export Citation Format

Share Document