scholarly journals Machine learning analysis plans for randomised controlled trials: detecting treatment effect heterogeneity with strict control of type I error

Trials ◽  
2020 ◽  
Vol 21 (1) ◽  
Author(s):  
James A. Watson ◽  
Chris C. Holmes
Keyword(s):  
Machine Learning ◽  
Treatment Effect ◽  
Randomised Controlled Trials ◽  
Type I Error ◽  
Controlled Trials ◽  
Type I ◽  
Treatment Effect Heterogeneity ◽  
Randomised Controlled ◽  
Effect Heterogeneity ◽  
Learning Analysis

scholarly journals Machine learning for randomised controlled trials: identifying treatment effectheterogeneity with strict control of type I error

2018 ◽  
Author(s):  
Chris C. Holmes ◽  
James A. Watson
Keyword(s):  
Machine Learning ◽  
Learning Strategies ◽  
Randomised Controlled Trials ◽  
Hypothesis Test ◽  
Data Partitioning ◽  
P Value ◽  
Controlled Trials ◽  
Type I ◽  
Strict Control ◽  
Randomised Controlled

AbstractBackgroundIt is widely acknowledged that retrospective exploratory analyses of randomised controlled trials (RCTs) seeking to identify treatment effect heterogeneity (TEH) are prone to bias and false positives. Yet the increasing availability of multiple data modalities on subjects and the desire to learn all we can from trial participants motivates the inclusion of such analyses within RCTs. Coupled to this, widespread advances in AI and machine learning (ML) methods hold great potential to utilise such data to characterise subjects exhibiting heterogeneous treatment response.MethodsWe present new learning strategies for RCT ML discovery methods that ensure strict control of the false positive reporting rate at a pre-specified level. Our approach uses randomised data partitioning and statistical or ML based prediction on held-out data. This can test for both crossover and non-crossover TEH. The former is done via a two-sample hypothesis test measuring overall predictive performance of the ML method. The latter is done via ‘stacking’ the ML predictors alongside a classical statistical model to formally test the added benefit of the ML algorithm. An adaptation of recent statistical theory allows for the construction of a valid aggregate p-value. This learning strategy is agnostic to the choice of ML method.ResultsWe demonstrate our approach with a re-analysis of the SEAQUAMAT trial. We find no evidence for any crossover subgroup who would benefit from a change in treatment from the current standard-of-care, artesunate, but strong evidence for significant noncrossover TEH within the artesunate treatment group. We find that artesunate provides a differential benefit to patients with high numbers of circulating ring stage parasites.ConclusionsOur ML approach combined with the use of computational notebooks and version control can improve the robustness and transparency of RCT exploratory analyses. The methods allow researchers to apply the latest ML techniques safe in the knowledge that any declared associations are statistically significant at a user defined level.

scholarly journals Assessing effect heterogeneity of a randomized treatment using conditional inference trees

2021 ◽  
pp. 096228022110528
Author(s):  
Ashwini Venkatasubramaniam ◽  
Brandon Koch ◽  
Lauren Erickson ◽  
Simone French ◽  
David Vock ◽  
...  
Keyword(s):  
Treatment Effect ◽  
Type I Error ◽  
Prognostic Score ◽  
Individual Characteristics ◽  
Conditional Inference ◽  
Type I ◽  
Novel Approach ◽  
Treatment Effect Heterogeneity ◽  
Conditional Inference Trees ◽  
Effect Heterogeneity

Treatment effect heterogeneity occurs when individual characteristics influence the effect of a treatment. We propose a novel approach that combines prognostic score matching and conditional inference trees to characterize effect heterogeneity of a randomized binary treatment. One key feature that distinguishes our method from alternative approaches is that it controls the Type I error rate, that is, the probability of identifying effect heterogeneity if none exists and retains the underlying subgroups. This feature makes our technique particularly appealing in the context of clinical trials, where there may be significant costs associated with erroneously declaring that effects differ across population subgroups. Treatment effect heterogeneity trees are able to identify heterogeneous subgroups, characterize the relevant subgroups and estimate the associated treatment effects. We demonstrate the efficacy of the proposed method using a comprehensive simulation study and illustrate our method using a nutrition trial dataset to evaluate effect heterogeneity within a patient population.

Treating depression with physical activity in adolescents and young adults: a systematic review and meta-analysis of randomised controlled trials

2017 ◽  
Vol 48 (7) ◽  
pp. 1068-1083 ◽  
Author(s):  
A. P. Bailey ◽  
S. E. Hetrick ◽  
S. Rosenbaum ◽  
R. Purcell ◽  
A. G. Parker
Keyword(s):  
Physical Activity ◽  
Young Adults ◽  
Treatment Effect ◽  
Randomised Controlled Trials ◽  
Meta Analysis ◽  
Risk Of Bias ◽  
Adolescents And Young Adults ◽  
Controlled Trials ◽  
Randomised Controlled

AbstractWe aimed to establish the treatment effect of physical activity for depression in young people through meta-analysis. Four databases were searched to September 2016 for randomised controlled trials of physical activity interventions for adolescents and young adults, 12–25 years, experiencing a diagnosis or threshold symptoms of depression. Random-effects meta-analysis was used to estimate the standardised mean difference (SMD) between physical activity and control conditions. Subgroup analysis and meta-regression investigated potential treatment effect modifiers. Acceptability was estimated using dropout. Trials were assessed against risk of bias domains and overall quality of evidence was assessed using GRADE criteria. Seventeen trials were eligible and 16 provided data from 771 participants showing a large effect of physical activity on depression symptoms compared to controls (SMD = −0.82, 95% CI = −1.02 to −0.61, p < 0.05, I2 = 38%). The effect remained robust in trials with clinical samples (k = 5, SMD = −0.72, 95% CI = −1.15 to −0.30), and in trials using attention/activity placebo controls (k = 7, SMD = −0.82, 95% CI = −1.05 to −0.59). Dropout was 11% across physical activity arms and equivalent in controls (k = 12, RD = −0.01, 95% CI = −0.04 to 0.03, p = 0.70). However, the quality of RCT-level evidence contributing to the primary analysis was downgraded two levels to LOW (trial-level risk of bias, suspected publication bias), suggesting uncertainty in the size of effect and caution in its interpretation. While physical activity appears to be a promising and acceptable intervention for adolescents and young adults experiencing depression, robust clinical effectiveness trials that minimise risk of bias are required to increase confidence in the current finding. The specific intervention characteristics required to improve depression remain unclear, however best candidates given current evidence may include, but are not limited to, supervised, aerobic-based activity of moderate-to-vigorous intensity, engaged in multiple times per week over eight or more weeks. Further research is needed. (Registration: PROSPERO-CRD 42015024388).

Who Benefits from Electroconvulsive Therapy?

1992 ◽  
Vol 160 (3) ◽  
pp. 355-359 ◽  
Author(s):  
Heather Buchan ◽  
Eve Johnstone ◽  
K. McPherson ◽  
R. L. Palmer ◽  
T. J. Crow ◽  
...  
Keyword(s):  
Electroconvulsive Therapy ◽  
Treatment Effect ◽  
Randomised Controlled Trials ◽  
Controlled Trials ◽  
Combining Data ◽  
Randomised Controlled ◽  
Improved Outcome

This paper describes the results obtained by combining data from the Northwick Park and Leicester randomised controlled trials of ECT. Patients who suffered from depression in which retardation and delusions were features and who received real ECT had a significantly improved outcome at the end of four weeks of treatment (as measured by improvement in the HRSD) in comparison with those who received simulated ECT. However, this treatment effect was not detectable at six-month follow-up. Patients who were neither retarded nor deluded did not benefit significantly from real as opposed to simulated ECT.

Adjusting for treatment switching in randomised controlled trials – A simulation study and a simplified two-stage method

2014 ◽  
Vol 26 (2) ◽  
pp. 724-751 ◽  
Author(s):  
Nicholas R Latimer ◽  
KR Abrams ◽  
PC Lambert ◽  
MJ Crowther ◽  
AJ Wailoo ◽  
...  
Keyword(s):  
Treatment Effect ◽  
Survival Benefit ◽  
Randomised Controlled Trials ◽  
Health Technology ◽  
Control Group ◽  
Controlled Trials ◽  
Inverse Probability ◽  
Treatment Switching ◽  
Overall Survival Benefit ◽  
Randomised Controlled

Estimates of the overall survival benefit of new cancer treatments are often confounded by treatment switching in randomised controlled trials (RCTs) – whereby patients randomised to the control group are permitted to switch onto the experimental treatment upon disease progression. In health technology assessment, estimates of the unconfounded overall survival benefit associated with the new treatment are needed. Several switching adjustment methods have been advocated in the literature, some of which have been used in health technology assessment. However, it is unclear which methods are likely to produce least bias in realistic RCT-based scenarios. We simulated RCTs in which switching, associated with patient prognosis, was permitted. Treatment effect size and time dependency, switching proportions and disease severity were varied across scenarios. We assessed the performance of alternative adjustment methods based upon bias, coverage and mean squared error, related to the estimation of true restricted mean survival in the absence of switching in the control group. We found that when the treatment effect was not time-dependent, rank preserving structural failure time models (RPSFTM) and iterative parameter estimation methods produced low levels of bias. However, in the presence of a time-dependent treatment effect, these methods produced higher levels of bias, similar to those produced by an inverse probability of censoring weights method. The inverse probability of censoring weights and structural nested models produced high levels of bias when switching proportions exceeded 85%. A simplified two-stage Weibull method produced low bias across all scenarios and provided the treatment switching mechanism is suitable, represents an appropriate adjustment method.

scholarly journals Examination of overall treatment effect and the proportion attributable to contextual effect in osteoarthritis: meta-analysis of randomised controlled trials

2016 ◽  
Vol 75 (11) ◽  
pp. 1964-1970 ◽  
Author(s):  
Kun Zou ◽  
Jean Wong ◽  
Natasya Abdullah ◽  
Xi Chen ◽  
Toby Smith ◽  
...  
Keyword(s):  
Treatment Effect ◽  
Randomised Controlled Trials ◽  
Specific Effect ◽  
Pain Reduction ◽  
Contextual Effect ◽  
Controlled Trials ◽  
Large Sample Size ◽  
Duration Of Treatment ◽  
Manual Search ◽  
Randomised Controlled

ObjectiveTo examine the overall treatment effect and the proportion attributable to contextual effect (PCE) in randomised controlled trials (RCTs) of diverse treatments for osteoarthritis (OA).MethodsWe searched Medline, Embase, Central, Science Citation Index, AMED and CINAHL through October 2014, supplemented with manual search of reference lists, published meta-analyses and systematic reviews. Included were RCTs in OA comparing placebo with representative complementary, pharmacological, non-pharmacological and surgical treatments. The primary outcome was pain. Secondary outcomes were function and stiffness. The effect size (ES) of overall treatment effect and the PCE were pooled using random-effects model. Subgroup analyses and meta-regression were conducted to examine determinants of the PCE.ResultsIn total, 215 trials (41 392 participants) were included. The overall treatment effect for pain ranged from the smallest with lavage (ES=0.46, 95% CI 0.24 to 0.68) to the largest with topical non-steroidal anti-inflammatory drugs (ES=1.37, 95% CI 1.19 to 1.55). On average, 75% (PCE=0.75, 95% CI 0.72 to 0.79) of pain reduction was attributable to contextual effect. It varied by treatment from 47% (PCE=0.47, 95% CI 0.32 to 0.70) for intra-articular corticosteroid to 91% (PCE=0.91, 95% CI 0.60 to 1.37) for joint lavage. Similar results were observed for function and stiffness. Treatment delivered by needle/injection and other means than oral medication, longer duration of treatment, large sample size (≥100 per arm) and public funding source were associated with increased PCE for pain reduction.ConclusionsThe majority (75%) of the overall treatment effect in OA RCTs is attributable to contextual effects rather than the specific effect of treatments. Reporting overall treatment effect and PCE, in addition to traditional ES, permits a more balanced, clinically meaningful interpretation of RCT results. This would help dispel the frequent discordance between conclusions from RCT evidence and clinical experience—the ‘efficacy paradox’.

Sign in / Sign up

Export Citation Format

Share Document