A Framework for Using Real-World Data and Health Outcomes Modeling to Evaluate Machine Learning–Based Risk Prediction Models

2021 ◽  
Author(s):  
Patricia J. Rodriguez ◽  
David L. Veenstra ◽  
Patrick J. Heagerty ◽  
Christopher H. Goss ◽  
Kathleen J. Ramos ◽  
...  
Keyword(s):  
Machine Learning ◽  
Health Outcomes ◽  
Risk Prediction ◽  
Real World ◽  
Prediction Models ◽  
Real World Data ◽  
Risk Prediction Models ◽  
World Data

Performance Metrics for the Comparative Analysis of Clinical Risk Prediction Models Employing Machine Learning

2021 ◽  
Author(s):  
Chenxi Huang ◽  
Shu-Xia Li ◽  
César Caraballo ◽  
Frederick A. Masoudi ◽  
John S. Rumsfeld ◽  
...  
Keyword(s):  
Machine Learning ◽  
Risk Prediction ◽  
Health Care Professionals ◽  
Clinical Decision Making ◽  
Performance Metrics ◽  
Prediction Models ◽  
Learning Models ◽  
Risk Prediction Models ◽  
Clinical Risk ◽  
Machine Learning Models

Background: New methods such as machine learning techniques have been increasingly used to enhance the performance of risk predictions for clinical decision-making. However, commonly reported performance metrics may not be sufficient to capture the advantages of these newly proposed models for their adoption by health care professionals to improve care. Machine learning models often improve risk estimation for certain subpopulations that may be missed by these metrics. Methods and Results: This article addresses the limitations of commonly reported metrics for performance comparison and proposes additional metrics. Our discussions cover metrics related to overall performance, discrimination, calibration, resolution, reclassification, and model implementation. Models for predicting acute kidney injury after percutaneous coronary intervention are used to illustrate the use of these metrics. Conclusions: We demonstrate that commonly reported metrics may not have sufficient sensitivity to identify improvement of machine learning models and propose the use of a comprehensive list of performance metrics for reporting and comparing clinical risk prediction models.

scholarly journals Developing a machine learning environmental allergy prediction model from real world data through a novel decentralized mobile study platform.

2020 ◽  
Author(s):  
Chethan Sarabu ◽  
Sandra Steyaert ◽  
Nirav Shah
Keyword(s):  
Machine Learning ◽  
Real World ◽  
Predictive Models ◽  
Clinical Care ◽  
Sensor Data ◽  
Real World Data ◽  
World Data ◽  
Demographic Groups ◽  
Prediction And Prevention ◽  
Wide Range

Environmental allergies cause significant morbidity across a wide range of demographic groups. This morbidity could be mitigated through individualized predictive models capable of guiding personalized preventive measures. We developed a predictive model by integrating smartphone sensor data with symptom diaries maintained by patients. The machine learning model was found to be highly predictive, with an accuracy of 0.801. Such models based on real-world data can guide clinical care for patients and providers, reduce the economic burden of uncontrolled allergies, and set the stage for subsequent research pursuing allergy prediction and prevention. Moreover, this study offers proof-of-principle regarding the feasibility of building clinically useful predictive models from 'messy,' participant derived real-world data.

scholarly journals An application of machine learning based on real-world data: Mining features of fibrinogen in clinical stages of lung cancer between sexes

2021 ◽  
Vol 9 (8) ◽  
pp. 623-623
Author(s):  
Fangtao Yin ◽  
Hongyu Zhu ◽  
Songlin Hong ◽  
Chen Sun ◽  
Jie Wang ◽  
...  
Keyword(s):  
Machine Learning ◽  
Lung Cancer ◽  
Data Mining ◽  
Real World ◽  
Real World Data ◽  
World Data ◽  
Clinical Stages

scholarly journals Predicting Response to Tocilizumab Monotherapy in Rheumatoid Arthritis: A Real-World Data Analysis Using Machine Learning

2020 ◽  
Author(s):  
Fredrik D Johansson ◽  
Jamie E Collins ◽  
Vincent Yau ◽  
Hongshu Guan ◽  
Seoyoung C Kim ◽  
...  
Keyword(s):  
Rheumatoid Arthritis ◽  
Logistic Regression ◽  
Real World ◽  
Prediction Models ◽  
External Validation ◽  
Similar Efficacy ◽  
Prediction Score ◽  
Real World Data ◽  
World Data ◽  
Biologic Dmard

Abstract Background Tocilizumab (TCZ) had similar efficacy when used as monotherapy or in combination with other treatments for rheumatoid arthritis (RA) in randomized controlled trials (RCT). Recently, we derived a remission prediction score for TCZ monotherapy (TCZm) using RCT data. Herein, we describe external validation and several extensions of the prediction score using “real world data” (RWD).MethodsWe identified patients in Corrona-RA who used TCZm (n=453), matching the design and patients from four RCTs used in previous work (n=853). Patients were followed to determine remission status at 24 weeks. We compared the performance of remission prediction models in RWD, first based on variables determined in our prior work in RCTs, and then using an extended variable set, comparing logistic regression and random forest models. We included patients on other biologic DMARD monotherapies (bDMARDm) to improve prediction.ResultsThe fraction of patients observed reaching remission on TCZm by their follow-up visit was 12% (n=53) in RWD vs 15% (n=127) in RCTs. Discrimination was good in RWD for the risk score developed in RCTS with AUROC of 0.70 (95% CI 0.64, 0.77). Fitting the same logistic regression model to all bDMARDm patients in the RWD improved the AUROC on TCZm patients to 0.73 (95% CI 0.64, 0.82). Extending the variable set and adding regularization further increased it to 0.77 (95% CI 0.68, 0.85).ConclusionThe remission prediction scores, derived in RCTs, discriminated patients in RWD about as well as in RCTs. Discrimination was further improved by retraining models on RWD, including a larger variable set and learning from patients on similar therapies.

Sign in / Sign up

Export Citation Format

Share Document