scholarly journals Model-Based ROC Curve: Examining the Effect of Case Mix and Model Calibration on the ROC Plot

2021 ◽  
pp. 0272989X2110509
Author(s):  
Mohsen Sadatsafavi ◽  
Paramita Saha-Chaudhuri ◽  
John Petkau
Keyword(s):  
Risk Prediction ◽  
Roc Curve ◽  
Model Calibration ◽  
Prediction Models ◽  
Roc Curves ◽  
Statistical Test ◽  
Case Mix ◽  
Model Discrimination ◽  
Risk Prediction Models ◽  
Model Based

Background The performance of risk prediction models is often characterized in terms of discrimination and calibration. The receiver-operating characteristic (ROC) curve is widely used for evaluating model discrimination. However, when comparing ROC curves across different samples, the effect of case mix makes the interpretation of discrepancies difficult. Further, compared with model discrimination, evaluating model calibration has not received the same level of attention. Current methods for examining model calibration require specification of smoothing or grouping factors. Methods We introduce the “model-based” ROC curve (mROC) to assess model calibration and the effect of case mix during external validation. The mROC curve is the ROC curve that should be observed if the prediction model is calibrated in the external population. We show that calibration-in-the-large and the equivalence of mROC and ROC curves are together sufficient conditions for the model to be calibrated. Based on this, we propose a novel statistical test for calibration that, unlike current methods, does not require any subjective specification of smoothing or grouping factors. Results Through a stylized example, we demonstrate how mROC separates the effect of case mix and model miscalibration when externally validating a risk prediction model. We present the results of simulation studies that confirm the properties of the new calibration test. A case study on predicting the risk of acute exacerbations of chronic obstructive pulmonary disease puts the developments in a practical context. R code for the implementation of this method is provided. Conclusion mROC can easily be constructed and used to interpret the effect of case mix and calibration on the ROC plot. Given the popularity of ROC curves among applied investigators, this framework can further promote assessment of model calibration. Highlights Compared with examining model discrimination, examining model calibration has not received the same level of attention among investigators who develop or examine risk prediction models. This article introduces the model-based ROC (mROC) curve as the basis for graphical and statistical examination of model calibration on the ROC plot. This article introduces a formal statistical test based on mROC for examining model calibration that does not require arbitrary smoothing or grouping factors. Investigators who develop or validate risk prediction models can now also use the popular ROC plot for examining model calibration, as a critical but often neglected component in predictive analytics.

scholarly journals Epidemic Models for Personalised COVID-19 Isolation and Exit Policies Using Clinical Risk Predictions

2020 ◽  
Author(s):  
Theodoros Evgeniou ◽  
Mathilde Fekom ◽  
Anton Ovchinnikov ◽  
Raphael Porcher ◽  
Camille Pouchol ◽  
...  
Keyword(s):  
Risk Prediction ◽  
Prediction Models ◽  
Epidemic Models ◽  
Sensitivity Analyses ◽  
Model Discrimination ◽  
Risk Models ◽  
Risk Prediction Models ◽  
Seir Model ◽  
Clinical Risk ◽  
Time Sensitivity

Background: In early May 2020, following social distancing measures due to COVID-19, governments consider relaxing lock-down. We combined individual clinical risk predictions with epidemic modelling to examine simulations of risk based differential isolation and exit policies. Methods: We extended a standard susceptible-exposed-infected-removed (SEIR) model to account for personalised predictions of severity, defined by the risk of an individual needing intensive care if infected, and simulated differential isolation policies using COVID-19 data and estimates in France as of early May 2020. We also performed sensitivity analyses. The framework may be used with other epidemic models, with other risk predictions, and for other epidemic outbreaks. Findings: Simulations indicated that, assuming everything else the same, an exit policy considering clinical risk predictions starting on May 11, as planned by the French government, could enable to immediately relax restrictions for an extra 10% (6 700 000 people) or more of the lowest-risk population, and consequently relax the restrictions on the remaining population significantly faster -- while abiding to the current ICU capacity. Similar exit policies without risk predictions would exceed the ICU capacity by a multiple. Sensitivity analyses showed that when the assumed percentage of severe patients among the population decreased, or the prediction model discrimination improved, or the ICU capacity increased, policies based on risk models had a greater impact on the results of epidemic simulations. At the same time, sensitivity analyses also showed that differential isolation policies require the higher risk individuals to comply with recommended restrictions. In general, our simulations demonstrated that risk prediction models could improve policy effectiveness, keeping everything else constant. Interpretation: Clinical risk prediction models can inform new personalised isolation and exit policies, which may lead to both safer and faster outcomes than what can be achieved without such prediction models.

scholarly journals Rare Germline Pathogenic Variants Identified by Multigene Panel Testing and the Risk of Aggressive Prostate Cancer

Cancers ◽  
2021 ◽  
Vol 13 (7) ◽  
pp. 1495
Author(s):  
Tú Nguyen-Dumont ◽  
James G. Dowty ◽  
Robert J. MacInnis ◽  
Jason A. Steen ◽  
Moeen Riaz ◽  
...  
Keyword(s):  
Prostate Cancer ◽  
Risk Prediction ◽  
Prediction Models ◽  
Risk Prediction Models ◽  
Aggressive Prostate Cancer ◽  
Panel Testing ◽  
Pathogenic Variants ◽  
Increased Risk ◽  
Predisposition Genes ◽  
Rare Genetic Variants

While gene panel sequencing is becoming widely used for cancer risk prediction, its clinical utility with respect to predicting aggressive prostate cancer (PrCa) is limited by our current understanding of the genetic risk factors associated with predisposition to this potentially lethal disease phenotype. This study included 837 men diagnosed with aggressive PrCa and 7261 controls (unaffected men and men who did not meet criteria for aggressive PrCa). Rare germline pathogenic variants (including likely pathogenic variants) were identified by targeted sequencing of 26 known or putative cancer predisposition genes. We found that 85 (10%) men with aggressive PrCa and 265 (4%) controls carried a pathogenic variant (p < 0.0001). Aggressive PrCa odds ratios (ORs) were estimated using unconditional logistic regression. Increased risk of aggressive PrCa (OR (95% confidence interval)) was identified for pathogenic variants in BRCA2 (5.8 (2.7–12.4)), BRCA1 (5.5 (1.8–16.6)), and ATM (3.8 (1.6–9.1)). Our study provides further evidence that rare germline pathogenic variants in these genes are associated with increased risk of this aggressive, clinically relevant subset of PrCa. These rare genetic variants could be incorporated into risk prediction models to improve their precision to identify men at highest risk of aggressive prostate cancer and be used to identify men with newly diagnosed prostate cancer who require urgent treatment.

scholarly journals Risk Prediction of Barrett’s Esophagus in a Taiwanese Health Examination Center Based on Regression Models

2021 ◽  
Vol 18 (10) ◽  
pp. 5332
Author(s):  
Po-Hsiang Lin ◽  
Jer-Guang Hsieh ◽  
Hsien-Chung Yu ◽  
Jyh-Horng Jeng ◽  
Chiao-Lin Hsu ◽  
...  
Keyword(s):  
Barrett’S Esophagus ◽  
Risk Prediction ◽  
Barrett's Esophagus ◽  
Gastrointestinal Endoscopy ◽  
Prediction Models ◽  
Upper Gastrointestinal Endoscopy ◽  
Health Examination ◽  
Endoscopic Screening ◽  
Risk Prediction Models ◽  
Upper Gastrointestinal

Determining the target population for the screening of Barrett’s esophagus (BE), a precancerous condition of esophageal adenocarcinoma, remains a challenge in Asia. The aim of our study was to develop risk prediction models for BE using logistic regression (LR) and artificial neural network (ANN) methods. Their predictive performances were compared. We retrospectively analyzed 9646 adults aged ≥20 years undergoing upper gastrointestinal endoscopy at a health examinations center in Taiwan. Evaluated by using 10-fold cross-validation, both models exhibited good discriminative power, with comparable area under curve (AUC) for the LR and ANN models (Both AUC were 0.702). Our risk prediction models for BE were developed from individuals with or without clinical indications of upper gastrointestinal endoscopy. The models have the potential to serve as a practical tool for identifying high-risk individuals of BE among the general population for endoscopic screening.

scholarly journals Evaluating risk prediction models for adults with heart failure: A systematic literature review

PLoS ONE ◽  
2020 ◽  
Vol 15 (1) ◽  
pp. e0224135 ◽  
Author(s):  
Gian Luca Di Tanna ◽  
Heidi Wirtz ◽  
Karen L. Burrows ◽  
Gary Globe
Keyword(s):  
Heart Failure ◽  
Literature Review ◽  
Risk Prediction ◽  
Systematic Literature Review ◽  
Prediction Models ◽  
Risk Prediction Models

scholarly journals RISK PREDICTION MODELS FOR SURVIVAL AFTER HEART TRANSPLANTATION: A SYSTEMATIC REVIEW

2019 ◽  
Vol 35 (10) ◽  
pp. S94-S95
Author(s):  
N. Aleksova ◽  
A. Alba ◽  
V. Molinero ◽  
K. Connolly ◽  
A. Orchanian-Cheff ◽  
...  
Keyword(s):  
Systematic Review ◽  
Heart Transplantation ◽  
Risk Prediction ◽  
Prediction Models ◽  
Risk Prediction Models
Sign in / Sign up

Export Citation Format

Share Document