Automatic development of clinical prediction models with genetic programming: A case study in cardiovascular disease

Cardiovascular disease (CVD) is one of the leading causes of death across the world. CVD can lead to angina, heart attacks, heart failure, strokes, and eventually, death; among many other serious conditions. The early intervention with those at a higher risk of developing CVD, typically with statin treatment, leads to better health outcomes. For this reason, clinical prediction models (CPMs) have been developed to identify those at a high risk of developing CVD so that treatment can begin at an earlier stage. Currently, CPMs are built around statistical analysis of factors linked to developing CVD, such as body mass index and family history. The emerging field of machine learning (ML) in healthcare, using computer algorithms that learn from a dataset without explicit programming, has the potential to outperform the CPMs available today. ML has already shown exciting progress in the detection of skin malignancies, bone fractures and many other medical conditions. In this review, we will analyse and explain the CPMs currently in use with comparisons to their developing ML counterparts. We have found that although the newest non-ML CPMs are effective, ML-based approaches consistently outperform them. However, improvements to the literature need to be made before ML should be implemented over current CPMs.

Download Full-text

Clinical Prediction Models for Primary Prevention of Cardiovascular Disease: Validity in Independent Cohorts

10.1101/2021.01.31.21250871 ◽

2021 ◽

Author(s):

Gaurav Gulati ◽

Riley J Brazil ◽

Jason Nelson ◽

David van Klaveren ◽

Christine M. Lundquist ◽

...

Keyword(s):

Cardiovascular Disease ◽

Decision Making ◽

Primary Prevention ◽

Prediction Models ◽

Model Updating ◽

Clinical Prediction ◽

Net Benefit ◽

Clinical Prediction Models ◽

C Statistic ◽

Prevention Of Cardiovascular Disease

AbstractBackgroundClinical prediction models (CPMs) are used to inform treatment decisions for the primary prevention of cardiovascular disease. We aimed to assess the performance of such CPMs in fully independent cohorts.Methods and Results63 models predicting outcomes for patients at risk of cardiovascular disease from the Tufts PACE CPM Registry were selected for external validation on publicly available data from up to 4 broadly inclusive primary prevention clinical trials. For each CPM-trial pair, we assessed model discrimination, calibration, and net benefit. Results were stratified based on the relatedness of derivation and validation cohorts, and net benefit was reassessed after updating model intercept, slope, or complete re-estimation. The median c statistic of the CPMs decreased from 0.77 (IQR 0.72-0.78) in the derivation cohorts to 0.63 (IQR 0.58-0.66) when externally validated. The validation c-statistic was higher when derivation and validation cohorts were considered related than when they were distantly related (0.67 vs 0.60, p < 0.001). The calibration slope was also higher in related cohorts than distantly related cohorts (0.69 vs 0.58, p < 0.001). Net benefit analysis suggested substantial likelihood of harm when models were externally applied, but this likelihood decreased after model updating.ConclusionsDiscrimination and calibration decrease significantly when CPMs for primary prevention of cardiovascular disease are tested in external populations, particularly when the population is only distantly related to the derivation population. Poorly calibrated predictions lead to poor decision making. Model updating can reduce the likelihood of harmful decision making, and is needed to realize the full potential of risk-based decision making in new settings.

Download Full-text

Dynamic clinical prediction models for discrete time‐to‐event data with competing risks—A case study on the OUTCOMEREA database

Biometrical Journal ◽

10.1002/bimj.201700259 ◽

2018 ◽

Vol 61 (3) ◽

pp. 514-534 ◽

Cited By ~ 2

Author(s):

Rachel Heyard ◽

Jean‐François Timsit ◽

Wafa Ibn Essaied ◽

Leonhard Held ◽

Keyword(s):

Competing Risks ◽

Discrete Time ◽

Prediction Models ◽

Event Data ◽

Clinical Prediction ◽

Time To Event ◽

Clinical Prediction Models ◽

Time To Event Data

Download Full-text

A nonparametric updating method to correct clinical prediction model drift

Journal of the American Medical Informatics Association ◽

10.1093/jamia/ocz127 ◽

2019 ◽

Vol 26 (12) ◽

pp. 1448-1457 ◽

Cited By ~ 8

Author(s):

Sharon E Davis ◽

Robert A Greevy ◽

Christopher Fonnesbeck ◽

Thomas A Lasko ◽

Colin G Walsh ◽

...

Keyword(s):

Prediction Models ◽

Testing Procedure ◽

Learning Approaches ◽

Department Of Veterans Affairs ◽

Clinical Prediction ◽

Clinical Prediction Models ◽

Subsequent Performance ◽

Nonparametric Models ◽

Population Shift

Abstract Objective Clinical prediction models require updating as performance deteriorates over time. We developed a testing procedure to select updating methods that minimizes overfitting, incorporates uncertainty associated with updating sample sizes, and is applicable to both parametric and nonparametric models. Materials and Methods We describe a procedure to select an updating method for dichotomous outcome models by balancing simplicity against accuracy. We illustrate the test’s properties on simulated scenarios of population shift and 2 models based on Department of Veterans Affairs inpatient admissions. Results In simulations, the test generally recommended no update under no population shift, no update or modest recalibration under case mix shifts, intercept correction under changing outcome rates, and refitting under shifted predictor-outcome associations. The recommended updates provided superior or similar calibration to that achieved with more complex updating. In the case study, however, small update sets lead the test to recommend simpler updates than may have been ideal based on subsequent performance. Discussion Our test’s recommendations highlighted the benefits of simple updating as opposed to systematic refitting in response to performance drift. The complexity of recommended updating methods reflected sample size and magnitude of performance drift, as anticipated. The case study highlights the conservative nature of our test. Conclusions This new test supports data-driven updating of models developed with both biostatistical and machine learning approaches, promoting the transportability and maintenance of a wide array of clinical prediction models and, in turn, a variety of applications relying on modern prediction tools.

Download Full-text

Field Synopsis of Sex in Clinical Prediction Models for Cardiovascular Disease

Circulation Cardiovascular Quality and Outcomes ◽

10.1161/circoutcomes.115.002473 ◽

2016 ◽

Vol 9 (2 suppl 1) ◽

pp. S8-S15 ◽

Cited By ~ 4

Author(s):

Jessica K. Paulus ◽

Benjamin S. Wessler ◽

Christine Lundquist ◽

Lana L.Y. Lai ◽

Gowri Raman ◽

...

Keyword(s):

Cardiovascular Disease ◽

Prediction Models ◽

Clinical Prediction ◽

Clinical Prediction Models

Download Full-text

Abstract 174: Clinical Prediction Models for Cardiovascular Disease: The Tufts PACE CPM Database

Circulation Cardiovascular Quality and Outcomes ◽

10.1161/circoutcomes.8.suppl_2.174 ◽

2015 ◽

Vol 8 (suppl_2) ◽

Author(s):

Benjamin S Wessler ◽

Lana Lai YH ◽

Whitney Kramer ◽

Michael Cangelosi ◽

Gowri Raman ◽

...

Keyword(s):

Cardiovascular Disease ◽

Prediction Models ◽

De Novo ◽

Prognostic Models ◽

Calibration Plot ◽

Clinical Prediction ◽

Clinical Prediction Models ◽

Coronary Syndrome ◽

Stage Of Development ◽

Distinct Index

Background: Clinical prediction models (CPMs) estimate the probability of clinical outcomes and hold the potential to improve decision making and individualize care. For patients with cardiovascular disease (CVD) there are numerous CPMs available though the extent of this literature is not well described. Methods and Results: We conducted a systematic review for articles containing CPMs for CVD published between January 1990 through May 2012. CVD includes coronary artery disease (CAD), congestive heart failure (CHF), arrhythmias, stroke, venous thromboembolism (VTE) and peripheral vascular disease (PVD). We created a novel database and characterized CPMs based on the stage of development, population under study, performance, covariates, and predicted outcomes. We included articles that describe newly developed CPMs that predict the risk of developing an outcome (prognostic models) or the probability of a specific diagnosis (diagnostic models). There are 796 models included in this database representing 31 distinct index conditions. 717 (90%) are de novo CPMs, 21 (3%) are CPM recalibrations, and 58 (7%) are CPM adaptations. There are 215 CPMs for patients with CAD, 168 CPMs for population samples at risk for incident CVD, and 79 models for patients with CHF (Figure). De novo CPMs predicting mortality were most commonly published for patients with known CAD (98 models) followed by HF (63 models) and stroke (24 models). There are 77 distinct index/ outcome (I/O) pairings and models are roughly evenly split between those predicting short term outcomes (< 3 months) and those predicting long term outcomes (< 6 months). There are 41 diagnostic CPMs included in this database, most commonly predicting diagnoses of CAD (11 models), VTE (10 models), and acute coronary syndrome (5 models). Of the de novo models in this database 450 (63%) report a c-statistic and 259 (36%) report either the Hosmer-Lemeshow statistic or show a calibration plot. Conclusions: There is an abundance of CPMs available for many CVD conditions, with substantial redundancy in the literature. The comparative performance of these models, the consistency of effects and risk estimates across models and the actual and potential clinical impact of this body of literature is poorly understood.

Download Full-text

Updating Clinical Prediction Models: An Illustrative Case Study

10.1007/978-3-030-85292-4_14 ◽

2021 ◽

pp. 109-113

Author(s):

Hendrik-Jan Mijderwijk ◽

Stefan van Beek ◽

Daan Nieboer

Keyword(s):

Prediction Models ◽

Illustrative Case ◽

Clinical Prediction ◽

Clinical Prediction Models ◽

Illustrative Case Study

Download Full-text

Application of clinical prediction modeling in pediatric neurosurgery: a case study

Child s Nervous System ◽

10.1007/s00381-021-05112-z ◽

2021 ◽

Author(s):

Hendrik-Jan Mijderwijk ◽

Thomas Beez ◽

Daniel Hänggi ◽

Daan Nieboer

Keyword(s):

Decision Making ◽

Prediction Models ◽

Pediatric Neurosurgery ◽

Medical Decision ◽

Clinical Prediction ◽

Clinical Prediction Models ◽

Current Form ◽

Related Risk ◽

Decision Making Processes

AbstractThere has been an increasing interest in articles reporting on clinical prediction models in pediatric neurosurgery. Clinical prediction models are mathematical equations that combine patient-related risk factors for the estimation of an individual’s risk of an outcome. If used sensibly, these evidence-based tools may help pediatric neurosurgeons in medical decision-making processes. Furthermore, they may help to communicate anticipated future events of diseases to children and their parents and facilitate shared decision-making accordingly. A basic understanding of this methodology is incumbent when developing or applying a prediction model. This paper addresses this methodology tailored to pediatric neurosurgery. For illustration, we use original pediatric data from our institution to illustrate this methodology with a case study. The developed model is however not externally validated, and clinical impact has not been assessed; therefore, the model cannot be recommended for clinical use in its current form.

Download Full-text