scholarly journals Clinical Prediction Models for Primary Prevention of Cardiovascular Disease: Validity in Independent Cohorts

2021 ◽  
Author(s):  
Gaurav Gulati ◽  
Riley J Brazil ◽  
Jason Nelson ◽  
David van Klaveren ◽  
Christine M. Lundquist ◽  
...  
Keyword(s):  
Cardiovascular Disease ◽  
Decision Making ◽  
Primary Prevention ◽  
Prediction Models ◽  
Model Updating ◽  
Clinical Prediction ◽  
Net Benefit ◽  
Clinical Prediction Models ◽  
C Statistic ◽  
Prevention Of Cardiovascular Disease

AbstractBackgroundClinical prediction models (CPMs) are used to inform treatment decisions for the primary prevention of cardiovascular disease. We aimed to assess the performance of such CPMs in fully independent cohorts.Methods and Results63 models predicting outcomes for patients at risk of cardiovascular disease from the Tufts PACE CPM Registry were selected for external validation on publicly available data from up to 4 broadly inclusive primary prevention clinical trials. For each CPM-trial pair, we assessed model discrimination, calibration, and net benefit. Results were stratified based on the relatedness of derivation and validation cohorts, and net benefit was reassessed after updating model intercept, slope, or complete re-estimation. The median c statistic of the CPMs decreased from 0.77 (IQR 0.72-0.78) in the derivation cohorts to 0.63 (IQR 0.58-0.66) when externally validated. The validation c-statistic was higher when derivation and validation cohorts were considered related than when they were distantly related (0.67 vs 0.60, p < 0.001). The calibration slope was also higher in related cohorts than distantly related cohorts (0.69 vs 0.58, p < 0.001). Net benefit analysis suggested substantial likelihood of harm when models were externally applied, but this likelihood decreased after model updating.ConclusionsDiscrimination and calibration decrease significantly when CPMs for primary prevention of cardiovascular disease are tested in external populations, particularly when the population is only distantly related to the derivation population. Poorly calibrated predictions lead to poor decision making. Model updating can reduce the likelihood of harmful decision making, and is needed to realize the full potential of risk-based decision making in new settings.

scholarly journals Performance of Heart Failure Clinical Prediction Models: A Systematic External Validation Study

2021 ◽  
Author(s):  
Jenica N. Upshaw ◽  
Jason Nelson ◽  
Benjamin Koethe ◽  
Jinny G. Park ◽  
Hannah McGinnes ◽  
...  
Keyword(s):  
Heart Failure ◽  
Prediction Models ◽  
Model Performance ◽  
Case Mix ◽  
Decision Threshold ◽  
Clinical Prediction ◽  
Net Benefit ◽  
Clinical Prediction Models ◽  
Model Based ◽  
C Statistic

BackgroundMost heart failure (HF) clinical prediction models (CPMs) have not been externally validated.MethodsWe performed a systematic review to identify CPMs predicting outcomes in HF, stratified by acute and chronic HF CPMs. External validations were performed using individual patient data from 8 large HF trials (1 acute, 7 chronic). CPM discrimination (c-statistic, % relative change in c-statistic), calibration (calibration slope, Harrell’s E, E90), and net benefit were evaluated for each CPM with and without recalibration.ResultsOf 135 HF CPMs screened, 24 (18%) were compatible with the population, predictors and outcomes to the trials and 42 external validations were performed (14 acute HF, 28 chronic HF). The median derivation c-statistic of acute HF CPMs was 0.76 (IQR, 0.75, 0.8), validation c-statistic was 0.67 (0.65, 0.68) and model-based c-statistic was 0.68 (0.66, 0.76), Hence, most of the apparent decrement in model performance was due to narrower case-mix in the validation cohort compared with the development cohort. The median derivation c-statistic for chronic HF CPMs was 0.76 (0.74, 0.8), validation c-statistic 0.61 (0.6, 0.63) and model-based c-statistic 0.68 (0.62, 0.71), suggesting that the decrement in model performance was only partially due to case-mix heterogeneity. Calibration was generally poor - median E (standardized by outcome rate) was 0.5 (0.4, 2.2) for acute HF CPMs and 0.5 (0.3, 0.7) for chronic HF CPMs. Updating the intercept alone led to a significant improvement in calibration in acute HF CPMs, but not in chronic HF CPMs. Net benefit analysis showed potential for harm in using CPMs when the decision threshold was not near the overall outcome rate but this improved with model recalibration.ConclusionsOnly a small minority of published CPMs contained variables and outcomes that were compatible with the clinical trial datasets. For acute HF CPMs, discrimination is largely preserved after adjusting for case-mix; however, the risk of net harm is substantial without model recalibration for both acute and chronic HF CPMs.

scholarly journals The Generalizability of Clinical Prediction Models for Patients with Acute Coronary Syndromes: Results from Independent External Validations

2021 ◽  
Author(s):  
Benjamin S. Wessler ◽  
Jason Nelson ◽  
Jinny G. Park ◽  
Hannah McGinnes ◽  
Jenica Upshaw ◽  
...  
Keyword(s):  
Prediction Models ◽  
Individual Patient Data ◽  
Model Performance ◽  
Clinical Prediction ◽  
Net Benefit ◽  
Clinical Prediction Models ◽  
Coronary Syndrome ◽  
C Statistic ◽  
Coronary Syndromes ◽  
Relative Change

AbstractPurposeIt is increasingly recognized that clinical prediction models (CPMs) often do not perform as expected when they are tested on new databases. Independent external validations of CPMs are recommended but often not performed. Here we conduct independent external validations of acute coronary syndrome (ACS) CPMs.MethodsA systematic review identified CPMs predicting outcomes for patients with ACS. Independent external validations were performed by evaluating model performance using individual patient data from 5 large clinical trials. CPM performance with and without various recalibration techniques was evaluated with a focus on CPM discrimination (c-statistic, % relative change in c-statistic) as well as calibration (Harrell’s Eavg, E90, Net Benefit).ResultsOf 269 ACS CPMs screened, 23 (8.5%) were compatible with at least one of the trials and 28 clinically appropriate external validations were performed. The median c statistic of the CPMs in the derivation cohorts was 0.76 (IQR, 0.74 to 0.78). The median c-statistic in these external validations was 0.70 (IQR, 0.66 to 0.71) reflecting a 24% decrement in discrimination. However, this decrement in discrimination was due mostly to narrower case-mix in the validation cohorts compared to derivation cohorts, as reflected in the median model based c-statistic [0.71 (IQR 0.66 to 0.75). The median calibration slope in external validations was 0.84 (IQR, 0.72 to 0.98) and the median Eavg (standardized by the outcome rate) was 0.4 (IQR, 0.3 to 0.8). Net benefit indicates that most CPMs had a high risk of causing net harm when not recalibrated, particularly for decision thresholds not near the overall outcome rate.ConclusionIndependent external validations of published ACS CPMs demonstrate that models tested in our sample had relatively well-preserved discrimination but poor calibration when externally validated. Applying ‘off-the-shelf’ CPMs often risks net harm unless models are recalibrated to the populations on which they are used.

Reviewing the use and quality of machine learning in developing clinical prediction models for cardiovascular disease

2021 ◽  
pp. postgradmedj-2020-139352
Author(s):  
Simon Allan ◽  
Raphael Olaiya ◽  
Rasan Burhan
Keyword(s):  
Machine Learning ◽  
Cardiovascular Disease ◽  
Prediction Models ◽  
Bone Fractures ◽  
Statin Treatment ◽  
Computer Algorithms ◽  
Clinical Prediction ◽  
Clinical Prediction Models ◽  
Heart Attacks

Cardiovascular disease (CVD) is one of the leading causes of death across the world. CVD can lead to angina, heart attacks, heart failure, strokes, and eventually, death; among many other serious conditions. The early intervention with those at a higher risk of developing CVD, typically with statin treatment, leads to better health outcomes. For this reason, clinical prediction models (CPMs) have been developed to identify those at a high risk of developing CVD so that treatment can begin at an earlier stage. Currently, CPMs are built around statistical analysis of factors linked to developing CVD, such as body mass index and family history. The emerging field of machine learning (ML) in healthcare, using computer algorithms that learn from a dataset without explicit programming, has the potential to outperform the CPMs available today. ML has already shown exciting progress in the detection of skin malignancies, bone fractures and many other medical conditions. In this review, we will analyse and explain the CPMs currently in use with comparisons to their developing ML counterparts. We have found that although the newest non-ML CPMs are effective, ML-based approaches consistently outperform them. However, improvements to the literature need to be made before ML should be implemented over current CPMs.

Statins for primary prevention: what is the regulator’s role?

2020 ◽  
pp. bmjebm-2019-111321 ◽  
Author(s):  
Tom Jefferson ◽  
Maryanne Demasi ◽  
Peter Doshi
Keyword(s):  
Cardiovascular Disease ◽  
Decision Making ◽  
Primary Prevention ◽  
General Public ◽  
Trial Data ◽  
Central Location ◽  
Decision Making Processes ◽  
Drug Approvals ◽  
Prevention Of Cardiovascular Disease

Globally, drug regulators have approved statins for the prevention of cardiovascular disease (CVD), although their use in primary prevention has been controversial. A highly publicised debate has ensued over whether the benefits outweigh the harms. Drug regulators, which are legally required to make independent judgements on drug approvals, have remained silent during the debate. Our aim was to navigate the decision-making processes of European drug regulators and ultimately request the data upon which statins were approved. Our findings revealed a system of fragmented regulation in which many countries licensed statins but did not analyse the data themselves. There is no easily accessible archive containing information about the licensing approval of statins or a central location for holding the trial data. This is an unsustainable model and serves neither the general public, nor researchers.

scholarly journals Field Synopsis of Sex in Clinical Prediction Models for Cardiovascular Disease

2016 ◽  
Vol 9 (2 suppl 1) ◽  
pp. S8-S15 ◽  
Author(s):  
Jessica K. Paulus ◽  
Benjamin S. Wessler ◽  
Christine Lundquist ◽  
Lana L.Y. Lai ◽  
Gowri Raman ◽  
...  
Keyword(s):  
Cardiovascular Disease ◽  
Prediction Models ◽  
Clinical Prediction ◽  
Clinical Prediction Models

Abstract 174: Clinical Prediction Models for Cardiovascular Disease: The Tufts PACE CPM Database

2015 ◽  
Vol 8 (suppl_2) ◽  
Author(s):  
Benjamin S Wessler ◽  
Lana Lai YH ◽  
Whitney Kramer ◽  
Michael Cangelosi ◽  
Gowri Raman ◽  
...  
Keyword(s):  
Cardiovascular Disease ◽  
Prediction Models ◽  
De Novo ◽  
Prognostic Models ◽  
Calibration Plot ◽  
Clinical Prediction ◽  
Clinical Prediction Models ◽  
Coronary Syndrome ◽  
Stage Of Development ◽  
Distinct Index

Background: Clinical prediction models (CPMs) estimate the probability of clinical outcomes and hold the potential to improve decision making and individualize care. For patients with cardiovascular disease (CVD) there are numerous CPMs available though the extent of this literature is not well described. Methods and Results: We conducted a systematic review for articles containing CPMs for CVD published between January 1990 through May 2012. CVD includes coronary artery disease (CAD), congestive heart failure (CHF), arrhythmias, stroke, venous thromboembolism (VTE) and peripheral vascular disease (PVD). We created a novel database and characterized CPMs based on the stage of development, population under study, performance, covariates, and predicted outcomes. We included articles that describe newly developed CPMs that predict the risk of developing an outcome (prognostic models) or the probability of a specific diagnosis (diagnostic models). There are 796 models included in this database representing 31 distinct index conditions. 717 (90%) are de novo CPMs, 21 (3%) are CPM recalibrations, and 58 (7%) are CPM adaptations. There are 215 CPMs for patients with CAD, 168 CPMs for population samples at risk for incident CVD, and 79 models for patients with CHF (Figure). De novo CPMs predicting mortality were most commonly published for patients with known CAD (98 models) followed by HF (63 models) and stroke (24 models). There are 77 distinct index/ outcome (I/O) pairings and models are roughly evenly split between those predicting short term outcomes (< 3 months) and those predicting long term outcomes (< 6 months). There are 41 diagnostic CPMs included in this database, most commonly predicting diagnoses of CAD (11 models), VTE (10 models), and acute coronary syndrome (5 models). Of the de novo models in this database 450 (63%) report a c-statistic and 259 (36%) report either the Hosmer-Lemeshow statistic or show a calibration plot. Conclusions: There is an abundance of CPMs available for many CVD conditions, with substantial redundancy in the literature. The comparative performance of these models, the consistency of effects and risk estimates across models and the actual and potential clinical impact of this body of literature is poorly understood.

Abstract 17185: Modest Performance of Heart Failure Clinical Prediction Models: A Systematic External Validation Study

Circulation ◽  
2018 ◽  
Vol 138 (Suppl_1) ◽  
Author(s):  
Jenica N Upshaw ◽  
Jason Nelson ◽  
Benjamin Wessler ◽  
Benjamin Koethe ◽  
Christine Lundquist ◽  
...  
Keyword(s):  
Heart Failure ◽  
Prediction Models ◽  
External Validation ◽  
Significant Heterogeneity ◽  
Clinical Prediction ◽  
Clinical Prediction Models ◽  
Derivation Cohort ◽  
C Statistic ◽  
Improved Model ◽  
Relative Change

Introduction: Most heart failure (HF) clinical prediction models (CPMs] have not been independently externally validated. We sought to test the performance of HF models in a diverse population using a systematic approach. Methods: A systematic review identified CPMs predicting outcomes for patients with HF. Individual patient data from 5 large publicaly available clinical trials enrolling patients with chronic HF were matched to published CPMs based on similarity in populations and available outcome and predictor variables in the clinical trial databases. CPM performance was evaluated for discrimination (c-statistic, % relative change in c-statistic) and calibration (Harrell’s E and E 90 , the mean and the 90% quantile of the error distribution from the smoothed loess observed value) for the original and recalibrated models. Results: Out of 135 HF CPMs reviewed, we identified 45 CPM-trial pairs including 13 unique CPMs. The outcome was mortality for all of the models with a trial match. During external validations, median c-statistic was 0.595 (IQR 0.563 to 0.630) with a median relative decrease in the c-statistic of -57 % (IQR, -49% to -71%) compared to the c-statistic reported in the derivation cohort. Overall, the median Harrell’s E was 0.09 (IQR, 0.04 to 0.135) and E 90 was 0.11 (IQR, 0.07 to 0.21). Recalibration of the intercept and slope led to substantially improved calibration with median change in Harrell’s E of -35% [IQR 0 to -75%] for the intercept and -56% [IQR -17% to -75%] for the intercept and slope. Refitting model covariates improved the median c-statistic by 38% to 0.629 [IQR 0.613 to 0.649]. Conclusion: For HF CPMs, independent external validations demonstrate that CPMs perform significantly worse than originally presented; however with significant heterogeneity. Recalibration of the intercept and slope improved model calibration. These results underscore the need to carefully consider the derivation cohort characteristics when using published CPMs.

scholarly journals Clinical prediction models to inform individualized decision-making in subfertile couples: a stratified medicine approach

2014 ◽  
Vol 29 (9) ◽  
pp. 1851-1858 ◽  
Author(s):  
D. J. McLernon ◽  
E. R. te Velde ◽  
E. W. Steyerberg ◽  
B. W. J. Mol ◽  
S. Bhattacharya
Keyword(s):  
Decision Making ◽  
Prediction Models ◽  
Stratified Medicine ◽  
Clinical Prediction ◽  
Clinical Prediction Models ◽  
Individualized Decision
Sign in / Sign up

Export Citation Format

Share Document