Assessment of risk model performance

Personal predictive models for disease development play important roles in chronic disease prevention. The performance of these models is evaluated by applying them to the baseline covariates of participants in external cohort studies, with model predictions compared to subjects' subsequent disease incidence. However, the covariate distribution among participants in a validation cohort may differ from that of the population for which the model will be used. Since estimates of predictive model performance depend on the distribution of covariates among the subjects to which it is applied, such differences can cause misleading estimates of model performance in the target population. We propose a method for addressing this problem by weighting the cohort subjects to make their covariate distribution better match that of the target population. Simulations show that the method provides accurate estimates of model performance in the target population, while un-weighted estimates may not. We illustrate the method by applying it to evaluate an ovarian cancer prediction model targeted to US women, using cohort data from participants in the California Teachers Study. The methods can be implemented using open-source code for public use as the R-package RMAP (Risk Model Assessment Package) available at http://stanford.edu/~ggong/rmap/ .

Download Full-text

Cardiovascular risk model performance in women with and without hypertensive disorders of pregnancy

Heart ◽

10.1136/heartjnl-2018-313439 ◽

2018 ◽

Vol 105 (4) ◽

pp. 330-336 ◽

Cited By ~ 3

Author(s):

Veerle Dam ◽

N Charlotte Onland-Moret ◽

W M Monique Verschuren ◽

Jolanda M A Boer ◽

Laura Benschop ◽

...

Keyword(s):

Risk Model ◽

Model Performance ◽

Predictive Performance ◽

Baseline Risk ◽

Hypertensive Disorders Of Pregnancy ◽

Cvd Risk ◽

Hypertensive Disorders ◽

Framingham Risk ◽

History Of ◽

Score Model

ObjectivesCompare the predictive performance of Framingham Risk Score (FRS), Pooled Cohort Equations (PCEs) and Systematic COronary Risk Evaluation (SCORE) model between women with and without a history of hypertensive disorders of pregnancy (hHDP) and determine the effects of recalibration and refitting on predictive performance.MethodsWe included 29 751 women, 6302 with hHDP and 17 369 without. We assessed whether models accurately predicted observed 10-year cardiovascular disease (CVD) risk (calibration) and whether they accurately distinguished between women developing CVD during follow-up and not (discrimination), separately for women with and without hHDP. We also recalibrated (updating intercept and slope) and refitted (recalculating coefficients) the models.ResultsOriginal FRS and PCEs overpredicted 10-year CVD risks, with expected:observed (E:O) ratios ranging from 1.51 (for FRS in women with hHDP) to 2.29 (for PCEs in women without hHDP), while E:O ratios were close to 1 for SCORE. Overprediction attenuated slightly after recalibration for FRS and PCEs in both hHDP groups. Discrimination was reasonable for all models, with C-statistics ranging from 0.70-0.81 (women with hHDP) and 0.72–0.74 (women without hHDP). C-statistics improved slightly after refitting 0.71–0.83 (with hHDP) and 0.73–0.80 (without hHDP). The E:O ratio of the original PCE model was statistically significantly better in women with hHDP compared with women without hHDP.ConclusionsSCORE performed best in terms of both calibration and discrimination, while FRS and PCEs overpredicted risk in women with and without hHDP, but improved after recalibrating and refitting the models. No separate model for women with hHDP seems necessary, despite their higher baseline risk.

Download Full-text

Dietary Information Improves Model Performance and Predictive Ability of a Noninvasive Type 2 Diabetes Risk Model

PLoS ONE ◽

10.1371/journal.pone.0166206 ◽

2016 ◽

Vol 11 (11) ◽

pp. e0166206 ◽

Cited By ~ 1

Author(s):

Tianshu Han ◽

Shuang Tian ◽

Li Wang ◽

Xi Liang ◽

Hongli Cui ◽

...

Keyword(s):

Type 2 Diabetes ◽

Risk Model ◽

Model Performance ◽

Predictive Ability ◽

Diabetes Risk

Download Full-text

Development and validation of a prognostic model based on comorbidities to predict Covid-19 severity. A population-based study

International Journal of Epidemiology ◽

10.1093/ije/dyaa209 ◽

2020 ◽

Author(s):

Francisco Gude-Sampedro ◽

Carmen Fernández-Merino ◽

Lucía Ferreiro ◽

Óscar Lado-Baleato ◽

Jenifer Espasandín-Domínguez ◽

...

Keyword(s):

Disease Severity ◽

Risk Model ◽

Model Performance ◽

Population Based ◽

Chronic Obstructive ◽

Operating Characteristics ◽

Obstructive Pulmonary Disease ◽

Logistic Regression Models ◽

Health Authorities ◽

Obesity Hypertension

Abstract Background The prognosis of patients with Covid-19 infection is uncertain. We derived and validated a new risk model for predicting progression to disease severity, hospitalization, admission to intensive care unit (ICU) and mortality in patients with Covid-19 infection (Gal-Covid-19 scores). Methods This is a retrospective cohort study of patients with Covid-19 infection confirmed by reverse transcription polymerase chain reaction (RT-PCR) in Galicia, Spain. Data were extracted from electronic health records of patients, including age, sex and comorbidities according to International Classification of Primary Care codes (ICPC-2). Logistic regression models were used to estimate the probability of disease severity. Calibration and discrimination were evaluated to assess model performance. Results The incidence of infection was 0.39% (10 454 patients). A total of 2492 patients (23.8%) required hospitalization, 284 (2.7%) were admitted to the ICU and 544 (5.2%) died. The variables included in the models to predict severity included age, gender and chronic comorbidities such as cardiovascular disease, diabetes, obesity, hypertension, chronic obstructive pulmonary disease, asthma, liver disease, chronic kidney disease and haematological cancer. The models demonstrated a fair–good fit for predicting hospitalization {AUC [area under the receiver operating characteristics (ROC) curve] 0.77 [95% confidence interval (CI) 0.76, 0.78]}, admission to ICU [AUC 0.83 (95%CI 0.81, 0.85)] and death [AUC 0.89 (95%CI 0.88, 0.90)]. Conclusions The Gal-Covid-19 scores provide risk estimates for predicting severity in Covid-19 patients. The ability to predict disease severity may help clinicians prioritize high-risk patients and facilitate the decision making of health authorities.

Download Full-text

Prognostic Model to Predict Postoperative Adverse Events in Pediatric Patients With Aortic Coarctation

Frontiers in Cardiovascular Medicine ◽

10.3389/fcvm.2021.672627 ◽

2021 ◽

Vol 8 ◽

Author(s):

Yan Gu ◽

Qianqian Li ◽

Rui Lin ◽

Wenxi Jiang ◽

Xue Wang ◽

...

Keyword(s):

Adverse Events ◽

Pediatric Patients ◽

Risk Model ◽

Model Performance ◽

Left Ventricular ◽

Left Ventricular Ejection ◽

Coarctation Of Aorta ◽

Ventricular Ejection Fraction ◽

Net Reclassification Improvement ◽

Postoperative Risk

Background: Postoperative adverse events remain excessively high in surgical patients with coarctation of aorta (CoA). Currently, there is no generally accepted strategy to predict these patients' individual outcomes.Objective: This study aimed to develop a risk model for the prediction of postoperative risk in pediatric patients with CoA.Methods: In total, 514 patients with CoA at two centers were enrolled. Using daily clinical practice data, we developed a model to predict 30-day or in-hospital adverse events after the operation. The least absolute shrinkage and selection operator approach was applied to select predictor variables and logistic regression was used to develop the model. Model performance was estimated using the receiver-operating characteristic curve, the Hosmer–Lemeshow test and the calibration plot. Net reclassification improvement (NRI) and integrated discrimination improvement (IDI) compared with existing risk strategies were assessed.Results: Postoperative adverse events occurred in 195 (37.9%) patients in the overall population. Nine predictive variables were identified, including incision of left thoracotomy, preoperative ventilation, concomitant ventricular septal defect, preoperative cardiac dysfunction, severe pulmonary hypertension, height, weight-for-age z-score, left ventricular ejection fraction and left ventricular posterior wall thickness. A multivariable logistic model [area under the curve = 0.8195 (95% CI: 0.7514–0.8876)] with adequate calibration was developed. Model performance was significantly improved compared with the existing Aristotle Basic Complexity (ABC) score (NRI = 47.3%, IDI = 11.5%) and the Risk Adjustment for Congenital Heart Surgery (RACHS-1) (NRI = 75.0%, IDI = 14.9%) in the validation set.Conclusion: Using daily clinical variables, we generated and validated an easy-to-apply postoperative risk model for patients with CoA. This model exhibited a remarkable improvement over the ABC score and the RACHS-1 method.

Download Full-text

Statistical Methods for Measuring Market Risk Model Performance - Comparison in View of the Fundamental Review of the Trading Book

SSRN Electronic Journal ◽

10.2139/ssrn.2722850 ◽

2015 ◽

Author(s):

Maximilian Dinse

Keyword(s):

Statistical Methods ◽

Risk Model ◽

Model Performance ◽

Market Risk ◽

Performance Comparison

Download Full-text

Development of a model for predicting the 4-year risk of symptomatic knee osteoarthritis in China: a longitudinal cohort study

Arthritis Research & Therapy ◽

10.1186/s13075-021-02447-5 ◽

2021 ◽

Vol 23 (1) ◽

Author(s):

Limin Wang ◽

Han Lu ◽

Hongbo Chen ◽

Shida Jin ◽

Mengqi Wang ◽

...

Keyword(s):

Knee Osteoarthritis ◽

Activities Of Daily Living ◽

Daily Living ◽

Risk Model ◽

Characteristic Curve ◽

Model Performance ◽

Moderate Physical Activity ◽

Symptomatic Knee Osteoarthritis ◽

Symptomatic Knee

Abstract Objectives We aimed to develop a model for predicting the 4-year risk of knee osteoarthritis (KOA) based on survey data obtained via a random, nationwide sample of Chinese individuals. Methods Data was analyzed from 8193 middle-aged and older adults included in the China Health and Retirement Longitudinal Study (CHARLS). The incident of symptomatic KOA was defined as participants who were free of symptomatic KOA at baseline (CHARLS2011) and diagnosed with symptomatic KOA at the 4-year follow-up (CHARLS2015). The effects of potential predictors on the incident of KOA were estimated using logistic regression models and the final model was internally validated using the bootstrapping technique. Model performance was assessed based on discrimination—area under the receiver operating characteristic curve (AUC)—and calibration. Results A total of 815 incidents of KOA were identified at the 4-year follow-up, resulting in a cumulative incidence of approximately 9.95%. The final multivariable model included age, sex, waist circumference, residential area, difficulty with activities of daily living (ADLs)/instrumental activities of daily living (IADLs), history of hip fracture, depressive symptoms, number of chronic comorbidities, self-rated health status, and level of moderate physical activity (MPA). The risk model showed good discrimination with AUC = 0.719 (95% confidence interval [CI] 0.700–0.737) and optimism-corrected AUC = 0.712 after bootstrap validation. A satisfactory agreement was observed between the observed and predicted probability of incident symptomatic KOA. And a simple clinical score model was developed for quantifying the risk of KOA. Conclusion Our prediction model may aid the early identification of individuals at the greatest risk of developing KOA within 4 years.

Download Full-text

Contemporary Clinical and Coronary Anatomic Risk Model for 30-Day Mortality After Percutaneous Coronary Intervention

Circulation Cardiovascular Interventions ◽

10.1161/circinterventions.121.010863 ◽

2021 ◽

Author(s):

Jacob A. Doll ◽

Colin I. O’Donnell ◽

Meg E. Plomondon ◽

Stephen W. Waldo

Keyword(s):

Percutaneous Coronary Intervention ◽

Risk Model ◽

Validation Cohort ◽

Model Performance ◽

Coronary Intervention ◽

Veterans Affairs ◽

Syntax Score ◽

Risk Of Death ◽

Percutaneous Coronary ◽

Anatomic Information

Background: Percutaneous coronary intervention (PCI) procedures are increasing in clinical and anatomic complexity, likely increasing the calculated risk of mortality. There is need for a real-time risk prediction tool that includes clinical and coronary anatomic information that is integrated into the electronic medical record system. Methods: We assessed 70 503 PCIs performed in 73 Veterans Affairs hospitals from 2008 to 2019. We used regression and machine-learning strategies to develop a prediction model for 30-day mortality following PCI. We assessed model performance with and without inclusion of the Veterans Affairs SYNTAX score (Synergy Between Percutaneous Coronary Intervention With Taxus and Cardiac Surgery), an assessment of anatomic complexity. Finally, the discriminatory ability of the Veterans Affairs model was compared with the CathPCI mortality model. Results: The overall 30-day morality rate was 1.7%. The final model included 14 variables. Presentation status (salvage, emergent, urgent), ST-segment–elevation myocardial infarction, cardiogenic shock, age, congestive heart failure, prior valve disease, chronic kidney disease, chronic lung disease, atrial fibrillation, elevated international normalized ratio, and the Veterans Affairs SYNTAX score were all associated with increased risk of death, while increasing body mass index, hemoglobin level, and prior coronary artery bypass graft surgery were associated with lower risk of death. C-index for the development cohort was 0.93 (95% CI, 0.92–0.94) and for the 2019 validation cohort and the site validation cohort was 0.87 (95% CI, 0.83–0.92) and 0.86 (95% CI, 0.83–0.89), respectively. The positive likelihood ratio of predicting a mortality event in the top decile was 2.87% more accurate than the CathPCI mortality model. Inclusion of anatomic information in the model resulted in significant improvement in model performance (likelihood ratio test P <0.01). Conclusions: This contemporary risk model accurately predicts 30-day post-PCI mortality using a combination of clinical and anatomic variables. This can be immediately implemented into clinical practice to promote personalized informed consent discussions and appropriate preparation for high-risk PCI cases.

Download Full-text

TESTING THE LONG-RUN RISK MODEL: A KALMAN FILTER APPROACH

Annals of Financial Economics ◽

10.1142/s2010495218500197 ◽

2018 ◽

Vol 13 (04) ◽

pp. 1850019

Author(s):

JIANQIU WANG ◽

KE WU

Keyword(s):

Kalman Filter ◽

Maximum Likelihood ◽

Risk Model ◽

Estimation Method ◽

Model Performance ◽

Empirical Support ◽

Likelihood Estimation ◽

Long Run ◽

Filter Approach ◽

Long Run Risk

This paper reevaluates the Long-Run Risk model proposed by Bansal and Yaron (2004) using the Kalman filter and Maximum Likelihood estimation method. Our findings show that the persistence of the small long-run predictable component in the consumption growth process is the key for the model performance. In our estimation exercises, if we relax the persistence restriction on the long-run risk parameter and adopt a Maximum Likelihood estimate, the Long-Run Risk model still requires a relative risk aversion at around 70 to fit the US data. However, we do not find strong empirical support for the persistence restriction from the data.

Download Full-text

Estimating wildfire risk on a Mojave Desert landscape using remote sensing and field sampling

International Journal of Wildland Fire ◽

10.1071/wf12158 ◽

2013 ◽

Vol 22 (6) ◽

pp. 770 ◽

Cited By ~ 10

Author(s):

Peter F. Van Linn ◽

Kenneth E. Nussear ◽

Todd C. Esque ◽

Lesley A. DeFalco ◽

Richard D. Inman ◽

...

Keyword(s):

Remote Sensing ◽

Spatial Data ◽

Mojave Desert ◽

Risk Model ◽

Model Performance ◽

Remote Sensing Data ◽

Fire Risk ◽

Wildfire Risk ◽

Fuel Loads ◽

Desert Landscape

Predicting wildfires that affect broad landscapes is important for allocating suppression resources and guiding land management. Wildfire prediction in the south-western United States is of specific concern because of the increasing prevalence and severe effects of fire on desert shrublands and the current lack of accurate fire prediction tools. We developed a fire risk model to predict fire occurrence in a north-eastern Mojave Desert landscape. First we developed a spatial model using remote sensing data to predict fuel loads based on field estimates of fuels. We then modelled fire risk (interactions of fuel characteristics and environmental conditions conducive to wildfire) using satellite imagery, our model of fuel loads, and spatial data on ignition potential (lightning strikes and distance to roads), topography (elevation and aspect) and climate (maximum and minimum temperatures). The risk model was developed during a fire year at our study landscape and validated at a nearby landscape; model performance was accurate and similar at both sites. This study demonstrates that remote sensing techniques used in combination with field surveys can accurately predict wildfire risk in the Mojave Desert and may be applicable to other arid and semiarid lands where wildfires are prevalent.

Download Full-text