Development and Validation of Elective and Nonelective Risk Prediction Models for In-Hospital Mortality in Proximal Aortic Surgery Using the National Institute for Cardiovascular Outcomes Research (NICOR) Database

Importance: Modern predictive models require large amounts of data for training and evaluation which can result in building models that are specific to certain locations, populations in them and clinical practices. Yet, best practices and guidelines for clinical risk prediction models have not yet considered such challenges to generalizability. Objectives: To investigate changes in measures of predictive discrimination, calibration, and algorithmic fairness when transferring models for predicting in-hospital mortality across ICUs in different populations. Also, to study the reasons for the lack of generalizability in these measures. Design, Setting, and Participants: In this multi-center cross-sectional study, electronic health records from 179 hospitals across the US with 70,126 hospitalizations were analyzed. Time of data collection ranged from 2014 to 2015. Main Outcomes and Measures: The main outcome is in-hospital mortality. Generalization gap, defined as difference between model performance metrics across hospitals, is computed for discrimination and calibration metrics, namely area under the receiver operating characteristic curve (AUC) and calibration slope. To assess model performance by race variable, we report differences in false negative rates across groups. Data were also analyzed using a causal discovery algorithm "Fast Causal Inference" (FCI) that infers paths of causal influence while identifying potential influences associated with unmeasured variables. Results: In-hospital mortality rates differed in the range of 3.9%-9.3% (1st-3rd quartile) across hospitals. When transferring models across hospitals, AUC at the test hospital ranged from 0.777 to 0.832 (1st to 3rd quartile; median 0.801); calibration slope from 0.725 to 0.983 (1st to 3rd quartile; median 0.853); and disparity in false negative rates from 0.046 to 0.168 (1st to 3rd quartile; median 0.092). When transferring models across geographies, AUC ranged from 0.795 to 0.813 (1st to 3rd quartile; median 0.804); calibration slope from 0.904 to 1.018 (1st to 3rd quartile; median 0.968); and disparity in false negative rates from 0.018 to 0.074 (1st to 3rd quartile; median 0.040). Distribution of all variable types (demography, vitals, and labs) differed significantly across hospitals and regions. Shifts in the race variable distribution and some clinical (vitals, labs and surgery) variables by hospital or region. Race variable also mediates differences in the relationship between clinical variables and mortality, by hospital/region. Conclusions and Relevance: Group-specific metrics should be assessed during generalizability checks to identify potential harms to the groups. In order to develop methods to improve and guarantee performance of prediction models in new environments for groups and individuals, better understanding and provenance of health processes as well as data generating processes by sub-group are needed to identify and mitigate sources of variation.

Download Full-text

P3429A novel prediction model for mortality after cardiac surgery using institutional case volume

European Heart Journal ◽

10.1093/eurheartj/ehz745.0303 ◽

2019 ◽

Vol 40 (Supplement_1) ◽

Author(s):

S Lee ◽

J B Park ◽

Y J Cho ◽

H G Ryu ◽

E J Jang

Keyword(s):

Cardiac Surgery ◽

Hospital Mortality ◽

Risk Prediction ◽

Prediction Models ◽

Validation Cohort ◽

Mortality Data ◽

P Value ◽

Case Volume ◽

Risk Prediction Models ◽

Derivation Cohort

Abstract Purpose A number of risk prediction models have been developed to identify short term mortality after cardiovascular surgery. Most models include patient characteristics, laboratory data, and type of surgery, but no consideration for the amount of surgical experience. With numerous reports on the impact of case volume on patient outcome after high risk procedures, we attempted to develop a risk prediction models for in-hospital and 1-year mortality that takes institutional case volume into account. Methods We identified adult patients who underwent cardiac surgery from January 2008 to December 2017 from the National Health Insurance Service (NHIS) database by searching for patients with procedure codes of coronary artery bypass grafting, valve surgery, and surgery on thoracic aorta during the hospitalization. Study subjects were randomly assigned to either the derivation cohort or the validation cohort. In-hospital mortality and 1-year mortality data were collected using the NHIS database. Risk prediction models were developed from the derivation cohort using Cox proportional hazards regression. The prediction performances of models were evaluated in the validation cohort. Results The models developed in this study demonstrated fair discrimination for derivation cohort (N=22,004, c-statistics, 0.75 for in-hospital mortality; 0.73 for 1-year mortality) and acceptable calibration in the validation cohort. (N=22,003, Hosmer-Lemeshow χ2-test, P=0.08 and 0.16, respectively). Case volume was the key factor of mortality prediction models after cardiac surgery. (50≤ x <100 case per year. 100≤ x <200 case per year, ≥200 case per year are correlated with OR 3.29, 2.49, 1.85 in in-hospital mortality, 2.76, 1.99, 1.69 in 1-year mortality respectively, P value <0.001.) Annual case volume as risk factor Variables In-hospital mortality 1-year mortality OR (95% CI) p-value OR (95% CI) p-value Annual case-volume (reference: ≥200) – – 100–200 1.69 (1.48, 1.93) <0.001 1.85 (1.58, 2.18) <0.001 50–100 1.99 (1.75, 2.25) <0.001 2.49 (2.15, 2.89) <0.001 <50 2.76 (2.44, 3.11) <0.001 3.29 (2.85, 3.79) <0.001 OR: Odds ratio; CI: confidence interval; Ref: Reference. Discrimination and calibration Conclusion We developed and validated new risk prediction models for in-hospital and 1-year mortality after cardiac surgery using the NHIS database. These models may provide useful guides to predict mortality risks of patients with basic information and without laboratory findings.

Download Full-text