scholarly journals Validation of Three Postoperative Risk Prediction Models for Intensive Care Unit Mortality after Cardiac Surgery

2018 ◽  
Vol 66 (08) ◽  
pp. 651-660 ◽  
Author(s):  
Camila Caiado ◽  
Charles McCollum ◽  
Michael Goldstein ◽  
Ignacio Malagon ◽  
Rajamiyer Venkateswaran ◽  
...  
Keyword(s):  
Intensive Care Unit ◽  
Cardiac Surgery ◽  
Risk Prediction ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
Icu Mortality ◽  
Risk Prediction Models ◽  
Postoperative Risk ◽  
Operating Characteristic Curve

Background Several cardiac surgery risk prediction models based on postoperative data have been developed. However, unlike preoperative cardiac surgery risk prediction models, postoperative models are rarely externally validated or utilized by clinicians. The objective of this study was to externally validate three postoperative risk prediction models for intensive care unit (ICU) mortality after cardiac surgery. Methods The logistic Cardiac Surgery Scores (logCASUS), Rapid Clinical Evaluation (RACE), and Sequential Organ Failure Assessment (SOFA) scores were calculated over the first 7 postoperative days for consecutive adult cardiac surgery patients between January 2013 and May 2015. Model discrimination was assessed using receiver operating characteristic curve analyses. Calibration was assessed using the Hosmer–Lemeshow (HL) test, calibration plots, and observed to expected ratios. Recalibration of the models was performed. Results A total of 2255 patients were included with an ICU mortality rate of 1.8%. Discrimination for all three models on each postoperative day was good with areas under the receiver operating characteristic curve of >0.8. Generally, RACE and logCASUS had better discrimination than SOFA. Calibration of the RACE score was better than logCASUS, but ratios of observed to expected mortality for both were generally <0.65. Locally recalibrated SOFA, logCASUS and RACE models all performed well. Conclusion All three models demonstrated good discrimination for the first 7 days after cardiac surgery. After recalibration, logCASUS and RACE scores appear to be most useful for daily risk prediction after cardiac surgery. If appropriately calibrated, postoperative cardiac surgery risk prediction models have the potential to be useful tools after cardiac surgery.

scholarly journals Traditional Cardiovascular Risk Factors and Individual Prediction of Cardiovascular Events in Childhood Cancer Survivors

2019 ◽  
Vol 112 (3) ◽  
pp. 256-265 ◽  
Author(s):  
Yan Chen ◽  
Eric J Chow ◽  
Kevin C Oeffinger ◽  
William L Border ◽  
Wendy M Leisenring ◽  
...  
Keyword(s):  
Risk Factors ◽  
Heart Failure ◽  
Cardiovascular Risk ◽  
Childhood Cancer ◽  
Cardiovascular Risk Factors ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
Childhood Cancer Survivors ◽  
Operating Characteristic Curve

Abstract Background Childhood cancer survivors have an increased risk of heart failure, ischemic heart disease, and stroke. They may benefit from prediction models that account for cardiotoxic cancer treatment exposures combined with information on traditional cardiovascular risk factors such as hypertension, dyslipidemia, and diabetes. Methods Childhood Cancer Survivor Study participants (n = 22 643) were followed through age 50 years for incident heart failure, ischemic heart disease, and stroke. Siblings (n = 5056) served as a comparator. Participants were assessed longitudinally for hypertension, dyslipidemia, and diabetes based on self-reported prescription medication use. Half the cohort was used for discovery; the remainder for replication. Models for each outcome were created for survivors ages 20, 25, 30, and 35 years at the time of prediction (n = 12 models). Results For discovery, risk scores based on demographic, cancer treatment, hypertension, dyslipidemia, and diabetes information achieved areas under the receiver operating characteristic curve and concordance statistics 0.70 or greater in 9 and 10 of the 12 models, respectively. For replication, achieved areas under the receiver operating characteristic curve and concordance statistics 0.70 or greater were observed in 7 and 9 of the models, respectively. Across outcomes, the most influential exposures were anthracycline chemotherapy, radiotherapy, diabetes, and hypertension. Survivors were then assigned to statistically distinct risk groups corresponding to cumulative incidences at age 50 years of each target outcome of less than 3% (moderate-risk) or approximately 10% or greater (high-risk). Cumulative incidence of all outcomes was 1% or less among siblings. Conclusions Traditional cardiovascular risk factors remain important for predicting risk of cardiovascular disease among adult-age survivors of childhood cancer. These prediction models provide a framework on which to base future surveillance strategies and interventions.

scholarly journals External Validation of Five Scores to Predict Stroke-Associated Pneumonia and the Role of Selected Blood Biomarkers

Stroke ◽  
2021 ◽  
Vol 52 (1) ◽  
pp. 325-330
Author(s):  
Benjamin Hotter ◽  
Sarah Hoffmann ◽  
Lena Ulm ◽  
Christian Meisel ◽  
Alejandro Bustamante ◽  
...  
Keyword(s):  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Literature Search ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
Data Set ◽  
Biomarker Analysis ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Background and Purpose: Several clinical scoring systems as well as biomarkers have been proposed to predict stroke-associated pneumonia (SAP). We aimed to externally and competitively validate SAP scores and hypothesized that 5 selected biomarkers would improve performance of these scores. Methods: We pooled the clinical data of 2 acute stroke studies with identical data assessment: STRAWINSKI and PREDICT. Biomarkers (ultrasensitive procalcitonin; mid-regional proadrenomedullin; mid-regional proatrionatriuretic peptide; ultrasensitive copeptin; C-terminal proendothelin) were measured from hospital admission serum samples. A literature search was performed to identify SAP prediction scores. We then calculated multivariate regression models with the individual scores and the biomarkers. Areas under receiver operating characteristic curves were used to compare discrimination of these scores and models. Results: The combined cohort consisted of 683 cases, of which 573 had available backup samples to perform the biomarker analysis. Literature search identified 9 SAP prediction scores. Our data set enabled us to calculate 5 of these scores. The scores had area under receiver operating characteristic curve of 0.543 to 0.651 for physician determined SAP, 0.574 to 0.685 for probable and 0.689 to 0.811 for definite SAP according to Pneumonia in Stroke Consensus group criteria. Multivariate models of the scores with biomarkers improved virtually all predictions, but mostly in the range of an area under receiver operating characteristic curve delta of 0.05. Conclusions: All SAP prediction scores identified patients who would develop SAP with fair to strong capabilities, with better discrimination when stricter criteria for SAP diagnosis were applied. The selected biomarkers provided only limited added predictive value, currently not warranting addition of these markers to prediction models. Registration: URL: https://www.clinicaltrials.gov . Unique identifier: NCT01264549 and NCT01079728.

scholarly journals Semiquantitative Cough Strength Score for Predicting Reintubation After Planned Extubation

2015 ◽  
Vol 24 (6) ◽  
pp. e86-e90 ◽  
Author(s):  
Jun Duan ◽  
Lintong Zhou ◽  
Meiling Xiao ◽  
Jinhua Liu ◽  
Xiangmei Yang
Keyword(s):  
Intensive Care Unit ◽  
Intensive Care ◽  
Spontaneous Breathing ◽  
Operating Characteristic ◽  
Peak Flow ◽  
Characteristic Curve ◽  
Spontaneous Breathing Trial ◽  
Extubation Outcome ◽  
Cough Strength ◽  
Operating Characteristic Curve

Background Semiquantitative cough strength score (SCSS, graded 0–5) and cough peak flow (CPF) have been used to predict extubation outcome in patients in whom extubation is planned; however, the correlation of the 2 assessments is unclear. Methods In the intensive care unit of a university-affiliated hospital, 186 patients who were ready for extubation after a successful spontaneous breathing trial were enrolled in the study. Both SCSS and CPF were assessed before extubation. Reintubation was recorded 72 hours after extubation. Results Reintubation rate was 15.1% within 72 hours after planned extubation. Patients in whom extubation was successful had higher SCSSs than did reintubated patients (mean [SD], 3.2 [1.6] vs 2.2 [1.6], P = .002) and CPF (74.3 [40.0] vs 51.7 [29.4] L/min, P = .005). The SCSS showed a positive correlation with CPF (r = 0.69, P &lt; .001). Mean CPFs were 38.36 L/min, 39.51 L/min, 44.67 L/min, 57.54 L/min, 78.96 L/min, and 113.69 L/min in patients with SCSSs of 0, 1, 2, 3, 4, and 5, respectively. The discriminatory power for reintubation, evidenced by area under the receiver operating characteristic curve, was similar: 0.677 for SCSS and 0.678 for CPF (P = .97). As SCSS increased (from 0 to 1 to 2 to 3 to 4 to 5), the reintubation rate decreased (from 29.4% to 25.0% to 19.4% to 16.1% to 13.2% to 4.1%). Conclusions SCSS was convenient to measure at the bedside. It was positively correlated with CPF and had the same accuracy for predicting reintubation after planned extubation.

scholarly journals Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database

2021 ◽  
Author(s):  
Harvineet Singh ◽  
Vishwali Mhasawade ◽  
Rumi Chunara
Keyword(s):  
Hospital Mortality ◽  
Risk Prediction ◽  
Performance Metrics ◽  
Prediction Models ◽  
Characteristic Curve ◽  
False Negative ◽  
Model Performance ◽  
Clinical Practices ◽  
Cross Sectional ◽  
Risk Prediction Models

Importance: Modern predictive models require large amounts of data for training and evaluation which can result in building models that are specific to certain locations, populations in them and clinical practices. Yet, best practices and guidelines for clinical risk prediction models have not yet considered such challenges to generalizability. Objectives: To investigate changes in measures of predictive discrimination, calibration, and algorithmic fairness when transferring models for predicting in-hospital mortality across ICUs in different populations. Also, to study the reasons for the lack of generalizability in these measures. Design, Setting, and Participants: In this multi-center cross-sectional study, electronic health records from 179 hospitals across the US with 70,126 hospitalizations were analyzed. Time of data collection ranged from 2014 to 2015. Main Outcomes and Measures: The main outcome is in-hospital mortality. Generalization gap, defined as difference between model performance metrics across hospitals, is computed for discrimination and calibration metrics, namely area under the receiver operating characteristic curve (AUC) and calibration slope. To assess model performance by race variable, we report differences in false negative rates across groups. Data were also analyzed using a causal discovery algorithm "Fast Causal Inference" (FCI) that infers paths of causal influence while identifying potential influences associated with unmeasured variables. Results: In-hospital mortality rates differed in the range of 3.9%-9.3% (1st-3rd quartile) across hospitals. When transferring models across hospitals, AUC at the test hospital ranged from 0.777 to 0.832 (1st to 3rd quartile; median 0.801); calibration slope from 0.725 to 0.983 (1st to 3rd quartile; median 0.853); and disparity in false negative rates from 0.046 to 0.168 (1st to 3rd quartile; median 0.092). When transferring models across geographies, AUC ranged from 0.795 to 0.813 (1st to 3rd quartile; median 0.804); calibration slope from 0.904 to 1.018 (1st to 3rd quartile; median 0.968); and disparity in false negative rates from 0.018 to 0.074 (1st to 3rd quartile; median 0.040). Distribution of all variable types (demography, vitals, and labs) differed significantly across hospitals and regions. Shifts in the race variable distribution and some clinical (vitals, labs and surgery) variables by hospital or region. Race variable also mediates differences in the relationship between clinical variables and mortality, by hospital/region. Conclusions and Relevance: Group-specific metrics should be assessed during generalizability checks to identify potential harms to the groups. In order to develop methods to improve and guarantee performance of prediction models in new environments for groups and individuals, better understanding and provenance of health processes as well as data generating processes by sub-group are needed to identify and mitigate sources of variation.

P3429A novel prediction model for mortality after cardiac surgery using institutional case volume

2019 ◽  
Vol 40 (Supplement_1) ◽  
Author(s):  
S Lee ◽  
J B Park ◽  
Y J Cho ◽  
H G Ryu ◽  
E J Jang
Keyword(s):  
Cardiac Surgery ◽  
Hospital Mortality ◽  
Risk Prediction ◽  
Prediction Models ◽  
Validation Cohort ◽  
Mortality Data ◽  
P Value ◽  
Case Volume ◽  
Risk Prediction Models ◽  
Derivation Cohort

Abstract Purpose A number of risk prediction models have been developed to identify short term mortality after cardiovascular surgery. Most models include patient characteristics, laboratory data, and type of surgery, but no consideration for the amount of surgical experience. With numerous reports on the impact of case volume on patient outcome after high risk procedures, we attempted to develop a risk prediction models for in-hospital and 1-year mortality that takes institutional case volume into account. Methods We identified adult patients who underwent cardiac surgery from January 2008 to December 2017 from the National Health Insurance Service (NHIS) database by searching for patients with procedure codes of coronary artery bypass grafting, valve surgery, and surgery on thoracic aorta during the hospitalization. Study subjects were randomly assigned to either the derivation cohort or the validation cohort. In-hospital mortality and 1-year mortality data were collected using the NHIS database. Risk prediction models were developed from the derivation cohort using Cox proportional hazards regression. The prediction performances of models were evaluated in the validation cohort. Results The models developed in this study demonstrated fair discrimination for derivation cohort (N=22,004, c-statistics, 0.75 for in-hospital mortality; 0.73 for 1-year mortality) and acceptable calibration in the validation cohort. (N=22,003, Hosmer-Lemeshow χ2-test, P=0.08 and 0.16, respectively). Case volume was the key factor of mortality prediction models after cardiac surgery. (50≤ x <100 case per year. 100≤ x <200 case per year, ≥200 case per year are correlated with OR 3.29, 2.49, 1.85 in in-hospital mortality, 2.76, 1.99, 1.69 in 1-year mortality respectively, P value <0.001.) Annual case volume as risk factor Variables In-hospital mortality 1-year mortality OR (95% CI) p-value OR (95% CI) p-value Annual case-volume (reference: ≥200) – – 100–200 1.69 (1.48, 1.93) <0.001 1.85 (1.58, 2.18) <0.001 50–100 1.99 (1.75, 2.25) <0.001 2.49 (2.15, 2.89) <0.001 <50 2.76 (2.44, 3.11) <0.001 3.29 (2.85, 3.79) <0.001 OR: Odds ratio; CI: confidence interval; Ref: Reference. Discrimination and calibration Conclusion We developed and validated new risk prediction models for in-hospital and 1-year mortality after cardiac surgery using the NHIS database. These models may provide useful guides to predict mortality risks of patients with basic information and without laboratory findings.

An evaluation of ML-based clinical risk prediction models in live EHR systems: a study on multiple use cases in different hospitals (Preprint)

2021 ◽  
Author(s):  
Hong Sun ◽  
Kristof Depraetere ◽  
Laurent Meesseman ◽  
Patricia Cabanillas Silva ◽  
Ralph Szymanowsky ◽  
...  
Keyword(s):  
Risk Prediction ◽  
Prediction Models ◽  
Characteristic Curve ◽  
Use Cases ◽  
Generic Model ◽  
Retrospective Data ◽  
Clinical Workflow ◽  
Risk Prediction Models ◽  
Clinical Risk ◽  
The Cross

BACKGROUND Machine learning (ML) algorithms are currently used in a wide array of clinical domains to produce models that can predict clinical risk events. Most models are developed and evaluated with retrospective data, very few are evaluated in a clinical workflow, and even fewer report performances in different hospitals. We provide detailed evaluations of clinical risk prediction models in live clinical workflows for three different use cases in three different hospitals. OBJECTIVE The main objective of this study is to evaluate the clinical risk prediction models in live clinical workflows and compare with their performance on retrospective data. We also aimed at generalizing the results by applying our investigation to three different use cases in three different hospitals. METHODS We trained clinical risk prediction models for three use cases (delirium, sepsis and acute kidney injury (AKI)) in three different hospitals with retrospective data. The models are deployed in these three hospitals and used in daily clinical practice. The predictions made by these models are logged and correlated with the diagnosis at discharge. We compared the performance with evaluations on retrospective data and conducted cross-hospital evaluations. RESULTS The performance of the prediction models in live clinical workflows is similar to the performance with retrospective data. The average value of area under the receiver-operating characteristic curve (AUROC) decreases slightly by 0.8 percentage point (from 89.4 % to 88.6%). The cross-hospital evaluations exhibit severe reduced performance, the averaged AUROC decreased by 8 percentage point (from 94.2% to 86.3%), which indicates the importance of model calibration with data from deployment hospitals. CONCLUSIONS Calibrating the prediction model with data from different deployment hospitals leads to a good performance in live settings. The performance degradation in the cross-hospital evaluation indicates limitations in developing a generic model for different hospitals. Designing a generic model development process to generate specialized prediction models for each hospital guarantees the model performance in different hospitals.

Sign in / Sign up

Export Citation Format

Share Document