Machine Learning-based Derivation and External Validation of a Tool to Predict Death and Development of Organ Failure in Hospitalized Patients with COVID-19

Abstract BackgroundCOVID-19 mortality risk stratification tools could improve care, inform accurate and rapid triage decisions, and guide family discussions regarding goals of care. A minority of COVID-19 prognostic tools have been tested in external cohorts. Our objective was to compare machine learning algorithms and develop a tool for predicting subsequent clinical outcomes in COVID-19. MethodsWe conducted a retrospective cohort study that included hospitalized patients with COVID-19 from March 2020 to March 2021. 712 consecutive patients from University of Washington (UW) and 345 patients from Tongji Hospital in China were included. We applied three different machine learning algorithms to clinical and laboratory data collected within the initial 24 hours of hospital admission to determine the risk of in-hospital mortality, transfer to the intensive care unit (ICU), shock requiring vasopressors, and receipt of renal replacement therapy (RRT). Mortality risk models were derived, internally validated in UW and externally validated in Tongji Hospital. The risk models for ICU transfer, shock and RRT were derived and internally validated in the UW dataset. ResultsAmong the UW dataset, 122 patients died (17%) during hospitalization and the mean days to hospital mortality was 15.7 +/- 21.5 (mean +/- SD). Elastic net logistic regression resulted in a C-statistic for in-hospital mortality of 0.72 (95% CI, 0.64 to 0.81) in the internal validation and 0.85 (95% CI, 0.81 to 0.89) in the external validation set. Age, platelet count, and white blood cell count were the most important predictors of mortality. In the sub-group of patients > 50 years of age, the mortality prediction model continued to perform with a C-statistic of 0.82 (95% CI:0.76,0.87). Mortality prediction models also performed well for shock and RRT in the UW dataset but functioned with lower accuracy for ICU transfer. ConclusionsWe trained, internally and externally validated a prediction model using data collected within 24 hours of hospital admission to predict in-hospital mortality on average two weeks prior to death. We also developed models to predict RRT and shock with high accuracy. These models could be used to improve triage decisions, resource allocation, and support clinical trial enrichment.

Download Full-text

Mortality Prediction in Cerebral Hemorrhage Patients Using Machine Learning Algorithms in Intensive Care Units

Frontiers in Neurology ◽

10.3389/fneur.2020.610531 ◽

2021 ◽

Vol 11 ◽

Author(s):

Ximing Nie ◽

Yuan Cai ◽

Jingyi Liu ◽

Xiran Liu ◽

Jiahui Zhao ◽

...

Keyword(s):

Machine Learning ◽

Intensive Care ◽

Random Forest ◽

Hospital Mortality ◽

Intensive Care Units ◽

Cerebral Hemorrhage ◽

Characteristic Curve ◽

Learning Algorithms ◽

Mortality Prediction ◽

Machine Learning Algorithms

Objectives: This study aims to investigate whether the machine learning algorithms could provide an optimal early mortality prediction method compared with other scoring systems for patients with cerebral hemorrhage in intensive care units in clinical practice.Methods: Between 2008 and 2012, from Intensive Care III (MIMIC-III) database, all cerebral hemorrhage patients monitored with the MetaVision system and admitted to intensive care units were enrolled in this study. The calibration, discrimination, and risk classification of predicted hospital mortality based on machine learning algorithms were assessed. The primary outcome was hospital mortality. Model performance was assessed with accuracy and receiver operating characteristic curve analysis.Results: Of 760 cerebral hemorrhage patients enrolled from MIMIC database [mean age, 68.2 years (SD, ±15.5)], 383 (50.4%) patients died in hospital, and 377 (49.6%) patients survived. The area under the receiver operating characteristic curve (AUC) of six machine learning algorithms was 0.600 (nearest neighbors), 0.617 (decision tree), 0.655 (neural net), 0.671(AdaBoost), 0.819 (random forest), and 0.725 (gcForest). The AUC was 0.423 for Acute Physiology and Chronic Health Evaluation II score. The random forest had the highest specificity and accuracy, as well as the greatest AUC, showing the best ability to predict in-hospital mortality.Conclusions: Compared with conventional scoring system and the other five machine learning algorithms in this study, random forest algorithm had better performance in predicting in-hospital mortality for cerebral hemorrhage patients in intensive care units, and thus further research should be conducted on random forest algorithm.

Download Full-text

Predicting nonroutine discharge after elective spine surgery: external validation of machine learning algorithms

Journal of Neurosurgery Spine ◽

10.3171/2019.5.spine1987 ◽

2019 ◽

Vol 31 (5) ◽

pp. 742-747 ◽

Cited By ~ 4

Author(s):

Brittany M. Stopa ◽

Faith C. Robertson ◽

Aditya V. Karhade ◽

Melissa Chua ◽

Marike L. D. Broekman ◽

...

Keyword(s):

Machine Learning ◽

Spine Surgery ◽

Learning Algorithm ◽

Lumbar Disc ◽

External Validation ◽

Machine Learning Algorithms ◽

Multidisciplinary Teams ◽

Machine Learning Algorithm ◽

Predictive Values ◽

C Statistic

OBJECTIVENonroutine discharge after elective spine surgery increases healthcare costs, negatively impacts patient satisfaction, and exposes patients to additional hospital-acquired complications. Therefore, prediction of nonroutine discharge in this population may improve clinical management. The authors previously developed a machine learning algorithm from national data that predicts risk of nonhome discharge for patients undergoing surgery for lumbar disc disorders. In this paper the authors externally validate their algorithm in an independent institutional population of neurosurgical spine patients.METHODSMedical records from elective inpatient surgery for lumbar disc herniation or degeneration in the Transitional Care Program at Brigham and Women’s Hospital (2013–2015) were retrospectively reviewed. Variables included age, sex, BMI, American Society of Anesthesiologists (ASA) class, preoperative functional status, number of fusion levels, comorbidities, preoperative laboratory values, and discharge disposition. Nonroutine discharge was defined as postoperative discharge to any setting other than home. The discrimination (c-statistic), calibration, and positive and negative predictive values (PPVs and NPVs) of the algorithm were assessed in the institutional sample.RESULTSOverall, 144 patients underwent elective inpatient surgery for lumbar disc disorders with a nonroutine discharge rate of 6.9% (n = 10). The median patient age was 50 years and 45.1% of patients were female. Most patients were ASA class II (66.0%), had 1 or 2 levels fused (80.6%), and had no diabetes (91.7%). The median hematocrit level was 41.2%. The neural network algorithm generalized well to the institutional data, with a c-statistic (area under the receiver operating characteristic curve) of 0.89, calibration slope of 1.09, and calibration intercept of −0.08. At a threshold of 0.25, the PPV was 0.50 and the NPV was 0.97.CONCLUSIONSThis institutional external validation of a previously developed machine learning algorithm suggests a reliable method for identifying patients with lumbar disc disorder at risk for nonroutine discharge. Performance in the institutional cohort was comparable to performance in the derivation cohort and represents an improved predictive value over clinician intuition. This finding substantiates initial use of this algorithm in clinical practice. This tool may be used by multidisciplinary teams of case managers and spine surgeons to strategically invest additional time and resources into postoperative plans for this population.

Download Full-text

Electrocardiogram-based mortality prediction in patients with COVID-19 using machine learning

EP Europace ◽

10.1093/europace/euab116.512 ◽

2021 ◽

Vol 23 (Supplement_3) ◽

Author(s):

H Bleijendaal ◽

RR Van Der Leur ◽

K Taha ◽

T Mast ◽

JMIH Gho ◽

...

Keyword(s):

Machine Learning ◽

Logistic Regression ◽

The Netherlands ◽

Hospital Mortality ◽

Regression Model ◽

Logistic Regression Model ◽

Prediction Models ◽

External Validation ◽

Mortality Prediction ◽

Receiver Operating Curve

Abstract Funding Acknowledgements Type of funding sources: Public hospital(s). Main funding source(s): The Netherlands Organisation for Health Research and Development (ZonMw) University of Amsterdam Research Priority Area Medical Integromics OnBehalf CAPACITY-COVID19 Registry Background The electrocardiogram (ECG) is an easy to assess, widely available and inexpensive tool that is frequently used during the work-up of hospitalized COVID-19 patients. So far, no study has been conducted to evaluate if ECG-based machine learning models are able to predict all-cause in-hospital mortality in COVID-19 patients. Purpose With this study, we aim to evaluate the value of using the ECG to predict in-hospital all-cause mortality of COVID-19 patients by analyzing the ECG at hospital admission, comparing a logistic regression based approach and a DNN based approach. Secondly, we aim to identify specific ECG features associated with mortality in patients diagnosed with COVID-19. Methods and results We studied 882 patients admitted with COVID-19 across seven hospitals in the Netherlands. Raw-format 12-lead ECGs recorded after admission (<72 hours) were collected, manually assessed, and annotated using pre-defined ECG features. Using data from five out of seven centers (n = 634), two mortality prediction models were developed: (a) a logistic regression model using manually annotated ECG features, and (b) a pre-trained deep neural network (DNN) using the raw ECG waveforms. Data from two other centers (n = 248) were used for external validation. Performance of both prediction models was similar, with a mean area under the receiver operating curve of 0.69 [95%CI 0.55–0.82] for the logistic regression model and 0.71 [95%CI 0.59–0.81] for the DNN in the external validation cohort. After adjustment for age and sex, ventricular rate (OR 1.13 [95% CI 1.01–1.27] per 10 ms increase), right bundle branch block (3.26 [95% CI 1.15–9.50]), ST-depression (2.78 [95% CI 1.03–7.70]) and low QRS voltages (3.09 [95% CI 1.02-9.38]) remained as significant predictors for mortality. Conclusion This study shows that ECG-based prediction models at admission may be a valuable addition to the initial risk stratification in admitted COVID-19 patients. The DNN model showed similar performance to the logistic regression that needs time-consuming manual annotation. Several ECG features associated with mortality were identified. Figure 1: Overview of methods, using and example case: (left) logistic regression and (right) deep learning. This specific case had a high probability of in-hospital mortality (above the threshold of 30%). Follow-up of this case showed that the patient had died during admission. Abstract Figure. Overview of ML methods used

Download Full-text

Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making

BMC Medical Informatics and Decision Making ◽

10.1186/s12911-021-01403-2 ◽

2021 ◽

Vol 21 (1) ◽

Author(s):

Alan Brnabic ◽

Lisa M. Hess

Keyword(s):

Machine Learning ◽

Decision Making ◽

Literature Review ◽

Systematic Literature Review ◽

Real World ◽

Learning Algorithms ◽

External Validation ◽

Machine Learning Algorithms ◽

Learning Methods ◽

Machine Learning Methods

Abstract Background Machine learning is a broad term encompassing a number of methods that allow the investigator to learn from the data. These methods may permit large real-world databases to be more rapidly translated to applications to inform patient-provider decision making. Methods This systematic literature review was conducted to identify published observational research of employed machine learning to inform decision making at the patient-provider level. The search strategy was implemented and studies meeting eligibility criteria were evaluated by two independent reviewers. Relevant data related to study design, statistical methods and strengths and limitations were identified; study quality was assessed using a modified version of the Luo checklist. Results A total of 34 publications from January 2014 to September 2020 were identified and evaluated for this review. There were diverse methods, statistical packages and approaches used across identified studies. The most common methods included decision tree and random forest approaches. Most studies applied internal validation but only two conducted external validation. Most studies utilized one algorithm, and only eight studies applied multiple machine learning algorithms to the data. Seven items on the Luo checklist failed to be met by more than 50% of published studies. Conclusions A wide variety of approaches, algorithms, statistical software, and validation strategies were employed in the application of machine learning methods to inform patient-provider decision making. There is a need to ensure that multiple machine learning approaches are used, the model selection strategy is clearly defined, and both internal and external validation are necessary to be sure that decisions for patient care are being made with the highest quality evidence. Future work should routinely employ ensemble methods incorporating multiple machine learning algorithms.

Download Full-text

Machine Learning for Mortality Prediction in Pediatric Myocarditis

Frontiers in Pediatrics ◽

10.3389/fped.2021.644922 ◽

2021 ◽

Vol 9 ◽

Author(s):

Fu-Sheng Chou ◽

Laxmi V. Ghimire

Keyword(s):

Machine Learning ◽

Linear Regression ◽

Regression Model ◽

Current Knowledge ◽

External Validation ◽

Model Development ◽

Kidney Injury ◽

High Specificity ◽

Mortality Prediction ◽

Linear Regression Models

Background: Pediatric myocarditis is a rare disease. The etiologies are multiple. Mortality associated with the disease is 5–8%. Prognostic factors were identified with the use of national hospitalization databases. Applying these identified risk factors for mortality prediction has not been reported.Methods: We used the Kids' Inpatient Database for this project. We manually curated fourteen variables as predictors of mortality based on the current knowledge of the disease, and compared performance of mortality prediction between linear regression models and a machine learning (ML) model. For ML, the random forest algorithm was chosen because of the categorical nature of the variables. Based on variable importance scores, a reduced model was also developed for comparison.Results: We identified 4,144 patients from the database for randomization into the primary (for model development) and testing (for external validation) datasets. We found that the conventional logistic regression model had low sensitivity (~50%) despite high specificity (>95%) or overall accuracy. On the other hand, the ML model struck a good balance between sensitivity (89.9%) and specificity (85.8%). The reduced ML model with top five variables (mechanical ventilation, cardiac arrest, ECMO, acute kidney injury, ventricular fibrillation) were sufficient to approximate the prediction performance of the full model.Conclusions: The ML algorithm performs superiorly when compared to the linear regression model for mortality prediction in pediatric myocarditis in this retrospective dataset. Prospective studies are warranted to further validate the applicability of our model in clinical settings.

Download Full-text

Abstract 196: Development and Validation of Machine Learning-based Race-specific Models to Predict 10-year Risk of Heart Failure: A Multi-cohort Analysis

Circulation ◽

10.1161/circ.142.suppl_3.196 ◽

2020 ◽

Vol 142 (Suppl_3) ◽

Author(s):

Matthew W Segar ◽

Byron Jaeger ◽

Kershaw V Patel ◽

Vijay Nambi ◽

Chiadi E Ndumele ◽

...

Keyword(s):

Machine Learning ◽

Risk Factors ◽

Heart Failure ◽

Risk Prediction ◽

Prediction Models ◽

External Validation ◽

Superior Performance ◽

Risk Models ◽

Black And White ◽

White Adults

Introduction: Heart failure (HF) risk and the underlying biological risk factors vary by race. Machine learning (ML) may improve race-specific HF risk prediction but this has not been fully evaluated. Methods: The study included participants from 4 cohorts (ARIC, DHS, JHS, and MESA) aged > 40 years, free of baseline HF, and with adjudicated HF event follow-up. Black adults from JHS and white adults from ARIC were used to derive race-specific ML models to predict 10-year HF risk. The ML models were externally validated in subgroups of black and white adults from ARIC (excluding JHS participants) and pooled MESA/DHS cohorts and compared to prior established HF risk scores developed in ARIC and MESA. Harrell’s C-index and Greenwood-Nam-D’Agostino chi-square were used to assess discrimination and calibration, respectively. Results: In the derivation cohorts, 288 of 4141 (7.0%) black and 391 of 8242 (4.7%) white adults developed HF over 10 years. The ML models had excellent discrimination in both black and white participants (C-indices = 0.88 and 0.89). In the external validation cohorts for black participants from ARIC (excluding JHS, N = 1072) and MESA/DHS pooled cohorts (N = 2821), 131 (12.2%) and 115 (4.1%) developed HF. The ML model had adequate calibration and demonstrated superior discrimination compared to established HF risk models (Fig A). A consistent pattern was also observed in the external validation cohorts of white participants from the MESA/DHS pooled cohorts (N=3236; 100 [3.1%] HF events) (Fig A). The most important predictors of HF in both races were NP levels. Cardiac biomarkers and glycemic parameters were most important among blacks while LV hypertrophy and prevalent CVD and traditional CV risk factors were the strongest predictors among whites (Fig B). Conclusions: Race-specific and ML-based HF risk models that integrate clinical, laboratory, and biomarker data demonstrated superior performance when compared to traditional risk prediction models.

Download Full-text

Machine Learning Algorithms to Predict Recurrence within 10 Years after Breast Cancer Surgery: A Prospective Cohort Study

Cancers ◽

10.3390/cancers12123817 ◽

2020 ◽

Vol 12 (12) ◽

pp. 3817

Author(s):

Shi-Jer Lou ◽

Ming-Feng Hou ◽

Hong-Tai Chang ◽

Chong-Chi Chiu ◽

Hao-Hsien Lee ◽

...

Keyword(s):

Breast Cancer ◽

Machine Learning ◽

Learning Algorithms ◽

External Validation ◽

Model Development ◽

Cancer Surgery ◽

Machine Learning Algorithms ◽

Breast Cancer Surgery ◽

Training Dataset

No studies have discussed machine learning algorithms to predict recurrence within 10 years after breast cancer surgery. This study purposed to compare the accuracy of forecasting models to predict recurrence within 10 years after breast cancer surgery and to identify significant predictors of recurrence. Registry data for breast cancer surgery patients were allocated to a training dataset (n = 798) for model development, a testing dataset (n = 171) for internal validation, and a validating dataset (n = 171) for external validation. Global sensitivity analysis was then performed to evaluate the significance of the selected predictors. Demographic characteristics, clinical characteristics, quality of care, and preoperative quality of life were significantly associated with recurrence within 10 years after breast cancer surgery (p < 0.05). Artificial neural networks had the highest prediction performance indices. Additionally, the surgeon volume was the best predictor of recurrence within 10 years after breast cancer surgery, followed by hospital volume and tumor stage. Accurate recurrence within 10 years prediction by machine learning algorithms may improve precision in managing patients after breast cancer surgery and improve understanding of risk factors for recurrence within 10 years after breast cancer surgery.

Download Full-text

Predictive model for acute respiratory distress syndrome events in ICU patients in China using machine learning algorithms: a secondary analysis of a cohort study

Journal of Translational Medicine ◽

10.1186/s12967-019-2075-0 ◽

2019 ◽

Vol 17 (1) ◽

Cited By ~ 4

Author(s):

Xian-Fei Ding ◽

Jin-Bo Li ◽

Huo-Yan Liang ◽

Zong-Yu Wang ◽

Ting-Ting Jiao ◽

...

Keyword(s):

Machine Learning ◽

Acute Respiratory Distress Syndrome ◽

Cohort Study ◽

Respiratory Distress Syndrome ◽

Respiratory Distress ◽

Distress Syndrome ◽

External Validation ◽

Secondary Analysis ◽

Machine Learning Algorithms ◽

Chinese Patients

Abstract Background To develop a machine learning model for predicting acute respiratory distress syndrome (ARDS) events through commonly available parameters, including baseline characteristics and clinical and laboratory parameters. Methods A secondary analysis of a multi-centre prospective observational cohort study from five hospitals in Beijing, China, was conducted from January 1, 2011, to August 31, 2014. A total of 296 patients at risk for developing ARDS admitted to medical intensive care units (ICUs) were included. We applied a random forest approach to identify the best set of predictors out of 42 variables measured on day 1 of admission. Results All patients were randomly divided into training (80%) and testing (20%) sets. Additionally, these patients were followed daily and assessed according to the Berlin definition. The model obtained an average area under the receiver operating characteristic (ROC) curve (AUC) of 0.82 and yielded a predictive accuracy of 83%. For the first time, four new biomarkers were included in the model: decreased minimum haematocrit, glucose, and sodium and increased minimum white blood cell (WBC) count. Conclusions This newly established machine learning-based model shows good predictive ability in Chinese patients with ARDS. External validation studies are necessary to confirm the generalisability of our approach across populations and treatment practices.

Download Full-text

A simple index predicting mortality in acutely hospitalized patients

QJM ◽

10.1093/qjmed/hcaa293 ◽

2020 ◽

Author(s):

Paul Froom ◽

Zvi Shimoni ◽

Jochanan Benbassat ◽

Bernard Silke

Keyword(s):

Hospital Mortality ◽

Medical Condition ◽

External Validation ◽

Laboratory Data ◽

Age Groups ◽

Mortality Rates ◽

Hospitalized Patients ◽

Outcome Variable ◽

Simple Index ◽

C Statistic

Abstract Background Mortality rates used to evaluate and improve the quality of hospital care are adjusted for comorbidity and disease severity. Comorbidity measured by International Classification of Diseases codes do not reflect the severity of the medical condition that requires clinical assessments not available in electronic databases, and/or laboratory data with clinically relevant ranges to permit extrapolation from one setting to the next. Aim To propose a simple index predicting mortality in acutely hospitalized patients. Design Retrospective cohort study with internal and external validation. Methods The study populations were all acutely admitted patients in 2015-6, and in January - November, 2019 to internal medicine, cardiology and intensive care departments at the Laniado Hospital in Israel, and in 2002-19, at St James Hospital, Ireland. Predictor variables were age and admission laboratory tests. The outcome variable was in-hospital mortality. Using logistic regression of the data in the 2015-6 Israeli cohort, we derived an index that included age groups and significant laboratory data. Results In the Israeli 2015-6 cohort the index predicted mortality rates from 0.2 to 32.0% with a c-statistic (area under the ROC curve) of 0.86. In the Israeli 2019 validation cohort, the index predicted mortality rates from 0.3 to 38.9% with a c-statistic of 0.87. An abbreviated index performed similarly in the Irish 2002-19 cohort. Conclusions Hospital mortality can be predicted by age and selected admission laboratory data without acquiring information from the patient’s medical records. This permits an inexpensive comparison of performance of hospital departments.

Download Full-text

Machine Learning to Predict In-Hospital Mortality in COVID-19 Patients Using Computed Tomography-Derived Pulmonary and Vascular Features

Journal of Personalized Medicine ◽

10.3390/jpm11060501 ◽

2021 ◽

Vol 11 (6) ◽

pp. 501

Author(s):

Simone Schiaffino ◽

Marina Codari ◽

Andrea Cozzi ◽

Domenico Albano ◽

Marco Alì ◽

...

Keyword(s):

Machine Learning ◽

Computed Tomography ◽

Pulmonary Artery ◽

Hospital Mortality ◽

Univariate Analysis ◽

Mortality Prediction ◽

Chest Ct ◽

Multilayer Perceptrons ◽

Emergency Department Admission ◽

Pulmonary Artery Diameter

Pulmonary parenchymal and vascular damage are frequently reported in COVID-19 patients and can be assessed with unenhanced chest computed tomography (CT), widely used as a triaging exam. Integrating clinical data, chest CT features, and CT-derived vascular metrics, we aimed to build a predictive model of in-hospital mortality using univariate analysis (Mann–Whitney U test) and machine learning models (support vectors machines (SVM) and multilayer perceptrons (MLP)). Patients with RT-PCR-confirmed SARS-CoV-2 infection and unenhanced chest CT performed on emergency department admission were included after retrieving their outcome (discharge or death), with an 85/15% training/test dataset split. Out of 897 patients, the 229 (26%) patients who died during hospitalization had higher median pulmonary artery diameter (29.0 mm) than patients who survived (27.0 mm, p < 0.001) and higher median ascending aortic diameter (36.6 mm versus 34.0 mm, p < 0.001). SVM and MLP best models considered the same ten input features, yielding a 0.747 (precision 0.522, recall 0.800) and 0.844 (precision 0.680, recall 0.567) area under the curve, respectively. In this model integrating clinical and radiological data, pulmonary artery diameter was the third most important predictor after age and parenchymal involvement extent, contributing to reliable in-hospital mortality prediction, highlighting the value of vascular metrics in improving patient stratification.

Download Full-text