Development and validation of a machine learning model to predict mortality risk in patients with COVID-19

New York City quickly became an epicentre of the COVID-19 pandemic. An ability to triage patients was needed due to a sudden and massive increase in patients during the COVID-19 pandemic as healthcare providers incurred an exponential increase in workload,which created a strain on the staff and limited resources. Further, methods to better understand and characterise the predictors of morbidity and mortality was needed.MethodsWe developed a prediction model to predict patients at risk for mortality using only laboratory, vital and demographic information readily available in the electronic health record on more than 3395 hospital admissions with COVID-19. Multiple methods were applied, and final model was selected based on performance. A variable importance algorithm was used for interpretability, and understanding of performance and predictors was applied to the best model. We built a model with an area under the receiver operating characteristic curve of 83–97 to identify predictors and patients with high risk of mortality due to COVID-19. Oximetry, respirations, blood urea nitrogen, lymphocyte per cent, calcium, troponin and neutrophil percentage were important features, and key ranges were identified that contributed to a 50% increase in patients’ mortality prediction score. With an increasing negative predictive value starting 0.90 after the second day of admission suggests we might be able to more confidently identify likely survivorsDiscussionThis study serves as a use case of a machine learning methods with visualisations to aide clinicians with a better understanding of the model and predictors of mortality.ConclusionAs we continue to understand COVID-19, computer assisted algorithms might be able to improve the care of patients.

Download Full-text

Mortality Prediction in Cerebral Hemorrhage Patients Using Machine Learning Algorithms in Intensive Care Units

Frontiers in Neurology ◽

10.3389/fneur.2020.610531 ◽

2021 ◽

Vol 11 ◽

Author(s):

Ximing Nie ◽

Yuan Cai ◽

Jingyi Liu ◽

Xiran Liu ◽

Jiahui Zhao ◽

...

Keyword(s):

Machine Learning ◽

Intensive Care ◽

Random Forest ◽

Hospital Mortality ◽

Intensive Care Units ◽

Cerebral Hemorrhage ◽

Characteristic Curve ◽

Learning Algorithms ◽

Mortality Prediction ◽

Machine Learning Algorithms

Objectives: This study aims to investigate whether the machine learning algorithms could provide an optimal early mortality prediction method compared with other scoring systems for patients with cerebral hemorrhage in intensive care units in clinical practice.Methods: Between 2008 and 2012, from Intensive Care III (MIMIC-III) database, all cerebral hemorrhage patients monitored with the MetaVision system and admitted to intensive care units were enrolled in this study. The calibration, discrimination, and risk classification of predicted hospital mortality based on machine learning algorithms were assessed. The primary outcome was hospital mortality. Model performance was assessed with accuracy and receiver operating characteristic curve analysis.Results: Of 760 cerebral hemorrhage patients enrolled from MIMIC database [mean age, 68.2 years (SD, ±15.5)], 383 (50.4%) patients died in hospital, and 377 (49.6%) patients survived. The area under the receiver operating characteristic curve (AUC) of six machine learning algorithms was 0.600 (nearest neighbors), 0.617 (decision tree), 0.655 (neural net), 0.671(AdaBoost), 0.819 (random forest), and 0.725 (gcForest). The AUC was 0.423 for Acute Physiology and Chronic Health Evaluation II score. The random forest had the highest specificity and accuracy, as well as the greatest AUC, showing the best ability to predict in-hospital mortality.Conclusions: Compared with conventional scoring system and the other five machine learning algorithms in this study, random forest algorithm had better performance in predicting in-hospital mortality for cerebral hemorrhage patients in intensive care units, and thus further research should be conducted on random forest algorithm.

Download Full-text

Development and validation of a machine learning-based prediction model for near-term in-hospital mortality among patients with COVID-19

BMJ Supportive & Palliative Care ◽

10.1136/bmjspcare-2020-002602 ◽

2020 ◽

pp. bmjspcare-2020-002602 ◽

Cited By ~ 1

Author(s):

Prathamesh Parchure ◽

Himanshu Joshi ◽

Kavita Dharmarajan ◽

Robert Freeman ◽

David L Reich ◽

...

Keyword(s):

Machine Learning ◽

Hospital Mortality ◽

Characteristic Curve ◽

Laboratory Data ◽

Mortality Prediction ◽

Outcome Variable ◽

Electronic Health Record Data ◽

Test Set ◽

Electronic Health ◽

Near Term

ObjectivesTo develop and validate a model for prediction of near-term in-hospital mortality among patients with COVID-19 by application of a machine learning (ML) algorithm on time-series inpatient data from electronic health records.MethodsA cohort comprised of 567 patients with COVID-19 at a large acute care healthcare system between 10 February 2020 and 7 April 2020 observed until either death or discharge. Random forest (RF) model was developed on randomly drawn 70% of the cohort (training set) and its performance was evaluated on the rest of 30% (the test set). The outcome variable was in-hospital mortality within 20–84 hours from the time of prediction. Input features included patients’ vital signs, laboratory data and ECG results.ResultsPatients had a median age of 60.2 years (IQR 26.2 years); 54.1% were men. In-hospital mortality rate was 17.0% and overall median time to death was 6.5 days (range 1.3–23.0 days). In the test set, the RF classifier yielded a sensitivity of 87.8% (95% CI: 78.2% to 94.3%), specificity of 60.6% (95% CI: 55.2% to 65.8%), accuracy of 65.5% (95% CI: 60.7% to 70.0%), area under the receiver operating characteristic curve of 85.5% (95% CI: 80.8% to 90.2%) and area under the precision recall curve of 64.4% (95% CI: 53.5% to 75.3%).ConclusionsOur ML-based approach can be used to analyse electronic health record data and reliably predict near-term mortality prediction. Using such a model in hospitals could help improve care, thereby better aligning clinical decisions with prognosis in critically ill patients with COVID-19.

Download Full-text

Abstract 10992: Machine Learning to Predict In-Hospital Cardiac Arrest in Patients Admitted from the Emergency Department with COVID-19 and Suspected Pneumonia

Circulation ◽

10.1161/circ.144.suppl_2.10992 ◽

2021 ◽

Vol 144 (Suppl_2) ◽

Author(s):

Tsung-Chien Lu ◽

Eric H Chou ◽

CHIH-HUNG WANG ◽

Amir Mostafavi ◽

Mario Tovar ◽

...

Keyword(s):

Machine Learning ◽

Cardiac Arrest ◽

Characteristic Curve ◽

Vital Signs ◽

Early Warning Score ◽

Gradient Boosting ◽

Patients At Risk ◽

End Of Life Decision ◽

Hospital Cardiac Arrest ◽

Suspected Pneumonia

Introduction: There are only scarce models developed for stratifying the risk of cardiac arrest from COVID-19 patients presenting to the ED with suspected pneumonia. By using the machine learning (ML) approach, we aimed to develop and validate the ML models to predict in-hospital cardiac arrest (IHCA) in patients admitted from the ED. Hypothesis: We hypothesized that ML approach can serve as a valuable tool in identifying patients at risk of IHCA in a timely fashion. Methods: We included the COVID-19 patients admitted from the EDs of five hospitals in Texas between March and November 2020. All adult (≥ 18 years) patients were included if they had positive RT-PCR for SARS-CoV-2 and also received CXR examination for suspected pneumonia. Patients’ demographic, past medical history, vital signs at ED triage, CXR findings, and laboratory results were retrieved from the EMR system. The primary outcome (IHCA) was identified via a resuscitation code. Patients presented as OHCA or without any blood testing were excluded. Nonrandom splitting strategy based on different location was used to divide the dataset into the training (one urban and two suburban hospitals) and testing cohort (one urban and one suburban hospital) at around 2-to-1 ratio. Three supervised ML models were trained and performances were evaluated and compared with the National Early Warning Score (NEWS) by the area under the receiver operating characteristic curve (AUC). Results: We included 1,485 records for analysis. Of them, 190 (12.8%) developed IHCA. Of the constructed ML models, Random Forest outperformed the others with the best AUC result (0.930, 95% CI: 0.896-0.958), followed by Gradient Boosting (0.929, 95% CI: 0.891-0.959) and Extra Trees classifier (0.909, 95% CI: 0.875-0.943). All constructed ML models performed significantly better than by using the NEWS scoring system (AUC: 0.787, 95% CI: 0.725-0.840). The top six important features selected were age, oxygen saturation at triage, and lab data of APTT, lactic acid, and LDH. Conclusions: The ML approach showed excellent discriminatory performance to identify IHCA for patients with COVID-19 and suspected pneumonia. It has the potential to save more life or provide end-of-life decision making if successfully implemented in the EMR system.

Download Full-text

An approach to predicting patient experience through machine learning and social network analysis

Journal of the American Medical Informatics Association ◽

10.1093/jamia/ocaa194 ◽

2020 ◽

Vol 27 (12) ◽

pp. 1834-1843

Author(s):

Vitej Bari ◽

Jamie S Hirsch ◽

Joseph Narvaez ◽

Robert Sardinia ◽

Kevin R Bock ◽

...

Keyword(s):

Machine Learning ◽

Social Network ◽

Social Network Analysis ◽

Network Analysis ◽

Patient Experience ◽

Operating Characteristic ◽

Characteristic Curve ◽

Healthcare Providers ◽

Hospital Consumer Assessment ◽

Consumer Assessment

Abstract Objective Improving the patient experience has become an essential component of any healthcare system’s performance metrics portfolio. In this study, we developed a machine learning model to predict a patient’s response to the Hospital Consumer Assessment of Healthcare Providers and Systems survey’s “Doctor Communications” domain questions while simultaneously identifying most impactful providers in a network. Materials and Methods This is an observational study of patients admitted to a single tertiary care hospital between 2016 and 2020. Using machine learning algorithms, electronic health record data were used to predict patient responses to Hospital Consumer Assessment of Healthcare Providers and Systems survey questions in the doctor domain, and patients who are at risk for responding negatively were identified. Model performance was assessed by area under receiver-operating characteristic curve. Social network analysis metrics were also used to identify providers most impactful to patient experience. Results Using a random forest algorithm, patients’ responses to the following 3 questions were predicted: “During this hospital stay how often did doctors. 1) treat you with courtesy and respect? 2) explain things in a way that you could understand? 3) listen carefully to you?” with areas under the receiver-operating characteristic curve of 0.876, 0.819, and 0.819, respectively. Social network analysis found that doctors with higher centrality appear to have an outsized influence on patient experience, as measured by rank in the random forest model in the doctor domain. Conclusions A machine learning algorithm identified patients at risk of a negative experience. Furthermore, a doctor social network framework provides metrics for identifying those providers that are most influential on the patient experience.

Download Full-text

Development and validation of a practical machine-learning triage algorithm for the detection of patients in need of critical care in the emergency department

Scientific Reports ◽

10.1038/s41598-021-03104-2 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Yecheng Liu ◽

Jiandong Gao ◽

Jihai Liu ◽

Joseph Harold Walline ◽

Xiaoying Liu ◽

...

Keyword(s):

Machine Learning ◽

Emergency Department ◽

Critically Ill ◽

Critically Ill Patients ◽

Reference Model ◽

Characteristic Curve ◽

Learning System ◽

Development And Validation ◽

Triage Algorithm ◽

Mls Method

AbstractIdentifying critically ill patients is a key challenge in emergency department (ED) triage. Mis-triage errors are still widespread in triage systems around the world. Here, we present a machine learning system (MLS) to assist ED triage officers better recognize critically ill patients and provide a text-based explanation of the MLS recommendation. To derive the MLS, an existing dataset of 22,272 patient encounters from 2012 to 2019 from our institution’s electronic emergency triage system (EETS) was used for algorithm training and validation. The area under the receiver operating characteristic curve (AUC) was 0.875 ± 0.006 (CI:95%) in retrospective dataset using fivefold cross validation, higher than that of reference model (0.843 ± 0.005 (CI:95%)). In the prospective cohort study, compared to the traditional triage system’s 1.2% mis-triage rate, the mis-triage rate in the MLS-assisted group was 0.9%. This MLS method with a real-time explanation for triage officers was able to lower the mis-triage rate of critically ill ED patients.

Download Full-text

Multicenter validation of a machine learning algorithm for 48 hour all-cause mortality prediction

10.1101/427054 ◽

2018 ◽

Author(s):

Hamid Mohamadlou ◽

Saarang Panchavati ◽

Jacob Calvert ◽

Anna Lynn-Palevsky ◽

Christopher Barton ◽

...

Keyword(s):

Machine Learning ◽

Operating Characteristic ◽

Learning Algorithm ◽

Clinical Care ◽

Characteristic Curve ◽

Mortality Prediction ◽

Health Centers ◽

Training Data ◽

All Cause Mortality ◽

Test Sets

AbstractPurposeThis study evaluates a machine-learning-based mortality prediction tool.Materials and MethodsWe conducted a retrospective study with data drawn from three academic health centers. Inpatients of at least 18 years of age and with at least one observation of each vital sign were included. Predictions were made at 12, 24, and 48 hours before death. Models fit to training data from each institution were evaluated on hold-out test data from the same institution and data from the remaining institutions. Predictions were compared to those of qSOFA and MEWS using area under the receiver operating characteristic curve (AUROC).ResultsFor training and testing on data from a single institution, machine learning predictions averaged AUROCs of 0.97, 0.96, and 0.95 across institutional test sets for 12-, 24-, and 48-hour predictions, respectively. When trained and tested on data from different hospitals, the algorithm achieved AUROC up to 0.95, 0.93, and 0.91, for 12-, 24-, and 48-hour predictions, respectively. MEWS and qSOFA had average 48-hour AUROCs of 0.86 and 0.82, respectively.ConclusionThis algorithm may help identify patients in need of increased levels of clinical care.

Download Full-text

Machine Learning–Based Mortality Prediction of Patients at Risk During Hospital Admission

Journal of Patient Safety ◽

10.1097/pts.0000000000000957 ◽

2022 ◽

Vol Publish Ahead of Print ◽

Author(s):

Kevin M. Trentino ◽

Karin Schwarzbauer ◽

Andreas Mitterecker ◽

Axel Hofmann ◽

Adam Lloyd ◽

...

Keyword(s):

Machine Learning ◽

At Risk ◽

Hospital Admission ◽

Mortality Prediction ◽

Patients At Risk

Download Full-text

Decline in stroke alerts and hospitalisations during the COVID-19 pandemic

Stroke and Vascular Neurology ◽

10.1136/svn-2020-000441 ◽

2020 ◽

Vol 5 (4) ◽

pp. 403-405 ◽

Cited By ~ 7

Author(s):

Malveeka Sharma ◽

Vasileios-Arsenios Lioutas ◽

Tracy Madsen ◽

Judith Clark ◽

Jillian O'Sullivan ◽

...

Keyword(s):

New York ◽

Symptom Onset ◽

Hospital Admissions ◽

Healthcare Access ◽

Healthcare Providers ◽

Neurological Complications ◽

Stroke Severity ◽

Hospital Arrival ◽

Hospital Systems ◽

Public Health Officials

IntroductionPatients with stroke-like symptoms may be underutilising emergency medical services and avoiding hospitalisation during the COVID-19 pandemic. We investigated a decline in admissions for stroke and transient ischaemic attack (TIA) and emergency department (ED) stroke alert activations.MethodsWe retrospectively compiled total weekly hospital admissions for stroke and TIA between 31 December 2018 and 21 April 2019 versus 30 December 2019 and 19 April 2020 at five US tertiary academic comprehensive stroke centres in cities with early COVID-19 outbreaks in Boston, New York City, Providence and Seattle. We collected available data on ED stroke alerts, stroke severity using the National Institutes of Health Stroke Scale (NIHSS) and time from symptom onset to hospital arrival.ResultsCompared with 31 December 2018 to 21 April 2019, a decline in stroke/TIA admissions and ED stroke alerts occurred during 30 December 2019 to 19 April 2020 (p trend <0.001 for each). The declines coincided with state stay-at-home recommendations in late March. The greatest decline in hospital admissions was observed between 23 March and 19 April 2020, with a 31% decline compared with the corresponding weeks in 2019. Three of the five centres with 2019 and 2020 stroke alert data had a 46% decline in ED stroke alerts in late March and April 2020, compared with 2019. Median baseline NIHSS during these 4 weeks was 10 in 2020 and 7 in 2019. There was no difference in time from symptom onset to hospital arrival.ConclusionAt these five large academic US hospitals, admissions for stroke and TIA declined during the COVID-19 pandemic. There was a trend for fewer ED stroke alerts at three of the five centres with available 2019 and 2020 data. Acute stroke therapies are time-sensitive, so decreased healthcare access or utilisation may lead to more disabling or fatal strokes, or more severe non-neurological complications related to stroke. Our findings underscore the indirect effects of this pandemic. Public health officials, hospital systems and healthcare providers must continue to encourage patients with stroke to seek acute care during this crisis.

Download Full-text

Automated Spleen Injury Detection Using 3D Active Contours and Machine Learning

Entropy ◽

10.3390/e23040382 ◽

2021 ◽

Vol 23 (4) ◽

pp. 382

Author(s):

Julie Wang ◽

Alexander Wood ◽

Chao Gao ◽

Kayvan Najarian ◽

Jonathan Gryak

Keyword(s):

Machine Learning ◽

Random Forest ◽

Active Contours ◽

Imaging Modality ◽

Characteristic Curve ◽

Current Method ◽

Ct Scans ◽

Computer Assisted ◽

K Nearest Neighbors ◽

Active Hemorrhage

The spleen is one of the most frequently injured organs in blunt abdominal trauma. Computed tomography (CT) is the imaging modality of choice to assess patients with blunt spleen trauma, which may include lacerations, subcapsular or parenchymal hematomas, active hemorrhage, and vascular injuries. While computer-assisted diagnosis systems exist for other conditions assessed using CT scans, the current method to detect spleen injuries involves the manual review of scans by radiologists, which is a time-consuming and repetitive process. In this study, we propose an automated spleen injury detection method using machine learning. CT scans from patients experiencing traumatic injuries were collected from Michigan Medicine and the Crash Injury Research Engineering Network (CIREN) dataset. Ninety-nine scans of healthy and lacerated spleens were split into disjoint training and test sets, with random forest (RF), naive Bayes, SVM, k-nearest neighbors (k-NN) ensemble, and subspace discriminant ensemble models trained via 5-fold cross validation. Of these models, random forest performed the best, achieving an Area Under the receiver operating characteristic Curve (AUC) of 0.91 and an F1 score of 0.80 on the test set. These results suggest that an automated, quantitative assessment of traumatic spleen injury has the potential to enable faster triage and improve patient outcomes.

Download Full-text

Development and Validation of a Practical Machine-learning Triage Algorithm for the Detection of Patients in Need of Critical Care in the Emergency Department

10.21203/rs.3.rs-418248/v1 ◽

2021 ◽

Author(s):

Yecheng Liu ◽

Jiandong Gao ◽

Jihai Liu ◽

Joseph Harold Walline ◽

Xiaoying Liu ◽

...

Keyword(s):

Machine Learning ◽

Emergency Department ◽

Cohort Study ◽

Critically Ill ◽

Critically Ill Patients ◽

Characteristic Curve ◽

Learning System ◽

Validation Dataset ◽

Development And Validation ◽

Triage Algorithm

Abstract Identifying critically ill patients is a key challenge in emergency department (ED) triage. Mis-triage errors are still widespread in triage systems around the world. Here, we present a machine learning system (MLS) to assist ED triage officers better recognize critically ill patients and provide a text-based explanation of the MLS recommendation. To derive the MLS, an existing dataset of 20,272 patient encounters from 2012 to 2019 from our institution’s electronic emergency triage system (EETS) was used for algorithm training. We then conducted a prospective randomized cohort study of ED patients between March and April 2020. The area under the receiver operating characteristic curve (AUC) was 0.86 in a retrospective validation dataset of 2,000 randomized database cases. In the prospective cohort study, compared to the traditional triage system’s 1.2% mis-triage rate, the mis-triage rate in the MLS-assisted group was 0.9%. This MLS method with a real-time explanation for triage officers was able to lower the mis-triage rate of critically ill ED patients.

Download Full-text