Evaluation of crowdsourced mortality prediction models as a framework for assessing AI in medicine

AbstractApplications of machine learning in healthcare are of high interest and have the potential to significantly improve patient care. Yet, the real-world accuracy and performance of these models on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate different methods that predict healthcare outcomes. To overcome patient privacy concerns, we employed a Model-to-Data approach, allowing citizen scientists and researchers to train and evaluate machine learning models on private health data without direct access to that data. We focused on the prediction of all-cause mortality as the community challenge question. In total, we had 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries. The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI 0.942, 0.951) and an area under the precision-recall curve of 0.487 (95% CI 0.458, 0.499) on patients prospectively collected over a one year observation of a large health system. Post-hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data and have similar accuracy on the population. This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.

Download Full-text

Predicting mortality in hemodialysis patients using machine learning analysis

Clinical Kidney Journal ◽

10.1093/ckj/sfaa126 ◽

2020 ◽

Author(s):

Victoria Garcia-Montemayor ◽

Alejandro Martin-Malo ◽

Carlo Barbieri ◽

Francesco Bellocchio ◽

Sagrario Soriano ◽

...

Keyword(s):

Machine Learning ◽

Logistic Regression ◽

Random Forest ◽

Prediction Models ◽

Initial Period ◽

Area Under The Curve ◽

Mortality Prediction ◽

Machine Learning Techniques ◽

Prediction Of Mortality ◽

Mortality Prediction Models

Abstract Background Besides the classic logistic regression analysis, non-parametric methods based on machine learning techniques such as random forest are presently used to generate predictive models. The aim of this study was to evaluate random forest mortality prediction models in haemodialysis patients. Methods Data were acquired from incident haemodialysis patients between 1995 and 2015. Prediction of mortality at 6 months, 1 year and 2 years of haemodialysis was calculated using random forest and the accuracy was compared with logistic regression. Baseline data were constructed with the information obtained during the initial period of regular haemodialysis. Aiming to increase accuracy concerning baseline information of each patient, the period of time used to collect data was set at 30, 60 and 90 days after the first haemodialysis session. Results There were 1571 incident haemodialysis patients included. The mean age was 62.3 years and the average Charlson comorbidity index was 5.99. The mortality prediction models obtained by random forest appear to be adequate in terms of accuracy [area under the curve (AUC) 0.68–0.73] and superior to logistic regression models (ΔAUC 0.007–0.046). Results indicate that both random forest and logistic regression develop mortality prediction models using different variables. Conclusions Random forest is an adequate method, and superior to logistic regression, to generate mortality prediction models in haemodialysis patients.

Download Full-text

Predicting lethal courses in critically ill COVID-19 patients using a machine learning model trained on patients with non-COVID-19 viral pneumonia

Scientific Reports ◽

10.1038/s41598-021-92475-7 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Gregor Lichtner ◽

Felix Balzer ◽

Stefan Haufe ◽

Niklas Giesa ◽

Fridtjof Schiefenhövel ◽

...

Keyword(s):

Machine Learning ◽

Critically Ill ◽

Prediction Models ◽

Predictive Performance ◽

Learning Model ◽

Mortality Prediction ◽

Viral Pneumonia ◽

Machine Learning Model ◽

Mortality Prediction Models ◽

Time Courses

AbstractIn a pandemic with a novel disease, disease-specific prognosis models are available only with a delay. To bridge the critical early phase, models built for similar diseases might be applied. To test the accuracy of such a knowledge transfer, we investigated how precise lethal courses in critically ill COVID-19 patients can be predicted by a model trained on critically ill non-COVID-19 viral pneumonia patients. We trained gradient boosted decision tree models on 718 (245 deceased) non-COVID-19 viral pneumonia patients to predict individual ICU mortality and applied it to 1054 (369 deceased) COVID-19 patients. Our model showed a significantly better predictive performance (AUROC 0.86 [95% CI 0.86–0.87]) than the clinical scores APACHE2 (0.63 [95% CI 0.61–0.65]), SAPS2 (0.72 [95% CI 0.71–0.74]) and SOFA (0.76 [95% CI 0.75–0.77]), the COVID-19-specific mortality prediction models of Zhou (0.76 [95% CI 0.73–0.78]) and Wang (laboratory: 0.62 [95% CI 0.59–0.65]; clinical: 0.56 [95% CI 0.55–0.58]) and the 4C COVID-19 Mortality score (0.71 [95% CI 0.70–0.72]). We conclude that lethal courses in critically ill COVID-19 patients can be predicted by a machine learning model trained on non-COVID-19 patients. Our results suggest that in a pandemic with a novel disease, prognosis models built for similar diseases can be applied, even when the diseases differ in time courses and in rates of critical and lethal courses.

Download Full-text

Machine learning-based in-hospital mortality prediction models for patients with acute coronary syndrome

The American Journal of Emergency Medicine ◽

10.1016/j.ajem.2021.12.070 ◽

2022 ◽

Author(s):

Jun Ke ◽

Yiwei Chen ◽

Xiaoping Wang ◽

Zhiyong Wu ◽

Qiongyao Zhang ◽

...

Keyword(s):

Machine Learning ◽

Acute Coronary Syndrome ◽

Hospital Mortality ◽

Prediction Models ◽

Mortality Prediction ◽

Coronary Syndrome ◽

Mortality Prediction Models

Download Full-text

Symptom Prediction and Mortality Risk Calculation for COVID-19 Using Machine Learning

Frontiers in Artificial Intelligence ◽

10.3389/frai.2021.673527 ◽

2021 ◽

Vol 4 ◽

Author(s):

Elham Jamshidi ◽

Amirhossein Asgary ◽

Nader Tavakoli ◽

Alireza Zali ◽

Farzaneh Dastan ◽

...

Keyword(s):

Machine Learning ◽

Prediction Model ◽

Prediction Models ◽

Comprehensive Evaluation ◽

Gastrointestinal Symptoms ◽

Risk Groups ◽

Mortality Prediction ◽

Healthcare Outcomes ◽

Consciousness Disorders ◽

Medical Histories

Background: Early prediction of symptoms and mortality risks for COVID-19 patients would improve healthcare outcomes, allow for the appropriate distribution of healthcare resources, reduce healthcare costs, aid in vaccine prioritization and self-isolation strategies, and thus reduce the prevalence of the disease. Such publicly accessible prediction models are lacking, however.Methods: Based on a comprehensive evaluation of existing machine learning (ML) methods, we created two models based solely on the age, gender, and medical histories of 23,749 hospital-confirmed COVID-19 patients from February to September 2020: a symptom prediction model (SPM) and a mortality prediction model (MPM). The SPM predicts 12 symptom groups for each patient: respiratory distress, consciousness disorders, chest pain, paresis or paralysis, cough, fever or chill, gastrointestinal symptoms, sore throat, headache, vertigo, loss of smell or taste, and muscular pain or fatigue. The MPM predicts the death of COVID-19-positive individuals.Results: The SPM yielded ROC-AUCs of 0.53–0.78 for symptoms. The most accurate prediction was for consciousness disorders at a sensitivity of 74% and a specificity of 70%. 2,440 deaths were observed in the study population. MPM had a ROC-AUC of 0.79 and could predict mortality with a sensitivity of 75% and a specificity of 70%. About 90% of deaths occurred in the top 21 percentile of risk groups. To allow patients and clinicians to use these models easily, we created a freely accessible online interface at www.aicovid.net.Conclusion: The ML models predict COVID-19-related symptoms and mortality using information that is readily available to patients as well as clinicians. Thus, both can rapidly estimate the severity of the disease, allowing shared and better healthcare decisions with regard to hospitalization, self-isolation strategy, and COVID-19 vaccine prioritization in the coming months.

Download Full-text

Symptom Prediction and Mortality Risk Calculation for COVID-19 Using Machine Learning

10.1101/2021.02.04.21251143 ◽

2021 ◽

Author(s):

Elham Jamshidi ◽

Amirhossein Asgary ◽

Nader Tavakoli ◽

Alireza Zali ◽

Farzaneh Dastan ◽

...

Keyword(s):

Machine Learning ◽

Prediction Model ◽

Prediction Models ◽

Comprehensive Evaluation ◽

Gastrointestinal Symptoms ◽

Risk Groups ◽

Mortality Prediction ◽

Healthcare Outcomes ◽

Consciousness Disorders ◽

Medical Histories

ABSTRACTBackgroundEarly prediction of symptoms and mortality risks for COVID-19 patients would improve healthcare outcomes, allow for the appropriate distribution of healthcare resources, reduce healthcare costs, aid in vaccine prioritization and self-isolation strategies, and thus reduce the prevalence of the disease. Such publicly accessible prediction models are lacking, however.MethodsBased on a comprehensive evaluation of existing machine learning (ML) methods, we created two models based solely on the age, gender, and medical histories of 23,749 hospital-confirmed COVID-19 patients from February to September 2020: a symptom prediction model (SPM) and a mortality prediction model (MPM). The SPM predicts 12 symptom groups for each patient: respiratory distress, consciousness disorders, chest pain, paresis or paralysis, cough, fever or chill, gastrointestinal symptoms, sore throat, headache, vertigo, loss of smell or taste, and muscular pain or fatigue. The MPM predicts the death of COVID-19-positive individuals.ResultsThe SPM yielded ROC-AUCs of 0.53-0.78 for symptoms. The most accurate prediction was for consciousness disorders at a sensitivity of 74% and a specificity of 70%. 2440 deaths were observed in the study population. MPM had a ROC-AUC of 0.79 and could predict mortality with a sensitivity of 75% and a specificity of 70%. About 90% of deaths occurred in the top 21 percentile of risk groups. To allow patients and clinicians to use these models easily, we created a freely accessible online interface at www.aicovid.org.ConclusionsThe ML models predict COVID-19-related symptoms and mortality using information that is readily available to patients as well as clinicians. Thus, both can rapidly estimate the severity of the disease, allowing shared and better healthcare decisions with regard to hospitalization, self-isolation strategy, and COVID-19 vaccine prioritization in the coming months.Abstract Figure

Download Full-text

Piloting a model-to-data approach to enable predictive analytics in health care through patient mortality prediction

Journal of the American Medical Informatics Association ◽

10.1093/jamia/ocaa083 ◽

2020 ◽

Vol 27 (9) ◽

pp. 1393-1400

Author(s):

Timothy Bergquist ◽

Yao Yan ◽

Thomas Schaffter ◽

Thomas Yu ◽

Vikas Pejaver ◽

...

Keyword(s):

Predictive Models ◽

Performance Metrics ◽

Prediction Models ◽

Predictive Analytics ◽

Characteristic Curve ◽

Mortality Prediction ◽

Patient Privacy ◽

University Of Washington ◽

Mortality Prediction Models ◽

The University

Abstract Objective The development of predictive models for clinical application requires the availability of electronic health record (EHR) data, which is complicated by patient privacy concerns. We showcase the “Model to Data” (MTD) approach as a new mechanism to make private clinical data available for the development of predictive models. Under this framework, we eliminate researchers’ direct interaction with patient data by delivering containerized models to the EHR data. Materials and Methods We operationalize the MTD framework using the Synapse collaboration platform and an on-premises secure computing environment at the University of Washington hosting EHR data. Containerized mortality prediction models developed by a model developer, were delivered to the University of Washington via Synapse, where the models were trained and evaluated. Model performance metrics were returned to the model developer. Results The model developer was able to develop 3 mortality prediction models under the MTD framework using simple demographic features (area under the receiver-operating characteristic curve [AUROC], 0.693), demographics and 5 common chronic diseases (AUROC, 0.861), and the 1000 most common features from the EHR’s condition/procedure/drug domains (AUROC, 0.921). Discussion We demonstrate the feasibility of the MTD framework to facilitate the development of predictive models on private EHR data, enabled by common data models and containerization software. We identify challenges that both the model developer and the health system information technology group encountered and propose future efforts to improve implementation. Conclusions The MTD framework lowers the barrier of access to EHR data and can accelerate the development and evaluation of clinical prediction models.

Download Full-text

Update and, internal and temporal-validation of the FRANCE-2 and ACC-TAVI early-mortality prediction models for Transcatheter aortic Valve Implantation (TAVI) using data from the Netherlands heart registration (NHR)

IJC Heart & Vasculature ◽

10.1016/j.ijcha.2021.100716 ◽

2021 ◽

Vol 32 ◽

pp. 100716 ◽

Cited By ~ 1

Author(s):

Hatem Al-Farra ◽

Bas A.J.M. de Mol ◽

Anita C.J. Ravelli ◽

W.J.P.P. ter Burg ◽

Saskia Houterman ◽

...

Keyword(s):

Aortic Valve ◽

Prediction Models ◽

Mortality Prediction ◽

Early Mortality ◽

Transcatheter Aortic Valve ◽

Temporal Validation ◽

Mortality Prediction Models ◽

Aortic Valve Implantation ◽

Using Data ◽

Valve Implantation

Download Full-text

High mortality rate of obstetric critically ill women in Rwanda and its predictability

BMC Pregnancy and Childbirth ◽

10.1186/s12884-021-03882-7 ◽

2021 ◽

Vol 21 (1) ◽

Author(s):

Alcade Rudakemwa ◽

Amyl Lucille Cassidy ◽

Théogène Twagirumugabe

Keyword(s):

Prediction Models ◽

Average Length ◽

Mortality Prediction ◽

Routine Practice ◽

Icu Mortality ◽

Resource Limited ◽

Obstetric Haemorrhage ◽

Tertiary Hospitals ◽

Mortality Prediction Models ◽

Qsofa Score

Abstract Background Reasons for admission to intensive care units (ICUs) for obstetric patients vary from one setting to another. Outcomes from ICU and prediction models are not well explored in Rwanda owing to lack of appropriate scores. This study aimed to assess reasons for admission and accuracy of prediction models for mortality of obstetric patients admitted to ICUs of two public tertiary hospitals in Rwanda. Methods We prospectively collected data from all obstetric patients admitted to the ICUs of the two public tertiary hospitals in Rwanda from March 2017 to February 2018 to identify reasons for admission, demographic and clinical characteristics, outcome including death and its predictability by both the Modified Early Obstetric Warning Score (MEOWS) and quick Sequential Organ Failure Assessment (qSOFA). We analysed the accuracy of mortality prediction models by MEOWS or qSOFA by using logistic regression adjusting for factors associated with mortality. Area under the Receiver Operating characteristic (AUROC) curves is used to show the predicting capacity for each individual tool. Results Obstetric patients (n = 94) represented 12.8 % of all 747 ICU admissions which is 1.8 % of all 4.999 admitted women for pregnancy or labor. Sepsis (n = 30; 31.9 %) and obstetric haemorrhage (n = 24; 25.5 %) were the two commonest reasons for ICU admission. Overall ICU mortality for obstetric patients was 54.3 % (n = 51) with average length of stay of 6.6 ± 7.525 days. MEOWS score was an independent predictor of mortality (adjusted (a)OR 1.25; 95 % CI 1.07–1.46) and so was qSOFA score (aOR 2.81; 95 % CI 1.25–6.30) with an adjusted AUROC of 0.773 (95 % CI 0.67–0.88) and 0.764 (95 % CI 0.65–0.87), indicating fair accuracy for ICU mortality prediction in these settings of both MEOWS and qSOFA scores. Conclusions Sepsis and obstetric haemorrhage were the commonest reasons for obstetric admissions to ICU in Rwanda. MEOWS and qSOFA scores could accurately predict ICU mortality of obstetric patients in resource-limited settings, but larger studies are needed before a recommendation for their use in routine practice in similar settings.

Download Full-text