Improving and Externally Validating Mortality Prediction Models for COVID-19 Using Publicly Available Data

We conducted a systematic survey of COVID-19 endpoint prediction literature to: (a) identify publications that include data that adhere to FAIR (findability, accessibility, interoperability, and reusability) principles and (b) develop and reuse mortality prediction models that best generalize to these datasets. The largest such cohort data we knew of was used for model development. The associated published prediction model was subjected to recursive feature elimination to find a minimal logistic regression model which had statistically and clinically indistinguishable predictive performance. This model could still not be applied to the four external validation sets that were identified, due to complete absence of needed model features in some external sets. Thus, a generalizable model (GM) was built which could be applied to all four external validation sets. An age-only model was used as a benchmark, as it is the simplest, effective, and robust predictor of mortality currently known in COVID-19 literature. While the GM surpassed the age-only model in three external cohorts, for the fourth external cohort, there was no statistically significant difference. This study underscores: (1) the paucity of FAIR data being shared by researchers despite the glut of COVID-19 prediction models and (2) the difficulty of creating any model that consistently outperforms an age-only model due to the cohort diversity of available datasets.

Predictive Models of Intensive Care Unit Mortality - Severity of Illness Scores or Artificial Intelligence instruments? - Literature Review and Metanalysis (Preprint)

BACKGROUND The Severity of Illness Scores (SIS)- Acute Physiology and Chronic Health Evaluation (APACHE), Simplified Acute Physiology Score (SAPS), and Sequential Organ Failure Assessment (SOFA) - are current risk stratification and mortality prediction tools used in Intensive Care Units (ICU) across the globe, and rely on scores that assess disease severity on admission. Developers of Artificial Intelligence (AI) or Machine Learning (ML) models predictive of ICU mortality use the SIS performance as a reference point when reporting the performance of these computational constructs. OBJECTIVE Using systematic review and meta-analysis, we evaluated studies that compare ML-based mortality prediction models to SIS-based models. The review should inform clinicians regarding the prognostic value of ML-based ICU mortality prediction models compared with SIS models and their validity in supporting clinical decision-making. METHODS We performed a systematic search using PubMed, Scopus, Embase, and IEEE databases. Studies that report the performance of newly developed ML models predictive of ICU mortality and compare it with the performance of SIS models on the same datasets were eligible for inclusion. ML and the SIS models with a reported Area Under the Receiver Operating Characteristic (AUROC) curve were included in the meta-analysis to identify the group with superior performance. Data were extracted with guidance from the CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) checklist[1] and was appraised for risk of bias and applicability using PROBAST (A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies ) [2]. RESULTS After screening the literature, we identified and included 20 papers containing 47 ML models based on seven types of algorithms that were compared with three types of SIS models. The AUROC for predicting ICU mortality ranged between 0.828-0.875 for ML-based models and between 0.707-0.760 for SI-based models. We noted substantial heterogeneity among the models reported, and considerable variation among the AUROC estimates for both ML and SIS model types. Due to the high degree of heterogeneity, we performed a limited random-effect meta-analysis of externally validated subgroups of ML models and the subgroups of SIS used for comparison. CONCLUSIONS ML-based models can accurately predict ICU mortality as an alternative to traditional scoring models. The high degree of heterogeneity observed within and between studies limit the assessment of pooled results. The differences in development strategies, validation, statistical, and computational methods that these models rely on impede a head-to-head comparison, and we cannot declare the superiority of one model over the other. Consequently, we make no recommendation regarding the ML-based ICU mortality prediction models’ performance in clinical practice. To bridge the knowledge gap from design to practice, ML model developers must provide explainer models and make those knowledge objects reproducible, interoperable, and transparent[3]. CLINICALTRIAL the review was registered and approved by the international prospective register of systematic reviews, PROSPERO (reference number CRD42021203871).

Addition of admission lactate levels to Baux score improves mortality prediction in severe burns

Risk adjustment and mortality prediction models are central in optimising care and for benchmarking purposes. In the burn setting, the Baux score and its derivatives have been the mainstay for predictions of mortality from burns. Other well-known measures to predict mortality stem from the ICU setting, where, for example, the Simplified Acute Physiology Score (SAPS 3) models have been found to be instrumental. Other attempts to further improve the prediction of outcome have been based on the following variables at admission: Sequential Organ Failure Assessment (aSOFA) score, determinations of aLactate or Neutrophil to Lymphocyte Ratio (aNLR). The aim of the present study was to examine if estimated mortality rate (EMR, SAPS 3), aSOFA, aLactate, and aNLR can, either alone or in conjunction with the others, improve the mortality prediction beyond that of the effects of age and percentage total body surface area (TBSA%) burned among patients with severe burns who need critical care. This is a retrospective, explorative, single centre, registry study based on prospectively gathered data. The study included 222 patients with median (25th–75th centiles) age of 55.0 (38.0 to 69.0) years, TBSA% burned was 24.5 (13.0 to 37.2) and crude mortality was 17%. As anticipated highest predicting power was obtained with age and TBSA% with an AUC at 0.906 (95% CI 0.857 to 0.955) as compared with EMR, aSOFA, aLactate and aNLR. The largest effect was seen thereafter by adding aLactate to the model, increasing AUC to 0.938 (0.898 to 0.979) (p < 0.001). Whereafter, adding EMR, aSOFA, and aNLR, separately or in combinations, only marginally improved the prediction power. This study shows that the prediction model with age and TBSA% may be improved by adding aLactate, despite the fact that aLactate levels were only moderately increased. Thereafter, adding EMR, aSOFA or aNLR only marginally affected the mortality prediction.

Automated, high-dimensional evaluation of physiological aging and resilience in outbred mice

Behavior and physiology are essential readouts in many studies but have not benefited from the high-dimensional data revolution that has transformed molecular and cellular phenotyping. To address this, we developed an approach that combines commercially available automated phenotyping hardware with a systems biology analysis pipeline to generate a high-dimensional readout of mouse behavior/physiology, as well as intuitive and health-relevant summary statistics (resilience and biological age). We used this platform to longitudinally evaluate aging in hundreds of outbred mice across an age range from 6 months to 3.4 years. In contrast to the assumption that aging can only be measured at the limits of animal ability via challenge-based tasks, we observed widespread physiological and behavioral aging starting in early life. Using network connectivity analysis, we found that organism-level resilience exhibited an accelerating decline with age that was distinct from the trajectory of individual phenotypes. We developed a method, Combined Aging and Survival Prediction of Aging Rate (CASPAR), for jointly predicting chronological age and survival time and showed that the resulting model is able to predict both variables simultaneously, a behavior that is not captured by separate age and mortality prediction models. This study provides a uniquely high-resolution view of physiological aging in mice and demonstrates that systems-level analysis of physiology provides insights not captured by individual phenotypes. The approach described here allows aging, and other processes that affect behavior and physiology, to be studied with sophistication and rigor.

Predicting lethal courses in critically ill COVID-19 patients using a machine learning model trained on patients with non-COVID-19 viral pneumonia

In a pandemic with a novel disease, disease-specific prognosis models are available only with a delay. To bridge the critical early phase, models built for similar diseases might be applied. To test the accuracy of such a knowledge transfer, we investigated how precise lethal courses in critically ill COVID-19 patients can be predicted by a model trained on critically ill non-COVID-19 viral pneumonia patients. We trained gradient boosted decision tree models on 718 (245 deceased) non-COVID-19 viral pneumonia patients to predict individual ICU mortality and applied it to 1054 (369 deceased) COVID-19 patients. Our model showed a significantly better predictive performance (AUROC 0.86 [95% CI 0.86–0.87]) than the clinical scores APACHE2 (0.63 [95% CI 0.61–0.65]), SAPS2 (0.72 [95% CI 0.71–0.74]) and SOFA (0.76 [95% CI 0.75–0.77]), the COVID-19-specific mortality prediction models of Zhou (0.76 [95% CI 0.73–0.78]) and Wang (laboratory: 0.62 [95% CI 0.59–0.65]; clinical: 0.56 [95% CI 0.55–0.58]) and the 4C COVID-19 Mortality score (0.71 [95% CI 0.70–0.72]). We conclude that lethal courses in critically ill COVID-19 patients can be predicted by a machine learning model trained on non-COVID-19 patients. Our results suggest that in a pandemic with a novel disease, prognosis models built for similar diseases can be applied, even when the diseases differ in time courses and in rates of critical and lethal courses.

High mortality rate of obstetric critically ill women in Rwanda and its predictability

Background Reasons for admission to intensive care units (ICUs) for obstetric patients vary from one setting to another. Outcomes from ICU and prediction models are not well explored in Rwanda owing to lack of appropriate scores. This study aimed to assess reasons for admission and accuracy of prediction models for mortality of obstetric patients admitted to ICUs of two public tertiary hospitals in Rwanda. Methods We prospectively collected data from all obstetric patients admitted to the ICUs of the two public tertiary hospitals in Rwanda from March 2017 to February 2018 to identify reasons for admission, demographic and clinical characteristics, outcome including death and its predictability by both the Modified Early Obstetric Warning Score (MEOWS) and quick Sequential Organ Failure Assessment (qSOFA). We analysed the accuracy of mortality prediction models by MEOWS or qSOFA by using logistic regression adjusting for factors associated with mortality. Area under the Receiver Operating characteristic (AUROC) curves is used to show the predicting capacity for each individual tool. Results Obstetric patients (n = 94) represented 12.8 % of all 747 ICU admissions which is 1.8 % of all 4.999 admitted women for pregnancy or labor. Sepsis (n = 30; 31.9 %) and obstetric haemorrhage (n = 24; 25.5 %) were the two commonest reasons for ICU admission. Overall ICU mortality for obstetric patients was 54.3 % (n = 51) with average length of stay of 6.6 ± 7.525 days. MEOWS score was an independent predictor of mortality (adjusted (a)OR 1.25; 95 % CI 1.07–1.46) and so was qSOFA score (aOR 2.81; 95 % CI 1.25–6.30) with an adjusted AUROC of 0.773 (95 % CI 0.67–0.88) and 0.764 (95 % CI 0.65–0.87), indicating fair accuracy for ICU mortality prediction in these settings of both MEOWS and qSOFA scores. Conclusions Sepsis and obstetric haemorrhage were the commonest reasons for obstetric admissions to ICU in Rwanda. MEOWS and qSOFA scores could accurately predict ICU mortality of obstetric patients in resource-limited settings, but larger studies are needed before a recommendation for their use in routine practice in similar settings.