scholarly journals Improving and Externally Validating Mortality Prediction Models for COVID-19 Using Publicly Available Data

BioMed ◽  
2022 ◽  
Vol 2 (1) ◽  
pp. 13-26
Author(s):  
Avishek Chatterjee ◽  
Guus Wilmink ◽  
Henry Woodruff ◽  
Philippe Lambin
Keyword(s):  
Logistic Regression Model ◽  
Prediction Models ◽  
External Validation ◽  
Model Development ◽  
Predictive Performance ◽  
Mortality Prediction ◽  
Recursive Feature Elimination ◽  
Significant Difference ◽  
Mortality Prediction Models ◽  
Model Features

We conducted a systematic survey of COVID-19 endpoint prediction literature to: (a) identify publications that include data that adhere to FAIR (findability, accessibility, interoperability, and reusability) principles and (b) develop and reuse mortality prediction models that best generalize to these datasets. The largest such cohort data we knew of was used for model development. The associated published prediction model was subjected to recursive feature elimination to find a minimal logistic regression model which had statistically and clinically indistinguishable predictive performance. This model could still not be applied to the four external validation sets that were identified, due to complete absence of needed model features in some external sets. Thus, a generalizable model (GM) was built which could be applied to all four external validation sets. An age-only model was used as a benchmark, as it is the simplest, effective, and robust predictor of mortality currently known in COVID-19 literature. While the GM surpassed the age-only model in three external cohorts, for the fourth external cohort, there was no statistically significant difference. This study underscores: (1) the paucity of FAIR data being shared by researchers despite the glut of COVID-19 prediction models and (2) the difficulty of creating any model that consistently outperforms an age-only model due to the cohort diversity of available datasets.

scholarly journals Developing and validating COVID-19 adverse outcome risk prediction models from a bi-national European cohort of 5594 patients

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Espen Jimenez-Solem ◽  
Tonny S. Petersen ◽  
Casper Hansen ◽  
Christian Hansen ◽  
Christina Lioma ◽  
...  
Keyword(s):  
Prediction Models ◽  
Severe Disease ◽  
External Validation ◽  
Area Under The Curve ◽  
Predictive Performance ◽  
Mortality Prediction ◽  
Icu Admission ◽  
Risk Of Death ◽  
Online Risk ◽  
The United Kingdom

AbstractPatients with severe COVID-19 have overwhelmed healthcare systems worldwide. We hypothesized that machine learning (ML) models could be used to predict risks at different stages of management and thereby provide insights into drivers and prognostic markers of disease progression and death. From a cohort of approx. 2.6 million citizens in Denmark, SARS-CoV-2 PCR tests were performed on subjects suspected for COVID-19 disease; 3944 cases had at least one positive test and were subjected to further analysis. SARS-CoV-2 positive cases from the United Kingdom Biobank was used for external validation. The ML models predicted the risk of death (Receiver Operation Characteristics—Area Under the Curve, ROC-AUC) of 0.906 at diagnosis, 0.818, at hospital admission and 0.721 at Intensive Care Unit (ICU) admission. Similar metrics were achieved for predicted risks of hospital and ICU admission and use of mechanical ventilation. Common risk factors, included age, body mass index and hypertension, although the top risk features shifted towards markers of shock and organ dysfunction in ICU patients. The external validation indicated fair predictive performance for mortality prediction, but suboptimal performance for predicting ICU admission. ML may be used to identify drivers of progression to more severe disease and for prognostication patients in patients with COVID-19. We provide access to an online risk calculator based on these findings.

P–419 On prognosis after unexplained recurrent pregnancy losses (RPL); a systematic review and external validation of clinical prediction models

2021 ◽  
Vol 36 (Supplement_1) ◽  
Author(s):  
A Youssef
Keyword(s):  
Systematic Review ◽  
Supportive Care ◽  
Prediction Model ◽  
Pregnancy Outcome ◽  
Prediction Models ◽  
External Validation ◽  
Model Development ◽  
Predictive Performance ◽  
Pregnancy Rates ◽  
Cochrane Library

Abstract Study question Which models that predict pregnancy outcome in couples with unexplained RPL exist and what is the performance of the most used model? Summary answer We identified seven prediction models; none followed the recommended prediction model development steps. Moreover, the most used model showed poor predictive performance. What is known already RPL remains unexplained in 50–75% of couples For these couples, there is no effective treatment option and clinical management rests on supportive care. Essential part of supportive care consists of counselling on the prognosis of subsequent pregnancies. Indeed, multiple prediction models exist, however the quality and validity of these models varies. In addition, the prediction model developed by Brigham et al is the most widely used model, but has never been externally validated. Study design, size, duration We performed a systematic review to identify prediction models for pregnancy outcome after unexplained RPL. In addition we performed an external validation of the Brigham model in a retrospective cohort, consisting of 668 couples with unexplained RPL that visited our RPL clinic between 2004 and 2019. Participants/materials, setting, methods A systematic search was performed in December 2020 in Pubmed, Embase, Web of Science and Cochrane library to identify relevant studies. Eligible studies were selected and assessed according to the TRIPOD) guidelines, covering topics on model performance and validation statement. The performance of predicting live birth in the Brigham model was evaluated through calibration and discrimination, in which the observed pregnancy rates were compared to the predicted pregnancy rates. Main results and the role of chance Seven models were compared and assessed according to the TRIPOD statement. This resulted in two studies of low, three of moderate and two of above average reporting quality. These studies did not follow the recommended steps for model development and did not calculate a sample size. Furthermore, the predictive performance of neither of these models was internally- or externally validated. We performed an external validation of Brigham model. Calibration showed overestimation of the model and too extreme predictions, with a negative calibration intercept of –0.52 (CI 95% –0.68 – –0.36), with a calibration slope of 0.39 (CI 95% 0.07 – 0.71). The discriminative ability of the model was very low with a concordance statistic of 0.55 (CI 95% 0.50 – 0.59). Limitations, reasons for caution None of the studies are specifically named prediction models, therefore models may have been missed in the selection process. The external validation cohort used a retrospective design, in which only the first pregnancy after intake was registered. Follow-up time was not limited, which is important in counselling unexplained RPL couples. Wider implications of the findings: Currently, there are no suitable models that predict on pregnancy outcome after RPL. Moreover, we are in need of a model with several variables such that prognosis is individualized, and factors from both the female as the male to enable a couple specific prognosis. Trial registration number Not applicable

scholarly journals Predicting lethal courses in critically ill COVID-19 patients using a machine learning model trained on patients with non-COVID-19 viral pneumonia

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Gregor Lichtner ◽  
Felix Balzer ◽  
Stefan Haufe ◽  
Niklas Giesa ◽  
Fridtjof Schiefenhövel ◽  
...  
Keyword(s):  
Machine Learning ◽  
Critically Ill ◽  
Prediction Models ◽  
Predictive Performance ◽  
Learning Model ◽  
Mortality Prediction ◽  
Viral Pneumonia ◽  
Machine Learning Model ◽  
Mortality Prediction Models ◽  
Time Courses

AbstractIn a pandemic with a novel disease, disease-specific prognosis models are available only with a delay. To bridge the critical early phase, models built for similar diseases might be applied. To test the accuracy of such a knowledge transfer, we investigated how precise lethal courses in critically ill COVID-19 patients can be predicted by a model trained on critically ill non-COVID-19 viral pneumonia patients. We trained gradient boosted decision tree models on 718 (245 deceased) non-COVID-19 viral pneumonia patients to predict individual ICU mortality and applied it to 1054 (369 deceased) COVID-19 patients. Our model showed a significantly better predictive performance (AUROC 0.86 [95% CI 0.86–0.87]) than the clinical scores APACHE2 (0.63 [95% CI 0.61–0.65]), SAPS2 (0.72 [95% CI 0.71–0.74]) and SOFA (0.76 [95% CI 0.75–0.77]), the COVID-19-specific mortality prediction models of Zhou (0.76 [95% CI 0.73–0.78]) and Wang (laboratory: 0.62 [95% CI 0.59–0.65]; clinical: 0.56 [95% CI 0.55–0.58]) and the 4C COVID-19 Mortality score (0.71 [95% CI 0.70–0.72]). We conclude that lethal courses in critically ill COVID-19 patients can be predicted by a machine learning model trained on non-COVID-19 patients. Our results suggest that in a pandemic with a novel disease, prognosis models built for similar diseases can be applied, even when the diseases differ in time courses and in rates of critical and lethal courses.

scholarly journals Assessing prognosis and prediction of treatment response in early rheumatoid arthritis: systematic reviews

2018 ◽  
Vol 22 (66) ◽  
pp. 1-294 ◽  
Author(s):  
Rachel Archer ◽  
Emma Hock ◽  
Jean Hamilton ◽  
John Stevens ◽  
Munira Essat ◽  
...  
Keyword(s):  
Rheumatoid Arthritis ◽  
Prediction Models ◽  
Meta Analysis ◽  
External Validation ◽  
Model Development ◽  
Predictive Performance ◽  
Assessment Tools ◽  
Clinical Prediction ◽  
Clinical Prediction Models ◽  
Early Ra

Background Rheumatoid arthritis (RA) is a chronic, debilitating disease associated with reduced quality of life and substantial costs. It is unclear which tests and assessment tools allow the best assessment of prognosis in people with early RA and whether or not variables predict the response of patients to different drug treatments. Objective To systematically review evidence on the use of selected tests and assessment tools in patients with early RA (1) in the evaluation of a prognosis (review 1) and (2) as predictive markers of treatment response (review 2). Data sources Electronic databases (e.g. MEDLINE, EMBASE, The Cochrane Library, Web of Science Conference Proceedings; searched to September 2016), registers, key websites, hand-searching of reference lists of included studies and key systematic reviews and contact with experts. Study selection Review 1 – primary studies on the development, external validation and impact of clinical prediction models for selected outcomes in adult early RA patients. Review 2 – primary studies on the interaction between selected baseline covariates and treatment (conventional and biological disease-modifying antirheumatic drugs) on salient outcomes in adult early RA patients. Results Review 1 – 22 model development studies and one combined model development/external validation study reporting 39 clinical prediction models were included. Five external validation studies evaluating eight clinical prediction models for radiographic joint damage were also included. c-statistics from internal validation ranged from 0.63 to 0.87 for radiographic progression (different definitions, six studies) and 0.78 to 0.82 for the Health Assessment Questionnaire (HAQ). Predictive performance in external validations varied considerably. Three models [(1) Active controlled Study of Patients receiving Infliximab for the treatment of Rheumatoid arthritis of Early onset (ASPIRE) C-reactive protein (ASPIRE CRP), (2) ASPIRE erythrocyte sedimentation rate (ASPIRE ESR) and (3) Behandelings Strategie (BeSt)] were externally validated using the same outcome definition in more than one population. Results of the random-effects meta-analysis suggested substantial uncertainty in the expected predictive performance of models in a new sample of patients. Review 2 – 12 studies were identified. Covariates examined included anti-citrullinated protein/peptide anti-body (ACPA) status, smoking status, erosions, rheumatoid factor status, C-reactive protein level, erythrocyte sedimentation rate, swollen joint count (SJC), body mass index and vascularity of synovium on power Doppler ultrasound (PDUS). Outcomes examined included erosions/radiographic progression, disease activity, physical function and Disease Activity Score-28 remission. There was statistical evidence to suggest that ACPA status, SJC and PDUS status at baseline may be treatment effect modifiers, but not necessarily that they are prognostic of response for all treatments. Most of the results were subject to considerable uncertainty and were not statistically significant. Limitations The meta-analysis in review 1 was limited by the availability of only a small number of external validation studies. Studies rarely investigated the interaction between predictors and treatment. Suggested research priorities Collaborative research (including the use of individual participant data) is needed to further develop and externally validate the clinical prediction models. The clinical prediction models should be validated with respect to individual treatments. Future assessments of treatment by covariate interactions should follow good statistical practice. Conclusions Review 1 – uncertainty remains over the optimal prediction model(s) for use in clinical practice. Review 2 – in general, there was insufficient evidence that the effect of treatment depended on baseline characteristics. Study registration This study is registered as PROSPERO CRD42016042402. Funding The National Institute for Health Research Health Technology Assessment programme.

scholarly journals Electrocardiogram-based mortality prediction in patients with COVID-19 using machine learning

EP Europace ◽  
2021 ◽  
Vol 23 (Supplement_3) ◽  
Author(s):  
H Bleijendaal ◽  
RR Van Der Leur ◽  
K Taha ◽  
T Mast ◽  
JMIH Gho ◽  
...  
Keyword(s):  
Machine Learning ◽  
Logistic Regression ◽  
The Netherlands ◽  
Hospital Mortality ◽  
Regression Model ◽  
Logistic Regression Model ◽  
Prediction Models ◽  
External Validation ◽  
Mortality Prediction ◽  
Receiver Operating Curve

Abstract Funding Acknowledgements Type of funding sources: Public hospital(s). Main funding source(s): The Netherlands Organisation for Health Research and Development (ZonMw) University of Amsterdam Research Priority Area Medical Integromics OnBehalf CAPACITY-COVID19 Registry Background The electrocardiogram (ECG) is an easy to assess, widely available and inexpensive tool that is frequently used during the work-up of hospitalized COVID-19 patients. So far, no study has been conducted to evaluate if ECG-based machine learning models are able to predict all-cause in-hospital mortality in COVID-19 patients. Purpose With this study, we aim to evaluate the value of using the ECG to predict in-hospital all-cause mortality of COVID-19 patients by analyzing the ECG at hospital admission, comparing a logistic regression based approach and a DNN based approach. Secondly, we aim to identify specific ECG features associated with mortality in patients diagnosed with COVID-19.  Methods and results We studied 882 patients admitted with COVID-19 across seven hospitals in the Netherlands. Raw-format 12-lead ECGs recorded after admission (<72 hours) were collected, manually assessed, and annotated using pre-defined ECG features. Using data from five out of seven centers (n = 634), two mortality prediction models were developed: (a) a logistic regression model using manually annotated ECG features, and (b) a pre-trained deep neural network (DNN) using the raw ECG waveforms. Data from two other centers (n = 248) were used for external validation. Performance of both prediction models was similar, with a mean area under the receiver operating curve of 0.69 [95%CI 0.55–0.82] for the logistic regression model and 0.71 [95%CI 0.59–0.81] for the DNN in the external validation cohort. After adjustment for age and sex, ventricular rate (OR 1.13 [95% CI 1.01–1.27] per 10 ms increase), right bundle branch block (3.26 [95% CI 1.15–9.50]), ST-depression (2.78 [95% CI 1.03–7.70]) and low QRS voltages (3.09 [95% CI 1.02-9.38]) remained as significant predictors for mortality. Conclusion This study shows that ECG-based prediction models at admission may be a valuable addition to the initial risk stratification in admitted COVID-19 patients. The DNN model showed similar performance to the logistic regression that needs time-consuming manual annotation. Several ECG features associated with mortality were identified. Figure 1:  Overview of methods, using and example case: (left) logistic regression and (right) deep learning. This specific case had a high probability of in-hospital mortality (above the threshold of 30%). Follow-up of this case showed that the patient had died during admission. Abstract Figure. Overview of ML methods used

scholarly journals High mortality rate of obstetric critically ill women in Rwanda and its predictability

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Alcade Rudakemwa ◽  
Amyl Lucille Cassidy ◽  
Théogène Twagirumugabe
Keyword(s):  
Prediction Models ◽  
Average Length ◽  
Mortality Prediction ◽  
Routine Practice ◽  
Icu Mortality ◽  
Resource Limited ◽  
Obstetric Haemorrhage ◽  
Tertiary Hospitals ◽  
Mortality Prediction Models ◽  
Qsofa Score

Abstract Background Reasons for admission to intensive care units (ICUs) for obstetric patients vary from one setting to another. Outcomes from ICU and prediction models are not well explored in Rwanda owing to lack of appropriate scores. This study aimed to assess reasons for admission and accuracy of prediction models for mortality of obstetric patients admitted to ICUs of two public tertiary hospitals in Rwanda. Methods We prospectively collected data from all obstetric patients admitted to the ICUs of the two public tertiary hospitals in Rwanda from March 2017 to February 2018 to identify reasons for admission, demographic and clinical characteristics, outcome including death and its predictability by both the Modified Early Obstetric Warning Score (MEOWS) and quick Sequential Organ Failure Assessment (qSOFA). We analysed the accuracy of mortality prediction models by MEOWS or qSOFA by using logistic regression adjusting for factors associated with mortality. Area under the Receiver Operating characteristic (AUROC) curves is used to show the predicting capacity for each individual tool. Results Obstetric patients (n = 94) represented 12.8 % of all 747 ICU admissions which is 1.8 % of all 4.999 admitted women for pregnancy or labor. Sepsis (n = 30; 31.9 %) and obstetric haemorrhage (n = 24; 25.5 %) were the two commonest reasons for ICU admission. Overall ICU mortality for obstetric patients was 54.3 % (n = 51) with average length of stay of 6.6 ± 7.525 days. MEOWS score was an independent predictor of mortality (adjusted (a)OR 1.25; 95 % CI 1.07–1.46) and so was qSOFA score (aOR 2.81; 95 % CI 1.25–6.30) with an adjusted AUROC of 0.773 (95 % CI 0.67–0.88) and 0.764 (95 % CI 0.65–0.87), indicating fair accuracy for ICU mortality prediction in these settings of both MEOWS and qSOFA scores. Conclusions Sepsis and obstetric haemorrhage were the commonest reasons for obstetric admissions to ICU in Rwanda. MEOWS and qSOFA scores could accurately predict ICU mortality of obstetric patients in resource-limited settings, but larger studies are needed before a recommendation for their use in routine practice in similar settings.

scholarly journals Development and validation of clinical prediction models for mortality, functional outcome and cognitive impairment after stroke: a study protocol

BMJ Open ◽  
2017 ◽  
Vol 7 (8) ◽  
pp. e014607 ◽  
Author(s):  
Marion Fahey ◽  
Anthony Rudd ◽  
Yannick Béjot ◽  
Charles Wolfe ◽  
Abdel Douiri
Keyword(s):  
Cognitive Impairment ◽  
Functional Outcome ◽  
Multilevel Models ◽  
Prediction Models ◽  
External Validation ◽  
Model Development ◽  
College Hospital ◽  
Stroke Registry ◽  
Temporal Validation ◽  
Stroke Register

IntroductionStroke is a leading cause of adult disability and death worldwide. The neurological impairments associated with stroke prevent patients from performing basic daily activities and have enormous impact on families and caregivers. Practical and accurate tools to assist in predicting outcome after stroke at patient level can provide significant aid for patient management. Furthermore, prediction models of this kind can be useful for clinical research, health economics, policymaking and clinical decision support.Methods2869 patients with first-ever stroke from South London Stroke Register (SLSR) (1995–2004) will be included in the development cohort. We will use information captured after baseline to construct multilevel models and a Cox proportional hazard model to predict cognitive impairment, functional outcome and mortality up to 5 years after stroke. Repeated random subsampling validation (Monte Carlo cross-validation) will be evaluated in model development. Data from participants recruited to the stroke register (2005–2014) will be used for temporal validation of the models. Data from participants recruited to the Dijon Stroke Register (1985–2015) will be used for external validation. Discrimination, calibration and clinical utility of the models will be presented.EthicsPatients, or for patients who cannot consent their relatives, gave written informed consent to participate in stroke-related studies within the SLSR. The SLSR design was approved by the ethics committees of Guy’s and St Thomas’ NHS Foundation Trust, Kings College Hospital, Queens Square and Westminster Hospitals (London). The Dijon Stroke Registry was approved by the Comité National des Registres and the InVS and has authorisation of the Commission Nationale de l’Informatique et des Libertés.

scholarly journals Machine Learning for Mortality Prediction in Pediatric Myocarditis

2021 ◽  
Vol 9 ◽  
Author(s):  
Fu-Sheng Chou ◽  
Laxmi V. Ghimire
Keyword(s):  
Machine Learning ◽  
Linear Regression ◽  
Regression Model ◽  
Current Knowledge ◽  
External Validation ◽  
Model Development ◽  
Kidney Injury ◽  
High Specificity ◽  
Mortality Prediction ◽  
Linear Regression Models

Background: Pediatric myocarditis is a rare disease. The etiologies are multiple. Mortality associated with the disease is 5–8%. Prognostic factors were identified with the use of national hospitalization databases. Applying these identified risk factors for mortality prediction has not been reported.Methods: We used the Kids' Inpatient Database for this project. We manually curated fourteen variables as predictors of mortality based on the current knowledge of the disease, and compared performance of mortality prediction between linear regression models and a machine learning (ML) model. For ML, the random forest algorithm was chosen because of the categorical nature of the variables. Based on variable importance scores, a reduced model was also developed for comparison.Results: We identified 4,144 patients from the database for randomization into the primary (for model development) and testing (for external validation) datasets. We found that the conventional logistic regression model had low sensitivity (~50%) despite high specificity (>95%) or overall accuracy. On the other hand, the ML model struck a good balance between sensitivity (89.9%) and specificity (85.8%). The reduced ML model with top five variables (mechanical ventilation, cardiac arrest, ECMO, acute kidney injury, ventricular fibrillation) were sufficient to approximate the prediction performance of the full model.Conclusions: The ML algorithm performs superiorly when compared to the linear regression model for mortality prediction in pediatric myocarditis in this retrospective dataset. Prospective studies are warranted to further validate the applicability of our model in clinical settings.

Sign in / Sign up

Export Citation Format

Share Document