scholarly journals Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study

PLoS ONE ◽  
2021 ◽  
Vol 16 (8) ◽  
pp. e0255748
Author(s):  
Kristin E. Wickstrøm ◽  
Valeria Vitelli ◽  
Ewan Carr ◽  
Aleksander R. Holten ◽  
Rebecca Bendayan ◽  
...  
Keyword(s):  
Cohort Study ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Severe Disease ◽  
External Validation ◽  
University Hospital ◽  
Good Discrimination ◽  
Prediction Of Mortality ◽  
Performance Variation ◽  
Observational Cohort

Background Prediction models should be externally validated to assess their performance before implementation. Several prediction models for coronavirus disease-19 (COVID-19) have been published. This observational cohort study aimed to validate published models of severity for hospitalized patients with COVID-19 using clinical and laboratory predictors. Methods Prediction models fitting relevant inclusion criteria were chosen for validation. The outcome was either mortality or a composite outcome of mortality and ICU admission (severe disease). 1295 patients admitted with symptoms of COVID-19 at Kings Cross Hospital (KCH) in London, United Kingdom, and 307 patients at Oslo University Hospital (OUH) in Oslo, Norway were included. The performance of the models was assessed in terms of discrimination and calibration. Results We identified two models for prediction of mortality (referred to as Xie and Zhang1) and two models for prediction of severe disease (Allenbach and Zhang2). The performance of the models was variable. For prediction of mortality Xie had good discrimination at OUH with an area under the receiver-operating characteristic (AUROC) 0.87 [95% confidence interval (CI) 0.79–0.95] and acceptable discrimination at KCH, AUROC 0.79 [0.76–0.82]. In prediction of severe disease, Allenbach had acceptable discrimination (OUH AUROC 0.81 [0.74–0.88] and KCH AUROC 0.72 [0.68–0.75]). The Zhang models had moderate to poor discrimination. Initial calibration was poor for all models but improved with recalibration. Conclusions The performance of the four prediction models was variable. The Xie model had the best discrimination for mortality, while the Allenbach model had acceptable results for prediction of severe disease.

scholarly journals Regional performance variation in external validation of four prediction models for severity of COVID-19 at hospital admission: An observational multi-centre cohort study

2021 ◽  
Author(s):  
Kristin Wickstrom ◽  
Valeria Vitelli ◽  
Ewan Carr ◽  
Aleksander Rygh Holten ◽  
Rebecca Bendayan ◽  
...  
Keyword(s):  
Cohort Study ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Severe Disease ◽  
External Validation ◽  
University Hospital ◽  
Good Discrimination ◽  
Prediction Of Mortality ◽  
Performance Variation ◽  
Observational Cohort

Background: Several prediction models for coronavirus disease-19 (COVID-19) have been published. Prediction models should be externally validated to assess their performance before implementation. This observational cohort study aimed to validate published models of severity for hospitalized patients with COVID-19 using clinical and laboratory predictors. Methods: Prediction models fitting relevant inclusion criteria were chosen for validation. The outcome was either mortality or a composite outcome of mortality and ICU admission (severe disease). 1295 patients admitted with symptoms of COVID-19 at Kings Cross Hospital (KCH) in London, United Kingdom, and 307 patients at Oslo University Hospital (OUH) in Oslo, Norway were included. The performance of the models was assessed in terms of discrimination and calibration. Results: We identified two models for prediction of mortality (referred to as Xie and Zhang1) and two models for prediction of severe disease (Allenbach and Zhang2). The performance of the models was variable. For prediction of mortality Xie had good discrimination at OUH with an area under the receiver-operating characteristic (AUROC) 0.87 [95 % confidence interval (CI) 0.79-0.95] and acceptable discrimination at KCH, AUROC 0.79 [0.76-0.82]. In prediction of severe disease, Allenbach had acceptable discrimination (OUH AUROC 0.81 [0.74-0.88] and KCH AUROC 0.72 [0.68-0.75]). The Zhang models had moderate to poor discrimination. Initial calibration was poor for all models but improved with recalibration. Conclusions: The performance of the four prediction models was variable. The Xie model had the best discrimination for mortality, while the Allenbach model had acceptable results for prediction of severe disease.

scholarly journals Development and validation of prediction models for mechanical ventilation, renal replacement therapy, and readmission in COVID-19 patients

2021 ◽  
Author(s):  
Victor Alfonso Rodriguez ◽  
Shreyas Bhave ◽  
Ruijun Chen ◽  
Chao Pang ◽  
George Hripcsak ◽  
...  
Keyword(s):  
At Risk ◽  
Mechanical Ventilation ◽  
Renal Replacement Therapy ◽  
Receiver Operating Characteristic ◽  
Replacement Therapy ◽  
Operating Characteristic ◽  
Prediction Models ◽  
External Validation ◽  
Electronic Health Record Data ◽  
Receiver Operating

Abstract Objective Coronavirus disease 2019 (COVID-19) patients are at risk for resource-intensive outcomes including mechanical ventilation (MV), renal replacement therapy (RRT), and readmission. Accurate outcome prognostication could facilitate hospital resource allocation. We develop and validate predictive models for each outcome using retrospective electronic health record data for COVID-19 patients treated between March 2 and May 6, 2020. Materials and Methods For each outcome, we trained 3 classes of prediction models using clinical data for a cohort of SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2)–positive patients (n = 2256). Cross-validation was used to select the best-performing models per the areas under the receiver-operating characteristic and precision-recall curves. Models were validated using a held-out cohort (n = 855). We measured each model’s calibration and evaluated feature importances to interpret model output. Results The predictive performance for our selected models on the held-out cohort was as follows: area under the receiver-operating characteristic curve—MV 0.743 (95% CI, 0.682-0.812), RRT 0.847 (95% CI, 0.772-0.936), readmission 0.871 (95% CI, 0.830-0.917); area under the precision-recall curve—MV 0.137 (95% CI, 0.047-0.175), RRT 0.325 (95% CI, 0.117-0.497), readmission 0.504 (95% CI, 0.388-0.604). Predictions were well calibrated, and the most important features within each model were consistent with clinical intuition. Discussion Our models produce performant, well-calibrated, and interpretable predictions for COVID-19 patients at risk for the target outcomes. They demonstrate the potential to accurately estimate outcome prognosis in resource-constrained care sites managing COVID-19 patients. Conclusions We develop and validate prognostic models targeting MV, RRT, and readmission for hospitalized COVID-19 patients which produce accurate, interpretable predictions. Additional external validation studies are needed to further verify the generalizability of our results.

scholarly journals Developing and validating COVID-19 adverse outcome risk prediction models from a bi-national European cohort of 5594 patients

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Espen Jimenez-Solem ◽  
Tonny S. Petersen ◽  
Casper Hansen ◽  
Christian Hansen ◽  
Christina Lioma ◽  
...  
Keyword(s):  
Prediction Models ◽  
Severe Disease ◽  
External Validation ◽  
Area Under The Curve ◽  
Predictive Performance ◽  
Mortality Prediction ◽  
Icu Admission ◽  
Risk Of Death ◽  
Online Risk ◽  
The United Kingdom

AbstractPatients with severe COVID-19 have overwhelmed healthcare systems worldwide. We hypothesized that machine learning (ML) models could be used to predict risks at different stages of management and thereby provide insights into drivers and prognostic markers of disease progression and death. From a cohort of approx. 2.6 million citizens in Denmark, SARS-CoV-2 PCR tests were performed on subjects suspected for COVID-19 disease; 3944 cases had at least one positive test and were subjected to further analysis. SARS-CoV-2 positive cases from the United Kingdom Biobank was used for external validation. The ML models predicted the risk of death (Receiver Operation Characteristics—Area Under the Curve, ROC-AUC) of 0.906 at diagnosis, 0.818, at hospital admission and 0.721 at Intensive Care Unit (ICU) admission. Similar metrics were achieved for predicted risks of hospital and ICU admission and use of mechanical ventilation. Common risk factors, included age, body mass index and hypertension, although the top risk features shifted towards markers of shock and organ dysfunction in ICU patients. The external validation indicated fair predictive performance for mortality prediction, but suboptimal performance for predicting ICU admission. ML may be used to identify drivers of progression to more severe disease and for prognostication patients in patients with COVID-19. We provide access to an online risk calculator based on these findings.

The Use of Prognostic Prediction Models for Mortality or Clinical Deterioration among Hospitalized and Non-hospitalized Adults with COVID-19: A Systematic Review

2021 ◽  
Author(s):  
Patricia Pauline M. Remalante-Rayco ◽  
Evelyn Osio-Salido
Keyword(s):  
Systematic Review ◽  
Emergency Department ◽  
Early Warning ◽  
Prediction Models ◽  
External Validation ◽  
Risk Of Bias ◽  
Clinical Deterioration ◽  
Early Warning Score ◽  
Good Prediction ◽  
Prediction Of Mortality

Objective. To assess the performance of prognostic models in predicting mortality or clinical deterioration among patients with COVID-19, both hospitalized and non-hospitalized Methods. We conducted a systematic review of the literature until March 8, 2021. We included models for the prediction of mortality or clinical deterioration in COVID-19 with external validation. We used the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and the GRADEpro Guideline Development Tool (GDT) to assess the evidence obtained. Results. We reviewed 33 cohort studies. Two studies had a low risk of bias, four unclear risks, and 27 with a high risk of bias due to participant selection and analysis. For the outcome of mortality, the QCOVID model had excellent prediction with high certainty of evidence but was specific for use in England. The COVID Outcome Prediction in the Emergency Department (COPE) model, the 4C Mortality Score, the Age, BUN, number of comorbidities, CRP, SpO2/FiO2 ratio, platelet count, heart rate (ABC2-SPH) risk score, the Confusion Urea Respiration Blood Pressure (CURB-65) severity score, the Rapid Emergency Medicine Score (REMS), and the Risk Stratification in the Emergency Department in Acutely Ill Older Patients (RISE UP) score had fair to good prediction of death among inpatients, while the quick Sepsis-related Organ Failure Assessment (qSOFA) score had poor to fair prediction. The certainty of evidence for these models was very low to low. For the outcome of clinical deterioration, the 4C Deterioration Score had fair prediction, the National Early Warning Score 2 (NEWS2) score poor to good, and the Modified Early Warning Score (MEWS) had poor prediction. The certainty of evidence for these three models was also very low to low. None of these models had been validated in the Philippine setting. Conclusion. The QCOVID, COPE, ABC2-SPH, 4C, CURB-65, REMS, RISE-UP models for prediction of mortality and the 4C Deterioration and NEWS2 models for prediction of clinical deterioration are potentially useful but need to be validated among patients with COVID-19 of varying severity in the Philippine setting.

scholarly journals External validation of EPIC’s Risk of Unplanned Readmission model, the LACE+ index and SQLape as predictors of unplanned hospital readmissions: A monocentric, retrospective, diagnostic cohort study in Switzerland

PLoS ONE ◽  
2021 ◽  
Vol 16 (11) ◽  
pp. e0258338
Author(s):  
Aljoscha Benjamin Hwang ◽  
Guido Schuepfer ◽  
Mario Pietrini ◽  
Stefan Boes
Keyword(s):  
Cohort Study ◽  
Quality Of Care ◽  
Prediction Models ◽  
Hospital Readmissions ◽  
External Validation ◽  
Superior Performance ◽  
Study Endpoint ◽  
Unplanned Readmission ◽  
S Model

Introduction Readmissions after an acute care hospitalization are relatively common, costly to the health care system, and are associated with significant burden for patients. As one way to reduce costs and simultaneously improve quality of care, hospital readmissions receive increasing interest from policy makers. It is only relatively recently that strategies were developed with the specific aim of reducing unplanned readmissions using prediction models to identify patients at risk. EPIC’s Risk of Unplanned Readmission model promises superior performance. However, it has only been validated for the US setting. Therefore, the main objective of this study is to externally validate the EPIC’s Risk of Unplanned Readmission model and to compare it to the internationally, widely used LACE+ index, and the SQLAPE® tool, a Swiss national quality of care indicator. Methods A monocentric, retrospective, diagnostic cohort study was conducted. The study included inpatients, who were discharged between the 1st of January 2018 and the 31st of December 2019 from the Lucerne Cantonal Hospital, a tertiary-care provider in Central Switzerland. The study endpoint was an unplanned 30-day readmission. Models were replicated using the original intercept and beta coefficients as reported. Otherwise, score generator provided by the developers were used. For external validation, discrimination of the scores under investigation were assessed by calculating the area under the receiver operating characteristics curves (AUC). Calibration was assessed with the Hosmer-Lemeshow X2 goodness-of-fit test This report adheres to the TRIPOD statement for reporting of prediction models. Results At least 23,116 records were included. For discrimination, the EPIC´s prediction model, the LACE+ index and the SQLape® had AUCs of 0.692 (95% CI 0.676–0.708), 0.703 (95% CI 0.687–0.719) and 0.705 (95% CI 0.690–0.720). The Hosmer-Lemeshow X2 tests had values of p<0.001. Conclusion In summary, the EPIC´s model showed less favorable performance than its comparators. It may be assumed with caution that the EPIC´s model complexity has hampered its wide generalizability—model updating is warranted.

scholarly journals DEVELOPING AND VALIDATING COVID-19 ADVERSE OUTCOME RISK PREDICTION MODELS FROM A BI-NATIONAL EUROPEAN COHORT OF 5594 PATIENTS

2020 ◽  
Author(s):  
Espen Jimenez-Solem ◽  
Tonny S Petersen ◽  
Casper Hansen ◽  
Christian Hansen ◽  
Christina Lioma ◽  
...  
Keyword(s):  
Hospital Admission ◽  
Organ Dysfunction ◽  
Prediction Models ◽  
Severe Disease ◽  
External Validation ◽  
Manuscript Preparation ◽  
Design Data ◽  
Prognostic Features ◽  
Icu Admission ◽  
Risk Of Death

ABSTRACTBackgroundPatients with severe COVID-19 have overwhelmed healthcare systems worldwide. We hypothesized that Machine Learning (ML) models could be used to predict risks at different stages of management (at diagnosis, hospital admission and ICU admission) and thereby provide insights into drivers and prognostic markers of disease progression and death.MethodsFrom a cohort of approx. 2.6 million citizens in the two regions of Denmark, SARS-CoV-2 PCR tests were performed on subjects suspected for COVID-19 disease; 3944 cases had at least one positive test and were subjected to further analysis. A cohort of SARS- CoV-2 positive cases from the United Kingdom Biobank was used for external validation.FindingsThe ML models predicted the risk of death (Receiver Operation Characteristics – Area Under the Curve, ROC-AUC) of 0.904 at diagnosis, 0.818, at hospital admission and 0.723 at Intensive Care Unit (ICU) admission. Similar metrics were achieved for predicted risks of hospital and ICU admission and use of mechanical ventilation. We identified some common risk factors, including age, body mass index (BMI) and hypertension as driving factors, although the top risk features shifted towards markers of shock and organ dysfunction in ICU patients. The external validation indicated fair predictive performance for mortality prediction, but suboptimal performance for predicting ICU admission.InterpretationML may be used to identify drivers of progression to more severe disease and for prognostication patients in patients with COVID-19. Prognostic features included age, BMI and hypertension, although markers of shock and organ dysfunction became more important in more severe cases.We provide access to an online risk calculator based on these findings.FundingThe study was funded by grants from the Novo Nordisk Foundation to MS (#NNF20SA0062879 and #NNF19OC0055183) and MN (#NNF20SA0062879). The foundation took no part in project design, data handling and manuscript preparation.

scholarly journals Title: Risk factors for severe disease in patients admitted with COVID-19 to a hospital in London, England: a retrospective cohort study

2020 ◽  
Author(s):  
Jack W Goodall ◽  
Thomas A N Reed ◽  
Maddalena Ardissino ◽  
Paul Bassett ◽  
Ashley M Whittington ◽  
...  
Keyword(s):  
Risk Factors ◽  
Cohort Study ◽  
Proportional Hazards ◽  
Severe Disease ◽  
Cox Proportional Hazards ◽  
Hospitalized Patients ◽  
Reactive Protein ◽  
Global Pandemic ◽  
Northwick Park Hospital ◽  
Observational Cohort

COVID-19 has caused a major global pandemic and necessitated unprecedented public health restrictions in almost every country. Understanding risk factors for severe disease in hospitalized patients is critical as the pandemic progresses. This observational cohort study aimed to characterize the independent associations between the clinical outcomes of hospitalized patients and their demographics, comorbidities, blood tests and bedside observations. All patients admitted to Northwick Park Hospital, London, United Kingdom between 12 March and 15 April 2020 with COVID-19 were retrospectively identified. The primary outcome was death. Associations were explored using Cox proportional hazards modelling. The study included 981 patients. The mortality rate was 36.0%. Age (adjusted hazard ratio (aHR) 1.53), respiratory disease (aHR 1.37), immunosuppression (aHR 2.23), respiratory rate (aHR 1.28), hypoxia (aHR 1.36), Glasgow Coma Score <15 (aHR 1.92), urea (aHR 2.67), alkaline phosphatase (aHR 2.53), C-reactive protein (aHR 1.15), lactate (aHR 2.67), platelet count (aHR 0.77) and infiltrates on chest radiograph (aHR 1.89) were all associated with mortality. These important data will aid clinical risk stratification and provide direction for further research.

scholarly journals Risk factors for severe disease in patients admitted with COVID-19 to a hospital in London, England: a retrospective cohort study

2020 ◽  
Vol 148 ◽  
Author(s):  
J. W. Goodall ◽  
T. A. N. Reed ◽  
M. Ardissino ◽  
P. Bassett ◽  
A. M. Whittington ◽  
...  
Keyword(s):  
Risk Factors ◽  
Cohort Study ◽  
Proportional Hazards ◽  
Severe Disease ◽  
Cox Proportional Hazards ◽  
Reactive Protein ◽  
Global Pandemic ◽  
Northwick Park Hospital ◽  
Hospitalised Patients ◽  
Observational Cohort

Abstract COVID-19 has caused a major global pandemic and necessitated unprecedented public health restrictions in almost every country. Understanding risk factors for severe disease in hospitalised patients is critical as the pandemic progresses. This observational cohort study aimed to characterise the independent associations between the clinical outcomes of hospitalised patients and their demographics, comorbidities, blood tests and bedside observations. All patients admitted to Northwick Park Hospital, London, UK between 12 March and 15 April 2020 with COVID-19 were retrospectively identified. The primary outcome was death. Associations were explored using Cox proportional hazards modelling. The study included 981 patients. The mortality rate was 36.0%. Age (adjusted hazard ratio (aHR) 1.53), respiratory disease (aHR 1.37), immunosuppression (aHR 2.23), respiratory rate (aHR 1.28), hypoxia (aHR 1.36), Glasgow Coma Scale <15 (aHR 1.92), urea (aHR 2.67), alkaline phosphatase (aHR 2.53), C-reactive protein (aHR 1.15), lactate (aHR 2.67), platelet count (aHR 0.77) and infiltrates on chest radiograph (aHR 1.89) were all associated with mortality. These important data will aid clinical risk stratification and provide direction for further research.

Sign in / Sign up

Export Citation Format

Share Document