scholarly journals Development and validation of prediction models for mechanical ventilation, renal replacement therapy, and readmission in COVID-19 patients

2021 ◽  
Author(s):  
Victor Alfonso Rodriguez ◽  
Shreyas Bhave ◽  
Ruijun Chen ◽  
Chao Pang ◽  
George Hripcsak ◽  
...  
Keyword(s):  
At Risk ◽  
Mechanical Ventilation ◽  
Renal Replacement Therapy ◽  
Receiver Operating Characteristic ◽  
Replacement Therapy ◽  
Operating Characteristic ◽  
Prediction Models ◽  
External Validation ◽  
Electronic Health Record Data ◽  
Receiver Operating

Abstract Objective Coronavirus disease 2019 (COVID-19) patients are at risk for resource-intensive outcomes including mechanical ventilation (MV), renal replacement therapy (RRT), and readmission. Accurate outcome prognostication could facilitate hospital resource allocation. We develop and validate predictive models for each outcome using retrospective electronic health record data for COVID-19 patients treated between March 2 and May 6, 2020. Materials and Methods For each outcome, we trained 3 classes of prediction models using clinical data for a cohort of SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2)–positive patients (n = 2256). Cross-validation was used to select the best-performing models per the areas under the receiver-operating characteristic and precision-recall curves. Models were validated using a held-out cohort (n = 855). We measured each model’s calibration and evaluated feature importances to interpret model output. Results The predictive performance for our selected models on the held-out cohort was as follows: area under the receiver-operating characteristic curve—MV 0.743 (95% CI, 0.682-0.812), RRT 0.847 (95% CI, 0.772-0.936), readmission 0.871 (95% CI, 0.830-0.917); area under the precision-recall curve—MV 0.137 (95% CI, 0.047-0.175), RRT 0.325 (95% CI, 0.117-0.497), readmission 0.504 (95% CI, 0.388-0.604). Predictions were well calibrated, and the most important features within each model were consistent with clinical intuition. Discussion Our models produce performant, well-calibrated, and interpretable predictions for COVID-19 patients at risk for the target outcomes. They demonstrate the potential to accurately estimate outcome prognosis in resource-constrained care sites managing COVID-19 patients. Conclusions We develop and validate prognostic models targeting MV, RRT, and readmission for hospitalized COVID-19 patients which produce accurate, interpretable predictions. Additional external validation studies are needed to further verify the generalizability of our results.

scholarly journals External validation of six existing multivariable clinical prediction models for short-term mortality in patients undergoing lung resection

2020 ◽  
Author(s):  
Marcus Taylor ◽  
Bartłomiej Szafron ◽  
Glen P Martin ◽  
Udo Abah ◽  
Matthew Smith ◽  
...  
Keyword(s):  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
Lung Resection ◽  
External Validation ◽  
Clinical Prediction ◽  
National Guidelines ◽  
Clinical Prediction Models ◽  
Receiver Operating

Abstract OBJECTIVES National guidelines advocate the use of clinical prediction models to estimate perioperative mortality for patients undergoing lung resection. Several models have been developed that may potentially be useful but contemporary external validation studies are lacking. The aim of this study was to validate existing models in a multicentre patient cohort. METHODS The Thoracoscore, Modified Thoracoscore, Eurolung, Modified Eurolung, European Society Objective Score and Brunelli models were validated using a database of 6600 patients who underwent lung resection between 2012 and 2018. Models were validated for in-hospital or 30-day mortality (depending on intended outcome of each model) and also for 90-day mortality. Model calibration (calibration intercept, calibration slope, observed to expected ratio and calibration plots) and discrimination (area under receiver operating characteristic curve) were assessed as measures of model performance. RESULTS Mean age was 66.8 years (±10.9 years) and 49.7% (n = 3281) of patients were male. In-hospital, 30-day, perioperative (in-hospital or 30-day) and 90-day mortality were 1.5% (n = 99), 1.4% (n = 93), 1.8% (n = 121) and 3.1% (n = 204), respectively. Model area under the receiver operating characteristic curves ranged from 0.67 to 0.73. Calibration was inadequate in five models and mortality was significantly overestimated in five models. No model was able to adequately predict 90-day mortality. CONCLUSIONS Five of the validated models were poorly calibrated and had inadequate discriminatory ability. The modified Eurolung model demonstrated adequate statistical performance but lacked clinical validity. Development of accurate models that can be used to estimate the contemporary risk of lung resection is required.

A Predictive Model of Intrinsic Factors Associated with Long-Stay Nursing Home Care After Hospitalization

2020 ◽  
pp. 105477382098527
Author(s):  
Jane Flanagan ◽  
Marie Boltz ◽  
Ming Ji
Keyword(s):  
At Risk ◽  
Logistic Regression ◽  
Receiver Operating Characteristic ◽  
Predictive Model ◽  
Operating Characteristic ◽  
Resource Planning ◽  
Term Care ◽  
Intrinsic Factors ◽  
Patients At Risk ◽  
Receiver Operating

We aimed to build a predictive model with intrinsic factors measured upon admission to skilled nursing facilities (SNFs) post-acute care (PAC) to identify older adults transferred from SNFs to long-term care (LTC) instead of home. We analyzed data from Massachusetts in 23,662 persons admitted to SNFs from PAC in 2013. Explanatory logistic regression analysis identified single “intrinsic predictors” related to LTC placement. To assess overfitting, the logistic regression predictive model was cross-validated and evaluated by its receiver operating characteristic (ROC) curve. A 12-variable predictive model with “intrinsic predictors” demonstrated both high in-sample and out-of-sample predictive accuracy in the receiver operating characteristic ROC and area under the ROC among patients at risk of LTC placement. This predictive model may be used for early identification of patients at risk for LTC after hospitalization in order to support targeted rehabilitative approaches and resource planning.

scholarly journals External Validation of Five Scores to Predict Stroke-Associated Pneumonia and the Role of Selected Blood Biomarkers

Stroke ◽  
2021 ◽  
Vol 52 (1) ◽  
pp. 325-330
Author(s):  
Benjamin Hotter ◽  
Sarah Hoffmann ◽  
Lena Ulm ◽  
Christian Meisel ◽  
Alejandro Bustamante ◽  
...  
Keyword(s):  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Literature Search ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
Data Set ◽  
Biomarker Analysis ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Background and Purpose: Several clinical scoring systems as well as biomarkers have been proposed to predict stroke-associated pneumonia (SAP). We aimed to externally and competitively validate SAP scores and hypothesized that 5 selected biomarkers would improve performance of these scores. Methods: We pooled the clinical data of 2 acute stroke studies with identical data assessment: STRAWINSKI and PREDICT. Biomarkers (ultrasensitive procalcitonin; mid-regional proadrenomedullin; mid-regional proatrionatriuretic peptide; ultrasensitive copeptin; C-terminal proendothelin) were measured from hospital admission serum samples. A literature search was performed to identify SAP prediction scores. We then calculated multivariate regression models with the individual scores and the biomarkers. Areas under receiver operating characteristic curves were used to compare discrimination of these scores and models. Results: The combined cohort consisted of 683 cases, of which 573 had available backup samples to perform the biomarker analysis. Literature search identified 9 SAP prediction scores. Our data set enabled us to calculate 5 of these scores. The scores had area under receiver operating characteristic curve of 0.543 to 0.651 for physician determined SAP, 0.574 to 0.685 for probable and 0.689 to 0.811 for definite SAP according to Pneumonia in Stroke Consensus group criteria. Multivariate models of the scores with biomarkers improved virtually all predictions, but mostly in the range of an area under receiver operating characteristic curve delta of 0.05. Conclusions: All SAP prediction scores identified patients who would develop SAP with fair to strong capabilities, with better discrimination when stricter criteria for SAP diagnosis were applied. The selected biomarkers provided only limited added predictive value, currently not warranting addition of these markers to prediction models. Registration: URL: https://www.clinicaltrials.gov . Unique identifier: NCT01264549 and NCT01079728.

scholarly journals Epidemiological and Clinical Predictors of COVID-19

2020 ◽  
Vol 71 (15) ◽  
pp. 786-792 ◽  
Author(s):  
Yinxiaohe Sun ◽  
Vanessa Koh ◽  
Kalisvar Marimuthu ◽  
Oon Tek Ng ◽  
Barnaby Young ◽  
...  
Keyword(s):  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Cross Validation ◽  
Prediction Models ◽  
Tertiary Care ◽  
Laboratory Data ◽  
Care Hospital ◽  
Laboratory Test Results ◽  
Leave One Out ◽  
Receiver Operating

Abstract Background Rapid identification of COVID-19 cases, which is crucial to outbreak containment efforts, is challenging due to the lack of pathognomonic symptoms and in settings with limited capacity for specialized nucleic acid–based reverse transcription polymerase chain reaction (PCR) testing. Methods This retrospective case-control study involves subjects (7–98 years) presenting at the designated national outbreak screening center and tertiary care hospital in Singapore for SARS-CoV-2 testing from 26 January to 16 February 2020. COVID-19 status was confirmed by PCR testing of sputum, nasopharyngeal swabs, or throat swabs. Demographic, clinical, laboratory, and exposure-risk variables ascertainable at presentation were analyzed to develop an algorithm for estimating the risk of COVID-19. Model development used Akaike’s information criterion in a stepwise fashion to build logistic regression models, which were then translated into prediction scores. Performance was measured using receiver operating characteristic curves, adjusting for overconfidence using leave-one-out cross-validation. Results The study population included 788 subjects, of whom 54 (6.9%) were SARS-CoV-2 positive and 734 (93.1%) were SARS-CoV-2 negative. The median age was 34 years, and 407 (51.7%) were female. Using leave-one-out cross-validation, all the models incorporating clinical tests (models 1, 2, and 3) performed well with areas under the receiver operating characteristic curve (AUCs) of 0.91, 0.88, and 0.88, respectively. In comparison, model 4 had an AUC of 0.65. Conclusions Rapidly ascertainable clinical and laboratory data could identify individuals at high risk of COVID-19 and enable prioritization of PCR testing and containment efforts. Basic laboratory test results were crucial to prediction models.

scholarly journals Hypercapnic Ventilatory Response in the Weaning of Patients with Prolonged Mechanical Ventilation

2017 ◽  
Vol 2017 ◽  
pp. 1-7 ◽  
Author(s):  
Chung-Shu Lee ◽  
Ning-Hung Chen ◽  
Li-Pang Chuang ◽  
Chih-Hao Chang ◽  
Li-Fu Li ◽  
...  
Keyword(s):  
Mechanical Ventilation ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Ventilatory Response ◽  
Prolonged Mechanical Ventilation ◽  
Ventilatory Responses ◽  
Hypercapnic Ventilatory Response ◽  
Weaning Failure ◽  
Successful Weaning ◽  
Receiver Operating

Objective. To investigate whether hypercapnic ventilatory response (defined as the ratio of the change in minute ventilation [ΔV˙E] to the change in end-tidal partial pressure of carbon dioxide [ΔPETCO2]) is a predictor of successful weaning in patients with prolonged mechanical ventilation (PMV) and to determine a reference value for clinical use.Methods. A hypercapnic challenge test was performed on 32 PMV subjects (average age: 74.3 years ± 14.9 years). The subjects were divided into two groups (i.e., weaning successes and weaning failures) and their hypercapnic ventilatory responses were compared.Results. PMV subjects had an overall weaning rate of 68.8%. The weaning-success and weaning-failure groups had hypercapnic ventilatory responses (ΔV˙E/ΔPETCO2) of0.40±0.16and0.28±0.12 L/min/mmHg, respectively (P=.036). The area under the receiver operating characteristic curve was 0.716 of the hypercapnic ventilatory response, and the practical hypercapnic ventilatory response cut-off point for successful weaning was 0.265 with 86.4% sensitivity and 50% specificity.Conclusions. PMV subjects who failed weaning had a lower hypercapnic ventilatory response than successfully weaned subjects. However, the prediction capacity of this test, assessed by the area under the receiver operating characteristic (ROC) curve, poorly predicted weaning outcome.

Applying an Evidence-Based Assessment Model to Identify Students at Risk for Perceived Academic Problems following Concussion

2016 ◽  
Vol 22 (10) ◽  
pp. 1038-1049 ◽  
Author(s):  
Danielle M. Ransom ◽  
Alison R. Burns ◽  
Eric A. Youngstrom ◽  
Christopher G. Vaughan ◽  
Maegan D. Sady ◽  
...  
Keyword(s):  
At Risk ◽  
Receiver Operating Characteristic ◽  
Symptom Severity ◽  
Operating Characteristic ◽  
Executive Dysfunction ◽  
Evidence Based ◽  
Academic Problems ◽  
Post Injury ◽  
Students At Risk ◽  
Receiver Operating

AbstractObjectives: The aim of this study was to demonstrate the utility of an evidence-based assessment (EBA) model to establish a multimodal set of tools for identifying students at risk for perceived post-injury academic problems. Methods: Participants included 142 students diagnosed with concussion (age: M=14.95; SD=1.80; 59% male), evaluated within 4 weeks of injury (median=16 days). Demographics, pre-injury history, self- and parent-report measures assessing symptom severity and executive functions, and cognitive test performance were examined as predictors of self-reported post-injury academic problems. Results: Latent class analysis categorized participants into “high” (44%) and “low” (56%) levels of self-reported academic problems. Receiver operating characteristic analyses revealed significant discriminative validity for self- and parent-reported symptom severity and executive dysfunction and self-reported exertional response for identifying students reporting low versus high academic problems. Parent-reported symptom ratings [area under the receiver operating characteristic curve (AUC)=.79] and executive dysfunction (AUC=.74), and self-reported ratings of executive dysfunction (AUC=.84), symptoms (AUC=.80), and exertional response (AUC=.70) each classified students significantly better than chance (ps<.001). Hierarchical logistic regression indicated that, of the above, self-reported symptoms and executive dysfunction accounted for the most variance in the prediction of self-reported academic problems. Conclusions: Post-concussion symptom severity and executive dysfunction significantly predict perceived post-injury academic problems. EBA modeling identified the strongest set of predictors of academic challenges, offering an important perspective in the management of concussion by applying traditional strengths of neuropsychological assessment to clinical decision making. (JINS, 2016, 22, 1038–1049)

scholarly journals Predicting Risk of Postoperative Lung Injury in High-risk Surgical Patients

2014 ◽  
Vol 120 (5) ◽  
pp. 1168-1181 ◽  
Author(s):  
Daryl J. Kor ◽  
Ravi K. Lingineni ◽  
Ognjen Gajic ◽  
Pauline K. Park ◽  
James M. Blum ◽  
...  
Keyword(s):  
At Risk ◽  
High Risk ◽  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Surgical Patients ◽  
Patients At Risk ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Abstract Background: Acute respiratory distress syndrome (ARDS) remains a serious postoperative complication. Although ARDS prevention is a priority, the inability to identify patients at risk for ARDS remains a barrier to progress. The authors tested and refined the previously reported surgical lung injury prediction (SLIP) model in a multicenter cohort of at-risk surgical patients. Methods: This is a secondary analysis of a multicenter, prospective cohort investigation evaluating high-risk patients undergoing surgery. Preoperative ARDS risk factors and risk modifiers were evaluated for inclusion in a parsimonious risk-prediction model. Multiple imputation and domain analysis were used to facilitate development of a refined model, designated SLIP-2. Area under the receiver operating characteristic curve and the Hosmer–Lemeshow goodness-of-fit test were used to assess model performance. Results: Among 1,562 at-risk patients, ARDS developed in 117 (7.5%). Nine independent predictors of ARDS were identified: sepsis, high-risk aortic vascular surgery, high-risk cardiac surgery, emergency surgery, cirrhosis, admission location other than home, increased respiratory rate (20 to 29 and ≥30 breaths/min), Fio2 greater than 35%, and Spo2 less than 95%. The original SLIP score performed poorly in this heterogeneous cohort with baseline risk factors for ARDS (area under the receiver operating characteristic curve [95% CI], 0.56 [0.50 to 0.62]). In contrast, SLIP-2 score performed well (area under the receiver operating characteristic curve [95% CI], 0.84 [0.81 to 0.88]). Internal validation indicated similar discrimination, with an area under the receiver operating characteristic curve of 0.84. Conclusions: In this multicenter cohort of patients at risk for ARDS, the SLIP-2 score outperformed the original SLIP score. If validated in an independent sample, this tool may help identify surgical patients at high risk for ARDS.

scholarly journals External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature

2021 ◽  
Author(s):  
Benjamin S. Wessler ◽  
Jason Nelson ◽  
Jinny G. Park ◽  
Hannah McGinnes ◽  
Gaurav Gulati ◽  
...  
Keyword(s):  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Prediction Models ◽  
Characteristic Curve ◽  
External Validation ◽  
Clinical Prediction ◽  
Clinical Prediction Models ◽  
Operating Characteristic Curve ◽  
Performance Heterogeneity

Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results: We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0–94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th–75th percentile [interquartile range (IQR)], 0.66–0.79), representing a median percent decrease in discrimination of −11.1% (IQR, −32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR, −13.2 to 3.1) for closely related validations (n=123), −9.0 (IQR, −27.6 to 3.9) for related validations (n=862), and −17.2% (IQR, −42.3 to 0) for distantly related validations (n=717; P <0.001). Conclusions: Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.

scholarly journals Comparison of Intensive Care Outcome Prediction Models Based on Admission Scores With those Based on 24-Hour Data

2008 ◽  
Vol 36 (6) ◽  
pp. 845-849 ◽  
Author(s):  
G. J. Duke ◽  
M. Piercy ◽  
D. Digiantomasso ◽  
J. V. Green
Keyword(s):  
Intensive Care ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Outcome Prediction ◽  
Prediction Models ◽  
Apache Ii ◽  
Standardised Mortality Ratio ◽  
Receiver Operating Characteristic Plot ◽  
Mortality Ratio ◽  
Receiver Operating

We compared the performance of six outcome prediction models - three based on 24-hour data and three based on admission-only data - in a metropolitan university-affiliated teaching hospital with a 10-bed intensive care unit. The Acute Physiology and Chronic Health Evaluation models, version II (APACHE II) and version III-J, and the Simplified Acute Physiology Score version II (SAPS II) are based on 24-hour data and were compared with the Mortality Prediction Model version II and the SAPS version III using international and Australian coefficients (SAPS IIIA). Data were collected prospectively according to the standard methodologies for each model. Calibration and discrimination for each model were assessed by the standardised mortality ratio, area under the receiver operating characteristic plot and Hosmer-Lemeshow contingency tables and chi-squared statistics (C10 and H10). Predetermined criteria were area under the receiver operating characteristic plot >0.8, standardised mortality ratio 95% confidence interval includes 1.0, and C10 and H10 P values >0.05. Between October 1, 2005 and December 31, 2007, 1843 consecutive admissions were screened and after the standard exclusions, 1741 were included in the analysis. The SAAPS II and SAPS IIIA models fulfilled and the APACHE II model failed all criteria. The other models satisfied the discrimination criterion but significantly over-predicted mortality risk and require recalibration. Outcome prediction models based on admission-only data compared favourably to those based on 24-hour data.

Sign in / Sign up

Export Citation Format

Share Document