Derivation and Validation of Clinical Prediction Rule for COVID-19 Mortality in Ontario, Canada

AbstractBackgroundSARS-CoV-2 is currently causing a high mortality global pandemic. However, the clinical spectrum of disease caused by this virus is broad, ranging from asymptomatic infection to cytokine storm with organ failure and death. Risk stratification of individuals with COVID-19 would be desirable for management, prioritization for trial enrollment, and risk stratification. We sought to develop a prediction rule for mortality due to COVID-19 in individuals with diagnosed infection in Ontario, Canada.MethodsData from Ontario’s provincial iPHIS system were extracted for the period from January 23 to May 15, 2020. Both logistic regression-based prediction rules, and a rule derived using a Cox proportional hazards model, were developed in half the study and validated in remaining patients. Sensitivity analyses were performed with varying approaches to missing data.Results21,922 COVID-19 cases were reported. Individuals assigned to the derivation and validation sets were broadly similar. Age and comorbidities (notably diabetes, renal disease and immune compromise) were strong predictors of mortality. Four point-based prediction rules were derived (base case, smoking excluded as a predictor, long-term care excluded as a predictor, and Cox model based). All rules displayed excellent discrimination (AUC for all rules > 0.92) and calibration (both by graphical inspection and P > 0.50 by Hosmer-Lemeshow test) in the derivation set. All rules performed well in the validation set and were robust to random replacement of missing variables, and to the assumption that missing variables indicated absence of the comorbidity or characteristic in question.ConclusionsWe were able to use a public health case-management data system to derive and internally validate four accurate, well-calibrated and robust clinical prediction rules for COVID-19 mortality in Ontario, Canada. While these rules need external validation, they may be a useful tool for clinical management, risk stratification, and clinical trials.

Download Full-text

Derivation and Validation of Clinical Prediction Rules for COVID-19 Mortality in Ontario, Canada

Open Forum Infectious Diseases ◽

10.1093/ofid/ofaa463 ◽

2020 ◽

Vol 7 (11) ◽

Cited By ~ 2

Author(s):

David N Fisman ◽

Amy L Greer ◽

Michael Hillmer ◽

R Tuite

Keyword(s):

Risk Stratification ◽

Cox Model ◽

External Validation ◽

Sensitivity Analyses ◽

Base Case ◽

Clinical Prediction ◽

Clinical Prediction Rules ◽

Term Care ◽

Prediction Rules ◽

Validation Set

Abstract Background Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is currently causing a high-mortality global pandemic. The clinical spectrum of disease caused by this virus is broad, ranging from asymptomatic infection to organ failure and death. Risk stratification of individuals with coronavirus disease 2019 (COVID-19) is desirable for management, and prioritization for trial enrollment. We developed a prediction rule for COVID-19 mortality in a population-based cohort in Ontario, Canada. Methods Data from Ontario’s provincial iPHIS system were extracted for the period from January 23 to May 15, 2020. Logistic regression–based prediction rules and a rule derived using a Cox proportional hazards model were developed and validated using split-halves validation. Sensitivity analyses were performed, with varying approaches to missing data. Results Of 21 922 COVID-19 cases, 1734 with complete data were included in the derivation set; 1796 were included in the validation set. Age and comorbidities (notably diabetes, renal disease, and immune compromise) were strong predictors of mortality. Four point-based prediction rules were derived (base case, smoking excluded, long-term care excluded, and Cox model–based). All displayed excellent discrimination (area under the curve for all rules > 0.92) and calibration (P > .50 by Hosmer-Lemeshow test) in the derivation set. All performed well in the validation set and were robust to varying approaches to replacement of missing variables. Conclusions We used a public health case management data system to build and validate 4 accurate, well-calibrated, robust clinical prediction rules for COVID-19 mortality in Ontario, Canada. While these rules need external validation, they may be useful tools for management, risk stratification, and clinical trials.

Download Full-text

External Validation of the PediBIRN Clinical Prediction Rule for Abusive Head Trauma

PEDIATRICS ◽

10.1542/peds.2017-3674 ◽

2018 ◽

Vol 141 (5) ◽

pp. e20173674 ◽

Cited By ~ 6

Author(s):

Helena Pfeiffer ◽

Anne Smith ◽

Alison Mary Kemp ◽

Laura Elizabeth Cowley ◽

John A. Cheek ◽

...

Keyword(s):

Head Trauma ◽

External Validation ◽

Prediction Rule ◽

Clinical Prediction Rule ◽

Abusive Head Trauma ◽

Clinical Prediction

Download Full-text

1493. Creation and External Validation of a Clinical Prediction Rule for Diarrheal Etiology Using Natural Language Processing

Open Forum Infectious Diseases ◽

10.1093/ofid/ofz360.1357 ◽

2019 ◽

Vol 6 (Supplement_2) ◽

pp. S544-S544

Author(s):

Joel Iverson Howard ◽

Joni Aoki ◽

Jeffrey Ferraro ◽

Ben Haaland ◽

Andrew Pavia ◽

...

Keyword(s):

Natural Language Processing ◽

Natural Language ◽

Language Processing ◽

External Validation ◽

Prediction Rule ◽

Bloody Diarrhea ◽

Stepwise Logistic Regression ◽

Clinical Prediction ◽

Clinical Variables ◽

Diarrheal Illness

Abstract Background Infectious diarrheal illness is a significant contributor to healthcare costs in the US pediatric population. New multi-pathogen PCR-based panels have shown increased sensitivity over previous methods; however, they are costly and clinical utility may be limited in many cases. Clinical Prediction Rules (CPRs) may help optimize the appropriate use of these tests. Furthermore, Natural Language Processing (NLP) is an emerging tool to extract clinical history for decision support. Here, we examine NLP for the validation of a CPR for pediatric diarrhea. Methods Using data from a prospective clinical trial at 5 US pediatric hospitals, 961 diarrheal cases were assessed for etiology and relevant clinical variables. Of 65 variables collected in that study, 42 were excluded in our models based on a scarcity of documentation in reviewed clinical charts. The remaining 23 variables were ranked by random forest (RF) variable importance and utilized in both an RF and stepwise logistic regression (LR) model for viral-only etiology. We investigated whether NLP could accurately extract data from clinical notes comparable to study questionnaires. We used the eHOST abstraction software to abstract 6 clinical variables from patient charts that were useful in our CPR. These data will be used to train an NLP algorithm to extract the same variables from additional charts, and be combined with data from 2 other variables coded in the EMR to externally validate our model. Results Both RF and LR models achieved cross-validated area under the receiver operating characteristic curves of 0.74 using the top 5 variables (season, age, bloody diarrhea, vomiting/nausea, and fever), which did not improve significantly with the addition of more variables. Of 270 charts abstracted for NLP training, there were 41 occurrences of bloody diarrhea annotated, 339 occurrences of vomiting, and 145 occurrences of fever. Inter-annotator agreement over 9 training sets ranged between 0.63 and 0.83. Conclusion We have constructed a parsimonious CPR involving only 5 inputs for the prediction of a viral-only etiology for pediatric diarrheal illness using prospectively collected data. With the training of an NLP algorithm for automated chart abstraction we will validate the CPR. NLP could allow a CPR to run without manual data entry to improve care. Disclosures All authors: No reported disclosures.

Download Full-text

Ruling out coronary heart disease in primary care: external validation of a clinical prediction rule

British Journal of General Practice ◽

10.3399/bjgp12x654470 ◽

2012 ◽

Vol 62 (602) ◽

pp. 460.1-460

Author(s):

Dewi Djasmo ◽

Michael Echteld ◽

Leo Spee

Keyword(s):

Primary Care ◽

Coronary Heart Disease ◽

Heart Disease ◽

External Validation ◽

Prediction Rule ◽

Clinical Prediction Rule ◽

Clinical Prediction

Download Full-text

The Use of a Clinical Prediction Model and D-Dimer (DD) Testing in the Diagnosis of Deep Vein Thrombosis (DVT): A Systematic Review.

Blood ◽

10.1182/blood.v104.11.1052.1052 ◽

2004 ◽

Vol 104 (11) ◽

pp. 1052-1052

Author(s):

Carolyn J. Owen ◽

Steve Doucette ◽

Philip S. Wells

Keyword(s):

High Risk ◽

High Probability ◽

Prediction Rule ◽

Clinical Prediction Rule ◽

Clinical Prediction ◽

Clinical Prediction Model ◽

Prediction Rules ◽

Clinical Probability ◽

Probability Estimates ◽

Low Probability

Abstract Background: The diagnosis of DVT can be made by determining pretest probability of disease and using this information in combination with DD testing and ultrasound imaging. A number of studies have evaluated the use of clinical probability but this literature has not been summarized. Purpose: To systematically review trials that evaluated DVT prevalence using clinical prediction rules either with or without DD for the diagnosis of DVT. Data Sources: English and French language studies were identified from a MEDLINE search from 1990 to March 2004 and were supplemented by a review of all relevant bibliographies. Study Selection: Prospective management studies of symptomatic outpatients with suspected DVT in which patients were followed for a minimum of 3 months were selected. Clinical prediction rules had to be employed prior to DD and diagnostic tests. Studies were excluded if patients with a history of prior DVT were enrolled or if insufficient information was presented to calculate the prevalence of DVT for each of the 3 clinical probability estimates (low, moderate and high risk). Data Extraction: Two reviewers assessed each study for inclusion/exclusion criteria and collected data on prevalence and on sensitivity, specificity and likelihood ratios of DD in each of the 3 clinical probability estimates (low, moderate and high risk). Data Synthesis: 14 management studies involving a clinical prediction model in the diagnosis of DVT in over 8000 patients were included, of which 11 utilized DD in the diagnostic algorithm. All studies employed the same clinical prediction rule. The inverse variance weighted average prevalence of DVT in the low, moderate and high probability subgroups were 4.9% (95% CI= 4.2% to 5.7%), 17.4% (95% CI= 16.2% to 18.8%), and 53.6% (95% CI= 51.1% to 56.2%), respectively. The overall weighted prevalence was 18.3% (95% CI= 17.4% to 19.2%). The sensitivity of DD for the diagnosis of DVT in the low, moderate and high probability subgroups were 90.4% (95% CI= 84.7% to 94.2%), 92.0 % (95% CI= 89.1% to 94.2%), 93.6% (95% CI= 91.2% to 94.3%); and the specificities were 69.9% (95% CI= 68.0% to 71.8%), 52.4% (95% CI= 49.8% to 55.0%), and 43.2% (95% CI= 38.8% to 47.6%), respectively. The Mantel-Haenszel pooled estimates for diagnostic odds ratios (DOR) were 17.4 (95%CI=10.4–29.1), 10.2 (95% CI=7.1–14.6), and 10.1 (95% CI=6.9–14.9) in low, moderate and high groups respectively. Conclusion: Accurate estimates of the prevalence of DVT can be achieved using the same clinical prediction rule. Using this rule, it is unlikely that low probability patients have a DVT probability of more than 5%. Specificity of the DD seems to have clinically relevant differences depending on pretest probability but the DORs (which incorporate sensitivity and specificity) are similar. The data suggest that DVT can be excluded if patients are low probability even when DDs of lower sensitivity are employed and that DD testing has lower utility in high probability patients since false positives are common.

Download Full-text