brier score
Recently Published Documents





2022 ◽  
Flavio Azevedo Figueiredo ◽  
Lucas Emanuel Ferreira Ramos ◽  
Rafael Tavares Silva ◽  
Magda Carvalho Pires ◽  
Daniela Ponce ◽  

Background: Acute kidney injury (AKI) is frequently associated with COVID–19 and the need for kidney replacement therapy (KRT) is considered an indicator of disease severity. This study aimed to develop a prognostic score for predicting the need for KRT in hospitalized COVID–19 patients. Methods: This study is part of the multicentre cohort, the Brazilian COVID–19 Registry. A total of 5,212 adult COVID–19 patients were included between March/2020 and September/2020. We evaluated four categories of predictor variables: (1) demographic data; (2) comorbidities and conditions at admission; (3) laboratory exams within 24 h; and (4) the need for mechanical ventilation at any time during hospitalization. Variable selection was performed using generalized additive models (GAM) and least absolute shrinkage and selection operator (LASSO) regression was used for score derivation. The accuracy was assessed using the area under the receiver operating characteristic curve (AUCROC). Risk groups were proposed based on predicted probabilities: non-high (up to 14.9%), high (15.0 to 49.9%), and very high risk (≥ 50.0%). Results: The median age of the model–derivation cohort was 59 (IQR 47–70) years, 54.5% were men, 34.3% required ICU admission, 20.9% evolved with AKI, 9.3% required KRT, and 15.1% died during hospitalization. The validation cohort had similar age, sex, ICU admission, AKI, required KRT distribution and in–hospital mortality. Thirty–two variables were tested and four important predictors of the need for KRT during hospitalization were identified using GAM: need for mechanical ventilation, male gender, higher creatinine at admission, and diabetes. The MMCD score had excellent discrimination in derivation (AUROC = 0.929; 95% CI 0.918–0.939) and validation (AUROC = 0.927; 95% CI 0.911–0.941) cohorts an good overall performance in both cohorts (Brier score: 0.057 and 0.056, respectively). The score is implemented in a freely available online risk calculator ( Conclusion: The use of the MMCD score to predict the need for KRT may assist healthcare workers in identifying hospitalized COVID–19 patients who may require more intensive monitoring, and can be useful for resource allocation.

2022 ◽  
Vol 12 (1) ◽  
You Li ◽  
Yuncong He ◽  
Yan Meng ◽  
Bowen Fu ◽  
Shuanglong Xue ◽  

AbstractVenous thromboembolism (VTE), clinically presenting as deep vein thrombosis (DVT) or pulmonary embolism (PE). Not all DVT patients carry the same risk of developing acute pulmonary embolism (APE). To develop and validate a prediction model to estimate risk of APE in DVT patients combined with past medical history, clinical symptoms, physical signs, and the sign of the electrocardiogram. We analyzed data from a retrospective cohort of patients who were diagnosed as symptomatic VTE from 2013 to 2018 (n = 1582). Among them, 122 patients were excluded. All enrolled patients confirmed by pulmonary angiography or computed tomography pulmonary angiography (CTPA) and compression venous ultrasonography. Using the LASSO and logistics regression, we derived a predictive model with 16 candidate variables to predict the risk of APE and completed internal validation. Overall, 52.9% patients had DVT + APE (773 vs 1460), 47.1% patients only had DVT (687 vs 1460). The APE risk prediction model included one pre-existing disease or condition (respiratory failure), one risk factors (infection), three symptoms (dyspnea, hemoptysis and syncope), five signs (skin cold clammy, tachycardia, diminished respiration, pulmonary rales and accentuation/splitting of P2), and six ECG indicators (SIQIIITIII, right axis deviation, left axis deviation, S1S2S3, T wave inversion and Q/q wave), of which all were positively associated with APE. The ROC curves of the model showed AUC of 0.79 (95% CI, 0.77–0.82) and 0.80 (95% CI, 0.76–0.84) in the training set and testing set. The model showed good predictive accuracy (calibration slope, 0.83 and Brier score, 0.18). Based on a retrospective single-center population study, we developed a novel prediction model to identify patients with different risks for APE in DVT patients, which may be useful for quickly estimating the probability of APE before obtaining definitive test results and speeding up emergency management processes.

2022 ◽  
Vol 12 (1) ◽  
Jelle Lamsma ◽  
Rongqin Yu ◽  
Seena Fazel ◽  
Therese van Amelsvoort ◽  
Agna Bartels-Velthuis ◽  

AbstractOxford Mental Illness and Violence (OxMIV) addresses the need in mental health services for a scalable, transparent and valid tool to predict violent behaviour in patients with severe mental illness. However, external validations are lacking. Therefore, we have used a Dutch sample of general psychiatric patients with schizophrenia spectrum disorders (N = 637) to evaluate the performance of OxMIV in predicting interpersonal violence over 3 years. The predictors and outcome were measured with standardized instruments and multiple sources of information. Patients were mostly male (n = 493, 77%) and, on average, 27 (SD = 7) years old. The outcome rate was 9% (n = 59). Discrimination, as measured by the area under the curve, was moderate at 0.67 (95% confidence interval 0.61–0.73). Calibration-in-the-large was adequate, with a ratio between predicted and observed events of 1.2 and a Brier score of 0.09. At the individual level, risks were systematically underestimated in the original model, which was remedied by recalibrating the intercept and slope of the model. Probability scores generated by the recalibrated model can be used as an adjunct to clinical decision-making in Dutch mental health services.

2022 ◽  
Vol 22 (1) ◽  
James H. McVittie ◽  
David B. Wolfson ◽  
Vittorio Addona ◽  
Zhaoheng Li

AbstractWhen modelling the survival distribution of a disease for which the symptomatic progression of the associated condition is insidious, it is not always clear how to measure the failure/censoring times from some true date of disease onset. In a prevalent cohort study with follow-up, one approach for removing any potential influence from the uncertainty in the measurement of the true onset dates is through the utilization of only the residual lifetimes. As the residual lifetimes are measured from a well-defined screening date (prevalence day) to failure/censoring, these observed time durations are essentially error free. Using residual lifetime data, the nonparametric maximum likelihood estimator (NPMLE) may be used to estimate the underlying survival function. However, the resulting estimator can yield exceptionally wide confidence intervals. Alternatively, while parametric maximum likelihood estimation can yield narrower confidence intervals, it may not be robust to model misspecification. Using only right-censored residual lifetime data, we propose a stacking procedure to overcome the non-robustness of model misspecification; our proposed estimator comprises a linear combination of individual nonparametric/parametric survival function estimators, with optimal stacking weights obtained by minimizing a Brier Score loss function.

2022 ◽  
Vol 12 (1) ◽  
Min Kim ◽  
Younghyun Kang ◽  
Seng Chan You ◽  
Hyung-Deuk Park ◽  
Sang-Soo Lee ◽  

AbstractTo assess the utility of machine learning (ML) algorithms in predicting clinically relevant atrial high-rate episodes (AHREs), which can be recorded by a pacemaker. We aimed to develop ML-based models to predict clinically relevant AHREs based on the clinical parameters of patients with implanted pacemakers in comparison to logistic regression (LR). We included 721 patients without known atrial fibrillation or atrial flutter from a prospective multicenter (11 tertiary hospitals) registry comprising all geographical regions of Korea from September 2017 to July 2020. Predictive models of clinically relevant AHREs were developed using the random forest (RF) algorithm, support vector machine (SVM) algorithm, and extreme gradient boosting (XGB) algorithm. Model prediction training was conducted by seven hospitals, and model performance was evaluated using data from four hospitals. During a median follow-up of 18 months, clinically relevant AHREs were noted in 104 patients (14.4%). The three ML-based models improved the discrimination of the AHREs (area under the receiver operating characteristic curve: RF: 0.742, SVM: 0.675, and XGB: 0.745 vs. LR: 0.669). The XGB model had a greater resolution in the Brier score (RF: 0.008, SVM: 0.008, and XGB: 0.021 vs. LR: 0.013) than the other models. The use of the ML-based models in patient classification was associated with improved prediction of clinically relevant AHREs after pacemaker implantation.

2022 ◽  
Vol 9 ◽  
Wei Zhang ◽  
Jianyun Gao ◽  
Qiaozhen Lai ◽  
Yanzhen Chi ◽  
Tonghua Su

Several probabilistic forecast methods for heatwave (HW) in extended-range scales over China are constructed using four models (ECMWF, CMA, UKMO, and NCEP) from the Subseasonal-to-Seasonal (S2S) database. The methods include four single-model ensembles (SME; ECMWF, CMA, UKMO, and NCEP), multi-model ensemble (MME), and Bayesian model averaging (BMA). The construction and verification of reforecasts are implemented by a defined heat wave index (HWI) which is not only able to reflect the actual occurrence of heatwaves, but also to facilitate forecast and verification. The performance is measured by traditional verification method at each grid point of the 105°E to 132°E; 20°N to 45°N domain for the July, August, and September (JAS) of 1999–2010. For deterministic evaluations of HWI forecast, BMA shows a better pattern correlation coefficient than SME and MME and comparable equitable threat score (ETS) with ECMWF and MME. The good performance of ECMWF and MME take advantage of setting the percentile thresholds for forecasting HW. For the probabilistic forecast, the Brier score of BMA is comparable (superior) to that of MME and ECMWF at short (long) lead-time. BMA also demonstrates an improvement on the reliability of probabilistic forecast, indicating that BMA method is a useful tool for an extended-range forecast of HW. Meanwhile, in the real-time extended-range probabilistic forecast, the beginning date, end date, and probability of HW event can be predicted by the HWI probabilistic forecast of BMA.

Wang Han ◽  
Nur Azizah Allameen ◽  
Irwani Ibrahim ◽  
Preeti Dhanasekaran ◽  
Feng Mengling ◽  

Abstract To characterise gout patients at high risk of hospitalisation and to develop a web-based prognostic model to predict the likelihood of gout-related hospital admissions. This was a retrospective single-centre study of 1417 patients presenting to the emergency department (ED) with a gout flare between 2015 and 2017 with a 1-year look-back period. The dataset was randomly divided, with 80% forming the derivation and the remaining forming the validation cohort. A multivariable logistic regression model was used to determine the likelihood of hospitalisation from a gout flare in the derivation cohort. The coefficients for the variables with statistically significant adjusted odds ratios were used for the development of a web-based hospitalisation risk estimator. The performance of this risk estimator model was assessed via the area under the receiver operating characteristic curve (AUROC), calibration plot, and brier score. Patients who were hospitalised with gout tended to be older, less likely male, more likely to have had a previous hospital stay with an inpatient primary diagnosis of gout, or a previous ED visit for gout, less likely to have been prescribed standby acute gout therapy, and had a significant burden of comorbidities. In the multivariable-adjusted analyses, previous hospitalisation for gout was associated with the highest odds of gout-related admission. Early identification of patients with a high likelihood of gout-related hospitalisation using our web-based validated risk estimator model may assist to target resources to the highest risk individuals, reducing the frequency of gout-related admissions and improving the overall health-related quality of life in the long term. Key points • We reported the characteristics of gout patients visiting a tertiary hospital in Singapore. • We developed a web-based prognostic model with non-invasive variables to predict the likelihood of gout-relatedhospital admissions.

2021 ◽  
pp. 1-10
Pragya Kumari ◽  
Gajendra K. Vishwakarma ◽  
Atanu Bhattacharjee

BACKGROUND: HER2, ER, PR, and ERBB2 play a vital role in treating breast cancer. These are significant predictive and prognosis biomarkers of breast cancer. OBJECTIVE: We aim to obtain a unique biomarker-specific prediction on overall survival to know their survival and death risk. METHODS: Survival analysis is performed on classified data using Classification and Regression Tree (CART) analysis. Hazard ratio and Confidence Interval are computed using MLE and the Bayesian approach with the CPH model for univariate and multivariable illustrations. Validation of CART is executed with the Brier score, and accuracy and sensitivity are obtained using the k-nn classifier. RESULTS: Utilizing CART analysis, the cut-off value of continuous-valued biomarkers HER2, ER, PR, and ERBB2 are obtained as 14.707, 8.128, 13.153, and 6.884, respectively. Brier score of CART is 0.16 towards validation of methodology. Survival analysis gives a demonstration of the survival estimates with significant statistical strategies. CONCLUSIONS: Patients with breast cancer are at low risk of death, whose HER2 value is below its cut-off value, and ER, PR, and ERBB2 values are greater than their cut-off values. This comparison is with the patient having the opposite side of these cut-off values for the same biomarkers.

2021 ◽  
Faisal Aziz ◽  
Alexander Christian Reisinger ◽  
Felix Aberer ◽  
Caren Sourij ◽  
Norbert Tripolt ◽  

Abstract Background: TheSimplified Acute Physiology Score 3 (SAPS 3) is routinely used in intensive care units (ICUs) to predict in-hospital mortality. However, its predictive performance has not been widely evaluated in Coronavirus disease 19 (COVID-19) patients.This studyevaluated and comparedthe performance of SAPS 3for predicting in-hospital mortalityinCOVID-19patients with and without diabetesin Austria.Methods: This study analyzed the Austrian national public health institute (GÖG) data ofCOVID-19patients admitted to ICUs (N=5,850)fromMarch 2020 to March 2021.The SAPS 3 score was calculated and the predicted in-hospital mortality was estimatedusingthreelogit regression equations: standard equation, Central European equation, and Austrian equation recalibrated for COVID-19 patients. Concordance between observed and predicted mortalities was assessed using the standardized mortality ratio (SMR). Discrimination was assessed using the C-statistic. The DeLong test was applied to compare discrimination between diabetes and non-diabetes patients. Accuracy was assessed using the Brier score andcalibration using the calibration plot and Hosmer-Lemeshow test. Results: Theobservedin-hospital mortality was 38.9% in all patients, 42.9% in diabetes, and 37.3% innon-diabetes patients. Themean ±SD SAPS 3 score was 57.4 ±13.2 in all patients,58.8 ±12.9 in diabetes, and 56.8 ±13.2 in non-diabetes patients.The SMR was significantly greater than 1 for standard and Central European equations, while it was close to 1 for the Austrian equation in all, diabetes, and non-diabetes patients. TheC-statistics was 0.69 with aninsignificant (P=0.193) difference between diabetes (0.70)and non-diabetes (0.68)patients. The Brier score was >0.20 for all SAPS 3 equations. Calibration was unsatisfactory for both standard and Central European equations in all cohorts, whereas it was satisfactory for the Austrian equation in diabetes patients.Conclusions:The SAPS 3 score demonstratedlow discrimination and accuracy in COVID-19 patients in Austria with aninsignificant difference between diabetes and non-diabetes patients. All three equations of SAPS 3 were miscalibrated particularly in non-diabetes patients, while the Austrian equation demonstrated satisfactory calibration in diabetes patients. These findingssuggest that both uncalibrated and calibrated versions ofSAPS 3 should be used with caution in COVID-19 patients.

2021 ◽  
Vol 12 ◽  
Xinping Lin ◽  
Shiteng Lin ◽  
XiaoLi Cui ◽  
Daizun Zou ◽  
FuPing Jiang ◽  

Background and Purpose: Treatment for mild stroke remains an open question. We aim to develop a decision support tool based on machine learning (ML) algorithms, called DAMS (Disability After Mild Stroke), to identify mild stroke patients who would be at high risk of post-stroke disability (PSD) if they only received medical therapy and, more importantly, to aid neurologists in making individual clinical decisions in emergency contexts.Methods: Ischemic stroke patients were prospectively recorded in the National Advanced Stroke Center of Nanjing First Hospital (China) between July 2016 and September 2020. The exclusion criteria were patients who received thrombolytic therapy, age <18 years, lack of 3-month modified Rankin Scale (mRS), disabled before the index stroke, with an admission National Institute of Health stroke scale (NIHSS) > 5. The primary outcome was PSD, corresponding to 3-month mRS ≥ 2. We developed five ML models and assessed the area under curve (AUC) of receiver operating characteristic, calibration curve, and decision curve analysis. The optimal ML model was selected to be DAMS. In addition, SHapley Additive exPlanations (SHAP) approach was introduced to rank the feature importance. Finally, rapid-DAMS (R-DAMS) was constructed for a more urgent situation based on DAMS.Results: A total of 1,905 mild stroke patients were enrolled in this study, and patients with PSD accounted for 23.4% (447). There was no difference in AUCs between the five models (ranged from 0.691 to 0.823). Although there was similar discriminative performance between ML models, the support vector machine model exhibited higher net benefit and better calibration (Brier score, 0.159, calibration slope, 0.935, calibration intercept, 0.035). Therefore, this model was selected for DAMS. In addition, SHAP approach showed that the most crucial feature was NIHSS on admission. Finally, R-DAMS was constructed and there was similar discriminative performance between R-DAMS and DAMS, but the former performed worse on calibration.Conclusions: DAMS and R-DAMS, as prediction-driven decision support tools, were designed to aid clinical decision-making for mild stroke patients in emergency contexts. In addition, even within a narrow range of baseline scores, NIHSS on admission is the strongest feature that contributed to the prediction.

Sign in / Sign up

Export Citation Format

Share Document