Validation of a Machine Learning Model for Early Shock Detection

2021 ◽  
Author(s):  
Yuliya Pinevich ◽  
Adam Amos-Binks ◽  
Christie S Burris ◽  
Gregory Rule ◽  
Marija Bogojevic ◽  
...  

ABSTRACT Objectives The objectives of this study were to test in real time a Trauma Triage, Treatment, and Training Decision Support (4TDS) machine learning (ML) model of shock detection in a prospective silent trial, and to evaluate specificity, sensitivity, and other estimates of diagnostic performance compared to the gold standard of electronic medical records (EMRs) review. Design We performed a single-center diagnostic performance study. Patients and setting A prospective cohort consisted of consecutive patients aged 18 years and older who were admitted from May 1 through September 30, 2020 to six Mayo Clinic intensive care units (ICUs) and five progressive care units. Measurements and main results During the study time, 5,384 out of 6,630 hospital admissions were eligible. During the same period, the 4TDS shock model sent 825 alerts and 632 were eligible. Among 632 hospital admissions with alerts, 287 were screened positive and 345 were negative. Among 4,752 hospital admissions without alerts, 78 were screened positive and 4,674 were negative. The area under the receiver operating characteristics curve for the 4TDS shock model was 0.86 (95% CI 0.85-0.87%). The 4TDS shock model demonstrated a sensitivity of 78.6% (95% CI 74.1-82.7%) and a specificity of 93.1% (95% CI 92.4-93.8%). The model showed a positive predictive value of 45.4% (95% CI 42.6-48.3%) and a negative predictive value of 98.4% (95% CI 98-98.6%). Conclusions We successfully validated an ML model to detect circulatory shock in a prospective observational study. The model used only vital signs and showed moderate performance compared to the gold standard of clinician EMR review when applied to an ICU patient cohort.

2020 ◽  
Vol 4 (Supplement_1) ◽  
Author(s):  
Julie Siersbæk ◽  
Annette R Larsen ◽  
Mads Nybo ◽  
Henrik Boye Thybo Christesen

Abstract Background: The diagnosis of congenital hyperinsulinism (CHI) is often hampered by a plasma insulin (p-insulin) detection limit of 2-3 mU/L (14-21 pmol/L) by RIA methods. Objective: To evaluate the diagnostic performance of a sensitive immunoassay for p-insulin and to find the optimal p-insulin cut-off for CHI versus other conditions with hypoglycaemia. Design: Single centre retrospective cohort study. Methods: Diagnostic tests with no medication, no i.v. glucose and under fasting conditions were performed in children with a clinical diagnosis of CHI. P-insulin concentrations determined at simultaneous p-glucose concentrations at least <3.2 mmol/L (57.5 mg/dL) were included in the analysis (n=61). The diagnosis of CHI was either clinical (n=61) or by gold standard criteria: hypoketotic hypoglycaemia plus disease-causing genetic mutations and/or diffuse, focal or atypical pancreatic histopathology (n=57). Samples from 15 children with idiopathic ketotic hypoglycaemia (IKH, diagnosis by exclusion, p-ketones >1.5 mmol/L during hypoglycaemia) were used as controls. P-insulin was measured by the high-sensitive assay (Cobas e411 immunoassay analyzer); lower detection limit 1.4 pmol/L (0.2 mU/L); normal range 18-173 pmol/L (2.57-24.7 mU/L). Concentrations <18 pmol/L were considered suppressed; ≥18 pmol/L un-suppressed. Receiver operating characteristics (ROC) curves with determination of area under the curve (AUC) values were performed for the diagnostic performance of p-insulin in the diagnosis of CHI. Results: In the 61 samples from CHI patients, the median (range) p-insulin was un-suppressed in all diagnostic samples [90; 20-758 pmol/L (12.9; 2.9-109.1 mU/L)], while p-insulin was suppressed in all 15 samples from IKH patients [1.5; 1.5-9 pmol/L (0.21; 0.21-1.3 mU/L)]. The ROC AUC was 1.0 (95%CI. 1.0-1.0) for the diagnosis of CHI defined both by the clinic and by gold standard. The optimal p-insulin cut-off was 14.5 pmol/L (2.1 mU/L) or 12.5 pmol/L (1.8 mU/L), for CHI patients by use of a simultaneous p-glucose cut-off of <3.2 mmol/L (57.5 mg/dL; n=61), or 3.0 mmol/L (55 mg/dL; n=49), respectively. Conclusions: The sensitive insulin assay performed excellent in diagnosing CHI with a ROC AUC of 1.0. The use of a p-insulin cut-off of 13 pmol/L (1.86 mU/L) during a diagnostic hypoketotic hypoglycaemia test may establish the diagnosis of CHI without further diagnostic testing.


Author(s):  
Isil Basara Akin ◽  
Hakan Abdullah Ozgul ◽  
Canan Altay ◽  
Merih Guray Durak ◽  
Suleyman Ozkan Aksoy ◽  
...  

Abstract Purpose Phyllodes tumors (PTs) are uncommon fibroepithelial breast lesions that are classified as three different forms as benign phyllodes tumor (BPT), borderline phyllodes tumor (BoPT), and malignant phyllodes tumor (MPT). Conventional radiologic methods make only a limited contribution to exact diagnosis, and texture analysis data increase the diagnostic performance. In this study, we aimed to evaluate the contribution of texture analysis of US images (TAUI) of PTs in order to discriminate between BPTs and BoPTs-MPTs. Methods The number of patients was 63 (41 BPTs, 12 BoPTs, and 10 MPTs). Patients were divided into two groups (Group 1-BPT, Group 2-BoPT/MPT). TAUI with LIFEx software was performed retrospectively. An independent machine learning approach, MATLAB R2020a (Math- Works, Natick, Massachusetts) was used with the dataset with p < 0.004. Two machine learning approaches were used to build prediction models for differentiating between Group 1 and Group 2. Receiver operating characteristics (ROC) curve analyses were performed to evaluate the diagnostic performance of statistically significant texture data between phyllodes subgroups. Results In TAUI, 10 statistically significant second order texture values were identified as significant factors capable of differentiating among the two groups (p < 0.05). Both of the models of our dataset make a diagnostic contribution to the discrimination between BopTs-MPTs and BPTs. Conclusion In PTs, US is the main diagnostic method. Adding machine learning-based TAUI to conventional US findings can provide optimal diagnosis, thereby helping to choose the correct surgical method. Consequently, decreased local recurrence rates can be achieved.


2021 ◽  
Vol 39 (28_suppl) ◽  
pp. 333-333
Author(s):  
Kevin Miao ◽  
Justice Dahle ◽  
Sasha Yousefi ◽  
Bilwa Buchake ◽  
Parambir Kaur ◽  
...  

333 Background: Patients undergoing outpatient infusion chemotherapy for cancer are at risk for potentially preventable, unplanned acute care in the form of emergency department (ED) visits and hospital admissions. This can impact outcomes, patient decisions, and costs to the patient and healthcare system. To address this need, the Centers for Medicare & Medicaid Services developed the Chemotherapy Measure (OP-35). Recent randomized controlled data indicate that electronic health record (EHR)-based machine learning (ML) approaches accurately direct supportive care to reduce acute care during radiotherapy. As this may extend to systemic therapy, this study aims to develop and evaluate ML approaches to predict the risk of OP-35 qualifying, potentially preventable acute care within 30 days of infusional systemic therapy. Methods: This study included data from UCSF cancer patients receiving infusional chemotherapy from July 1, 2017, to February 11, 2021, (total 7,068 patients over 84,174 treatments). The data incorporated into the ML included 430 EHR-derived variables, including cancer diagnosis, therapeutic agents, laboratory values, vital signs, medications, and encounter history. Three ML approaches were trained to predict an OP-35 acute care risk following a systemic therapy infusion with least absolute shrinkage selection operator (LASSO), random forest, and gradient boosted trees (GBT; XGBoost) approaches. The models were trained on a subset (75% of patients; before October 12, 2019) of the dataset and validated on a mutually exclusive subset (25% patients; after October 12, 2019) based on the receiver operating characteristic (ROC) curves and calibration plots. Results: There were 1,651 total acute care visits (244 ED visits and 1,407 ED visits converted into hospitalization); 1,310 infusions included a qualifying acute care visit (200 with ED visits only, 0 direct hospital admissions, and 1,110 with both ED visit and hospitalization). Each ML approach demonstrated good performance in the internal validation cohort, with GBT (AUC 0.805) outpacing the random forest (0.750) and LASSO logistic regression (0.755) approaches. Visualization of calibration plots verified concordance between predicted and observed rates of acute care. All three models shared patient age and days elapsed since last treatment as important contributors. Conclusions: EHR-based ML approaches demonstrate high predictive ability for OP-35 qualifying acute care rates on a per-infusion basis, identifying 30-day potentially preventable acute care risk for patients undergoing chemotherapy. Prospective validation of these models is ongoing. Early prediction can facilitate interventional strategies which may reduce acute care, improve health outcomes, and reduce costs.


2020 ◽  
Vol 93 (1113) ◽  
pp. 20191028 ◽  
Author(s):  
Meng Chen ◽  
Ximing Wang ◽  
Guangyu Hao ◽  
Xujie Cheng ◽  
Chune Ma ◽  
...  

Objective: To investigate the diagnostic performance of deep learning (DL)-based vascular extraction and stenosis detection technology in assessing coronary artery disease (CAD). Methods: The diagnostic performance of DL technology was evaluated by retrospective analysis of coronary computed tomography angiography in 124 suspected CAD patients, using invasive coronary angiography as reference standard. Lumen diameter stenosis ≥50% was considered obstructive, and the diagnostic performances were evaluated at per-patient, per-vessel and per-segment levels. The diagnostic performances between DL model and reader model were compared by the areas under the receiver operating characteristics curves (AUCs). Results: In patient-based analysis, AUC of 0.78 was obtained by DL model to detect obstructive CAD [sensitivity of 94%, specificity of 63%, positive predictive value of 94%, and negative predictive value of 59%], While AUC by reader model was 0.74 (sensitivity of 97%, specificity of 50%, positive predictive value of 93%, negative predictive value of 73%). In vessel-based analysis, the AUCs of DL model and reader model were 0.87 and 0.89 respectively. In segment-based analysis, the AUCs of 0.84 and 0.89 were obtained by DL model and reader model respectively. It took 0.47 min to analyze all segments per patient by DL model, which is significantly less than reader model (29.65 min) (p < 0.001). Conclusion: The DL technology can accurately and effectively identify obstructive CAD, with less time-consuming, and it could be a reliable diagnostic tool to detect CAD. Advances in knowledge: The DL technology has valuable prospect with the diagnostic ability to detect CAD.


2019 ◽  
Author(s):  
Awoke Derbie ◽  
Daniel Mekonnen ◽  
Yimtubeznash Woldeamanuel ◽  
Xaveer Van Ostade ◽  
Tamrat Abebe

Abstract Background: Genital infection with certain types of Human papillomavirus (HPV) is a major cause of cervical cancer globally. For early detection of premalignant dysplasia, evidences are coming out on the usefulness of HPV E6/E7 mRNA test as a potential tool compared with cytology and HPV DNA testing. Taking into account shortage of compiled data on this field, the aim of this systematic review was to describe the latest diagnostic performance of HPV E6/E7 mRNA testing to detect high grade cervical lesions (CIN2+) where by histology as was taken as a ‘gold standard’. Methods: Articles published in English were systematically searched using key words from PubMed/Medline and SCOPUS. In addition, Google Scholar and the Google database were searched manually for grey literature. Two reviewers independently assessed study eligibility, risk of bias and extracted the data. We performed a descriptive presentation of the performance of E6/E7 mRNA testings (interims of sensitivity, specificity, negative and positive predictive values) for the detection of CIN2+. Results: Out of 231 applicable citations, we have included 29 articles with a total of 23,576 study participants (age range, 15-84) who had different cervical pathologies. Among the participants who had cervical histology, the proportion of CIN2+ was between 10.6% and 90.6%. Using histology as a gold standard, 11 studies evaluated the PreTect HPV Proofer, 7 studies evaluated the APTIMA HPV assay (Gen-Probe) and 6 studies evaluated the Quantivirus® HPV assay. The diagnostic performance of those three most common mRNA testing tools to detect CIN2+ was; 1) PreTect Proofer; median sensitivity 83%, median specificity 73%, median PPV 70 and median NPV 88.9%. 2) APTIMA assay; median sensitivity 91.4%, median specificity 46.2%, median PPV 34.3% and median NPV 96.3%. 3) Quantivirus®: median sensitivity 86.1%, median specificity 54.6%, median PPV 54.3%, median NPV 89.3%. Further, the area under the receiver operating characteristics (AU-ROC) curve varied between 63.8% and 90.9%. Conclusions: The reported diagnostic accuracy implies that mRNA tests possess diagnostic relevance to detect CIN2+ and could potentially be considered in areas where there is no histology facility. Further studies including its cost should be considered.


2018 ◽  
Author(s):  
Angela Tam ◽  
Christian Dansereau ◽  
Yasser Itturia-Medina ◽  
Sebastian Urchs ◽  
Pierre Orban ◽  
...  

AbstractClinical trials in Alzheimer’s disease need to enroll patients whose cognition will decline over time, if left untreated, in order to demonstrate the efficacy of an intervention. Machine learning models used to screen for patients at risk of progression to dementia should therefore favor specificity (detecting only progressors) over sensitivity (detecting all progressors), especially when the prevalence of progressors is low. Here, we explore whether such high-risk patients can be identified using cognitive assessments and structural neuroimaging, by training machine learning tools in a high specificity regime. A multimodal signature of Alzheimer’s dementia was first extracted from ADNI1. We then validated the predictive value of this signature on ADNI1 patients with mild cognitive impairment (N=235). The signature was optimized to predict progression to dementia over three years with low sensitivity (55.1%) but high specificity (95.6%), resulting in only moderate accuracy (69.3%) but high positive predictive value (80.4%, adjusted for a “typical” 33% prevalence rate of true progressors). These results were replicated in ADNI2 (N=235), with 87.8% adjusted positive predictive value (96.7% specificity, 47.3% sensitivity, 85.1% accuracy). We found that cognitive measures alone could identify high-risk individuals, with structural measurements providing a slight improvement. The signature had comparable receiver operating characteristics to standard machine learning tools, yet a marked improvement in positive predictive value was achieved over the literature by selecting a high specificity operating point. The multimodal signature can be readily applied for the enrichment of clinical trials.


2020 ◽  
Vol 9 (6) ◽  
pp. 1668 ◽  
Author(s):  
Fu-Yuan Cheng ◽  
Himanshu Joshi ◽  
Pranai Tandon ◽  
Robert Freeman ◽  
David L Reich ◽  
...  

Objectives: Approximately 20–30% of patients with COVID-19 require hospitalization, and 5–12% may require critical care in an intensive care unit (ICU). A rapid surge in cases of severe COVID-19 will lead to a corresponding surge in demand for ICU care. Because of constraints on resources, frontline healthcare workers may be unable to provide the frequent monitoring and assessment required for all patients at high risk of clinical deterioration. We developed a machine learning-based risk prioritization tool that predicts ICU transfer within 24 h, seeking to facilitate efficient use of care providers’ efforts and help hospitals plan their flow of operations. Methods: A retrospective cohort was comprised of non-ICU COVID-19 admissions at a large acute care health system between 26 February and 18 April 2020. Time series data, including vital signs, nursing assessments, laboratory data, and electrocardiograms, were used as input variables for training a random forest (RF) model. The cohort was randomly split (70:30) into training and test sets. The RF model was trained using 10-fold cross-validation on the training set, and its predictive performance on the test set was then evaluated. Results: The cohort consisted of 1987 unique patients diagnosed with COVID-19 and admitted to non-ICU units of the hospital. The median time to ICU transfer was 2.45 days from the time of admission. Compared to actual admissions, the tool had 72.8% (95% CI: 63.2–81.1%) sensitivity, 76.3% (95% CI: 74.7–77.9%) specificity, 76.2% (95% CI: 74.6–77.7%) accuracy, and 79.9% (95% CI: 75.2–84.6%) area under the receiver operating characteristics curve. Conclusions: A ML-based prediction model can be used as a screening tool to identify patients at risk of imminent ICU transfer within 24 h. This tool could improve the management of hospital resources and patient-throughput planning, thus delivering more effective care to patients hospitalized with COVID-19.


2020 ◽  
Vol 10 (1) ◽  
Author(s):  
Domenico Scrutinio ◽  
Carlo Ricciardi ◽  
Leandro Donisi ◽  
Ernesto Losavio ◽  
Petronilla Battista ◽  
...  

AbstractStroke is among the leading causes of death and disability worldwide. Approximately 20–25% of stroke survivors present severe disability, which is associated with increased mortality risk. Prognostication is inherent in the process of clinical decision-making. Machine learning (ML) methods have gained increasing popularity in the setting of biomedical research. The aim of this study was twofold: assessing the performance of ML tree-based algorithms for predicting three-year mortality model in 1207 stroke patients with severe disability who completed rehabilitation and comparing the performance of ML algorithms to that of a standard logistic regression. The logistic regression model achieved an area under the Receiver Operating Characteristics curve (AUC) of 0.745 and was well calibrated. At the optimal risk threshold, the model had an accuracy of 75.7%, a positive predictive value (PPV) of 33.9%, and a negative predictive value (NPV) of 91.0%. The ML algorithm outperformed the logistic regression model through the implementation of synthetic minority oversampling technique and the Random Forests, achieving an AUC of 0.928 and an accuracy of 86.3%. The PPV was 84.6% and the NPV 87.5%. This study introduced a step forward in the creation of standardisable tools for predicting health outcomes in individuals affected by stroke.


Author(s):  
Laila Abib ◽  
Renato Sá ◽  
Fernando Peixoto-Filho

Objective The main objective of this study was to examine the diagnostic performance of the first-trimester combined test for aneuploidies in unselected pregnancies from Rio de Janeiro and compare it with the examples available in the literature. Methods We investigated 3,639 patients submitted to aneuploidy screening from February 2009 to September 2015. The examination is composed of the Fetal Medicine Foundation risk evaluation based on nuchal translucency evaluation, mother's age, presence of risk factors, presence of the nasal bone and Doppler of the ductus venous in addition to biochemical analysis of pregnancy-associated plasma protein A (PAPP-A) and beta-human chorionic gonadotropin (β−hCG) markers. The cut-off point for high risk for aneuploidies was defined as greater than 1:100, with intermediate risk defined between 1:100 and 1:1,000, and low risk defined as less than 1:1,000. The variable aneuploidy was considered as a result not only of trisomy of chromosome 21 but also trisomy of chromosomes 13 and 18. Results Excluding the losses, the results of 2,748 patients were analyzed. The first-trimester combined test achieved 71.4% sensitivity with a 7.4% false-positive (FP) rate, specificity of 92.6%, positive predictive value (PPV) of 6.91% and negative predictive value (NPV) of 99.76%, when the cut-off point considered was greater than 1:1,000. Through a receiving operating characteristics (ROC) curve, the cut-off point that maximized the sensitivity and specificity for the diagnosis of aneuploidies was defined as 1:1,860. When we adjusted the false-positive (FP) rate to 5%, the detection rate for this analysis is 72.7%, with a cut-off point of 1:610. Conclusion The combined test of aneuploidy screening showed a detection rate inferior to those described in the literature for a higher FP rate.


Stroke ◽  
2018 ◽  
Vol 49 (12) ◽  
pp. 2866-2871 ◽  
Author(s):  
Philip Chang ◽  
Ilana Ruff ◽  
Scott J. Mendelson ◽  
Fan Caprio ◽  
Deborah L. Bergman ◽  
...  

Background and Purpose— A quarter of acute strokes occur in patients hospitalized for another reason. A stroke recognition instrument may be useful for non-neurologists to discern strokes from mimics such as seizures or delirium. We aimed to derive and validate a clinical score to distinguish stroke from mimics among inhospital suspected strokes. Methods— We reviewed consecutive inpatient stroke alerts in a single academic center from January 9, 2014, to December 7, 2016. Data points, including demographics, stroke risk factors, stroke alert reason, postoperative status, neurological examination, vital signs and laboratory values, and final diagnosis, were collected. Using multivariate logistic regression, we derived a weighted scoring system in the first half of patients (derivation cohort) and validated it in the remaining half of patients (validation cohort) using receiver operating characteristics testing. Results— Among 330 subjects, 116 (35.2%) had confirmed stroke, 43 (13.0%) had a neurological mimic (eg, seizure), and 171 (51.8%) had a non-neurological mimic (eg, encephalopathy). Four risk factors independently predicted stroke: clinical deficit score (clinical deficit score 1: 1 point; clinical deficit score ≥2: 3 points), recent cardiac procedure (1 point), history of atrial fibrillation (1 point), and being a new patient (<24 hours from admission: 1 point). The score showed excellent discrimination in the first 165 patients (derivation cohort, area under the curve=0.93) and remaining 165 patients (validation cohort, area under the curve=0.88). A score of ≥2 had 92.2% sensitivity, 69.6% specificity, 62.2% positive predictive value, and 94.3% negative predictive value for identifying stroke. Conclusions— The 2CAN score for recognizing inpatient stroke performs well in a single-center study. A future prospective multicenter study would help validate this score.


Sign in / Sign up

Export Citation Format

Share Document