Prediction of extubation failure in the paediatric cardiac ICU using machine learning and high-frequency physiologic data

2021 ◽  
pp. 1-8
Author(s):  
Sydney R. Rooney ◽  
Evan L. Reynolds ◽  
Mousumi Banerjee ◽  
Sara K. Pasquali ◽  
John R. Charpie ◽  
...  
Keyword(s):  
Machine Learning ◽  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Extubation Failure ◽  
Machine Learning Methods ◽  
Operating Characteristic Curve ◽  
Physiologic Data ◽  
Receiver Operating

Abstract Background: Cardiac intensivists frequently assess patient readiness to wean off mechanical ventilation with an extubation readiness trial despite it being no more effective than clinician judgement alone. We evaluated the utility of high-frequency physiologic data and machine learning for improving the prediction of extubation failure in children with cardiovascular disease. Methods: This was a retrospective analysis of clinical registry data and streamed physiologic extubation readiness trial data from one paediatric cardiac ICU (12/2016-3/2018). We analysed patients’ final extubation readiness trial. Machine learning methods (classification and regression tree, Boosting, Random Forest) were performed using clinical/demographic data, physiologic data, and both datasets. Extubation failure was defined as reintubation within 48 hrs. Classifier performance was assessed on prediction accuracy and area under the receiver operating characteristic curve. Results: Of 178 episodes, 11.2% (N = 20) failed extubation. Using clinical/demographic data, our machine learning methods identified variables such as age, weight, height, and ventilation duration as being important in predicting extubation failure. Best classifier performance with this data was Boosting (prediction accuracy: 0.88; area under the receiver operating characteristic curve: 0.74). Using physiologic data, our machine learning methods found oxygen saturation extremes and descriptors of dynamic compliance, central venous pressure, and heart/respiratory rate to be of importance. The best classifier in this setting was Random Forest (prediction accuracy: 0.89; area under the receiver operating characteristic curve: 0.75). Combining both datasets produced classifiers highlighting the importance of physiologic variables in determining extubation failure, though predictive performance was not improved. Conclusion: Physiologic variables not routinely scrutinised during extubation readiness trials were identified as potential extubation failure predictors. Larger analyses are necessary to investigate whether these markers can improve clinical decision-making.

Revisiting performance metrics for prediction with rare outcomes

2021 ◽  
Vol 30 (10) ◽  
pp. 2352-2366
Author(s):  
Samrachana Adhikari ◽  
Sharon-Lise Normand ◽  
Jordan Bloom ◽  
David Shahian ◽  
Sherri Rose
Keyword(s):  
Machine Learning ◽  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
True Positive ◽  
Post Surgery ◽  
Operating Characteristic Curve ◽  
Prediction Problems ◽  
Receiver Operating

Machine learning algorithms are increasingly used in the clinical literature, claiming advantages over logistic regression. However, they are generally designed to maximize the area under the receiver operating characteristic curve. While area under the receiver operating characteristic curve and other measures of accuracy are commonly reported for evaluating binary prediction problems, these metrics can be misleading. We aim to give clinical and machine learning researchers a realistic medical example of the dangers of relying on a single measure of discriminatory performance to evaluate binary prediction questions. Prediction of medical complications after surgery is a frequent but challenging task because many post-surgery outcomes are rare. We predicted post-surgery mortality among patients in a clinical registry who received at least one aortic valve replacement. Estimation incorporated multiple evaluation metrics and algorithms typically regarded as performing well with rare outcomes, as well as an ensemble and a new extension of the lasso for multiple unordered treatments. Results demonstrated high accuracy for all algorithms with moderate measures of cross-validated area under the receiver operating characteristic curve. False positive rates were [Formula: see text]1%, however, true positive rates were [Formula: see text]7%, even when paired with a 100% positive predictive value, and graphical representations of calibration were poor. Similar results were seen in simulations, with the addition of high area under the receiver operating characteristic curve ([Formula: see text]90%) accompanying low true positive rates. Clinical studies should not primarily report only area under the receiver operating characteristic curve or accuracy.

scholarly journals Premio de Investigación SCHOT 2020: desarrollo y validación de un modelo multivariables de predicción de estadía hospitalaria en pacientes mayores de 65 años sometidos artroplastia total de cadera electiva en Chile utilizando aprendizaje de máquinas

2021 ◽  
Vol 62 (03) ◽  
pp. e180-e192
Author(s):  
Claudio Díaz-Ledezma ◽  
David Díaz-Solís ◽  
Raúl Muñoz-Reyes ◽  
Jonathan Torres Castro
Keyword(s):  
Machine Learning ◽  
Support Vector Machine ◽  
Big Data ◽  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Support Vector ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Resumen Introducción La predicción de la estadía hospitalaria luego de una artroplastia total de cadera (ATC) electiva es crucial en la evaluación perioperatoria de los pacientes, con un rol determinante desde el punto de vista operacional y económico. Internacionalmente, se han empleado macrodatos (big data, en inglés) e inteligencia artificial para llevar a cabo evaluaciones pronósticas de este tipo. El objetivo del presente estudio es desarrollar y validar, con el empleo del aprendizaje de máquinas (machine learning, en inglés), una herramienta capaz de predecir la estadía hospitalaria de pacientes chilenos mayores de 65 años sometidos a ATC por artrosis. Material y Métodos Empleando los registros electrónicos de egresos hospitalarios anonimizados del Departamento de Estadísticas e Información de Salud (DEIS), se obtuvieron los datos de 8.970 egresos hospitalarios de pacientes sometidos a ATC por artrosis entre los años 2016 y 2018. En total, 15 variables disponibles en el DEIS, además del porcentaje de pobreza de la comuna de origen del paciente, fueron incluidos para predecir la probabilidad de que un paciente presentara una estadía acortada (< 3 días) o prolongada (> 3 días) luego de la cirugía. Utilizando técnicas de aprendizaje de máquinas, 8 algoritmos de predicción fueron entrenados con el 80% de la muestra. El 20% restante se empleó para validar las capacidades predictivas de los modelos creados a partir de los algoritmos. La métrica de optimización se evaluó y ordenó en un ranking utilizando el área bajo la curva de característica operativa del receptor (area under the receiver operating characteristic curve, AUC-ROC, en inglés), que corresponde a cuan bien un modelo puede distinguir entre dos grupos. Resultados El algoritmo XGBoost obtuvo el mejor desempeño, con una AUC-ROC promedio de 0,86 (desviación estándar [DE]: 0,0087). En segundo lugar, observamos que el algoritmo lineal de máquina de vector de soporte (support vector machine, SVM, en inglés) obtuvo una AUC-ROC de 0,85 (DE: 0,0086). La importancia relativa de las variables explicativas demostró que la región de residencia, el servicio de salud, el establecimiento de salud donde se operó el paciente, y la modalidad de atención son las variables que más determinan el tiempo de estadía de un paciente. Discusión El presente estudio desarrolló algoritmos de aprendizaje de máquinas basados en macrodatos chilenos de libre acceso, y logró desarrollar y validar una herramienta que demuestra una adecuada capacidad discriminatoria para predecir la probabilidad de estadía hospitalaria acortada versus prolongada en adultos mayores sometidos a ATC por artrosis. Conclusión Los algoritmos creados a traves del empleo del aprendizaje de máquinas permiten predecir la estadía hospitalaria en pacientes chilenos operado de artroplastia total de cadera electiva.

scholarly journals Claims‐Based Algorithms for Identifying Patients With Pulmonary Hypertension: A Comparison of Decision Rules and Machine‐Learning Approaches

2020 ◽  
Vol 9 (19) ◽  
Author(s):  
Mei‐Sing Ong ◽  
Jeffrey G. Klann ◽  
Kueiyu Joshua Lin ◽  
Bradley A. Maron ◽  
Shawn N. Murphy ◽  
...  
Keyword(s):  
Machine Learning ◽  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Decision Rules ◽  
Machine Learning Algorithms ◽  
Rule Based ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Background Real‐world healthcare data are an important resource for epidemiologic research. However, accurate identification of patient cohorts—a crucial first step underpinning the validity of research results—remains a challenge. We developed and evaluated claims‐based case ascertainment algorithms for pulmonary hypertension (PH), comparing conventional decision rules with state‐of‐the‐art machine‐learning approaches. Methods and Results We analyzed an electronic health record‐Medicare linked database from two large academic tertiary care hospitals (years 2007–2013). Electronic health record charts were reviewed to form a gold standard cohort of patients with (n=386) and without PH (n=164). Using health encounter data captured in Medicare claims (including patients’ demographics, diagnoses, medications, and procedures), we developed and compared 2 approaches for identifying patients with PH: decision rules and machine‐learning algorithms using penalized lasso regression, random forest, and gradient boosting machine. The most optimal rule‐based algorithm—having ≥3 PH‐related healthcare encounters and having undergone right heart catheterization—attained an area under the receiver operating characteristic curve of 0.64 (sensitivity, 0.75; specificity, 0.48). All 3 machine‐learning algorithms outperformed the most optimal rule‐based algorithm ( P <0.001). A model derived from the random forest algorithm achieved an area under the receiver operating characteristic curve of 0.88 (sensitivity, 0.87; specificity, 0.70), and gradient boosting machine achieved comparable results (area under the receiver operating characteristic curve, 0.85; sensitivity, 0.87; specificity, 0.70). Penalized lasso regression achieved an area under the receiver operating characteristic curve of 0.73 (sensitivity, 0.70; specificity, 0.68). Conclusions Research‐grade case identification algorithms for PH can be derived and rigorously validated using machine‐learning algorithms. Simple decision rules commonly applied in published literature performed poorly; more complex rule‐based algorithms may potentially address the limitation of this approach. PH research using claims data would be considerably strengthened through the use of validated algorithms for cohort ascertainment.

scholarly journals Can machine learning improve mortality prediction following cardiac surgery?

2020 ◽  
Vol 58 (6) ◽  
pp. 1130-1136
Author(s):  
Umberto Benedetto ◽  
Shubhra Sinha ◽  
Matt Lyon ◽  
Arnaldo Dimagli ◽  
Tom R Gaunt ◽  
...  
Keyword(s):  
Machine Learning ◽  
Cardiac Surgery ◽  
Random Forest ◽  
Hospital Mortality ◽  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Abstract OBJECTIVES Interest in the clinical usefulness of machine learning for risk prediction has bloomed recently. Cardiac surgery patients are at high risk of complications and therefore presurgical risk assessment is of crucial relevance. We aimed to compare the performance of machine learning algorithms over traditional logistic regression (LR) model to predict in-hospital mortality following cardiac surgery. METHODS A single-centre data set of prospectively collected information from patients undergoing adult cardiac surgery from 1996 to 2017 was split into 70% training set and 30% testing set. Prediction models were developed using neural network, random forest, naive Bayes and retrained LR based on features included in the EuroSCORE. Discrimination was assessed using area under the receiver operating characteristic curve, and calibration analysis was undertaken using the calibration belt method. Model calibration drift was assessed by comparing Goodness of fit χ2 statistics observed in 2 equal bins from the testing sample ordered by procedure date. RESULTS A total of 28 761 cardiac procedures were performed during the study period. The in-hospital mortality rate was 2.7%. Retrained LR [area under the receiver operating characteristic curve 0.80; 95% confidence interval (CI) 0.77–0.83] and random forest model (0.80; 95% CI 0.76–0.83) showed the best discrimination. All models showed significant miscalibration. Retrained LR proved to have the weakest calibration drift. CONCLUSIONS Our findings do not support the hypothesis that machine learning methods provide advantage over LR model in predicting operative mortality after cardiac surgery.

A validated novel preoperative index to predict the extent of intraperitoneal contamination in patients with acute abdominal pathology: A cohort study

2019 ◽  
Vol 30 (7-8) ◽  
pp. 221-228
Author(s):  
Shahab Hajibandeh ◽  
Shahin Hajibandeh ◽  
Nicholas Hobbs ◽  
Jigar Shah ◽  
Matthew Harris ◽  
...  
Keyword(s):  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Validation Cohort ◽  
Characteristic Curve ◽  
Curve Analysis ◽  
Emergency Laparotomy ◽  
Contamination Index ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Aims To investigate whether an intraperitoneal contamination index (ICI) derived from combined preoperative levels of C-reactive protein, lactate, neutrophils, lymphocytes and albumin could predict the extent of intraperitoneal contamination in patients with acute abdominal pathology. Methods Patients aged over 18 who underwent emergency laparotomy for acute abdominal pathology between January 2014 and October 2018 were randomly divided into primary and validation cohorts. The proposed intraperitoneal contamination index was calculated for each patient in each cohort. Receiver operating characteristic curve analysis was performed to determine discrimination of the index and cut-off values of preoperative intraperitoneal contamination index that could predict the extent of intraperitoneal contamination. Results Overall, 468 patients were included in this study; 234 in the primary cohort and 234 in the validation cohort. The analyses identified intraperitoneal contamination index of 24.77 and 24.32 as cut-off values for purulent contamination in the primary cohort (area under the curve (AUC): 0.73, P < 0.0001; sensitivity: 84%, specificity: 60%) and validation cohort (AUC: 0.83, P < 0.0001; sensitivity: 91%, specificity: 69%), respectively. Receiver operating characteristic curve analysis also identified intraperitoneal contamination index of 33.70 and 33.41 as cut-off values for feculent contamination in the primary cohort (AUC: 0.78, P < 0.0001; sensitivity: 87%, specificity: 64%) and validation cohort (AUC: 0.79, P < 0.0001; sensitivity: 86%, specificity: 73%), respectively. Conclusions As a predictive measure which is derived purely from biomarkers, intraperitoneal contamination index may be accurate enough to predict the extent of intraperitoneal contamination in patients with acute abdominal pathology and to facilitate decision-making together with clinical and radiological findings.

A conditional approach for the receiver operating characteristic curve construction to evaluate diagnostic test performance in a family-matched case–control design

2021 ◽  
pp. 096228022199595
Author(s):  
Yalda Zarnegarnia ◽  
Shari Messinger
Keyword(s):  
Receiver Operating Characteristic Curve ◽  
Receiver Operating Characteristic ◽  
Operating Characteristic ◽  
Characteristic Curve ◽  
Case Control ◽  
Paired Data ◽  
Characteristic Curves ◽  
Receiver Operating Characteristic Curves ◽  
Operating Characteristic Curve ◽  
Receiver Operating

Receiver operating characteristic curves are widely used in medical research to illustrate biomarker performance in binary classification, particularly with respect to disease or health status. Study designs that include related subjects, such as siblings, usually have common environmental or genetic factors giving rise to correlated biomarker data. The design could be used to improve detection of biomarkers informative of increased risk, allowing initiation of treatment to stop or slow disease progression. Available methods for receiver operating characteristic construction do not take advantage of correlation inherent in this design to improve biomarker performance. This paper will briefly review some developed methods for receiver operating characteristic curve estimation in settings with correlated data from case–control designs and will discuss the limitations of current methods for analyzing correlated familial paired data. An alternative approach using conditional receiver operating characteristic curves will be demonstrated. The proposed approach will use information about correlation among biomarker values, producing conditional receiver operating characteristic curves that evaluate the ability of a biomarker to discriminate between affected and unaffected subjects in a familial paired design.

Sign in / Sign up

Export Citation Format

Share Document