Derivation and implementation of a machine learning approach to prompt serious illness conversations among outpatients with cancer.

2019 ◽  
Vol 37 (31_suppl) ◽  
pp. 131-131
Author(s):  
Ravi Bharat Parikh ◽  
Chris Manz ◽  
Corey Chivers ◽  
Susan B Regli ◽  
Jennifer Braun ◽  
...  
Keyword(s):  
Machine Learning ◽  
At Risk ◽  
High Risk ◽  
Risk Group ◽  
Gradient Boosting ◽  
Short Term ◽  
Serious Illness ◽  
Patients With Cancer ◽  
Short Term Mortality ◽  
The Impact

131 Background: Machine learning (ML) algorithms can accurately identify patients with cancer at risk of short-term mortality and facilitate timely conversations about treatment and end-of-life preferences. We developed, validated, and implemented a ML algorithm to predict mortality in a general oncology setting, using electronic health record (EHR) data prior to a clinic visit. Methods: Our cohort consisted of patients aged ≥18 years who had an encounter in outpatient oncology practices within a large academic health system between February 1st and July 1st, 2016. We randomly split the sample into training (70%) and validation (30%) cohorts at the patient-encounter level. We trained three ML algorithms to predict 180-day mortality and describe performance in the holdout validation cohort. From October 2018 to February 2019, we used the best-performing algorithm to generate weekly lists of high-risk patients at a single community oncology practice and studied the impact on rates of documented serious illness conversations (SICs). Results: Among 62,377 encounters used to train the algorithms, 7.4% involved a patient who died within 180 days. Gradient boosting and/or random forest outperformed logistic regression in all metrics (Table), and the gradient boosting model had superior discrimination and calibration. In the gradient boosting model, observed 180-day mortality was 45.5% (95% CI 39.0-52.3%) in the high-risk group vs. 3.3% (95% CI 2.9-3.7%) in the low-risk group. In a survey of oncology clinicians, 59% of patients flagged as high-risk were appropriate for a serious illness conversation in the upcoming week (response rate 52%). Five months after implementing the intervention, average monthly documented SICs increased by 23% (31.7 to 39). Conclusions: A ML algorithm based on EHR data accurately identified patients with cancer at risk of short-term mortality, was concordant with oncologists’ assessments, and was associated with more SICs. [Table: see text]

Outcomes among patients with cancer previously identified as being at risk for 30-day mortality using augmented intelligence.

2021 ◽  
Vol 39 (15_suppl) ◽  
pp. 12031-12031
Author(s):  
Ajeet Gajra ◽  
Marjorie E. Zettler ◽  
Amy R. Ellis ◽  
Kelly A. Miller ◽  
John G. Frownfelter ◽  
...  
Keyword(s):  
At Risk ◽  
High Risk ◽  
Hospital Admissions ◽  
Short Term ◽  
Patients With Cancer ◽  
Short Term Mortality ◽  
Medium Risk ◽  
Ed Visits ◽  
Oncology Practice

12031 Background: An augmented intelligence (AI) tool using a machine learning algorithm was developed and validated to generate insights into risk for short-term mortality among patients with cancer. The algorithm, which scores patients every week as being at low, medium or high risk for death within 30 days, allowing providers to potentially intervene and modify care of those at medium to high risk based on established practice pathways. Deployment of the algorithm increased palliative care referrals in a large community hematology/oncology practice in the United States (Gajra et al, JCO 2020). The objective of this retrospective analysis was to evaluate the differences in survival and healthcare utilization (HCU) outcomes of patients previously scored as medium or high risk by the AI tool. Methods: Between 6/2018 – 10/2019, the AI tool scored patients on a weekly basis at the hematology/oncology practice. In 9/2020, a chart review was conducted for the 886 patients who had been identified by the algorithm as being at medium or high risk for 30-day mortality during the index period, to determine outcomes (including death, emergency department [ED] visits, and hospital admissions). Data are presented using descriptive statistics. Results: Of the 886 at-risk patients, 450 (50.8%) were deceased at the time of follow-up. Of these, 244 (54.2%) died within the first 180 days of scoring as at-risk, with median time to death 68 days (IQR 99). Among the 255 patients scored as high risk, 171 (67.1%) had died, vs. 279 (44.2%) of the 631 patients who were scored as medium risk (p < 0.001). Of the 601 patients who were scored more than once during the index period as medium or high risk, 342 (56.9%) had died, vs. 108 (37.9%) of the 285 who were scored as at risk only once (p < 0.001). A total of 363 patients (43.1%) had at least 1 ED visit, and 346 patients (41.1%) had at least 1 hospital admission. There was no difference in the proportion of patients scored as high risk compared with those scored as medium risk in ED visits (104 of 237 [43.9%] vs. 259 of 605 [42.8%], p = 0.778) or hospital admissions (100 of 237 [42.2%] vs. 246 of 605 [40.7%], p = 0.684, respectively). Compared with patients scored as medium or high risk only once during the index period, patients who were scored as at-risk more than once had more ED visits (282 of 593 [47.6%] vs. 81 of 249 [32.5%], p < 0.001) and hospital admissions (269 of 593 [45.4%] vs. 77 of 249 [30.9%], p < 0.001). Conclusions: This follow-up study found that half of the patients identified as at-risk for short-term mortality during the index period were deceased, with greater likelihood associated with high risk score and being scored more than once. Over 40% had visited an ED or were admitted to hospital. These findings have important implications for the use of the algorithm to guide treatment discussions, prevent acute HCU and to plan ahead for end of life care in patients with cancer.

Prospective validation of a machine learning algorithm to predict short-term mortality among outpatients with cancer.

2020 ◽  
Vol 38 (15_suppl) ◽  
pp. 2009-2009
Author(s):  
Chris Manz ◽  
Corey Chivers ◽  
Manqing Liu ◽  
Susan B Regli ◽  
Sujatha Changolkar ◽  
...  
Keyword(s):  
Prospective Cohort ◽  
Performance Metrics ◽  
Risk Group ◽  
Gynecologic Oncology ◽  
Receiver Operating Curve ◽  
Brier Score ◽  
Gradient Boosting ◽  
Short Term ◽  
Mortality Risks ◽  
Short Term Mortality

2009 Background: Oncologists accurately identify only 35% of patients with cancer who will die in six months. There is an urgent need for automated, accurate prognostic systems to inform treatment and advance care planning in oncology. We assessed the prospective performance of a previously described ML algorithm (Parikh et al, JAMA Netw Open, 2019) to predict short-term mortality in a cohort of general oncology outpatients. Methods: Our prospective cohort consisted of patients aged ≥18 years who had a medical or gynecologic oncology encounter between March 1 and April 30, 2019 in either a tertiary academic practice or one of twelve community practices within a large academic cancer system. We used a retrospectively validated gradient-boosting ML algorithm, based on 559 structured electronic health record (EHR) variables, to predict 180-day mortality prior to each oncology encounter. For patients with multiple encounters, we selected the last encounter to assess performance. We assessed several performance metrics, including area under the receiver operating curve (AUC), area under the precision-recall curve (AUPRC), scaled Brier score (sBrier; a measure of calibration ranging from 0 [random] to 1 [perfect]), and positive predictive value (PPV). Results: Of 25,537 unique patients, median age was 64.4 (interquartile range 53.3 – 73.0), 76.8% were White, 56.5% were treated at a community center, and 4.1% died within 180 days. The ML algorithm had an AUC of 0.89 (95% confidence interval [CI] 0.88-0.90), AUPRC 0.34, and sBrier 0.29. At a prespecified threshold of 40%, observed 180-day mortality was 44.5% (95% CI 40.7 – 48.4%) in the high-risk group vs. 3.0% (95% CI 2.8% – 3.3%) in the low-risk group. There was an 85-fold difference in mortality (13.6% vs. 0.16%) in the top vs. bottom risk quartiles. The model was well-calibrated for mortality risks ≤40% and slightly under-calibrated for mortality risks > 40%. Performance varied across cancer types in the tertiary hospital but did not vary by race or practice type (Table). Conclusions: In this prospective cohort study among outpatients with cancer, a ML prognostic algorithm based on EHR data had better discrimination and calibration that published cancer-specific models. This is one of the first ML prognostic models to be prospectively validated in oncology. [Table: see text]

scholarly journals Trajectories of mortality risk among patients with cancer and associated end-of-life utilization

2021 ◽  
Vol 4 (1) ◽  
Author(s):  
Ravi B. Parikh ◽  
Manqing Liu ◽  
Eric Li ◽  
Runze Li ◽  
Jinbo Chen
Keyword(s):  
Machine Learning ◽  
End Of Life ◽  
Mortality Risk ◽  
Machine Learning Algorithms ◽  
Short Term ◽  
Patients With Cancer ◽  
All Cause Mortality ◽  
Short Term Mortality ◽  
Patients At Risk ◽  
To Receive

AbstractMachine learning algorithms may address prognostic inaccuracy among clinicians by identifying patients at risk of short-term mortality and facilitating earlier discussions about hospice enrollment, discontinuation of therapy, or other management decisions. In the present study, we used prospective predictions from a real-time machine learning prognostic algorithm to identify two trajectories of all-cause mortality risk for decedents with cancer. We show that patients with an unpredictable trajectory, where mortality risk rises only close to death, are significantly less likely to receive guideline-based end-of-life care and may not benefit from the integration of prognostic algorithms in practice.

A systematic evaluation of models predicting short-term mortality for cancer patients to highlight uncertainty about model performance and high risk of bias.

2021 ◽  
Vol 39 (15_suppl) ◽  
pp. e13559-e13559
Author(s):  
Sheng-Chieh Lu ◽  
Cai Xu ◽  
Chandler Nguyen ◽  
Larissa Meyer ◽  
Chris Sidey-Gibbons
Keyword(s):  
Machine Learning ◽  
High Risk ◽  
Prediction Model ◽  
Cancer Mortality ◽  
Model Performance ◽  
Risk Of Bias ◽  
Short Term ◽  
Electronic Health Record Data ◽  
Bias Assessment ◽  
Short Term Mortality

e13559 Background: Short-term cancer mortality prediction has many implications concerning care planning. An accurate prognosis allows healthcare providers to adjust care plans and take appropriate actions, such as initiating end-of-life conversations. Machine learning (ML) techniques demonstrated promising capability to support clinical decision-making via providing reliable predictions for a variety of clinical outcomes, including cancer mortality. However, the evidence has not yet been systematically synthesized and evaluated. The objective of this review was to examine the performance and risk-of-bias for ML models trained to predict short-term (≤ 12 months) cancer mortality. Methods: We identified relevant literature from five electronic databases: Ovid Medline, Ovid EMBASE, Scopus, Web of Science, and IEEE Xplore. We searched each database with predefined MeSH terms and keywords of oncology, machine learning, and mortality using AND/OR statements. Inclusion criteria included: 1) developed/validated ML models for predicting oncology patient mortality within one year using electronic health record data; 2) reported model performance within a dataset that was not used to train the models; 3) original research; 4) peer-reviewed full paper in English; 5) published before 1/10/2020. We conducted risk of bias assessment using prediction model risk of bias assessment tool (PROBAST). Results: Ten articles were included in this review. Most studies focused on predicting 1-year mortality (n = 6) for multiple types of cancer (n = 5). Most studies (n = 7) used a single metric, the area under the receiver operating characteristic curve (AUROC), to examine their models. The AUROC ranged from .69 to .91, with a median of .85. Information on samples (n = 10), resampling methods (n = 6), model tuning approaches (n = 9), censoring (n = 10), and sample size determinations (n = 10) were incomplete or absent. Six studies have a high risk of bias for the analysis domain in the PROBAST. Conclusions: The performance of ML models for short-term cancer mortality appears promising. However, most studies report only a single performance metric that obfuscates evaluation of a model’s true performance. This is especially problematic when predicting rare events such as short-term mortality. We found little-to-no information on a given model’s ability to correctly identify patients at high risk of mortality. The incomplete reporting of model development poses challenges to risk of bias assessment and reduces the confidence in the results. Our findings suggest that future studies should report comprehensive performance metrics using a standard reporting guideline, such as transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD), to ensure sufficient information for replication, justification, and adoption.

scholarly journals Derivation of an electronic frailty index for short-term mortality in heart failure: a machine learning approach

2021 ◽  
Author(s):  
Chengsheng Ju ◽  
Jiandong Zhou ◽  
Sharen Lee ◽  
Martin Sebastian Tan ◽  
Ying Liu ◽  
...  
Keyword(s):  
Machine Learning ◽  
Heart Failure ◽  
Logistic Regression ◽  
Decision Tree ◽  
Frailty Index ◽  
Gradient Boosting ◽  
Importance Score ◽  
Multivariate Logistic Regression ◽  
Short Term ◽  
Short Term Mortality

AbstractObjectiveFrailty may be found in heart failure patients especially in the elderly and is associated with a poor prognosis. However, assessment of frailty status is time-consuming and the electronic frailty indices developed using health records have served as useful surrogates. We hypothesized that an electronic frailty index developed using machine learning can improve short-term mortality prediction in patients with heart failure.MethodsThis was a retrospective observational study included patients admitted to nine public hospitals for heart failure from Hong Kong between 2013 and 2017. Age, sex, variables in the modified frailty index, Deyo’s Charlson comorbidity index (≥2), neutrophil-to-lymphocyte ratio (NLR) and prognostic nutritional index (PNI) were analyzed. Gradient boosting, which is a supervised sequential ensemble learning algorithm with weak prediction submodels (typically decision trees), was applied to predict mortality. Comparisons were made with decision tree and multivariate logistic regression.ResultsA total of 8893 patients (median: age 81, Q1-Q3: 71-87 years old) were included, in whom 9% had 30-day mortality and 17% had 90-day mortality. PNI, age and NLR were the most important variables predicting 30-day mortality (importance score: 37.4, 32.1, 20.5, respectively) and 90-day mortality (importance score: 35.3, 36.3, 14.6, respectively). Gradient boosting significantly outperformed decision tree and multivariate logistic regression (area under the curve: 0.90, 0.86 and 0.86 for 30-day mortality; 0.92, 0.89 and 0.86 for 90-day mortality).ConclusionsThe electronic frailty index based on comorbidities, inflammation and nutrition information can readily predict mortality outcomes. Their predictive performances were significantly improved by gradient boosting techniques.

Forecasting US movies box office performances in Turkey using machine learning algorithms

2020 ◽  
Vol 39 (5) ◽  
pp. 6579-6590
Author(s):  
Sandy Çağlıyor ◽  
Başar Öztayşi ◽  
Selime Sezgin
Keyword(s):  
Machine Learning ◽  
Global Economy ◽  
Learning Algorithms ◽  
Forecast Model ◽  
Machine Learning Algorithms ◽  
Gradient Boosting ◽  
High Stakes ◽  
Box Office ◽  
Industry Forecast ◽  
The Impact

The motion picture industry is one of the largest industries worldwide and has significant importance in the global economy. Considering the high stakes and high risks in the industry, forecast models and decision support systems are gaining importance. Several attempts have been made to estimate the theatrical performance of a movie before or at the early stages of its release. Nevertheless, these models are mostly used for predicting domestic performances and the industry still struggles to predict box office performances in overseas markets. In this study, the aim is to design a forecast model using different machine learning algorithms to estimate the theatrical success of US movies in Turkey. From various sources, a dataset of 1559 movies is constructed. Firstly, independent variables are grouped as pre-release, distributor type, and international distribution based on their characteristic. The number of attendances is discretized into three classes. Four popular machine learning algorithms, artificial neural networks, decision tree regression and gradient boosting tree and random forest are employed, and the impact of each group is observed by compared by the performance models. Then the number of target classes is increased into five and eight and results are compared with the previously developed models in the literature.

Short-term risk stratification using CADILLAC risk score in patients with ST elevation myocardial infarction

2020 ◽  
Vol 41 (Supplement_2) ◽  
Author(s):  
T Satou ◽  
H Kitahara ◽  
K Ishikawa ◽  
T Nakayama ◽  
Y Fujimoto ◽  
...  
Keyword(s):  
Myocardial Infarction ◽  
High Risk ◽  
Adverse Events ◽  
Risk Score ◽  
Risk Group ◽  
Risk Groups ◽  
Low Risk ◽  
Short Term ◽  
Recurrent Myocardial Infarction ◽  
St Elevation

Abstract Background The recent reperfusion therapy for ST-elevation myocardial infarction (STEMI) has made the length of hospital stay shorter without adverse events. CADILLAC risk score is reportedly one of the risk scores predicting the long-term prognosis in STEMI patients. Purpose To invenstigate the usefulness of CADILLAC risk score for predicting short-term outcomes in STEMI patients. Methods Consecutive patients admitted to our university hospital and our medical center with STEMI (excluding shock, arrest case) who underwent primary PCI between January 2012 and April 2018 (n=387) were enrolled in this study. The patients were classified into 3 groups according to the CADILLAC risk score: low risk (n=176), intermediate risk (n=87), and high risk (n=124). Data on adverse events within 30 days after hospitalization, including in-hospital death, sustained ventricular arrhythmia, recurrent myocardial infarction, heart failure requiring intravenous treatment, stroke, or clinical hemorrhage, were collected. Results In the low risk group, adverse events within 30 days were significantly less observed, compared to the intermediate and high risk groups (n=13, 7.4% vs. n=13, 14.9% vs. n=58, 46.8%, p&lt;0.001). In particular, all adverse events occurred within 3 days in the low risk group, although adverse events, such as heart failure (n=4), recurrent myocardial infarction (n=1), stroke (n=1), and gastrointestinal bleeding (n=1), were substantially observed after day 4 of hospitalization in the intermediate and high risk groups. Conclusions In STEMI patients with low CADILLAC risk score, better short-term prognosis was observed compared to the intermediate and high risk groups, and all adverse events occurred within 3 days of hospitalization, suggesting that discharge at day 4 might be safe in this study population. CADILLAC risk score may help stratify patient risk for short-term prognosis and adjust management of STEMI patients. Initial event occurrence timing Funding Acknowledgement Type of funding source: None

scholarly journals Explainable machine learning can outperform Cox regression predictions and provide insights in breast cancer survival

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Arturo Moncada-Torres ◽  
Marissa C. van Maaren ◽  
Mathijs P. Hendriks ◽  
Sabine Siesling ◽  
Gijs Geleijnse
Keyword(s):  
Breast Cancer ◽  
Machine Learning ◽  
Explicit Knowledge ◽  
Cox Regression ◽  
Metastatic Breast ◽  
Gradient Boosting ◽  
Support Vector ◽  
Netherlands Cancer Registry ◽  
Extreme Gradient Boosting ◽  
The Impact

AbstractCox Proportional Hazards (CPH) analysis is the standard for survival analysis in oncology. Recently, several machine learning (ML) techniques have been adapted for this task. Although they have shown to yield results at least as good as classical methods, they are often disregarded because of their lack of transparency and little to no explainability, which are key for their adoption in clinical settings. In this paper, we used data from the Netherlands Cancer Registry of 36,658 non-metastatic breast cancer patients to compare the performance of CPH with ML techniques (Random Survival Forests, Survival Support Vector Machines, and Extreme Gradient Boosting [XGB]) in predicting survival using the $$c$$ c -index. We demonstrated that in our dataset, ML-based models can perform at least as good as the classical CPH regression ($$c$$ c -index $$\sim \,0.63$$ ∼ 0.63 ), and in the case of XGB even better ($$c$$ c -index $$\sim 0.73$$ ∼ 0.73 ). Furthermore, we used Shapley Additive Explanation (SHAP) values to explain the models’ predictions. We concluded that the difference in performance can be attributed to XGB’s ability to model nonlinearities and complex interactions. We also investigated the impact of specific features on the models’ predictions as well as their corresponding insights. Lastly, we showed that explainable ML can generate explicit knowledge of how models make their predictions, which is crucial in increasing the trust and adoption of innovative ML techniques in oncology and healthcare overall.

Sign in / Sign up

Export Citation Format

Share Document