scholarly journals Use of a Machine Learning Program to Correctly Triage Incoming Text Messaging Replies From a Cardiovascular Text–Based Secondary Prevention Program: Feasibility Study (Preprint)

2020 ◽  
Author(s):  
Nicole Lowres ◽  
Andrew Duckworth ◽  
Julie Redfern ◽  
Aravinda Thiagalingam ◽  
Clara K Chow

BACKGROUND SMS text messaging programs are increasingly being used for secondary prevention, and have been shown to be effective in a number of health conditions including cardiovascular disease. SMS text messaging programs have the potential to increase the reach of an intervention, at a reduced cost, to larger numbers of people who may not access traditional programs. However, patients regularly reply to the SMS text messages, leading to additional staffing requirements to monitor and moderate the patients’ SMS text messaging replies. This additional staff requirement directly impacts the cost-effectiveness and scalability of SMS text messaging interventions. OBJECTIVE This study aimed to test the feasibility and accuracy of developing a machine learning (ML) program to triage SMS text messaging replies (ie, identify which SMS text messaging replies require a health professional review). METHODS SMS text messaging replies received from 2 clinical trials were manually coded (1) into “Is staff review required?” (binary response of yes/no); and then (2) into 12 general categories. Five ML models (Naïve Bayes, OneVsRest, Random Forest Decision Trees, Gradient Boosted Trees, and Multilayer Perceptron) and an ensemble model were tested. For each model run, data were randomly allocated into training set (2183/3118, 70.01%) and test set (935/3118, 29.98%). Accuracy for the yes/no classification was calculated using area under the receiver operating characteristics curve (AUC), false positives, and false negatives. Accuracy for classification into 12 categories was compared using multiclass classification evaluators. RESULTS A manual review of 3118 SMS text messaging replies showed that 22.00% (686/3118) required staff review. For determining need for staff review, the Multilayer Perceptron model had highest accuracy (AUC 0.86; 4.85% false negatives; and 4.63% false positives); with addition of heuristics (specified keywords) fewer false negatives were identified (3.19%), with small increase in false positives (7.66%) and AUC 0.79. Application of this model would result in 26.7% of SMS text messaging replies requiring review (true + false positives). The ensemble model produced the lowest false negatives (1.43%) at the expense of higher false positives (16.19%). OneVsRest was the most accurate (72.3%) for the 12-category classification. CONCLUSIONS The ML program has high sensitivity for identifying the SMS text messaging replies requiring staff input; however, future research is required to validate the models against larger data sets. Incorporation of an ML program to review SMS text messaging replies could significantly reduce staff workload, as staff would not have to review all incoming SMS text messages. This could lead to substantial improvements in cost-effectiveness, scalability, and capacity of SMS text messaging–based interventions.

10.2196/19200 ◽  
2020 ◽  
Vol 8 (6) ◽  
pp. e19200
Author(s):  
Nicole Lowres ◽  
Andrew Duckworth ◽  
Julie Redfern ◽  
Aravinda Thiagalingam ◽  
Clara K Chow

Background SMS text messaging programs are increasingly being used for secondary prevention, and have been shown to be effective in a number of health conditions including cardiovascular disease. SMS text messaging programs have the potential to increase the reach of an intervention, at a reduced cost, to larger numbers of people who may not access traditional programs. However, patients regularly reply to the SMS text messages, leading to additional staffing requirements to monitor and moderate the patients’ SMS text messaging replies. This additional staff requirement directly impacts the cost-effectiveness and scalability of SMS text messaging interventions. Objective This study aimed to test the feasibility and accuracy of developing a machine learning (ML) program to triage SMS text messaging replies (ie, identify which SMS text messaging replies require a health professional review). Methods SMS text messaging replies received from 2 clinical trials were manually coded (1) into “Is staff review required?” (binary response of yes/no); and then (2) into 12 general categories. Five ML models (Naïve Bayes, OneVsRest, Random Forest Decision Trees, Gradient Boosted Trees, and Multilayer Perceptron) and an ensemble model were tested. For each model run, data were randomly allocated into training set (2183/3118, 70.01%) and test set (935/3118, 29.98%). Accuracy for the yes/no classification was calculated using area under the receiver operating characteristics curve (AUC), false positives, and false negatives. Accuracy for classification into 12 categories was compared using multiclass classification evaluators. Results A manual review of 3118 SMS text messaging replies showed that 22.00% (686/3118) required staff review. For determining need for staff review, the Multilayer Perceptron model had highest accuracy (AUC 0.86; 4.85% false negatives; and 4.63% false positives); with addition of heuristics (specified keywords) fewer false negatives were identified (3.19%), with small increase in false positives (7.66%) and AUC 0.79. Application of this model would result in 26.7% of SMS text messaging replies requiring review (true + false positives). The ensemble model produced the lowest false negatives (1.43%) at the expense of higher false positives (16.19%). OneVsRest was the most accurate (72.3%) for the 12-category classification. Conclusions The ML program has high sensitivity for identifying the SMS text messaging replies requiring staff input; however, future research is required to validate the models against larger data sets. Incorporation of an ML program to review SMS text messaging replies could significantly reduce staff workload, as staff would not have to review all incoming SMS text messages. This could lead to substantial improvements in cost-effectiveness, scalability, and capacity of SMS text messaging–based interventions.


2019 ◽  
Vol 40 (Supplement_1) ◽  
Author(s):  
N Lowres ◽  
A Duckworth ◽  
C K Chow ◽  
A Thiagalingam ◽  
J Redfern

Abstract Background Cardiovascular SMS text programs are effective alternate secondary prevention programs for cardiac risk factor reduction and can be delivered as one-way or two-way communication. However, people text back regularly, leading to staffing costs to monitor replies. If you could reduce the need for staff review by 60–70%, costs and scalability of text programs would substantially improve. Purpose To develop and assess accuracy of a machine-learning (ML) program to “triage” and identify texts requiring review/action. Methods We manually reviewed and classified all replies received from two “TEXT ME” cardiovascular secondary prevention programs. Simultaneously a ML model was developed to classify texts and determine those needing a reply (figure). Comparison of ML models included “Naïve Bayes”, “random forest decision trees”, and “gradient boosted trees”, along with comparison to “convolutional neural network” and “recurrent neural network” classification approaches. “Natural language programming” was evaluated however this presented challenges in relation to text content due to non-standard English grammar, frequent use of non-standard abbreviations, and spelling errors. The ML program was trained with 70% of the data-set and accuracy was tested with 30%. Results Manual review of 3118 text replies revealed that only one text was considered urgent, and only 21% required review/action: categorisation was not straight forward due to complexity of texts often containing more than one sentiment (table). The ML program was able to correctly classify 84% of texts into the designated 12 categories. The sensitivity for correctly identifing the need for health professional review was 94% (6.4% false negatives; 3.6% false positives); but with addition of “heuristics” (e.g. searching for specified keywords, question marks etc) sensitivity increased to 97% (2.9% false negatives; 7.3% false positives). Therefore, health professionals would only have to review 27% (true + false positives) of all text replies. Table 1. SMS manual categorisation (n=3118) REVIEW REQUIRED Health Question/concern Admin request Request to STOP Ceased smoking SMS not delivered Urgent/ distress (13%) (4.5%) (3%) (0.8%) (0.4%) (0.03%) NO REVIEW REQUIRED General statement Statement of thanks Reporting good health Blank message Unrelated/ accidental Emoticon only (33%) (23%) (11%) (6%) (4%) (2.4%) Figure 1. Development process Conclusions The ML program has high sensitivity identifying text replies requiring health professional input and a low false negative rate indicating few messages needing response would be missed. Thus, introduction of the program could significantly reduce the workload of health professionals, leading to substantial improvements in scalability and capacity of text-based programs. The future implications for this technology are vast, including utilisation in other interactive mHealth interfaces and cardiovascular health “apps”. Acknowledgement/Funding National Heart Foundation Vanguard Grant; National Health and Medical Research Council Project Grant


10.2196/15890 ◽  
2020 ◽  
Vol 8 (6) ◽  
pp. e15890
Author(s):  
Jessica L Watterson ◽  
Diego Castaneda ◽  
Caricia Catalani

Background Antenatal care (ANC) has the potential to improve maternal health, but it remains underutilized and unevenly implemented in many low- and middle-income countries. Increasingly, text messaging programs for pregnant women show evidence that they can improve the utilization of ANC during pregnancy; however, gaps remain regarding how implementation affects outcomes. Objective This study aimed to assess facilitators and barriers to implementation of an SMS text messaging intervention for pregnant women in Samoa and to assess its impact on ANC attendance. Methods This study took place in Upolu, Samoa, from March to August 2014 and employed a quasi-experimental design. Half (n=3) of the public antenatal clinics on the island offered adult pregnant women the SMS text messaging intervention, with 552 women registering for the messages. At the comparison clinics (n=3), 255 women registered and received usual care. The intervention consisted of unidirectional text messages containing health tips and appointment reminders. The outcome of interest was the number of attended antenatal visits. Implementation data were also collected through a survey of the participating midwives (n=7) and implementation notes. Data analysis included a comparison of women’s baseline characteristics between the two groups, followed by the use of negative binomial regressions to test for associations between participation in the intervention and increased ANC attendance, controlling for individual characteristics and accounting for the clustering of women within clinics. Results The comparison of ANC attendance rates found that women receiving the SMS text messaging intervention attended 15% fewer ANC visits than the comparison group (P=.004), controlling for individual characteristics and clustering. Data analysis of the implementation process suggests that barriers to successful implementation include women registering very late in pregnancy, sharing their phone with others, and inconsistent explanation of the intervention to women. Conclusions These results suggest that unidirectional text messages do not encourage, and might even discourage, ANC attendance in Samoa. Interpreted with other evidence in the literature, these results suggest that SMS text messaging interventions are more effective when they facilitate better communication between patients and health workers. This study is an important contribution to our understanding of when SMS text messaging interventions are and are not effective in improving maternal health care utilization.


2019 ◽  
Vol 7 ◽  
pp. 29
Author(s):  
Emma M.A.L. Beauxis-Aussalet ◽  
Joost Van Doorn ◽  
Lynda Hardman

Classifiers are applied in many domains where classification errors have significant implications. However, end-users may not always understand the errors and their impact, as error visualizations are typically designed for experts and for improving classifiers. We discuss the specific needs of classifiers' end-users, and a simplified visualization designed to address them. We evaluate this design with users from three levels of expertise, and compare it with ROC curves and confusion matrices. We identify key difficulties with understanding the classification errors, and how visualizations addressed or aggravated them. The main issues concerned confusions of the actual and predicted classes (e.g., confusion of False Positives and False Negatives). The machine learning terminology, complexity of ROC curves, and symmetry of confusion matrices aggravated the confusions. The end-user-oriented visualization reduced the difficulties by using several visual features to clarify the actual and predicted classes, and more tangible metrics and representation. Our results contribute to supporting end-users' understanding of classification errors, and informed decisions when choosing or tuning classifiers.


2019 ◽  
Author(s):  
Dickson Shey Nsagha ◽  
Vincent Verla Siysi ◽  
Same Ekobo ◽  
Thomas Obinchemti Egbe ◽  
Odette Dzemo Kibu

BACKGROUND Incomplete adherence to antiretroviral therapy (ART) is one of the factors that contribute to HIV drug resistance, and it is a major problem for the public health system in controlling the HIV pandemic. There is emerging evidence that SMS can play an important role in health care delivery among patients with HIV on ART, especially in resource-limited settings. OBJECTIVE This paper aims to assess the impact of two-way and one-way SMS text messaging on adherence to HIV treatment. We hypothesized that sending weekly text messages through the one-way and two-way SMS text messaging approach will improve adherence to ART among patients with HIV and improve associated clinical outcomes (quality of life). METHODS A randomized controlled trial is being carried out among participants with HIV who have been on ART for at least one month from an accredited treatment center, namely the Buea Regional Hospital and Kumba District Hospital of South West Region, Cameroon. Participants with HIV, both male and female, aged 21 years and older make up a sample size of 207. The interventions involved the use of mobile phone text messages. Before commencing the intervention, a focus group discussion was carried out among the participants to understand their perception about the use of SMS-based interventions to improve adherence. A total of 246 participants were randomized to receive either a one-way text message (SMS sent to a recipient without recipient sending a reply) or two-way text message (SMS sent to a recipient and recipient sends a reply) or the control (no SMS, only standard care). Data on adherence and quality of life were collected at baseline and after 6 months and will be analyzed using SPSS version 21, while qualitative data will be analyzed using Atlas.ti 7.5. RESULTS Data collection began in September 2019 with focus group discussions and baseline data collection. After 1 month of baseline data collection, the intervention began in October 2019, and postintervention data were collected after 6 months (March 2020). At the end of the study, we will be able to understand the perception of patients toward SMS text messaging–based interventions and also assess the impact of one-way and two-way SMS text messages on treatment adherence among patients with HIV and on associated clinical outcomes (quality of life). CONCLUSIONS The impact of SMS text messaging varies across different settings. The results from this study will determine the perception of patients toward an SMS text messaging–based intervention and its impact on adherence to ART. INTERNATIONAL REGISTERED REPORT DERR1-10.2196/16127


10.2196/21592 ◽  
2020 ◽  
Vol 9 (12) ◽  
pp. e21592
Author(s):  
Maulika Kohli ◽  
Elizabeth C Pasipanodya ◽  
Jessica L Montoya ◽  
Maria Marquine ◽  
Martin Hoenigl ◽  
...  

Background African Americans are disproportionally affected by HIV and have poorer rates of antiretroviral therapy (ART) adherence compared to other racial or ethnic groups in the United States. Factors associated with poor HIV disease outcomes are commonly associated with sociostructural barriers that prevent engagement with and retention in HIV care. SMS text messaging interventions to promote ART adherence among predominantly non-Hispanic White persons with HIV (PWH) have been shown to be efficacious; however, limited research has been devoted to culturally tailoring interventions for underrepresented racial/ethnic groups. Considering African Americans show poorer engagement along the HIV care continuum, we developed an individualized and culturally tailored two-way SMS text messaging intervention to improve ART adherence and associated virologic suppression among African American PWH. Objective In this paper we describe the protocol of a culturally tailored individualized Texting for Adherence Building (iTAB) intervention in a 24- to 48-week, single-arm study. Methods We developed a culturally tailored iTAB intervention, which we are implementing in a 24- to 48-week, single-arm study. Participants were recruited from the Family Health Centers of San Diego (FHCSD), a federally qualified health center. Patient inclusion criteria were (1) receiving care at the FHCSD, (2) living with HIV, (3) self-identification as Black, African American, or of African ancestry, (4) English speaking, (5) age 18 or older, (6) currently on ART, and (7) able to provide informed consent. Study enrollment began in November 2017 and closed in July 2019. A total of 90 participants from the FHCSD enrolled in the iTAB intervention, and we anticipate completing data collection in July 2020. Participants were assisted in individualizing and customizing their SMS text message preferences at the baseline study visit. Self-assessment measures are collected at baseline, interim, and final study visits. Problems related to sending/receiving SMS text messages and barriers to ART adherence are assessed at each interim study visit. The FHCSD staff monitors and tracks participants’ daily SMS text message responses to ART adherence reminders using a clinical dashboard. Results We hypothesize that the proportion of individuals achieving HIV virologic suppression (viral load <40 copies/mL) will be greater at the end of the intervention period compared to the proportion prior to study implementation. Additionally, we anticipate that rates of virologic suppression at the end of the intervention among participants receiving iTAB will be comparable to those among the general FHCSD non-African American population who did not receive iTAB. Finally, we anticipate a high response rate to iTAB SMS text messages as well as positive participant feedback at the end of the intervention with regard to the acceptability of, satisfaction with, and perceived efficacy of iTAB. Conclusions The iTAB intervention is a novel individualized two-way SMS text messaging intervention that has been culturally tailored for use among African Americans with HIV. We anticipate that iTAB will demonstrate efficacy in future randomized control trials and will be supportive of medication adherence among other populations facing health disparities. International Registered Report Identifier (IRRID) DERR1-10.2196/21592


2019 ◽  
Author(s):  
Panpan Zhai ◽  
Khezar Hayat ◽  
Wenjing Ji ◽  
Qian Li ◽  
Li Shi ◽  
...  

BACKGROUND Hypertension is one of the leading risk factors for ischemic heart diseases, and high rates of hypertension prevalence have either remained the same or increased in developing countries in recent years. Unfortunately, about 20% to 50% of patients with chronic diseases have been nonadherent to their drug therapy. SMS text messaging and pharmacy student–led consultations have the potential to help patients manage their blood pressure (BP). OBJECTIVE The aim of this study was to assess the effectiveness, feasibility, and acceptability of SMS text messaging and consultation to manage the BP of Chinese patients with hypertension. METHODS We conducted a two-arm cluster randomized controlled trial among patients with hypertension in Xi’an City, Shaanxi Province, China, and recruited 384 patients from 8 community health care centers. Patients were randomized into an intervention group to receive SMS text messages and consultations or into a control group to receive usual care for 3 months. We sent SMS text messages at 3-day intervals and collected data at baseline (including demographics, clinical outcomes, medication complexity, side effects, patient behavior, knowledge about hypertension, BP, and medication adherence) and the 3-month follow-up (including BP, medication adherence, and knowledge about hypertension). RESULTS We assessed 445 patients with hypertension and excluded 61 patients who were not eligible or who had not filled out their questionnaires. The mean age of the patients was 68.5 (SD 7.9) years in the intervention group and 69.4 (SD 9.7) years in the control group, and the sample was primarily female (265/384, 69.0%). Patients in the intervention group showed significant improvements in systolic BP (SBP; mean 134.5 mm Hg, SD 15.5 mm Hg vs mean 140.7 mm Hg, SD 15.2 mm Hg; <i>P</i>=.001), medication adherence (mean 7.4, SD 1.2 vs mean 7.0, SD 1.3; <i>P</i>=.04), and knowledge about hypertension (mean 6.3, SD 0.9 vs mean 5.9, SD 1.2; <i>P</i>=.004) compared with those in the control group. In measures of diastolic BP (DBP), the two arms showed nonsignificant improvements (mean 78.2 mm Hg, SD 9.0 mm Hg vs mean 77.2 mm Hg, SD 10.3 mm Hg; <i>P</i>=.06). In total, 176 patients had controlled BP at the 3-month follow-up (98 patients in the intervention group vs 78 patients in the control group), but it was nonsignificant (<i>P</i>=.08). CONCLUSIONS The use of SMS text messaging and consultation to manage SBP and improve medication adherence is effective, feasible, and acceptable among Chinese patients with hypertension, although a significant difference was not observed with regard to DBP. It is important to maximize the potential of SMS text messaging and consultation by increasing the feasibility and acceptance of mobile interventions and conduct a cost-effectiveness analysis on this method. CLINICALTRIAL Chinese Clinical Trial Registry ChiCTR1900026862; http://www.chictr.org.cn/showproj.aspx?proj=42717.


2020 ◽  
Author(s):  
Raquel Cobos-Campos ◽  
Javier Mar ◽  
Antxon Apiñaniz ◽  
Arantza Sáez de Lafuente ◽  
Naiara Parraza ◽  
...  

Abstract Background: Smoking in one of the most serious public health problems. It is well known that it constitutes a major risk factor for chronic diseases and the leading cause of preventable death worldwide.Due to high prevalence of smokers, new cost-effective strategies seeking to increase smoking cessation rates are needed. Methods:We performed a cost-effectiveness analysis comparing two treatments: health advice provided by general practitioners and nurses in primary care, and health advice reinforced by sending motivational text messages to patients’ mobile phones. A Markov model was used in which patients transitioned between three mutually exclusive health states (smoker, former smoker and dead) after 6-month cycles. We calculated the cost-effectiveness ratio associated with the sending of motivational messages throughout a patient’s life. Health care and society perspectives (separately) was adopted. Costs taken into account were direct health care costs and direct health care cost and costsfor lost productivity, respectively.Additionally, deterministic sensitivity analysis was performed modifying the probability of smoking cessation with each option. Results:Sending of text messages as a tool to support health advice was found to be cost-effective as it was associated with increases in costs of €7.4 and €1,327 per QALY gained for men and women respectively from a healthcare perspective, significantly far from the published cost-effectiveness threshold. From a societal perspective, the combined programmed was dominant. Conclusions: Sending text messages is a cost-effective approach. These findings support the implantation of the combined program across primary care health centres.


2020 ◽  
Author(s):  
Caroline Astrid Figueroa ◽  
Rosa Hernandez-Ramos ◽  
Claire Elizabeth Boone ◽  
Laura Gómez-Pathak ◽  
Vivian Yip ◽  
...  

BACKGROUND Social distancing is a crucial intervention to slow down person-to-person transmission of COVID-19. However, social distancing has negative consequences including increases in depression and anxiety. Digital interventions, like text-messaging, can provide accessible support on a population wide scale. We developed text messages in English and Spanish to help individuals manage their depressive mood and anxiety during the COVID-19 pandemic. OBJECTIVE In a two-arm randomized controlled trial we aim to examine the effect of our 60 days text-messaging intervention. Additionally, we aim to assess if the use of machine learning to adapt the messaging frequency and content improves the effectiveness of the intervention. Finally, we will examine the differences in daily mood ratings between the message categories and time windows. METHODS The messages are designed within two different categories: behavioral activation and coping skills. Participants will be randomized into 1) a random messaging arm, where message category and timing will be chosen with equal probabilities, and 2) a reinforcement-learning arm, with a learned decision mechanism for choosing the messages. Participants in both arms will receive one message per day within three different time windows and will be asked to provide their mood rating 3 hours later. We will compare self-reported daily mood ratings, self-reported Patient Health Questionnaire depression scale 8-item (PHQ-8) and Generalized Anxiety Disorder 7-item (GAD-7) at baseline and at intervention completion. RESULTS The Institutional Review Board at the University of California Berkeley approved this study (CPHS: 2020-04-13162) in April 2020. Data collection runs from April 2020 to April 2021. As of August 24th 2020, we have enrolled 229 participants. We plan to submit manuscripts describing the main results of the trial and results from the micro-randomized trial for publication in peer-reviewed journals and presentations at (inter)-national scientific meetings. CONCLUSIONS Results will contribute to our knowledge of effective psychological tools to alleviate the negative effects of social distancing, and the benefit of using machine learning to personalize digital mental health interventions. CLINICALTRIAL Clinicaltrials.gov: NCT04473599; pre-results.


Sign in / Sign up

Export Citation Format

Share Document