Usability testing of the PRO-CTCAE measurement system in patients with cancer.

2013 ◽  
Vol 31 (15_suppl) ◽  
pp. e17560-e17560 ◽  
Author(s):  
Maria R. Fawzy ◽  
Amy Pickar Abernethy ◽  
Martin W. Schoen ◽  
Lauren J. Rogak ◽  
Tito R. Mendoza ◽  
...  
Keyword(s):  
Adverse Events ◽  
Measurement System ◽  
Patient Reported Outcomes ◽  
Interactive Voice Response ◽  
Usability Testing ◽  
Ease Of Use ◽  
Cooperative Groups ◽  
Web Based ◽  
Oncology Research ◽  
Patient Reported

e17560 Background: Use of electronic Patient-Reported Outcomes (ePROs) are increasingly employed in patient care and oncology research. However, these systems can be difficult to use, preventing adoption and deployment. The objective is to evaluate the patient usability of the National Cancer Institute’s (NCI) Patient Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) Measurement System. Methods: PRO-CTCAE is an item bank of 124 questions evaluating 78 symptomatic adverse events, as well as a software system that includes a web-based and interactive voice response system [IVRS] to administer these items to patients, and electronic features to customize survey creation, schedule survey administration, export data, and generate alerts and reports. A protocol was developed for two rounds of usability testing to evaluate understanding and ease of navigation reported by patients undergoing cancer treatment. Software was refined between rounds and retested. Patients were asked to login and complete PRO-CTCAE items either in clinic or at home, then were immediately debriefed (semi-scripted) by a trained interviewer. Results: 173 patients at 3 cancer centers and affiliated community network sites were enrolled in the study between 1/11-5/12 (45% male; 32% non-white; 26% high school or less). Usability issues identified in Round 1 (N=37) included difficulty in using radio buttons, absence of survey progress indicators, and problems with system login. Modifications included increasing size of buttons, improving the progress bar feature, and simplifying the interface to focus users on survey questions. In Round 2 (N=136), remaining issues were related to general difficulty with Internet use, such as navigating to the URL. IVRS testing indicated high scores of ease of use and completion of items, but users had difficulty adding unique symptoms to the system. Conclusions: Iterative usability testing identified improvements to web-based and IVRS for patient self-reporting and achieved a more favorable user experience. The feasibility of integrating PRO-CTCAE into trials to gather symptomatic adverse events directly from patients is currently being evaluated in the cancer cooperative groups.

scholarly journals Usability testing of a Web-based solution for the collection of electronic patient-reported outcomes in people living with HIV in Nouvelle Aquitaine, France (Preprint)

2019 ◽  
Author(s):  
Diana Barger ◽  
Olivier Leleux ◽  
Valérie Conte ◽  
Vincent Sapparrart ◽  
Marie Gapillout ◽  
...  
Keyword(s):  
Patient Reported Outcomes ◽  
Usability Testing ◽  
Target Population ◽  
Ease Of Use ◽  
Hiv Care ◽  
People Living With Hiv ◽  
Web Based ◽  
Visualization System ◽  
Patient Reported ◽  
Living With Hiv

BACKGROUND Collecting patient-reported outcomes can be of great value for both research and chronic diseases management. We endeavoured to develop a new facet of the ANRS CO3 Aquitaine cohort study’s web-based data capture and visualization system (APPEGE ® 2.0) for the collection of electronic patient-reported outcomes in people living with HIV care for in Aquitaine, France. OBJECTIVE Given the novelty of the proposed data collection method for our setting and specific characteristics of the target population, we sought to evaluate the initial usability of a prototype of an electronic patient-reported outcomes (ePRO) information system (ARPEGE® 2.0). METHODS Two successive rounds of empirical, task-based, usability testing were conducted, involving eight “experts” and then six people living with HIV. Evaluators provided written feedback during round 1 and oral feedback during round 2. Evaluators who completed the full set of tasks responded to the Systems Usability Scale. We assessed changes in SUS scores between rounds and concluded usability testing when SUS scores reached a ceiling effect, defining “good” usability a priori as a usability score of 70. RESULTS Insights were generated regarding the visibility of system status and the match between the system and the real world that improved the solution’s usability markedly. Experts reported mean SUS scores of 65 +- 18.87 and patients reported mean SUS scores of 85 +- 5.4 (p=0.032). CONCLUSIONS Software modifications, informed by successive rounds of usability testing, resulted in sufficient gains in usability to undertake piloting. Usability testing also prompted us to find the appropriate balance between optimal security and ease of use. CLINICALTRIAL https://clinicaltrials.gov/ct2/show/NCT03296202 (Archived by WebCite at http://www.webcitation.org/6zgOBArps)

scholarly journals Web-Based Module for the Collection of Electronic Patient-Reported Outcomes in People Living With HIV in Nouvelle Aquitaine, France: Usability Evaluation

10.2196/15013 ◽  
2019 ◽  
Vol 3 (4) ◽  
pp. e15013 ◽  
Author(s):  
Diana Barger ◽  
Olivier Leleux ◽  
Valérie Conte ◽  
Vincent Sapparrart ◽  
Marie Gapillout ◽  
...  
Keyword(s):  
Patient Reported Outcomes ◽  
Usability Testing ◽  
Target Population ◽  
Ease Of Use ◽  
People Living With Hiv ◽  
Research Staff ◽  
Web Based ◽  
System Usability Scale ◽  
Patient Reported ◽  
Living With Hiv

Background Patient-reported outcomes (PROs) can be of great value for both research and chronic disease management. We developed a new module of the ANRS CO3 Aquitaine cohort study’s Web-based data capture and visualization solution (APPEGE 2.0) for the collection of electronic PROs among people living with HIV cared for in Nouvelle Aquitaine, France. Objective This study aimed to evaluate the usability of 2 successively developed prototypes of ARPEGE 2.0’s electronic PROs module before launching a pilot study, owing to the novelty of the proposed data collection method for our setting and specific characteristics of the target population. Methods A total of 2 sequential rounds of empirical, task-based usability evaluations were conducted, involving 8 research staff and then 7 people living with HIV. Evaluators provided written feedback during round 1 and oral feedback during round 2. Evaluators who completed the full set of tasks responded to the System Usability Scale (SUS). We assessed changes in SUS scores between rounds and concluded usability testing when SUS scores reached a ceiling effect, defining good usability a priori as a usability score of 70. Results Insights were generated regarding the visibility of system status and the match between the system and the real world that improved the module’s usability. Research staff evaluators reported mean SUS scores of 65 (SD 18.87) and patient evaluators reported mean SUS scores of 85 (SD 5.4; P=.032). Conclusions Software modifications, informed by successive rounds of usability testing, resulted in sufficient gains in usability to undertake piloting. Insights generated during evaluations prompted us to find the appropriate balance between optimal security and ease of use. Trial Registration ClinicalTrials.gov NCT03296202; https://clinicaltrials.gov/ct2/show/NCT03296202 International Registered Report Identifier (IRRID) RR2-10.2196/10.2196/resprot.9439

scholarly journals AO Patient Outcomes Center: Design, Implementation, and Evaluation of a Software Application for the Collection of Patient-Reported Outcome Measures in Orthopedic Outpatient Clinics (Preprint)

2018 ◽  
Author(s):  
Nan E Rothrock ◽  
Michael Bass ◽  
Andrea Blumenthal ◽  
Richard C Gershon ◽  
Beate Hanson ◽  
...  
Keyword(s):  
Patient Outcomes ◽  
User Satisfaction ◽  
Patient Reported Outcomes ◽  
Clinical Care ◽  
Patient Reported Outcome ◽  
Ease Of Use ◽  
Measurement Information ◽  
Web Based ◽  
Software Application ◽  
Patient Reported

BACKGROUND Patient-reported outcomes are increasingly utilized in routine orthopedic clinical care. Computer adaptive tests (CATs) from the Patient-Reported Outcomes Measurement Information System (PROMIS) offer a brief and precise assessment that is well suited for collection within busy clinical environments. However, software apps that support the administration and scoring of CATs, provide immediate access to patient-reported outcome (PRO) scores, and minimize clinician burden are not widely available. OBJECTIVE Our objective was to design, implement, and test the feasibility and usability of a Web-based system for collecting CATs in orthopedic clinics. METHODS AO Patient Outcomes Center (AOPOC) was subjected to 2 rounds of testing. Alpha testing was conducted in 3 orthopedic clinics to evaluate ease of use and feasibility of integration in clinics. Patients completed an assessment of PROMIS CATs and a usability survey. Clinicians participated in a brief semistructured interview. Beta-phase testing evaluated system performance through load testing and usability of the updated version of AOPOC. In both rounds of testing, user satisfaction, bugs, change requests, and performance of PROMIS CATs were captured. RESULTS Patient feedback supported the ease of use in completing an assessment in AOPOC. Across both phases of testing, clinicians rated AOPOC as easy to use but noted difficulties in integrating a Web-based software application within their clinics. PROMIS CATs performed well; the default assessment of 2 CATs was completed quickly (mean 9.5 items) with a satisfactory range of measurement. CONCLUSION AOPOC was demonstrated to be an easy-to-learn and easy-to-use software application for patients and clinicians that can be integrated into orthopedic clinical care. The workflow disruption in integrating any type of PRO collection must be addressed if patients’ voices are to be better integrated in clinical care.

scholarly journals Software for Administering the National Cancer Institute�s Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events: Usability Study (Preprint)

2018 ◽  
Author(s):  
Martin W Schoen ◽  
Ethan Basch ◽  
Lori L Hudson ◽  
Arlene E Chung ◽  
Tito R Mendoza ◽  
...  
Keyword(s):  
Clinical Trials ◽  
Adverse Events ◽  
National Cancer Institute ◽  
Patient Reported Outcomes ◽  
Task Completion ◽  
Heuristic Evaluation ◽  
Web Based ◽  
System Usability Scale ◽  
Patient Reported ◽  
The Common

BACKGROUND The US National Cancer Institute (NCI) developed software to gather symptomatic adverse events directly from patients participating in clinical trials. The software administers surveys to patients using items from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) through Web-based or automated telephone interfaces and facilitates the management of survey administration and the resultant data by professionals (clinicians and research associates). OBJECTIVE The purpose of this study was to iteratively evaluate and improve the usability of the PRO-CTCAE software. METHODS Heuristic evaluation of the software functionality was followed by semiscripted, think-aloud protocols in two consecutive rounds of usability testing among patients with cancer, clinicians, and research associates at 3 cancer centers. We conducted testing with patients both in clinics and at home (remotely) for both Web-based and telephone interfaces. Furthermore, we refined the software between rounds and retested. RESULTS Heuristic evaluation identified deviations from the best practices across 10 standardized categories, which informed initial software improvement. Subsequently, we conducted user-based testing among 169 patients and 47 professionals. Software modifications between rounds addressed identified issues, including difficulty using radio buttons, absence of survey progress indicators, and login problems (for patients) as well as scheduling of patient surveys (for professionals). The initial System Usability Scale (SUS) score for the patient Web-based interface was 86 and 82 (P=.22) before and after modifications, respectively, whereas the task completion score was 4.47, which improved to 4.58 (P=.39) after modifications. Following modifications for professional users, the SUS scores improved from 71 to 75 (P=.47), and the mean task performance improved significantly (4.40 vs 4.02; P=.001). CONCLUSIONS Software modifications, informed by rigorous assessment, rendered a usable system, which is currently used in multiple NCI-sponsored multicenter cancer clinical trials. CLINICALTRIAL ClinicalTrials.gov NCT01031641; https://clinicaltrials.gov/ct2/show/NCT01031641 (Archived by WebCite at http://www.webcitation.org/708hTjlTl)

scholarly journals Electronic patient-reported outcomes and machine learning in predicting immune-related adverse events of immune checkpoint inhibitor therapies

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Sanna Iivanainen ◽  
Jussi Ekstrom ◽  
Henri Virtanen ◽  
Vesa V. Kataja ◽  
Jussi P. Koivunen
Keyword(s):  
Machine Learning ◽  
Early Detection ◽  
Adverse Events ◽  
Cancer Patients ◽  
Immune Checkpoint ◽  
Patient Reported Outcomes ◽  
Prediction Models ◽  
Accuracy Score ◽  
Patient Reported ◽  
Auc Value

Abstract Background Immune-checkpoint inhibitors (ICIs) have introduced novel immune-related adverse events (irAEs), arising from various organ systems without strong timely dependency on therapy dosing. Early detection of irAEs could result in improved toxicity profile and quality of life. Symptom data collected by electronic (e) patient-reported outcomes (PRO) could be used as an input for machine learning (ML) based prediction models for the early detection of irAEs. Methods The utilized dataset consisted of two data sources. The first dataset consisted of 820 completed symptom questionnaires from 34 ICI treated advanced cancer patients, including 18 monitored symptoms collected using the Kaiku Health digital platform. The second dataset included prospectively collected irAE data, Common Terminology Criteria for Adverse Events (CTCAE) class, and the severity of 26 irAEs. The ML models were built using extreme gradient boosting algorithms. The first model was trained to detect the presence and the second the onset of irAEs. Results The model trained to predict the presence of irAEs had an excellent performance based on four metrics: accuracy score 0.97, Area Under the Curve (AUC) value 0.99, F1-score 0.94 and Matthew’s correlation coefficient (MCC) 0.92. The prediction of the irAE onset was more difficult with accuracy score 0.96, AUC value 0.93, F1-score 0.66 and MCC 0.64 but the model performance was still at a good level. Conclusion The current study suggests that ML based prediction models, using ePRO data as an input, can predict the presence and onset of irAEs with a high accuracy, indicating that ePRO follow-up with ML algorithms could facilitate the detection of irAEs in ICI-treated cancer patients. The results should be validated with a larger dataset. Trial registration Clinical Trials Register (NCT3928938), registration date the 26th of April, 2019

scholarly journals Linguistic validation of the simplified Chinese version of the US National Cancer Institute’s patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE™)

BMC Cancer ◽  
2020 ◽  
Vol 20 (1) ◽  
Author(s):  
Cheng KKF ◽  
S. A. Mitchell ◽  
N. Chan ◽  
E. Ang ◽  
W. Tam ◽  
...  
Keyword(s):  
Adverse Events ◽  
Cancer Patients ◽  
Patient Reported Outcomes ◽  
A Priori ◽  
Cognitive Interviews ◽  
Simplified Chinese ◽  
Patient Reported ◽  
The Us ◽  
The Common ◽  
Response Choices

Abstract Background The aim of this study was to translate and linguistically validate the U.S. National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE™) into Simplified Chinese for use in Singapore. Methods All 124 items of the English source PRO-CTCAE item library were translated into Simplified Chinese using internationally established translation procedures. Two rounds of cognitive interviews were conducted with 96 cancer patients undergoing adjuvant treatment to determine if the translations adequately captured the PRO-CTCAE source concepts, and to evaluate comprehension, clarity and ease of judgement. Interview probes addressed the 78 PRO-CTCAE symptom terms (e.g. fatigue), as well as the attributes (e.g. severity), response choices, and phrasing of ‘at its worst’. Items that met the a priori threshold of ≥20% of participants with comprehension difficulties were considered for rephrasing and retesting. Items where < 20% of the sample experienced comprehension difficulties were also considered for rephrasing if better phrasing options were available. Results A majority of PRO-CTCAE-Simplified Chinese items were well comprehended by participants in Round 1. One item posed difficulties in ≥20% and was revised. Two items presented difficulties in < 20% but were revised as there were preferred alternative phrasings. Twenty-four items presented difficulties in < 10% of respondents. Of these, eleven items were revised to an alternative preferred phrasing, four items were revised to include synonyms. Revised items were tested in Round 2 and demonstrated satisfactory comprehension. Conclusions PRO-CTCAE-Simplified Chinese has been successfully developed and linguistically validated in a sample of cancer patients residing in Singapore.

scholarly journals Establishing a common metric for patient-reported outcomes in cancer patients: linking patient reported outcomes measurement information system (PROMIS), numerical rating scale, and patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE)

2020 ◽  
Vol 4 (1) ◽  
Author(s):  
Minji K. Lee ◽  
Benjamin D. Schalet ◽  
David Cella ◽  
Kathleen J. Yost ◽  
Amy C. Dueck ◽  
...  
Keyword(s):  
Information System ◽  
Adverse Events ◽  
Patient Reported Outcomes ◽  
Rating Scale ◽  
Measurement Information ◽  
Outcomes Measurement ◽  
T Score ◽  
Patient Reported ◽  
Measurement Information System ◽  
The Common

Abstract Background Researchers and clinicians studying symptoms experienced by people with cancer must choose from various scales. It would be useful to know how the scores on one measure translate to another. Methods Using item response theory (IRT) with the single-group design, in which the same sample answers all measures, we produced crosswalk tables linking five 0–10 numeric rating scale (NRS) and 15 items from Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE, scored on a 1–5 scale) to the T-Score metric of six different scales from the NIH Patient reported Outcomes Measurement Information System (PROMIS®). The constructs, for which we conducted linking, include emotional distress-anxiety, emotional distress-depression, fatigue, sleep disturbance, pain intensity, and pain interference. We tested the IRT linking assumption of construct similarity between measures by comparing item content and testing unidimensionality of item sets comprising each construct. We also investigated the correlation of the measures to be linked and, by inspecting standardized mean differences, whether the linkage is invariant across age and gender subgroups. For measures that satisfied the assumptions, we conducted linking. Results In general, an NRS score of 0 corresponded to about 38.2 on the PROMIS T-Score scale (mean = 50; SD = 10); whereas an NRS score of 10 corresponded to a PROMIS T-Score of approximately 72.7. Similarly, the lowest/best score of 1 on PRO-CTCAE corresponded to 39.8 on T-score scale and the highest/worst score of 5 corresponded to 72.0. Conclusion We produced robust linking between single item symptom measures and PROMIS short forms.

scholarly journals Web-Based Patient-Reported Outcome Measures for Personalized Treatment and Care (PROMPT-Care): Multicenter Pragmatic Nonrandomized Trial

10.2196/19685 ◽  
2020 ◽  
Vol 22 (10) ◽  
pp. e19685
Author(s):  
Afaf Girgis ◽  
Ivana Durcinoska ◽  
Anthony Arnold ◽  
Joseph Descallar ◽  
Nasreen Kaadan ◽  
...  
Keyword(s):  
Emergency Department ◽  
Patient Reported Outcomes ◽  
Well Being ◽  
Patient Reported Outcome Measures ◽  
Patient Reported Outcome ◽  
Self Management ◽  
Diverse Population ◽  
Web Based ◽  
Patients With Cancer ◽  
Patient Reported

Background Despite the acceptability and efficacy of e–patient-reported outcome (ePRO) systems, implementation in routine clinical care remains challenging. Objective This pragmatic trial implemented the PROMPT-Care (Patient Reported Outcome Measures for Personalized Treatment and Care) web-based system into existing clinical workflows and evaluated its effectiveness among a diverse population of patients with cancer. Methods Adult patients with solid tumors receiving active treatment or follow-up care in four cancer centers were enrolled. The PROMPT-Care intervention supported patient management through (1) monthly off-site electronic PRO physical symptom and psychosocial well-being assessments, (2) automated electronic clinical alerts notifying the care team of unresolved clinical issues following two consecutive assessments, and (3) tailored online patient self-management resources. Propensity score matching was used to match controls with intervention patients in a 4:1 ratio for patient age, sex, and treatment status. The primary outcome was a reduction in emergency department presentations. Secondary outcomes were time spent on chemotherapy and the number of allied health service referrals. Results From April 2016 to October 2018, 328 patients from four public hospitals received the intervention. Matched controls (n=1312) comprised the general population of patients with cancer, seen at the participating hospitals during the study period. Emergency department visits were significantly reduced by 33% (P=.02) among patients receiving the intervention compared with patients in the matched controls. No significant associations were found in allied health referrals or time to end of chemotherapy. At baseline, the most common patient reported outcomes (above-threshold) were fatigue (39%), tiredness (38.4%), worry (32.9%), general wellbeing (32.9%), and sleep (24.1%), aligning with the most frequently accessed self-management domain pages of physical well-being (36%) and emotional well-being (23%). The majority of clinical feedback reports were reviewed by nursing staff (729/893, 82%), largely in response to the automated clinical alerts (n=877). Conclusions Algorithm-supported web-based systems utilizing patient reported outcomes in clinical practice reduced emergency department presentations among a diverse population of patients with cancer. This study also highlighted the importance of (1) automated triggers for reviewing above-threshold results in patient reports, rather than passive manual review of patient records; (2) the instrumental role nurses play in managing alerts; and (3) providing patients with resources to support guided self-management, where appropriate. Together, these factors will inform the integration of web-based PRO systems into future models of routine cancer care. Trial Registration Australian New Zealand Clinical Trials Registry ACTRN12616000615482; https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=370633 International Registered Report Identifier (IRRID) RR2-10.1186/s12885-018-4729-3

Targeted identification of adverse events in coronary artery disease patients based on patient-reported outcomes

2017 ◽  
Vol 6 (7) ◽  
pp. 583-589
Author(s):  
Jari Heiskanen ◽  
Anna-Maija Tolppanen ◽  
Janne Martikainen ◽  
Juha Hartikainen ◽  
Heikki Miettinen ◽  
...  
Keyword(s):  
Coronary Artery Disease ◽  
Coronary Artery ◽  
Adverse Events ◽  
Patient Reported Outcomes ◽  
Patient Reported ◽  
Artery Disease
Sign in / Sign up

Export Citation Format

Share Document