scholarly journals The Cognitive Walkthough for Implementation Strategies (CWIS): A Pragmatic Method for Assessing Implementation Strategy Usability

2020 ◽  
Author(s):  
Aaron R Lyon ◽  
Jessica Coifman ◽  
Heather Cook ◽  
Erin McRee ◽  
Freda F. Liu ◽  
...  
Keyword(s):  
Group Discussion ◽  
Group Testing ◽  
Implementation Strategies ◽  
Preliminary Evidence ◽  
Assessment Method ◽  
Usability Problems ◽  
Specific Direction ◽  
Usability Issue ◽  
Cognitive Walkthrough ◽  
Usability Data

Abstract Background. Implementation strategies have flourished over the last decade in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially-mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which implementation strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to surface aspects of strategies that may inhibit their usability and, ultimately, their effectiveness. This article presents a novel cognitive walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support mental health clinicians in the education sector to adopt measurement-based care.Method. The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially-mediated implementation strategies in health. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed a quantitative assessment of strategy usability.Results. Average subtask success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale: 0-100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative coding and classified by severity and problem type to explain the ratings. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes.Conclusions. Ratings indicated that usability of the consultation protocol was at the low end of the “acceptable” range. Collectively, the 21 usability issues explained the ISUS quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign.

scholarly journals The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability

2021 ◽  
Vol 2 (1) ◽  
Author(s):  
Aaron R. Lyon ◽  
Jessica Coifman ◽  
Heather Cook ◽  
Erin McRee ◽  
Freda F. Liu ◽  
...  
Keyword(s):  
Qualitative Content Analysis ◽  
School Mental Health ◽  
Group Discussion ◽  
Group Testing ◽  
Implementation Strategies ◽  
Preliminary Evidence ◽  
Assessment Method ◽  
Usability Problems ◽  
Measurement Based Care ◽  
Cognitive Walkthrough

Abstract Background Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to identify aspects of strategies that may inhibit their usability and, ultimately, effectiveness. This article presents a novel walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support school mental health clinicians to adopt measurement-based care. Method The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially mediated implementation strategies. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed an adapted quantitative assessment of strategy usability. Results Average anticipated success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale 0–100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative content analysis with consensus coding, and classified by severity and problem type. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes. Conclusions CWIS quantitative usability ratings indicated that the consultation protocol was at the low end of the “acceptable” range (based on norms from the unadapted scale). Collectively, the 21 resulting usability issues explained the quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign.

scholarly journals Enhanced Cognitive Walkthrough: Development of the Cognitive Walkthrough Method to Better Predict, Identify, and Present Usability Problems

2013 ◽  
Vol 2013 ◽  
pp. 1-17 ◽  
Author(s):  
Lars-Ola Bligård ◽  
Anna-Lisa Osvalder
Keyword(s):  
Medical Equipment ◽  
Development Process ◽  
Product Development Process ◽  
Infusion Pumps ◽  
Usability Problems ◽  
Cognitive Walkthrough ◽  
High Level ◽  
High Degree ◽  
Further Development ◽  
Potential Use

To avoid use errors when handling medical equipment, it is important to develop products with a high degree of usability. This can be achieved by performing usability evaluations in the product development process to detect and mitigate potential usability problems. A commonly used method is cognitive walkthrough (CW), but this method shows three weaknesses: poor high-level perspective, insufficient categorisation of detected usability problems, and difficulties in overviewing the analytical results. This paper presents a further development of CW with the aim of overcoming its weaknesses. The new method is called enhanced cognitive walkthrough (ECW). ECW is a proactive analytical method for analysis of potential usability problems. The ECW method has been employed to evaluate user interface designs of medical equipment such as home-care ventilators, infusion pumps, dialysis machines, and insulin pumps. The method has proved capable of identifying several potential use problems in designs.

scholarly journals Collaborating to co-produce strategies for delivering adolescent sexual and reproductive health interventions: processes and experiences from an implementation research project in Nigeria

2020 ◽  
Vol 35 (Supplement_2) ◽  
pp. ii84-ii97
Author(s):  
Chinyere Ojiugo Mbachu ◽  
Ifunanya Clara Agu ◽  
Obinna Onwujekwe
Keyword(s):  
Reproductive Health ◽  
Sexual And Reproductive Health ◽  
Qualitative Interviews ◽  
Group Discussion ◽  
Implementation Strategies ◽  
Design Strategies ◽  
Secondary Sources ◽  
Key Stakeholders ◽  
Academic Researchers ◽  
Document Review

Abstract Implementation science embraces collaboration between academic researchers and key stakeholders/implementers for the dual purpose of capacity building and context-adaptation. Co-production ensures that knowledge created with inputs from various groups of stakeholders is more reflective of local contexts. This paper highlights the experiences of academic researchers and non-academic implementers in collaborating to design implementation strategies for improving access to sexual and reproductive information and services for adolescents. Data were collected through primary and secondary sources. Detailed review of project documents such as minutes of research meetings, reports of workshops and outputs of group work activities enabled detailed description of the processes and steps of co-designing implementation strategies. Information on experiences and perspectives of benefits of the collaborative were collected through in-depth interviews of non-academic partners and focus group discussion with academic researchers. Narrative synthesis was done for information extracted through document review. Thematic analysis of qualitative interviews was done. The process of designing implementation strategies happened in three chronological steps of setting up the collaborative, selecting intervention areas and convening partners’ meetings to design strategies. Specific activities include stakeholder engagement, situation analysis, selection of intervention areas, designing the implementation strategies and pre-testing implementation tools. The process of analysing and selecting collaborators was iterative, and facilitated by having an ‘insider’ key informant. Working with key stakeholders enabled knowledge sharing and exchange among partners. Information sharing within the collaborative facilitated shifting of mindsets about adolescent sexual and reproductive health, and contextual adaptation of names and labels given to strategies. Co-producing implementation strategies with non-academic implementers enabled stakeholder ownership of implementation strategies and set the scene for their adoption in implementation settings. Some challenges of co-production of knowledge are that it is time consuming; involves several iterations that may influence coherence of strategies; involves multiple interests and priorities and poses a threat to fidelity.

scholarly journals PILOT OUTCOMES OF A COMMUNITY-BASED FALLS PREVENTION PROGRAM TARGETING OLDER ARAB AMERICANS

2019 ◽  
Vol 3 (Supplement_1) ◽  
pp. S868-S868
Author(s):  
Lan Yao ◽  
Suha Kridli ◽  
Amne Talab
Keyword(s):  
Focus Group ◽  
Tai Chi ◽  
Group Discussion ◽  
Preliminary Evidence ◽  
Falls Prevention ◽  
Community Center ◽  
Post Intervention ◽  
Community Based ◽  
For Profit ◽  
Tai Chi Exercise

Abstract Arab American (AA) is the 3rd largest ethnic population in the state of Michigan. Previous studies found that Michigan Arabs were less healthy than the general population in Michigan. Older AAs have higher mortality risk than non-Arab and non-Hispanic Whites, particularly due to chronic diseases. Community-based programs are an effective approach to prevent disease and injury, improve health, and enhance quality of life. While evidence for functional gains resulting from Tai Chi exercise is accumulating, there is little research and support for its feasibility and effectiveness that target older AAs, who are not culturally related to Tai Chi. Participants in this report were 8 older female AAs (mean age 62.4±3.2, range 58-66) who sought services at a not-for-profit Arab Community Center, which aims to enable and empower residents and communities to lead informed, productive and culturally sensitive lives. A certified Tai Chi instructor led the classes using Yang-style Tai Chi moves. The participants completed a 12-week twice-weekly 1-hour Tai Chi classes and post-program focus group discussion, held in a classroom of the Community Center. Post-intervention improvement in timed up & go test (p>.05), comfortable & fast gait speed (p>.05), unipedal stance time (p>.05) were observed. Themes identified from focus group supported Tai Chi’s benefits in balance and salient mental health benefits. The pilot data provides preliminary evidence for therapeutic gains resulting from Tai Chi practices. Agencies serving older AAs play important roles by creating and promoting evidence-based health promotion practices to address the growing needs among older adults.

scholarly journals Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems

2016 ◽  
Vol 24 (e1) ◽  
pp. e55-e60 ◽  
Author(s):  
Reza Khajouei ◽  
Misagh Zahiri Esfahani ◽  
Yunes Jahani
Keyword(s):  
Information Systems ◽  
Evaluation Methods ◽  
Usability Evaluation ◽  
Health Care Information ◽  
Office Management ◽  
Usability Problems ◽  
Significant Difference ◽  
Health Care Information Systems ◽  
Cognitive Walkthrough ◽  
Usability Evaluation Methods

Objective: There are several user-based and expert-based usability evaluation methods that may perform differently according to the context in which they are used. The objective of this study was to compare 2 expert-based methods, heuristic evaluation (HE) and cognitive walkthrough (CW), for evaluating usability of health care information systems. Materials and methods: Five evaluators independently evaluated a medical office management system using HE and CW. We compared the 2 methods in terms of the number of identified usability problems, their severity, and the coverage of each method. Results: In total, 156 problems were identified using the 2 methods. HE identified a significantly higher number of problems related to the “satisfaction” attribute (P = .002). The number of problems identified using CW concerning the “learnability” attribute was significantly higher than those identified using HE (P = .005). There was no significant difference between the number of problems identified by HE, based on different usability attributes (P = .232). Results of CW showed a significant difference between the number of problems related to usability attributes (P < .0001). The average severity of problems identified using CW was significantly higher than that of HE (P < .0001). Conclusion: This study showed that HE and CW do not differ significantly in terms of the number of usability problems identified, but they differ based on the severity of problems and the coverage of some usability attributes. The results suggest that CW would be the preferred method for evaluating systems intended for novice users and HE for users who have experience with similar systems. However, more studies are needed to support this finding.

scholarly journals COMPARISON OF HEURISTIC EVALUATION AND COGNITIVE WALKTHROUGH METHODS IN DOING USABILITY EVALUATION OF MOBILE-BASED DEL EGOV CENTRE HOSPITAL INFORMATION SYSTEM

SEMINASTIKA ◽  
2021 ◽  
Vol 3 (1) ◽  
pp. 99-106
Author(s):  
Gracella Tambunan ◽  
Lit Malem Ginting
Keyword(s):  
Mobile Application ◽  
Evaluation Method ◽  
Likert Scale ◽  
Usability Evaluation ◽  
Heuristic Evaluation ◽  
Usability Problems ◽  
Interactive Product ◽  
Cognitive Walkthrough ◽  
Centre Hospital ◽  
Usability Evaluation Methods

Usability is a factor that indicates the success of an interactive product or system, such as a mobile application. The increasing use of smartphones demands a more accurate and effective usability evaluation method to find usability problems, so that they can be used for product improvement in the development process. This study compares the Cognitive Walkthrough method with Heuristic Evaluation in evaluating the usability of the SIRS Del eGov Center mobile application. Evaluation with these two methods will be carried out by three evaluators who act as experts. Finding problems and recommending improvements from each method will produce an improvement prototype made in the form of a high-fidelity prototype. Each prototype will be tested against ten participants using the Usability Testing method, which will generate scores through the SUS table. From the test scores, the percentage of Likert scale and the success rate of each prototype will be found. The results show that between the two usability evaluation methods, the Heuristic Evaluation method is the more effective method, finds more usability problems, and has a higher Likert scale percentage, which is 66.5%, while Cognitive Walkthrough is 64.75%.

scholarly journals Usability Testing and Feedback Collection in a School Context: Case Poetry Machine

2019 ◽  
Vol 27 (3) ◽  
pp. 17-23 ◽  
Author(s):  
Anna Kantosalo ◽  
Sirpa Riihiaho
Keyword(s):  
Artificial Intelligence ◽  
Product Development ◽  
Peer Tutoring ◽  
School Context ◽  
Group Testing ◽  
Evaluation Criteria ◽  
Writing System ◽  
Natural Interaction ◽  
Usability Problems

Evaluation methods emphasizing children’s natural interaction help in getting feedback from children for product development. This paper presents a case study with 9- to 10-year-old children in a school context. The evaluated system was an artificial intelligence–based poetry writing system, and the methods used were peer tutoring with paired interviews and group testing with a new Feedback Game. The evaluation criteria included the number of usability problems, usefulness, and fun. The applicability of the methods is discussed along with the resources required. Peer tutoring revealed numerous problems, whereas group testing and the Feedback Game were quick to conduct and analyze.

Determining the Effectiveness of the Usability Problem Inspector: A Theory-Based Model and Tool for Finding Usability Problems

2003 ◽  
Vol 45 (3) ◽  
pp. 455-482 ◽  
Author(s):  
Terence S. Andre ◽  
H. Rex Hartson ◽  
Robert C. Williges
Keyword(s):  
Interface Design ◽  
Cost Effective ◽  
Usability Evaluation ◽  
Heuristic Evaluation ◽  
Evaluation Tool ◽  
Stage Of Development ◽  
Usability Problems ◽  
Usability Problem ◽  
Potential Applications ◽  
Cognitive Walkthrough

Despite the increased focus on usability and on the processes and methods used to increase usability, a substantial amount of software is unusable and poorly designed. Much of this is attributable to the lack of cost-effective usability evaluation tools that provide an interaction-based framework for identifying problems. We developed the user action framework and a corresponding evaluation tool, the usability problem inspector (UPI), to help organize usability concepts and issues into a knowledge base. We conducted a comprehensive comparison study to determine if our theory-based framework and tool could be effectively used to find important usability problems in an interface design, relative to two other established inspection methods (heuristic evaluation and cognitive walkthrough). Results showed that the UPI scored higher than heuristic evaluation in terms of thoroughness, validity, and effectiveness and was consistent with cognitive walkthrough for these same measures. We also discuss other potential advantages of the UPI over heuristic evaluation and cognitive walkthrough when applied in practice. Potential applications of this work include a cost-effective alternative or supplement to lab-based formative usability evaluation during any stage of development.

scholarly journals Medication Management Apps: Usable by Older Adults?

2017 ◽  
Vol 61 (1) ◽  
pp. 1141-1144 ◽  
Author(s):  
Rachel E. Stuck ◽  
Amy W. Chong ◽  
L. Mitzner Tracy ◽  
Wendy A. Rogers
Keyword(s):  
Older Adults ◽  
Older Adult ◽  
Medication Management ◽  
Healthcare Providers ◽  
Heuristic Evaluation ◽  
User Reviews ◽  
Usability Problems ◽  
Cognitive Walkthrough ◽  
App Design ◽  
Medication Reminder

For older adults, managing medications can be a burden and could lead to medication non-adherence. To decrease risks associated with medication non-adherence, healthcare providers may recommend medication reminder apps as an assistive tool. However, these apps are often not designed with consideration of older adults’ needs, capabilities, and limitations. To identify whether available apps are suitable for older adults, we conducted an in-depth cognitive walkthrough and a heuristic evaluation of the most commonly downloaded medication reminder app. Findings revealed three main issues: 1) difficulty in navigation, 2) poor visibility, and 3) a lack of transparency. We also selected the top five downloaded medication reminder apps and categorized user reviews to assess app functionality and usability problems. The results of our analysis provide guidance for app design for older adult users to provide effective tools for managing medications and supporting patient/user health.

Sign in / Sign up

Export Citation Format

Share Document