Applying Clinical Decision Support Design Best Practices With the Practical Robust Implementation and Sustainability Model Versus Reliance on Commercially Available Clinical Decision Support Tools: Randomized Controlled Trial

Background Limited consideration of clinical decision support (CDS) design best practices, such as a user-centered design, is often cited as a key barrier to CDS adoption and effectiveness. The application of CDS best practices is resource intensive; thus, institutions often rely on commercially available CDS tools that are created to meet the generalized needs of many institutions and are not user centered. Beyond resource availability, insufficient guidance on how to address key aspects of implementation, such as contextual factors, may also limit the application of CDS best practices. An implementation science (IS) framework could provide needed guidance and increase the reproducibility of CDS implementations. Objective This study aims to compare the effectiveness of an enhanced CDS tool informed by CDS best practices and an IS framework with a generic, commercially available CDS tool. Methods We conducted an explanatory sequential mixed methods study. An IS-enhanced and commercial CDS alert were compared in a cluster randomized trial across 28 primary care clinics. Both alerts aimed to improve beta-blocker prescribing for heart failure. The enhanced alert was informed by CDS best practices and the Practical, Robust, Implementation, and Sustainability Model (PRISM) IS framework, whereas the commercial alert followed vendor-supplied specifications. Following PRISM, the enhanced alert was informed by iterative, multilevel stakeholder input and the dynamic interactions of the internal and external environment. Outcomes aligned with PRISM’s evaluation measures, including patient reach, clinician adoption, and changes in prescribing behavior. Clinicians exposed to each alert were interviewed to identify design features that might influence adoption. The interviews were analyzed using a thematic approach. Results Between March 15 and August 23, 2019, the enhanced alert fired for 61 patients (106 alerts, 87 clinicians) and the commercial alert fired for 26 patients (59 alerts, 31 clinicians). The adoption and effectiveness of the enhanced alert were significantly higher than those of the commercial alert (62% vs 29% alerts adopted, P<.001; 14% vs 0% changed prescribing, P=.006). Of the 21 clinicians interviewed, most stated that they preferred the enhanced alert. Conclusions The results of this study suggest that applying CDS best practices with an IS framework to create CDS tools improves implementation success compared with a commercially available tool. Trial Registration ClinicalTrials.gov NCT04028557; http://clinicaltrials.gov/ct2/show/NCT04028557

Download Full-text

Applying Clinical Decision Support Design Best Practices With the Practical Robust Implementation and Sustainability Model Versus Reliance on Commercially Available Clinical Decision Support Tools: Randomized Controlled Trial (Preprint)

10.2196/preprints.24359 ◽

2020 ◽

Author(s):

Katy E Trinkley ◽

Miranda E Kroehl ◽

Michael G Kahn ◽

Larry A Allen ◽

Tellen D Bennett ◽

...

Keyword(s):

Decision Support ◽

Best Practices ◽

Clinical Decision Support ◽

Controlled Trial ◽

Clinical Decision ◽

User Centered Design ◽

Support Design ◽

Robust Implementation ◽

Sustainability Model ◽

User Centered

BACKGROUND Limited consideration of clinical decision support (CDS) design best practices, such as a user-centered design, is often cited as a key barrier to CDS adoption and effectiveness. The application of CDS best practices is resource intensive; thus, institutions often rely on commercially available CDS tools that are created to meet the generalized needs of many institutions and are not user centered. Beyond resource availability, insufficient guidance on how to address key aspects of implementation, such as contextual factors, may also limit the application of CDS best practices. An implementation science (IS) framework could provide needed guidance and increase the reproducibility of CDS implementations. OBJECTIVE This study aims to compare the effectiveness of an enhanced CDS tool informed by CDS best practices and an IS framework with a generic, commercially available CDS tool. METHODS We conducted an explanatory sequential mixed methods study. An IS-enhanced and commercial CDS alert were compared in a cluster randomized trial across 28 primary care clinics. Both alerts aimed to improve beta-blocker prescribing for heart failure. The enhanced alert was informed by CDS best practices and the Practical, Robust, Implementation, and Sustainability Model (PRISM) IS framework, whereas the commercial alert followed vendor-supplied specifications. Following PRISM, the enhanced alert was informed by iterative, multilevel stakeholder input and the dynamic interactions of the internal and external environment. Outcomes aligned with PRISM’s evaluation measures, including patient reach, clinician adoption, and changes in prescribing behavior. Clinicians exposed to each alert were interviewed to identify design features that might influence adoption. The interviews were analyzed using a thematic approach. RESULTS Between March 15 and August 23, 2019, the enhanced alert fired for 61 patients (106 alerts, 87 clinicians) and the commercial alert fired for 26 patients (59 alerts, 31 clinicians). The adoption and effectiveness of the enhanced alert were significantly higher than those of the commercial alert (62% vs 29% alerts adopted, <i>P</i><.001; 14% vs 0% changed prescribing, <i>P</i>=.006). Of the 21 clinicians interviewed, most stated that they preferred the enhanced alert. CONCLUSIONS The results of this study suggest that applying CDS best practices with an IS framework to create CDS tools improves implementation success compared with a commercially available tool. CLINICALTRIAL ClinicalTrials.gov NCT04028557; http://clinicaltrials.gov/ct2/show/NCT04028557

Download Full-text

Integrating the Practical Robust Implementation and Sustainability Model With Best Practices in Clinical Decision Support Design: Implementation Science Approach

Journal of Medical Internet Research ◽

10.2196/19676 ◽

2020 ◽

Vol 22 (10) ◽

pp. e19676

Author(s):

Katy E Trinkley ◽

Michael G Kahn ◽

Tellen D Bennett ◽

Russell E Glasgow ◽

Heather Haugen ◽

...

Keyword(s):

Decision Support ◽

Best Practices ◽

Implementation Science ◽

Clinical Decision Support ◽

Integrated Approach ◽

Clinical Decision ◽

Lessons Learned ◽

Support Design ◽

Robust Implementation ◽

Sustainability Model

Background Clinical decision support (CDS) design best practices are intended to provide a narrative representation of factors that influence the success of CDS tools. However, they provide incomplete direction on evidence-based implementation principles. Objective This study aims to describe an integrated approach toward applying an existing implementation science (IS) framework with CDS design best practices to improve the effectiveness, sustainability, and reproducibility of CDS implementations. Methods We selected the Practical Robust Implementation and Sustainability Model (PRISM) IS framework. We identified areas where PRISM and CDS design best practices complemented each other and defined methods to address each. Lessons learned from applying these methods were then used to further refine the integrated approach. Results Our integrated approach to applying PRISM with CDS design best practices consists of 5 key phases that iteratively interact and inform each other: multilevel stakeholder engagement, designing the CDS, design and usability testing, thoughtful deployment, and performance evaluation and maintenance. The approach is led by a dedicated implementation team that includes clinical informatics and analyst builder expertise. Conclusions Integrating PRISM with CDS design best practices extends user-centered design and accounts for the multilevel, interacting, and dynamic factors that influence CDS implementation in health care. Integrating PRISM with CDS design best practices synthesizes the many known contextual factors that can influence the success of CDS tools, thereby enhancing the reproducibility and sustainability of CDS implementations. Others can adapt this approach to their situation to maximize and sustain CDS implementation success.

Download Full-text

Integrating the Practical Robust Implementation and Sustainability Model With Best Practices in Clinical Decision Support Design: Implementation Science Approach (Preprint)

10.2196/preprints.19676 ◽

2020 ◽

Author(s):

Katy E Trinkley ◽

Michael G Kahn ◽

Tellen D Bennett ◽

Russell E Glasgow ◽

Heather Haugen ◽

...

Keyword(s):

Decision Support ◽

Best Practices ◽

Implementation Science ◽

Clinical Decision Support ◽

Integrated Approach ◽

Clinical Decision ◽

Lessons Learned ◽

Support Design ◽

Robust Implementation ◽

Sustainability Model

BACKGROUND Clinical decision support (CDS) design best practices are intended to provide a narrative representation of factors that influence the success of CDS tools. However, they provide incomplete direction on evidence-based implementation principles. OBJECTIVE This study aims to describe an integrated approach toward applying an existing implementation science (IS) framework with CDS design best practices to improve the effectiveness, sustainability, and reproducibility of CDS implementations. METHODS We selected the Practical Robust Implementation and Sustainability Model (PRISM) IS framework. We identified areas where PRISM and CDS design best practices complemented each other and defined methods to address each. Lessons learned from applying these methods were then used to further refine the integrated approach. RESULTS Our integrated approach to applying PRISM with CDS design best practices consists of 5 key phases that iteratively interact and inform each other: multilevel stakeholder engagement, designing the CDS, design and usability testing, thoughtful deployment, and performance evaluation and maintenance. The approach is led by a dedicated implementation team that includes clinical informatics and analyst builder expertise. CONCLUSIONS Integrating PRISM with CDS design best practices extends user-centered design and accounts for the multilevel, interacting, and dynamic factors that influence CDS implementation in health care. Integrating PRISM with CDS design best practices synthesizes the many known contextual factors that can influence the success of CDS tools, thereby enhancing the reproducibility and sustainability of CDS implementations. Others can adapt this approach to their situation to maximize and sustain CDS implementation success.

Download Full-text

Developing Clinical Decision Support System for Sleep Staging Tasks by Formulating Explanations from Artificial Intelligence: User-Centered Design and Evaluation. (Preprint)

10.2196/preprints.28659 ◽

2021 ◽

Author(s):

Jeonghwan Hwang ◽

Taeheon Lee ◽

Honggu Lee ◽

Seonjeong Byun

Keyword(s):

Artificial Intelligence ◽

Decision Support ◽

Design Process ◽

Clinical Decision Support ◽

User Study ◽

Clinical Decision ◽

Sleep Staging ◽

User Centered Design ◽

Iterative Design ◽

User Centered

BACKGROUND Despite the unprecedented performances of deep learning algorithms in clinical domains, full reviews of algorithmic predictions by human experts remain mandatory. Under these circumstances, artificial intelligence (AI) models are primarily designed as clinical decision support systems (CDSSs). However, from the perspective of clinical practitioners, the lack of clinical interpretability and user-centered interfaces block the adoption of these AI systems in practice. OBJECTIVE The aim of this study was to develop an AI-based CDSS for assisting polysomnographic technicians in reviewing AI-predicted sleep staging results. This study proposed and evaluated a CDSS that provides clinically sound explanations for AI predictions in a user-centered fashion. METHODS User needs for the system were identified during interviews with polysomnographic technicians. User observation sessions were conducted to understand the workflow of the practitioners during sleep scoring. Iterative design process was performed to ensure easy integration of the tool into clinical workflows. Then, we evaluated the system with polysomnographic technicians. We measured the improvements in sleep staging accuracies after adopting our tool and assessed qualitatively how the participants perceived and used the tool. RESULTS The user study revealed that technicians desire explanations relevant to key electroencephalogram (EEG) patterns for sleep staging when assessing the correctness of the AI predictions. Here, technicians could evaluate whether AI models properly locate and use those patterns during prediction. Based on this, information in AI models that is closely related to sleep EEG patterns was formulated and visualized during the iterative design process. Furthermore, we developed a different visualization strategy for each pattern based on the way the technicians interpreted the EEG recordings with these patterns during their workflows. Generally, the tool evaluation results from the nine polysomnographic technicians were positive. Quantitatively, technicians achieved better classification performances after reviewing the AI-generated predictions with the proposed system; classification accuracies measured with Macro-F1 scores improved from 60.20 to 62.71. Qualitatively, participants reported that the provided information from the tool effectively supported them, and they were able to develop notable adoption strategies for the tool. CONCLUSIONS Our findings indicate that formulating clinical explanations for automated predictions using the information in the AI with a user-centered design process is an effective strategy for developing a CDSS for sleep staging.

Download Full-text

Computerized Clinical Decision Support System for Emergency Department–Initiated Buprenorphine for Opioid Use Disorder: User-Centered Design (Preprint)

10.2196/preprints.13121 ◽

2018 ◽

Cited By ~ 1

Author(s):

Jessica M Ray ◽

Osama M Ahmed ◽

Yauheni Solad ◽

Matthew Maleska ◽

Shara Martel ◽

...

Keyword(s):

Decision Support ◽

Clinical Decision Support ◽

Emergency Care ◽

Clinical Decision ◽

Opioid Use Disorder ◽

Care Pathways ◽

User Centered Design ◽

Opioid Use ◽

Prototype Development ◽

User Centered

BACKGROUND Emergency departments (EDs) frequently care for individuals with opioid use disorder (OUD). Buprenorphine (BUP) is an effective treatment option for patients with OUD that can safely be initiated in the ED. At present, BUP is rarely initiated as a part of routine ED care. Clinical decision support (CDS) could accelerate adoption of ED-initiated BUP into routine emergency care. OBJECTIVE This study aimed to design and formatively evaluate a user-centered decision support tool for ED initiation of BUP for patients with OUD. METHODS User-centered design with iterative prototype development was used. Initial observations and interviews identified workflows and information needs. The design team and key stakeholders reviewed prototype designs to ensure accuracy. A total of 5 prototypes were evaluated and iteratively refined based on input from 26 attending and resident physicians. RESULTS Early feedback identified concerns with the initial CDS design: an alert with several screens. The timing of the alert led to quick dismissal without using the tool. User feedback on subsequent iterations informed the development of a flexible tool to support clinicians with varied levels of experience with the intervention by providing both one-click options for direct activation of care pathways and user-activated support for critical decision points. The final design resolved challenging navigation issues through targeted placement, color, and design of the decision support modules and care pathways. In final testing, users expressed that the tool could be easily learned without training and was reasonable for use during routine emergency care. CONCLUSIONS A user-centered design process helped designers to better understand users’ needs for a Web-based clinical decision tool to support ED initiation of BUP for OUD. The process identified varying needs across user experience and familiarity with the protocol, leading to a flexible design supporting both direct care pathways and user-initiated decision support.

Download Full-text

Utilizing a user-centered approach to develop and assess pharmacogenomic clinical decision support for thiopurine methyltransferase

BMC Medical Informatics and Decision Making ◽

10.1186/s12911-019-0919-4 ◽

2019 ◽

Vol 19 (1) ◽

Cited By ~ 3

Author(s):

Khoa A. Nguyen ◽

Himalaya Patel ◽

David A. Haggstrom ◽

Alan J. Zillich ◽

Thomas F. Imperiale ◽

...

Keyword(s):

Decision Support ◽

User Interface ◽

Phase I ◽

Clinical Decision Support ◽

Phase Ii ◽

Usability Testing ◽

Clinical Decision ◽

Thiopurine Methyltransferase ◽

User Centered Design ◽

User Centered

Abstract Background A pharmacogenomic clinical decision support tool (PGx-CDS) for thiopurine medications can help physicians incorporate pharmacogenomic results into prescribing decisions by providing up-to-date, real-time decision support. However, the PGx-CDS user interface may introduce errors and promote alert fatigue. The objective of this study was to develop and evaluate a prototype of a PGx-CDS user interface for thiopurine medications with user-centered design methods. Methods This study had two phases: In phase I, we conducted qualitative interviews to assess providers’ information needs. Interview transcripts were analyzed through a combination of inductive and deductive qualitative analysis to develop design requirements for a PGx-CDS user interface. Using these requirements, we developed a user interface prototype and evaluated its usability (phase II). Results In total, 14 providers participated: 10 were interviewed in phase I, and seven providers completed usability testing in phase II (3 providers participated in both phases). Most (90%) participants were interested in PGx-CDS systems to help improve medication efficacy and patient safety. Interviews yielded 11 themes sorted into two main categories: 1) health care providers’ views on PGx-CDS and 2) important design features for PGx-CDS. We organized these findings into guidance for PGx-CDS content and display. Usability testing of the PGx-CDS prototype showed high provider satisfaction. Conclusion This is one of the first studies to utilize a user-centered design approach to develop and assess a PGx-CDS interface prototype for Thiopurine Methyltransferase (TPMT). This study provides guidance for the development of a PGx-CDS, and particularly for biomarkers such as TPMT.

Download Full-text