“It’s good to feel like you’re doing something”: a qualitative study examining state health department employees’ views on why ineffective programs continue to be implemented in the USA

Background Mis-implementation, the inappropriate continuation of programs or policies that are not evidence-based or the inappropriate termination of evidence-based programs and policies, can lead to the inefficient use of scarce resources in public health agencies and decrease the ability of these agencies to deliver effective programs and improve population health. Little is known about why mis-implementation occurs, which is needed to understand how to address it. This study sought to understand the state health department practitioners’ perspectives about what makes programs ineffective and the reasons why ineffective programs continue. Methods Eight state health departments (SHDs) were selected to participate in telephone-administered qualitative interviews about decision-making around ending or continuing programs. States were selected based on geographic representation and on their level of mis-implementation (low and high) categorized from our previous national survey. Forty-four SHD chronic disease staff participated in interviews, which were audio-recorded and transcribed verbatim. Transcripts were consensus coded, and themes were identified and summarized. This paper presents two sets of themes, related to (1) what makes a program ineffective and (2) why ineffective programs continue to be implemented according to SHD staff. Results Participants considered programs ineffective if they were not evidence-based or if they did not fit well within the population; could not be implemented well due to program restraints or a lack of staff time and resources; did not reach those who could most benefit from the program; or did not show the expected program outcomes through evaluation. Practitioners described several reasons why ineffective programs continued to be implemented, including concerns about damaging the relationships with partner organizations, the presence of program champions, agency capacity, and funding restrictions. Conclusions The continued implementation of ineffective programs occurs due to a number of interrelated organizational, relational, human resources, and economic factors. Efforts should focus on preventing mis-implementation since it limits public health agencies’ ability to conduct evidence-based public health, implement evidence-based programs effectively, and reduce the high burden of chronic diseases. The use of evidence-based decision-making in public health agencies and supporting adaptation of programs to improve their fit may prevent mis-implementation. Future work should identify effective strategies to reduce mis-implementation, which can optimize public health practice and improve population health.

Barriers to implementation of pediatric emergency department interventions for parental tobacco use and dependence: a qualitative study using the theoretical domains framework

Background Pediatric emergency department (PED) and urgent care (UC) professionals can play a key role in delivering evidence-based guidelines to address parental tobacco use and child tobacco smoke exposure (TSE). Understanding PED/UC professionals’ perceptions regarding these guidelines is the first step in developing and implementing a TSE screening and counseling intervention in these settings. This study aimed to use the theoretical domains framework (TDF) to identify current screening and counseling behaviors of PED/UC professionals related to parental tobacco use and child TSE, and determine barriers and enablers that influence these behaviors. Methods Semi-structured, focused interviews were conducted with 29 actively practicing PED/UC clinical staff who worked at one large, Midwestern children’s hospital. The interview guide was informed by the TDF and included open-ended questions. Content analysis of interview transcripts was guided by the TDF. Nurses, physicians, and healthcare administrators were assessed overall and by group membership to ensure each group was represented based on their varying PED/UC roles. Results Fifty-one percent were nurses, 38% were physicians, and 11% were healthcare administrators. Most PED/UC professionals did not currently follow the guidelines, but perceived addressing parental tobacco use as part of their role. All 14 TDF domains were identified by nurses, physicians, and administrators in relation to counseling for parental tobacco use and child TSE. Domains with the most sub-themes were (1) knowledge: lack of knowledge about tobacco counseling, including implementing counseling, cessation resources/referrals, and thirdhand smoke; (2) beliefs about capabilities: not comfortable counseling parents, easier to discuss with parents who are receptive and to ask and advise when patients have a TSE-related complaint, and more likely to discuss if there were resources/referrals; and (3) environmental context and resources: barriers include lack of time, training, and resources and referral information to give to parents, and an enabler is using TSE-related complaints as a context to offer counseling. Conclusions Study findings provide a strong foundation for developing and implementing clinical practice guidelines regarding parental tobacco use and child TSE in the PED/UC setting. Future intervention development will address all TDF domains and test the implementation of the intervention in the PED/UC setting.

Rates of sustainment in the Universal Stages of Implementation Completion

Background Sustainment is a desirable outcome of implementation, but its precise definition remains unclear, contributing to the difficulty of identifying a generalized rate of sustainment. Several studies and reviews on the topic differ on both definition and levels of analysis. Furthermore, methodological limitations might have influenced the results, including the unknown quality with which some interventions were delivered. The Universal Stages of Implementation Completion (UniSIC) is a standardized measurement tool that tracks the implementation process and milestone completion across a wide range of real-world implementations—this provides a unique opportunity to identify a generalized rate of sustainment. Methods UniSIC data was captured from the SIC website on 27 September 2020 and included data from all sites (n = 1778) that had been tracked to date. Data were restricted to sites that achieved competency in program delivery, and thus had a newly adopted program worthy of sustainment. Dates and indicator variables of implementation activities were combined to form two alternate definitions of sustainment: sustained (start-up) was achieved if sites continued to deliver services 2 years past their program start-up date; sustained (competent) was achieved if sites continued to deliver services 2 years past their competence and/or certification date. Of sites eligible for inclusion based on these definitions (N = 208), descriptive analyses were conducted to determine a rate of sustainment for all programs that successfully started a program. These definitions were also applied to a combined sample for a general rate of sustainment among all sites. Rates of competency among both a sample of sites that started up and a combined sample were also identified. Results The rate of competence was 58.5% and the rate of sustained (start-up) was 37.1%, while the rate of sustained (competent) was 25.1%. The rates of competence and sustainment among the combined samples were far lower: 15.6% for competence, 6.8% for sustained (start-up), and 4.4% for sustained (competent). Conclusions These identified rates of sustainment are accurate initial estimates of sustainment of community-based practices, or in general. Future research on rates of sustainment should carefully define measures of sustainment and be transparent about the real-world conditions on which analyses are centered.

Coordination of sustainable financing for evidence-based youth mental health treatments: protocol for development and evaluation of the fiscal mapping process

Background Sustained delivery of evidence-based treatments (EBTs) is essential to addressing the public health and economic impacts of youth mental health problems, but is complicated by the limited and fragmented funding available to youth mental health service agencies (hereafter, “service agencies”). Strategic planning tools are needed that can guide these service agencies in their coordination of sustainable funding for EBTs. This protocol describes a mixed-methods research project designed to (1) develop and (2) evaluate our novel fiscal mapping process that guides strategic planning efforts to finance the sustainment of EBTs in youth mental health services. Method Participants will be 48 expert stakeholder participants, including representatives from ten service agencies and their partners from funding agencies (various public and private sources) and intermediary organizations (which provide guidance and support on the delivery of specific EBTs). Aim 1 is to develop the fiscal mapping process: a multi-step, structured tool that guides service agencies in selecting the optimal combination of strategies for financing their EBT sustainment efforts. We will adapt the fiscal mapping process from an established intervention mapping process and will incorporate an existing compilation of 23 financing strategies. We will then engage participants in a modified Delphi exercise to achieve consensus on the fiscal mapping process steps and gather information that can inform the selection of strategies. Aim 2 is to evaluate preliminary impacts of the fiscal mapping process on service agencies’ EBT sustainment capacities (i.e., structures and processes that support sustainment) and outcomes (e.g., intentions to sustain). The ten agencies will pilot test the fiscal mapping process. We will evaluate how the fiscal mapping process impacts EBT sustainment capacities and outcomes using a comparative case study approach, incorporating data from focus groups and document review. After pilot testing, the stakeholder participants will conceptualize the process and outcomes of fiscal mapping in a participatory modeling exercise to help inform future use and evaluation of the tool. Discussion This project will generate the fiscal mapping process, which will facilitate the coordination of an array of financing strategies to sustain EBTs in community youth mental health services. This tool will promote the sustainment of youth-focused EBTs.

Moving beyond Aim Three: a need for a transdisciplinary approach to build capacity for economic evaluations in implementation science

Background Understanding the costs and economic benefits of implementation has been identified by policymakers and researchers as critical to increase the uptake and sustainment of evidence-based practices, but this topic remains relatively understudied. Conducting team science with health economists has been proposed as a solution to increase economic evaluation in implementation science; however, these recommendations ignore the differences in goals and perspectives in these two fields. Our recent qualitative research identified that implementation researchers predominantly approach health economists to examine costs, whereas the majority of health economists expressed limited interest in conducting economic evaluations and a desire to be more integrated within implementation science initiatives. These interviews pointed to challenges in establishing fruitful partnerships when health economists are relegated to the “Third Aim” (i.e., lowest-priority research objective) in implementation science projects by their research partners. Discussion In this debate paper, we argue that implementation researchers and health economists need to focus on team science research principles to expand capacity to address pressing research questions that cut across the two fields. Specifically, we use the four-phase model of transdisciplinary research to outline the goals and processes needed to build capacity in this area (Hall et al., Transl Behav Med 2:415–30, 2012). The first phase focuses on the development of transdisciplinary research teams, including identifying appropriate partners (e.g., considering policy or public health researchers in addition to health economists) and building trust. The conceptual phase focuses on strategies to consider when developing joint research questions and methodology across fields. In the implementation phase, we outline the effective processes for conducting research projects, such as team learning. Finally, in the translation phase, we highlight how a transdisciplinary approach between health economists and implementation researchers can impact real-world practice and policy. Summary The importance of investigating the economic impact of evidence-based practice implementation is widely recognized, but efforts have been limited due to the challenges in conducting team science across disciplines. Training in team science can help advance transdisciplinary efforts, which has the potential to increase the rigor and impact of economic evaluations in implementation science while expanding the roles taken by health economists.

Implementation of evidence-based guidance for dementia palliative care using participatory action research: examining implementation through the Consolidated Framework for Implementation Research (CFIR)

Background The importance of providing evidence-based palliative care for people with dementia is increasingly acknowledged as important for patient outcomes. In Ireland, evidence-based guidance has been developed in order to address key features of dementia palliative care, including the management of pain, medications and hydration and nutrition. The aim of this study was to identify and explore the factors affecting the implementation of evidence-based guidance on dementia palliative care. Methods The Consolidated Framework for Implementation (CFIR) guided a mixed-method pre-post study. One guidance document pertaining to the management of pain, medication or hydration and nutrition was implemented in three long-term care facilities. Participatory action research in the form of work-based learning groups was used to implement the guidance, drawing on a situational analysis (pre-implementation). Staff questionnaires and audits were conducted pre- and post-implementation while champion interviews were also conducted post-implementation. Results Features of the guidance, the inner setting components such as readiness to change, and the process of implementation were most frequently identified as impacting implementation. Components of the outer setting, such as external policy incentives and individual characteristics, featured less commonly. Data from qualitative interviews revealed that the guidance was perceived as advantageous or complimentary to previous care provided. Within the inner setting, leadership and support from other colleagues facilitated implementation. However, limited availability of other healthcare professionals to assist with carrying out guidance actions presented a barrier in some facilities. The external facilitators of the work-based learning groups (WBLGs) were perceived as experienced and encouraged active participation and reflection on practices. Despite the challenge of releasing staff to attend the WBLGs, quantitative data demonstrated reduced staff de-motivation amongst those who did attend was noted post-implementation (pre-Mdn = 19.50 versus post-Mdn = 22.00, U = 497.00, p = 0.07). Conclusions A situational analysis informed by the CFIR framework in conjunction with a participatory action research approach helped to advance the implementation of the guidance. The progress of implementation depended on the extent to which evidence-based care was previously being implemented at each site. Post-implementation analysis using CFIR identified challenges to address in future projects such as staff cover and timing of training to facilitate attendance for staff with different working hours. Facilitators included multidisciplinary engagement with the intervention and champions at each site to support the implementation process.

A theory-based exploration of antibiotic use in primary healthcare in Gezira state, Sudan

Background Inappropriate use of antibiotics is a major contributing factor to the emergence of antimicrobial resistance globally, including in Sudan. Objectives The project aimed to develop a theory-driven behaviour change strategy addressing both prescribers and patients based on factors that are driving antibiotic use in primary healthcare settings in Gezira state in Sudan. Methods The strategy was designed based on the Theoretical Domains Framework (TDF) to identify behavioural domains and the Behaviour Change Wheel (BCW) to select appropriate intervention functions. The process included (1) a formative qualitative research study and (2) a knowledge co-production workshop that utilized the results of the qualitative study to design a salient, appropriate, and credible behaviour change strategy. Results The TDF domains related to prescribers that emerged from the study included knowledge, skills, and intention. The selected BCW intervention functions included education, training, modelling, and persuasion. The main TDF domains related to patients included social influences and intention. The selected BCW intervention functions included enablement and education. Conclusion Using the TDF and BCW intervention functions, the study identified behavioural domains that influence antibiotic prescription and consumption in rural primary healthcare settings in Gezira state in Sudan and appropriate intervention functions to modify these behaviours. Knowledge co-production ensured that the evidence-based strategy was acceptable and practical in the local context.

A mixed-method evaluation of the adoption and implementation of the College Alcohol Intervention Matrix among prevention experts: a study protocol

Background Risky drinking among college campuses has been a long-standing concern and there have been dedicated efforts to develop evidence-based prevention and treatment strategies (EBSs) to decrease alcohol use and increase healthy behaviors among college students. Further, the College Alcohol Intervention Matrix (CollegeAIM) was developed as a tool with accompanying resources, to assist institutions of higher education in selecting EBSs that are appropriate and a good fit for their campuses. However, the CollegeAIM tool and selection of prevention strategies from stakeholders’ perspectives has yet to be evaluated. This study protocol describes the methodology for a research project evaluating CollegeAIM from an implementation science perspective using the Exploration, Preparation, Implementation, and Sustainment framework. Methods The aims of this study will be accomplished with a mixed-method design comprised of reviews of strategic planning documents, quantitative surveys and interviews with prevention experts, and focus groups to identify key components of a decision-support program for prevention experts to support the use of CollegeAIM. Participants are members of the multi-site Missouri Partners in Prevention coalition to reduce risky substance use on college campuses across the state. Discussion The results of this study will provide key information to support the development of additional supportive tools for campuses that can improve their selection and implementation of EBSs that fit the needs of their respective campuses. This work is important to further advance the implementation and sustainment of extant EBSs for risky college alcohol use.

Strategies to implement SARS-CoV-2 point-of-care testing into primary care settings: a qualitative secondary analysis guided by the Behaviour Change Wheel

Background The purpose of this study is to develop a theory-driven understanding of the barriers and facilitators underpinning physicians’ attitudes and capabilities to implementing SARS-CoV-2 point-of-care (POC) testing into primary care practices. Methods We used a secondary qualitative analysis approach to re-analyse data from a qualitative, interview study of 22 primary care physicians from 21 primary care practices across three regions in England. We followed the three-step method based on the Behaviour Change Wheel to identify the barriers to implementing SARS-CoV-2 POC testing and identified strategies to address these challenges. Results Several factors underpinned primary care physicians’ attitudes and capabilities to implement SARS-CoV-2 POC testing into practice. First, limited knowledge of the SARS-CoV-2 POC testing landscape and a demanding workload affected physicians’ willingness to use the tests. Second, there was scepticism about the insufficient evidence pertaining to the clinical efficacy and utility of POC tests, which affected physicians’ confidence in the accuracy of tests. Third, physicians would adopt POC tests if they were prescribed and recommended by authorities. Fourth, physicians required professional education and training to increase their confidence in using POC tests but also suggested that healthcare assistants should administer the tests. Fifth, physicians expressed concerns about their limited workload capacity and that extra resources are needed to accommodate any anticipated changes. Sixth, information sharing across practices shaped perceptions of POC tests and the quality of information influenced physician perceptions. Seventh, financial incentives could motivate physicians and were also needed to cover the associated costs of testing. Eighth, physicians were worried that society will view primary care as an alternative to community testing centres, which would change perceptions around their professional identity. Ninth, physicians’ perception of assurance/risk influenced their willingness to use POC testing if it could help identify infectious individuals, but they were also concerned about the risk of occupational exposure and potentially losing staff members who would need to self-isolate. Conclusions Improving primary care physicians’ knowledgebase of SARS-CoV-2 POC tests, introducing policies to embed testing into practice, and providing resources to meet the anticipated demands of testing are critical to implementing testing into practice.

A systematic review of randomized control trials of HPV self-collection studies among women in sub-Saharan Africa using the RE-AIM framework

Introduction Self-collection of samples for HPV testing may increase women’s access to cervical cancer screening in low- and middle-income settings. However, implementation remains poor in many regions. The purpose of this systematic review was to examine implementation data from randomized controlled trials evaluating human papillomavirus (HPV) self-collection testing among women in sub-Saharan Africa using the RE-AIM (Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance) framework. Methods We searched four electronic databases (PubMed, CINAHL, Web of Science, and Global Health) for pragmatic randomized controlled trials that promote HPV self-collection among women in sub-Saharan Africa. Study selection and data extraction were conducted according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) checklist. Two researchers independently extracted information from each article using a RE-AIM data extraction tool. The reporting of RE-AIM dimensions was summarized and synthesized across included interventions. Results We identified 2008 citations, and eight studies were included. These reported on five unique interventions. The five interventions were conducted in five countries: Cameroon, Ethiopia, Kenya, Nigeria, and Uganda. Intervention reach (80%) was the most commonly reported RE-AIM dimension, followed by adoption (56%), efficacy/effectiveness (52%), implementation (47%), and maintenance (0%). All the interventions described increased uptake of HPV testing among study participants (effectiveness). However, the majority of the studies focused on reporting internal validity indicators such as inclusion criteria (100%) and exclusion criteria (100%), and few reported on external validity indicators such as participation rate (40%), intervention cost (40%), staff selection (20%), and cost of maintenance (0%). Conclusions Our review highlights the under-reporting of external validity indicators such as participation rate, intervention, and maintenance costs in studies of self-collection for HPV testing among women in SSA. Future research should focus on including factors that highlight internal validity factors and external validity factors to develop a greater understanding of ways to increase not only reach but also implementation and long-term maintenance of these interventions. Such data may advance the translation of HPV interventions into practice and reduce health disparities in SSA. Findings highlight the need for innovative tools such as participatory learning approaches or open challenges to expand knowledge and assessment of external validity indicators to ultimately increase the uptake of HPV testing among women in SSA.