Assessing and Strengthening Evidence-Based Program Registries’ Usefulness for Social Service Program Replication and Adaptation

Background: Government and private funders increasingly require social service providers to adopt program models deemed “evidence based,” particularly as defined by evidence-based program registries, such as What Works Clearinghouse and National Registry of Evidence-Based Programs and Practices. These registries summarize the evidence about programs’ effectiveness, giving near-exclusive priority to evidence from experimental-design evaluations. The registries’ goal is to aid decision making about program replication, but critics suspect the emphasis on evidence from experimental-design evaluations, while ensuring strong internal validity, may inadvertently undermine that goal, which requires strong external validity as well. Objective: The objective of this study is to determine the extent to which the registries’ reports provide information about context-specific program implementation factors that affect program outcomes and would thus support decision making about program replication and adaptation. Method: A research-derived rubric was used to rate the extent of context-specific reporting in the population of seven major registries’ evidence summaries ( N = 55) for youth development programs. Findings: Nearly all (91%) of the reports provide context-specific information about program participants, but far fewer provide context-specific information about implementation fidelity and other variations in program implementation (55%), the program’s environment (37%), costs (27%), quality assurance measures (22%), implementing agencies (19%), or staff (15%). Conclusion: Evidence-based program registries provide insufficient information to guide context-sensitive decision making about program replication and adaptation. Registries should supplement their evidence base with nonexperimental evaluations and revise their methodological screens and synthesis-writing protocols to prioritize reporting—by both evaluators and the registries themselves—of context-specific implementation factors that affect program outcomes.

Download Full-text

Poor availability of context-specific evidence hampers decision-making in conservation

10.1101/2020.02.13.946954 ◽

2020 ◽

Author(s):

Alec P. Christie ◽

Tatsuya Amano ◽

Philip A. Martin ◽

Silviu O. Petrovan ◽

Gorm E. Shackelford ◽

...

Keyword(s):

Decision Making ◽

Step Change ◽

Evidence Based ◽

Bird Conservation ◽

Robust Designs ◽

Relevant Evidence ◽

Local Contexts ◽

Context Specific ◽

Study Designs ◽

Conservation Interventions

AbstractEvidence-based conservation relies on robust and relevant evidence. Practitioners often prefer locally relevant studies whose results are more likely to be transferable to the context of planned conservation interventions. To quantify the availability of relevant evidence for amphibian and bird conservation we reviewed Conservation Evidence, a database of quantitative tests of conservation interventions. Studies were geographically clustered and found at extremely low densities - fewer than one study was present within a 2,000 km radius of a given location. The availability of relevant evidence was extremely low when we restricted studies to those studying biomes or taxonomic orders containing high percentages of threatened species, compared to the most frequently studied biomes and taxonomic orders. Further constraining the evidence by study design showed that only 17-20% of amphibian and bird studies used robust designs. Our results highlight the paucity of evidence on the effectiveness of conservation interventions, and the disparity in evidence for local contexts that are frequently studied and those where conservation needs are greatest. Addressing the serious global shortfall in context-specific evidence requires a step change in the frequency of testing conservation interventions, greater use of robust study designs and standardized metrics, and methodological advances to analyze patchy evidence bases.

Download Full-text

Rugged landscapes: complexity and implementation science

Implementation Science ◽

10.1186/s13012-020-01028-5 ◽

2020 ◽

Vol 15 (1) ◽

Author(s):

Joseph T. Ornstein ◽

Ross A. Hammond ◽

Margaret Padek ◽

Stephanie Mazzucca ◽

Ross C. Brownson

Keyword(s):

Public Health ◽

Decision Making ◽

Quality Improvement ◽

Evolutionary Biology ◽

Public Health Practice ◽

Evidence Based ◽

Health Practice ◽

Context Specific ◽

Evidence Based Programs ◽

Evidence Based Decision Making

Abstract Background Mis-implementation—defined as failure to successfully implement and continue evidence-based programs—is widespread in public health practice. Yet the causes of this phenomenon are poorly understood. Methods We develop an agent-based computational model to explore how complexity hinders effective implementation. The model is adapted from the evolutionary biology literature and incorporates three distinct complexities faced in public health practice: dimensionality, ruggedness, and context-specificity. Agents in the model attempt to solve problems using one of three approaches—Plan-Do-Study-Act (PDSA), evidence-based interventions (EBIs), and evidence-based decision-making (EBDM). Results The model demonstrates that the most effective approach to implementation and quality improvement depends on the underlying nature of the problem. Rugged problems are best approached with a combination of PDSA and EBI. Context-specific problems are best approached with EBDM. Conclusions The model’s results emphasize the importance of adapting one’s approach to the characteristics of the problem at hand. Evidence-based decision-making (EBDM), which combines evidence from multiple independent sources with on-the-ground local knowledge, is a particularly potent strategy for implementation and quality improvement.

Download Full-text

Facilitators and Barriers to Adopting Evidence-Based Physical Education in Elementary Schools

Journal of Physical Activity and Health ◽

10.1123/jpah.8.s1.s17 ◽

2011 ◽

Vol 8 (s1) ◽

pp. S17-S25 ◽

Cited By ~ 37

Author(s):

Monica A.F. Lounsbery ◽

Thomas L. McKenzie ◽

Stewart Trost ◽

Nicole J. Smith

Keyword(s):

Decision Making ◽

Program Evaluation ◽

Physical Education ◽

Teacher Perceptions ◽

Evidence Based ◽

Program Outcomes ◽

Chi Square ◽

Adoption Decision ◽

Signed Rank ◽

Facilitators And Barriers

Background:Evidence-based physical education (EBPE) programs have increased physical activity (PA) by as much as 18%, yet widespread adoption has not occurred. Understanding school facilitators and barriers to PE should prove useful to EBPE dissemination efforts.Methods:Pairs of principals and PE teachers from 154 schools (75 Adopters and 79 Non-Adopters) from 34 states completed questionnaires. Differences between Adopter and Non-Adopter schools were tested using t tests or Wilcoxon Signed Rank Tests and chi-square analyses.Results:Principals and teachers reported distinct PE curriculum adoption decision making roles, but few viewed themselves as very involved in program evaluation. Teachers in Adopter schools were more satisfied with PE program outcomes and had greater involvement in teacher evaluation and program decision making. Compared with teachers, principals were generally more satisfied with their school’s PE program outcomes and did not share the same perceptions of PE barriers. However, principals also demonstrated a general lack of PE program familiarity.Conclusions:To facilitate EBPE adoption, dissemination efforts should target both principals and PE teachers. Increasing principal’s knowledge may be instrumental in addressing some teacher perceptions of barriers to PE. Strategic advocacy efforts, including targeting policies that require PE program evaluation, are needed.

Download Full-text

Supporting Climate-Resilient Planning at National and District Levels: A Pathway to Multi-stakeholder Decision-Making in Uganda

Climate Risk in Africa ◽

10.1007/978-3-030-61160-6_8 ◽

2021 ◽

pp. 131-145

Author(s):

Rosalind J. Cornforth ◽

Celia Petty ◽

Grady Walker

Keyword(s):

Decision Making ◽

Rural Communities ◽

Specific Information ◽

Visual Methods ◽

Informed Decision Making ◽

Multi Stakeholder ◽

Context Specific ◽

Qualitative Methodologies ◽

Inform Decision Making ◽

Insight Into

AbstractIf rural adaptation is to be effective, then it cannot take the form of prescriptive actions determined by outsiders and subsequently imposed upon rural communities. Our focus in this chapter is to reflect on the effectiveness of rural adaptation in the context of food security and agriculture in Uganda and provide insight into a way forward using learning from the HyCRISTAL project rural pilot. We critically explore the boundaries of ‘adaptation’ and ‘resilience’ as policy responses to climate change in poor rural communities through the interdisciplinary use of quantitative and qualitative methodologies, including innovative visual methods and action research. We identify some of the limits to building adaptive communities and explore potential solutions for enabling informed decision-making for rural adaptation that are linked to investment in sustainable development. We highlight the importance of multi-stakeholder approaches and the generation of a ‘knowledge ecosystem’ that combines physical and social science methods and data to generate context-specific information to inform decision-making.

Download Full-text

Using Mixed Methods to Adapt and Evaluate the Implementation of a Multicomponent Tobacco-Free Workplace Program within Behavioral Health Centers

10.21203/rs.3.rs-18558/v1 ◽

2020 ◽

Author(s):

Isabel Martinez Leal ◽

Tzu-An Chen ◽

Virmarie Correa-Fernández ◽

Kathy Le ◽

Daniel P. O’Connor ◽

...

Keyword(s):

Mixed Methods ◽

Behavioral Health ◽

Program Implementation ◽

Organizational Capacity ◽

Tobacco Dependence ◽

Routine Practice ◽

Evidence Based ◽

Program Outcomes ◽

Evidence Based Practices ◽

Local Contexts

Abstract Background: Behavioral health treatment centers (BHTCs) rarely implement existing evidence-based practices for treating tobacco dependence, despite high rates of tobacco use among their clients. Taking Texas Tobacco Free (TTTF) has successfully targeted this disparity by delivering an evidence-based, multilevel, tobacco-free workplace program providing policy implementation and enforcement, education, provider training in tobacco screenings and treatments, and nicotine replacement therapies (NRT) to BHTCs across Texas. We describe a mixed methods design used to conduct a formative evaluation process to adapt implementation strategies to local contexts, evaluate program outcomes and characterize processes influencing program implementation in two BHTCs serving 17 clinics.Methods: Varied data collection included pre and post-implementation leader, provider, and staff surveys; and pre, mid, and post-implementation provider, staff and consumer focus groups. During implementation, data were collected via various logs (tobacco screenings, NRT delivery) to monitor program content delivery. The RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework guided translation of behavioral interventions into sustainable practice.Results: While program implementation varied between each BHTC, all clinics adopted a 100% tobacco-free workplace policy, integrated tobacco screenings into routine practice, delivered evidence-based interventions, dispensed NRT to consumers and staff, and increased provider knowledge on how to address tobacco dependence. Pre, mid, and post-implementation qualitative findings served to: 1) develop program strategies and materials adapted to local contexts and populations and address barriers; 2) adjust delivery systems of key components to enhance implementation; 3) understand reasons for success or failure to implement specific practices; and 4) reveal program integration into clinic culture, enhancing sustainability.Conclusions: Implementation of TTTF at both BHTCs increased organizational capacity in the provision of evidence-based practices to treat tobacco dependence through successfully meeting most of our RE-AIM targets. Mixing methods involved program adopters and recipients as collaborators directly shaping core interventions to their individual context and needs, thus increasing program fit, ownership, adoption and sustainability; closing the gap between research and practice. These findings contribute to the development of flexible strategies and interventions capable of addressing variable implementation contexts and barriers, thus enhancing the effectiveness and sustainability of a tobacco-free workplace program.

Download Full-text

Abstraction: An alternative neurocognitive account of recognition, prediction, and decision making

Behavioral and Brain Sciences ◽

10.1017/s0140525x19003017 ◽

2020 ◽

Vol 43 ◽

Author(s):

Valerie F. Reyna ◽

David A. Broniatowski

Keyword(s):

Decision Making ◽

Software Engineering ◽

Systems Engineering ◽

Evidence Based ◽

Extensive Literature ◽

Trace Theory ◽

Fuzzy Trace Theory ◽

Abstract Representations ◽

Technical Systems

Abstract Gilead et al. offer a thoughtful and much-needed treatment of abstraction. However, it fails to build on an extensive literature on abstraction, representational diversity, neurocognition, and psychopathology that provides important constraints and alternative evidence-based conceptions. We draw on conceptions in software engineering, socio-technical systems engineering, and a neurocognitive theory with abstract representations of gist at its core, fuzzy-trace theory.

Download Full-text

Debates in Dysphagia Management: How Do You Use Evidence-Based Practice in Your Dysphagia Patient Care?

Perspectives on Swallowing and Swallowing Disorders (Dysphagia) ◽

10.1044/sasd20.4.121 ◽

2011 ◽

Vol 20 (4) ◽

pp. 121-123

Author(s):

Jeri A. Logemann

Keyword(s):

Decision Making ◽

Patient Care ◽

Clinical Outcomes ◽

Clinical Judgment ◽

Clinical Decision Making ◽

Evidence Based Practice ◽

Clinical Decision ◽

Evidence Based ◽

Clinical Method ◽

Do So

Evidence-based practice requires astute clinicians to blend our best clinical judgment with the best available external evidence and the patient's own values and expectations. Sometimes, we value one more than another during clinical decision-making, though it is never wise to do so, and sometimes other factors that we are unaware of produce unanticipated clinical outcomes. Sometimes, we feel very strongly about one clinical method or another, and hopefully that belief is founded in evidence. Some beliefs, however, are not founded in evidence. The sound use of evidence is the best way to navigate the debates within our field of practice.

Download Full-text