Using Implementation Fidelity to Aid in Interpreting Program Impacts: A Brief Review

2019 ◽  
Vol 48 (9) ◽  
pp. 590-598 ◽  
Author(s):  
Heather C. Hill ◽  
Anna Erickson
Keyword(s):  
Program Implementation ◽  
Student Outcomes ◽  
Implementation Fidelity ◽  
Educational Interventions ◽  
Fidelity Level ◽  
Program Fidelity ◽  
Null Results ◽  
Size Type ◽  
Two Measures ◽  
Cluster Randomized

Poor program implementation constitutes one explanation for null results in trials of educational interventions. For this reason, researchers often collect data about implementation fidelity when conducting such trials. In this article, we document whether and how researchers report and measure program fidelity in recent cluster-randomized trials. We then create two measures—one describing the level of fidelity reported by authors and another describing whether the study reports null results—and examine the correspondence between the two. We also explore whether fidelity is influenced by study size, type of fidelity measured and reported, and features of the intervention. We find that as expected, fidelity level relates to student outcomes; we also find that the presence of new curriculum materials positively predicts fidelity level.

scholarly journals Making Every Study Count: Learning From Replication Failure to Improve Intervention Research

2019 ◽  
Vol 48 (9) ◽  
pp. 599-607 ◽  
Author(s):  
James S. Kim
Keyword(s):  
Program Implementation ◽  
Student Outcomes ◽  
Positive Impact ◽  
Educational Interventions ◽  
Intervention Research ◽  
Theory Of Change ◽  
Future Research ◽  
Efficacy Trials ◽  
Replication Failure

Why, when so many educational interventions demonstrate positive impact in tightly controlled efficacy trials, are null results common in follow-up effectiveness trials? Using case studies from literacy, this article suggests that replication failure can surface hidden moderators—contextual differences between an efficacy and an effectiveness trial—and generate new hypotheses and questions to guide future research. First, replication failure can reveal systemic barriers to program implementation. Second, it can highlight for whom and in what contexts a program theory of change works best. Third, it suggests that a fidelity first and adaptation second model of program implementation can enhance the effectiveness of evidence-based interventions and improve student outcomes. Ultimately, researchers can make every study count by learning from both replication success and failure to improve the rigor, relevance, and reproducibility of intervention research.

After the Race to the Top: State and District Capacity to Sustain Professional Development Innovation in Florida

2021 ◽  
pp. 089590482110156
Author(s):  
Motoko Akiba ◽  
Cassandra Howard
Keyword(s):  
Professional Development ◽  
Lesson Study ◽  
Student Outcomes ◽  
Instructional System ◽  
Implementation Fidelity ◽  
Race To The Top ◽  
Mixed Methods Study ◽  
Policy Implications ◽  
Federally Funded ◽  
Strategic Adaptation

The Race to the Top (RTTT) program incentivized states to use innovation for systemwide improvement of student outcomes, but little is known about how RTTT-funded innovation was sustained after the RTTT program ended. This mixed-methods study examined state and district approaches to sustaining an international innovation called lesson study, a teacher-driven, collaborative, inquiry-based teacher learning process imported from Japan and promoted statewide in Florida. While the state’s role in sustaining lesson study was limited, we found that districts that integrated lesson study into the district instructional system through a clear expectation and strategic adaptation, supported school and teacher ownership of lesson study practice, and provided necessary support and funding were more likely to sustain lesson study. In contrast, the districts that focused on implementation fidelity and district-led facilitation eventually phased out lesson study. Policy implications for sustaining federally funded professional development innovations are discussed.

Implementation Fidelity and the Design of a Fractions Intervention

2019 ◽  
Vol 42 (4) ◽  
pp. 217-230 ◽  
Author(s):  
Lindy Crawford ◽  
Barbara Freeman ◽  
Jacqueline Huscroft-D’Angelo ◽  
Sarah Quebec Fuentes ◽  
Kristina N. Higgins
Keyword(s):  
Learning Disabilities ◽  
Conceptual Understanding ◽  
Student Outcomes ◽  
Implementation Fidelity ◽  
Learning Opportunities ◽  
Students With Learning Disabilities ◽  
Mathematics Intervention ◽  
Tier Ii ◽  
Higher Order Learning ◽  
And Mathematics

Interventions are implemented with greater fidelity when their core intent is made explicit. The core intent of this intervention was to increase access to higher order learning opportunities for students with learning disabilities or difficulties in mathematics through use of research and practice from the fields of special education and mathematics education. Four steps undertaken in the development of a Tier II fraction-based mathematics intervention designed to improve the conceptual understanding of students with learning disabilities or difficulties are described in this article: (a) articulation of a logic model, (b) delineation of intervention components, (c) analysis of reliability data related to implementation fidelity, and (d) pilot testing to measure implementation fidelity and student outcomes. Results of the pilot study demonstrated no significant effect for the component of technology; however, significant pre–post differences were found in the performance of all groups on their conceptual understanding of fractions as numbers.

The Impact of Implementation Fidelity on Student Outcomes in the Life Skills Training Program

2019 ◽  
Vol 12 (1) ◽  
pp. 113-123
Author(s):  
Enya B. Vroom ◽  
Oliver T. Massey ◽  
Svetlana Yampolskaya ◽  
Bruce L. Levin
Keyword(s):  
Training Program ◽  
Student Outcomes ◽  
Life Skills ◽  
Implementation Fidelity ◽  
Skills Training ◽  
Life Skills Training ◽  
The Impact

scholarly journals Learning From Cluster Randomized Trials in Education: An Assessment of the Capacity of Studies to Determine What Works, For Whom, and Under What Conditions

2020 ◽  
Vol 42 (3) ◽  
pp. 354-374
Author(s):  
Jessaca Spybrook ◽  
Qi Zhang ◽  
Ben Kelcey ◽  
Nianbo Dong
Keyword(s):  
Randomized Trials ◽  
Educational Interventions ◽  
School Type ◽  
Cluster Randomized Trials ◽  
Individual Level ◽  
The Past ◽  
Moderator Effects ◽  
What Works ◽  
Cluster Randomized ◽  
Cluster Level

Over the past 15 years, we have seen an increase in the use of cluster randomized trials (CRTs) to test the efficacy of educational interventions. These studies are often designed with the goal of determining whether a program works, or answering the what works question. Recently, the goals of these studies expanded to include for whom and under what conditions an intervention is effective. In this study, we examine the capacity of a set of CRTs to provide rigorous evidence about for whom and under what conditions an intervention is effective. The findings suggest that studies are more likely to be designed with the capacity to detect potentially meaningful individual-level moderator effects, for example, gender, than cluster-level moderator effects, for example, school type.

Teacher-Led Math Inquiry: A Cluster Randomized Trial in Belize

2018 ◽  
Vol 40 (3) ◽  
pp. 336-358 ◽  
Author(s):  
Darrell M. Hull ◽  
Krystal M. Hinerman ◽  
Sarah L. Ferguson ◽  
Qi Chen ◽  
Emma I. Näslund-Hadley
Keyword(s):  
Primary Schools ◽  
Multilevel Models ◽  
Implementation Fidelity ◽  
Cluster Randomized Trial ◽  
Policy Implications ◽  
Intention To Treat ◽  
School Year ◽  
Teacher Professional ◽  
Cluster Randomized ◽  
Fidelity Measures

Teacher professional development and in-class mentors were used to support structured inquiry with math manipulatives. Twenty-four primary schools (n = 6,628 students) were randomly assigned to treatment and control groups as an experimental field trial to examine the effectiveness of this instructional approach in a scaled-up application in Belize for the duration of a school year. Implementation fidelity measures were collected permitting evaluation of two separate multilevel models: intention-to-treat and test-of-treatment. Both quantitative and qualitative evidence suggest students within this culture respond well to this relatively simple and inexpensive intervention that departs from traditional, expository math instruction in many developing countries. Policy implications are discussed that supported nationwide rollout of the intervention.

Cancer screening, patient characteristics, and shared decision making (SDM): A complicated equation.

2017 ◽  
Vol 35 (8_suppl) ◽  
pp. 164-164
Author(s):  
James Austin Talcott ◽  
Maureen Bezuhly ◽  
Gina Aharanoff ◽  
Danielle Seiden ◽  
Michael P. Osborne ◽  
...  
Keyword(s):  
Cancer Screening ◽  
Randomized Trial ◽  
Specific Antigen ◽  
Cluster Randomized Trial ◽  
Patient Characteristics ◽  
Clinical Decision ◽  
Educational Interventions ◽  
Highly Educated ◽  
Screening Guideline ◽  
Cluster Randomized

164 Background: SDM may vary by patient and clinical decision. In a randomized trial of physician (MD) educational interventions to improve cancer screening guideline compliance, we studied patient (PT) and MD recall of discussions and factors that affected recall. Methods: In a cluster-randomized trial of educational supports for MDs, we are enrolling an age- (30-89 years) and sex-stratified sample of 216 PTs who underwent a physical examination at 2 urban hospitals, 18 for each of 12 primary care MDs. Screening guideline formatting (color-coding) and academic detailing were randomly assigned in a 2x2 design. Immediate post-encounter surveys recorded PT and MD recall of screening discussions. Results: The first 174 participants were diverse (63% white) and highly educated (77% college degree). PTs and MDs differed in recall of screening discussions, and the differences varied by screening test. When MDs reported a colorectal cancer (CRC) screening discussion, 21% of PTs did not; 20% of MDs disagreed when PTs reported the discussion. The discrepancies were greater for prostate specific antigen (PSA) screening, 29% and 29%, respectively, but much less for mammograms (MAM), 8% and 5%, respectively. Recall of the MD recommendation also differed: 15% of PTs disagreed when their MD reported it, and 33% of MDs when their PT reported it. For PSA, disagreement was 26% and 33%, respectively, and for MAM, disagreement was 17% and 10%, respectively. Overall, agreement between all PTs and MDs on whether screening was recommended was fair for CRC, PSA and MAM: kappa = 0.33, 0.34 and 0.29, respectively. Older PT age ( > 70) sharply eroded agreement on recalled CRC and PSA recommendations (kappa = -0.02 and -0.03, respectively) but much higher for MAM (kappa = 0.39). Recall concordance improved when elements of SDM were recalled. For CRC, kappa rose from -0.12 to 0.52 if the MD recalled any MDM element. Conclusions: In a highly educated, diverse PT population, PTs and MDs surveyed immediately after their encounter often disagreed on what had occurred in screening discussions. Disagreement was greater for CRC and PSA vs. MAM, greater for older PTs but improved when SDM was recalled. Clinical trial information: NCT02430948.

Dividing by Zero: Exploring Null Results in a Mathematics Professional Development Program

2018 ◽  
Vol 120 (6) ◽  
pp. 1-42
Author(s):  
Heather C. Hill ◽  
Douglas Lyman Corey ◽  
Robin T. Jacob
Keyword(s):  
Professional Development ◽  
Student Outcomes ◽  
Large Fraction ◽  
Research Question ◽  
Development Program ◽  
Cluster Randomized Trial ◽  
Professional Development Program ◽  
Multiple Sources ◽  
Null Results

Background/Context Since 2002, U.S. federal funding for educational research has favored the development and rigorous testing of interventions designed to improve student outcomes. However, recent reviews suggest that a large fraction of the programs developed and rigorously tested in the past decade have shown null results on student outcomes and, often, intermediate variables. Scholars reporting on null results often explain such results by citing factors they informally identified while they either delivered or observed the program. Purpose/Objective/Research Question/Focus of Study In this paper, we argue for a more systematic approach to examining null results, and develop a framework for evaluating null results based on the policy implementation literature. We then illustrate this approach by examining why one professional development program failed to show impacts on measures of teaching and student learning in a recent study. Setting The professional development program took place in a mid-sized urban school district on the East Coast. The provider was fully scaled up, capable of providing professional development in most U.S. locations. Research Design The main study of this program was conducted as a cluster randomized trial with 105 teachers in 18 schools. Here, we engage in a qualitative case study, using multiple sources of evidence to assess the likelihood that specific reasons for null results are valid. Data Collection and Analysis The case study sources of evidence include observations of professional development; teacher surveys and logs; transcribed videos of teachers’ mathematics instruction; and teacher interviews. Findings/Results Our analysis suggested that null impacts could result from district priorities and instructional guidance that compete with professional development goals; weaknesses in the intervention as well as its fit to teachers’ needs; and the difficulty of implementing ambitious instructional practice. Conclusions/Recommendations Our findings suggest the need for further elaboration of the null-results framework. They also suggest that professional development providers consider both (a) both the organizations in which programs operate and (b) fit of the program to teachers’ needs as potential barriers to success.

Sign in / Sign up

Export Citation Format

Share Document