An examination of the association between MTSS implementation fidelity measures and student outcomes

There are several widely used Positive Behavioral Interventions and Supports (PBIS) implementation fidelity measures, including the Schoolwide Evaluation Tool (SET), but there is limited empirical investigation substantiating the association between specific fidelity cut point scores and student outcomes. This article examined different SET cut points in reference to school-level outcomes across 180 elementary, middle, and high schools participating in randomized trials of PBIS. Specifically, we examined the implementation fidelity achieved in the sample, as well as within each school level (i.e., elementary, middle, and high). Using the binary complier index, we examined the association of each of three SET cut points with behavioral and academic outcomes using t tests and receiver operating characteristic (ROC) curve analyses. Although the t tests did not provide confirmation of one specific cut point, post hoc analyses suggested that the cut point may need to be tailored to school levels. There was moderate convergence between implementation on one scale (i.e., responses to behavioral violations) and academic achievement, based on ROC curve analyses for the 70% and 80% cut points. Implications for research and practice are discussed.

Download Full-text

After the Race to the Top: State and District Capacity to Sustain Professional Development Innovation in Florida

Educational Policy ◽

10.1177/08959048211015619 ◽

2021 ◽

pp. 089590482110156

Author(s):

Motoko Akiba ◽

Cassandra Howard

Keyword(s):

Professional Development ◽

Lesson Study ◽

Student Outcomes ◽

Instructional System ◽

Implementation Fidelity ◽

Race To The Top ◽

Mixed Methods Study ◽

Policy Implications ◽

Federally Funded ◽

Strategic Adaptation

The Race to the Top (RTTT) program incentivized states to use innovation for systemwide improvement of student outcomes, but little is known about how RTTT-funded innovation was sustained after the RTTT program ended. This mixed-methods study examined state and district approaches to sustaining an international innovation called lesson study, a teacher-driven, collaborative, inquiry-based teacher learning process imported from Japan and promoted statewide in Florida. While the state’s role in sustaining lesson study was limited, we found that districts that integrated lesson study into the district instructional system through a clear expectation and strategic adaptation, supported school and teacher ownership of lesson study practice, and provided necessary support and funding were more likely to sustain lesson study. In contrast, the districts that focused on implementation fidelity and district-led facilitation eventually phased out lesson study. Policy implications for sustaining federally funded professional development innovations are discussed.

Download Full-text

Implementation Fidelity and the Design of a Fractions Intervention

Learning Disability Quarterly ◽

10.1177/0731948719840774 ◽

2019 ◽

Vol 42 (4) ◽

pp. 217-230 ◽

Cited By ~ 1

Author(s):

Lindy Crawford ◽

Barbara Freeman ◽

Jacqueline Huscroft-D’Angelo ◽

Sarah Quebec Fuentes ◽

Kristina N. Higgins

Keyword(s):

Learning Disabilities ◽

Conceptual Understanding ◽

Student Outcomes ◽

Implementation Fidelity ◽

Learning Opportunities ◽

Students With Learning Disabilities ◽

Mathematics Intervention ◽

Tier Ii ◽

Higher Order Learning ◽

And Mathematics

Interventions are implemented with greater fidelity when their core intent is made explicit. The core intent of this intervention was to increase access to higher order learning opportunities for students with learning disabilities or difficulties in mathematics through use of research and practice from the fields of special education and mathematics education. Four steps undertaken in the development of a Tier II fraction-based mathematics intervention designed to improve the conceptual understanding of students with learning disabilities or difficulties are described in this article: (a) articulation of a logic model, (b) delineation of intervention components, (c) analysis of reliability data related to implementation fidelity, and (d) pilot testing to measure implementation fidelity and student outcomes. Results of the pilot study demonstrated no significant effect for the component of technology; however, significant pre–post differences were found in the performance of all groups on their conceptual understanding of fractions as numbers.

Download Full-text

The Impact of Implementation Fidelity on Student Outcomes in the Life Skills Training Program

School Mental Health ◽

10.1007/s12310-019-09333-1 ◽

2019 ◽

Vol 12 (1) ◽

pp. 113-123

Author(s):

Enya B. Vroom ◽

Oliver T. Massey ◽

Svetlana Yampolskaya ◽

Bruce L. Levin

Keyword(s):

Training Program ◽

Student Outcomes ◽

Life Skills ◽

Implementation Fidelity ◽

Skills Training ◽

Life Skills Training ◽

The Impact

Download Full-text

Teacher-Led Math Inquiry: A Cluster Randomized Trial in Belize

Educational Evaluation and Policy Analysis ◽

10.3102/0162373718768430 ◽

2018 ◽

Vol 40 (3) ◽

pp. 336-358 ◽

Cited By ~ 4

Author(s):

Darrell M. Hull ◽

Krystal M. Hinerman ◽

Sarah L. Ferguson ◽

Qi Chen ◽

Emma I. Näslund-Hadley

Keyword(s):

Primary Schools ◽

Multilevel Models ◽

Implementation Fidelity ◽

Cluster Randomized Trial ◽

Policy Implications ◽

Intention To Treat ◽

School Year ◽

Teacher Professional ◽

Cluster Randomized ◽

Fidelity Measures

Teacher professional development and in-class mentors were used to support structured inquiry with math manipulatives. Twenty-four primary schools (n = 6,628 students) were randomly assigned to treatment and control groups as an experimental field trial to examine the effectiveness of this instructional approach in a scaled-up application in Belize for the duration of a school year. Implementation fidelity measures were collected permitting evaluation of two separate multilevel models: intention-to-treat and test-of-treatment. Both quantitative and qualitative evidence suggest students within this culture respond well to this relatively simple and inexpensive intervention that departs from traditional, expository math instruction in many developing countries. Policy implications are discussed that supported nationwide rollout of the intervention.

Download Full-text

Using Implementation Fidelity to Aid in Interpreting Program Impacts: A Brief Review

Educational Researcher ◽

10.3102/0013189x19891436 ◽

2019 ◽

Vol 48 (9) ◽

pp. 590-598 ◽

Cited By ~ 4

Author(s):

Heather C. Hill ◽

Anna Erickson

Keyword(s):

Program Implementation ◽

Student Outcomes ◽

Implementation Fidelity ◽

Educational Interventions ◽

Fidelity Level ◽

Program Fidelity ◽

Null Results ◽

Size Type ◽

Two Measures ◽

Cluster Randomized

Poor program implementation constitutes one explanation for null results in trials of educational interventions. For this reason, researchers often collect data about implementation fidelity when conducting such trials. In this article, we document whether and how researchers report and measure program fidelity in recent cluster-randomized trials. We then create two measures—one describing the level of fidelity reported by authors and another describing whether the study reports null results—and examine the correspondence between the two. We also explore whether fidelity is influenced by study size, type of fidelity measured and reported, and features of the intervention. We find that as expected, fidelity level relates to student outcomes; we also find that the presence of new curriculum materials positively predicts fidelity level.

Download Full-text

Promoting Learning for a Student with a Severe Disability Through Paraprofessional Training

Research and Practice for Persons with Severe Disabilities ◽

10.1177/1540796917729682 ◽

2017 ◽

Vol 42 (4) ◽

pp. 211-224 ◽

Cited By ~ 2

Author(s):

Matthew E. Brock ◽

Rachel L. Seaman ◽

Courtney Downing

Keyword(s):

Performance Feedback ◽

Challenging Behavior ◽

Student Outcomes ◽

Positive Reinforcement ◽

Implementation Fidelity ◽

Severe Disabilities ◽

Severe Disability ◽

Elementary Student ◽

Evidence Based ◽

Paraprofessional Training

Most paraprofessionals are not well trained to implement evidence-based practices that can improve student outcomes. In this study, we trained a paraprofessional to use evidence-based instructional practices with an elementary student with a severe disability who exhibited challenging behavior. Through functional analysis, we determined the function of the student’s problem behavior was to escape instructional demands. We used a multiple probe across behavior design to test the efficacy of video modeling and performance feedback on the paraprofessional’s implementation of three evidence-based instructional plans that incorporated high rates of positive reinforcement. Training strategies resulted in high rates of paraprofessional implementation fidelity, and paraprofessional implementation translated into improved student outcomes. These results provide an illustration of how effective paraprofessional training can promote learning for a student with severe disabilities and challenging behavior.

Download Full-text

An Investigation of Concurrent Validity of Fidelity of Implementation Measures at Initial Years of Implementation

Remedial and Special Education ◽

10.1177/0741932518795639 ◽

2019 ◽

Vol 40 (1) ◽

pp. 25-31

Author(s):

Caitlin Rasplica Khoury ◽

Kent McIntosh ◽

Robert Hoselton

Keyword(s):

Student Outcomes ◽

Behavioral Interventions ◽

Fidelity Of Implementation ◽

Self Assessment ◽

Cross Sectional ◽

School Practices ◽

Positive Behavioral Interventions ◽

Self Assessments ◽

Positive Correlations ◽

Fidelity Measures

Fidelity of implementation of school practices is crucial to student outcomes. Several types of tools, including self-assessments, are available for measuring fidelity, but little is known regarding the relation of self-assessments of fidelity to fidelity instruments completed with the support of external experts, specifically, during the first few years of implementation. The present study used cross-sectional data from 1,438 schools to examine relations between fidelity self-assessment and team-based fidelity measures in the first 4 years of implementation of School-Wide Positive Behavioral Interventions and Supports (SWPBIS). Results showed strong positive correlations between fidelity self-assessments and a team-based measure of fidelity at each year of implementation.

Download Full-text

District Practices Associated With Successful SWPBIS Implementation

Behavioral Disorders ◽

10.1177/0198742917753612 ◽

2018 ◽

Vol 43 (3) ◽

pp. 393-406 ◽

Cited By ~ 3

Author(s):

Heather Peshak George ◽

Karen Elfner Cox ◽

Devon Minch ◽

Therese Sandomierski

Keyword(s):

Student Outcomes ◽

Empirical Support ◽

Implementation Fidelity ◽

Student Discipline ◽

Behavior Interventions ◽

Structured Interviews ◽

Data Infrastructure ◽

School Practices ◽

Positive Behavior Interventions ◽

District Personnel

Schoolwide positive behavior interventions and supports (SWPBIS) is a widely implemented model for systematically supporting the social and behavioral development of students with and without disabilities, including those with and at risk for emotional and behavioral disorders. Identifying district factors associated with SWPBIS implementation fidelity and improved student outcomes can assist district personnel with appropriate allocation of resources, including professional development and school-based implementation support. Due to the limited empirical support for district-level factors that influence school practices and student outcomes, this exploratory study was conducted with the goal of identifying characteristics associated with school districts that have a high proportion of schools implementing SWPBIS with fidelity and sustained positive student discipline outcomes. Six high-implementing districts were identified, and semi-structured interviews with district staff were then conducted to identify common features staff attributed to their district’s positive outcomes. Analysis of those interviews revealed eight themes including District Coordinator, Coaches, District Teaming, Internal Implementation Drivers, Leadership Buy-In and Support, District Data Infrastructure, Direct Support to Schools, and Communication. Limitations and implications are discussed.

Download Full-text

PBIS Implementation Fidelity and Student Outcomes in an Urban School District

10.32623/3.10007 ◽

2020 ◽

Vol 3 (2) ◽

pp. 96-117

Author(s):

C. Angelique Scherer ◽

W. Kyle Ingle

Keyword(s):

School District ◽

Repeated Measures ◽

Student Outcomes ◽

Standardized Test ◽

Action Plan ◽

Implementation Fidelity ◽

Urban School ◽

Future Research ◽

Urban School District ◽

Positive Behavior Interventions

The purpose of our study was to examine Positive Behavior Interventions and Supports (PBIS) implementation fidelity and student outcomes for secondary schools in a large, urban school district placed under a corrective action plan due to disproportionate suspension practices for students of color. Drawing upon data over a four-year period on PBIS implementation fidelity, discipline referrals, suspensions, and standardized test scores, we employed repeated measures analysis of variance (ANOVA) and revealed improved outcomes in the first year, but improvements did not continue over time. Additionally, academic outcomes did not show any significant increases. The study concludes with implications for policy, practice, and future research.

Download Full-text