Short Comparative Interrupted Time Series Using Aggregate School-Level Data in Education Research

Short comparative interrupted times series (CITS) designs are increasingly being used in education research to assess the effectiveness of school-level interventions. These designs can be implemented relatively inexpensively, often drawing on publicly available data on aggregate school performance. However, the validity of this approach hinges on a variety of assumptions and design decisions that are not clearly outlined in the literature. This article aims to serve as a practice guide for applied researchers when deciding how and whether to use this approach. We begin by providing an overview of the assumptions needed to estimate causal effects using school-level data, common threats to validity faced in practice and what effects can and cannot be estimated using school-level data. We then examine two analytic decisions researchers face in practice when implementing the design: correctly modeling the pretreatment functional form, which is modeling the preintervention trend, and selecting comparison cases. We then illustrate the use of this design in practice drawing on data from the implementation of the school improvement grant (SIG) program in Ohio. We conclude with advice for applied researchers implementing this design.

School Improvement Grants in Ohio: Effects on Student Achievement and School Administration

The federal School Improvement Grant (SIG) program allocated US$7 billion over nearly a decade in an effort to produce rapid and lasting improvements in schools identified as low performing. In this article, we use a regression discontinuity design to estimate the effect of Ohio’s SIG turnaround efforts on student achievement and school administration. The results indicate that Ohio’s SIG program significantly increased reading and math achievement, with effects in both subjects of up to 0.20 standard deviations in the second year after SIG eligibility identification. Estimates for the third year are somewhat larger, in the range of one quarter of a standard deviation. We provide evidence that these effects were primarily attributable to schools that implemented the SIG Turnaround model. We also show that SIG eligibility had a positive effect on per-pupil spending, but no average effect on administrative outcomes, including staff turnover, the number of staff members in the school, and school closure. These null overall effects mask heterogeneity across SIG models, however. Most notably, Turnaround schools experienced more turnover than they otherwise would have, whereas Transformation schools experienced less.

Teacher Migration: Extension and Application of the Population Ecology Model to Explore Teacher Transfers in a Reform Environment

This article assesses the association between the Title I School Improvement Grant (SIG) program’s personnel replacement policy and teacher employment patterns within an urban school district. Hannan and Freeman’s population ecology model allowed the authors to consider schools within districts as individual organizations nested within a larger organization. The data are drawn from employment records of 2,470 teachers who worked in 19 high schools in a single school district from 2006 to 2011. The personnel replacement policy of the Title I SIG program appears to have reinforced, and in some cases intensified, existing patterns of teacher selection, retention, and migration.

Irrational Exuberance for Market-based Reform: How Federal Turnaround Policies Thwart Democratic Schooling

Background In 2009, the Obama Administration announced its intention to rapidly “turn around” 5,000 of the nation's lowest-performing schools. To do so, it relied on the School Improvement Grant (SIG) program to provide temporary funding for states and schools, and to mandate drastic, school-level reforms. Most of these reforms require massive administrative and teacher layoffs, especially under the “turnaround option.”In the public debate about the SIG program, reforms such as turnarounds have been described as new and innovative. In reality, the nation has significant experience with them, particularly over the past 40 years. Turnaround-style reforms are not only based on unwarranted claims; they ignore contrary research evidence about the potential of mass firings to improve organizational performance. Purpose This paper considers the tensions with democratic education inherent in the federal SIG program's market-based school reforms. It examines the evolution of and intent behind the 2009 federal SIG program. From there, it considers the lessons of forty years of research on educational effectiveness and high-stakes accountability. It builds on this evidence, as well as the growing literature on communities’ engagement in reform, in its analysis of the school turnaround research and practice. The paper culminates in a set of recommendations that are intended to re-center the purposes of public education for low-income students, students of color, and local communities in developing more equitable, democratic school turnarounds. Research Design This article synthesizes forty years of research on school and district effectiveness, high-stakes accountability, and community engagement in school reform to evaluate the federal School Improvement Grant program's potential to cultivate democratic, equitable public schools. It also reviews the small, but rapidly growing literature on school turnarounds, paying particular attention to the ways in which this new field reproduces or departs from earlier literature that examined reform models that are analogous to the current SIG-funded school turnarounds. Conclusions: Based on the provisional lessons that are emerging from current SIG-inspired turnarounds, from research on earlier efforts to improve school and district effectiveness, and from pockets of promising community-based practices that are developing at local and national levels, we propose five steps that federal, state, and local policymakers can take toward fostering more equitable, democratic turnaround processes. First, increase current federal and state spending for public education, particularly as it is allocated for more democratic turnarounds. Second, focus turnaround policies on improving the quality of teaching and learning rather than on technical-structural changes. Third, engage a broad cross-section of schools’ communities—teachers, students, parents, and community organizations—in planning and implementing turnaround strategies that are tailored to each school and district context. Fourth, incorporate multiple indicators of effectiveness—apart from test scores—that reflect the range of purposes for schools. Fifth, support ongoing, systematic research, evaluation, and dissemination examining all aspects of turnaround processes in schools and districts.

All Hands on Deck: A Comprehensive, Results-Driven Counseling Model

A graduation rate of 49% alarmed Sunnyside High School in 2009. With graduation rates in the bottom 5% statewide, Sunnyside was awarded a federally funded School Improvement Grant. The “turnaround” principal and the school counselors aligned goals with the ASCA National Model through the program All Hands On Deck (AHOD), based on academic press, social support, and relational trust. In 2012, 78.8% of students graduated. This case study describes student success resulting from the counselor-led program AHOD.