Purpose: Concerns about variation in the quality of preservice preparation provided by many university-based principal preparation programs (PPPs) has led to calls to use outcomes of program graduates to hold PPPs accountable. Little research, however, has assessed the degree to which different outcomes for PPP graduates in fact vary systematically by program. Research Methods: Using administrative data from Tennessee, we link approximately a decade’s worth of PPP graduates to their schools, licensure examination scores, and multiple measures of job performance in their first 3 years as principals, including supervisors’ practice ratings on the state evaluation system, teacher and assistant principal ratings of school leadership on a statewide survey, and measures of student achievement growth. We use which PPP a principal completed to predict these outcomes using a regression approach with different sets of covariates. Findings: Although we are able to associate PPPs with high and low principal performance, we find that programs’ rankings vary by outcome measure, and we are unable to identify PPPs that perform consistently well or poorly across outcomes. Moreover, we find that Tennessee PPPs vary substantially in the characteristics of the schools into which their principals are hired and that taking these characteristics into account is important in ordering PPPs based on outcomes. In addition, even over a fairly long time frame, some programs produce too few graduates who later become school leaders to allow for reliable estimates. Implications: Although the use of graduates’ outcomes to differentiate PPPs holds promise, the methodological challenges to drawing valid and reliable conclusions about PPPs from graduates’ job outcomes are substantial. Policymakers and researchers may arrive at very different assessments of which PPPs are successful depending on which outcomes are chosen and what modeling approaches are employed.
