Is the National Study of Learning Mindsets Nationally-Representative?
The National Study of Learning Mindsets (NSLM) is a randomized trial evaluating an intervention in a national sample of schools that were selected to participate via probability sampling methods. The response rate for this study was 56%. This paper evaluates whether site-level non-response compromises the generalizability of the results from the achieved sample of schools in the NSLM. Comparisons of characteristics of schools taking part in the NSLM relative to national benchmarks shows that the NSLM sample has a high degree of similarity to the population of all regular, U.S. public high schools with at least 25 students in 9th grade and in which 9th grade is the lowest grade, via two metrics. First, comparisons of school- and district-level characteristics between the NSLM and the national population of inference show few statistically significant differences. Second, applying an empirical method to quantify the degree of generalizability—the Tipton (2014) generalizability index—found that the analytic sample is generalizable to the population overall (generalizability index = .98 on a 0 to 1 scale) and to four other theoretically-relevant inference populations identified based on school achievement level and school minority concentration measures (generalizability indices > .93). Thus, full-sample estimates and conditional estimates (within school achievement and racial composition subgroups) are likely to be highly generalizable to the corresponding populations of inference.