Using Pooled Heteroskedastic Ordered Probit Models to Improve Small-Sample Estimates of Latent Test Score Distributions

2020 ◽  
pp. 107699862092291
Author(s):  
Benjamin R. Shear ◽  
Sean F. Reardon
Keyword(s):  
Test Score ◽  
Sampling Error ◽  
Small Sample ◽  
Ordered Probit ◽  
Sample Sizes ◽  
Ordered Categorical Data ◽  
Ordered Probit Models ◽  
Ordered Categorical ◽  
Real Test ◽  
Score Distributions

This article describes an extension to the use of heteroskedastic ordered probit (HETOP) models to estimate latent distributional parameters from grouped, ordered-categorical data by pooling across multiple waves of data. We illustrate the method with aggregate proficiency data reporting the number of students in schools or districts scoring in each of a small number of ordered “proficiency” levels. HETOP models can be used to estimate means and standard deviations of the underlying (latent) test score distributions but may yield biased or very imprecise estimates when group sample sizes are small. A simulation study demonstrates that the pooled HETOP models described here can reduce the bias and sampling error of standard deviation estimates when group sample sizes are small. Analyses of real test score data demonstrate the use of the models and suggest the pooled models are likely to improve estimates in applied contexts.

Using Heteroskedastic Ordered Probit Models to Recover Moments of Continuous Test Score Distributions From Coarsened Data

2016 ◽  
Vol 42 (1) ◽  
pp. 3-45 ◽  
Author(s):  
Sean F. Reardon ◽  
Benjamin R. Shear ◽  
Katherine E. Castellano ◽  
Andrew D. Ho
Keyword(s):  
Test Score ◽  
Mean Squared Error ◽  
Population Distribution ◽  
Ordered Probit ◽  
Parameter Estimates ◽  
Demographic Groups ◽  
Continuous Test ◽  
Standard Deviations ◽  
Ordered Probit Models ◽  
Score Distributions

Test score distributions of schools or demographic groups are often summarized by frequencies of students scoring in a small number of ordered proficiency categories. We show that heteroskedastic ordered probit (HETOP) models can be used to estimate means and standard deviations of multiple groups’ test score distributions from such data. Because the scale of HETOP estimates is indeterminate up to a linear transformation, we develop formulas for converting the HETOP parameter estimates and their standard errors to a scale in which the population distribution of scores is standardized. We demonstrate and evaluate this novel application of the HETOP model with a simulation study and using real test score data from two sources. We find that the HETOP model produces unbiased estimates of group means and standard deviations, except when group sample sizes are small. In such cases, we demonstrate that a “partially heteroskedastic” ordered probit (PHOP) model can produce estimates with a smaller root mean squared error than the fully heteroskedastic model.

Ordinal Approaches to Decomposing Between-Group Test Score Disparities

2020 ◽  
pp. 107699862096772
Author(s):  
David M. Quinn ◽  
Andrew D. Ho
Keyword(s):  
Test Score ◽  
Educational Inequality ◽  
Decomposition Methods ◽  
Ordered Probit ◽  
Group Differences ◽  
Scale Invariant ◽  
Gap Statistic ◽  
The Difference ◽  
Ordered Probit Models ◽  
Test Score Gaps

The estimation of test score “gaps” and gap trends plays an important role in monitoring educational inequality. Researchers decompose gaps and gap changes into within- and between-school portions to generate evidence on the role schools play in shaping these inequalities. However, existing decomposition methods assume an equal-interval test scale and are a poor fit to coarsened data such as proficiency categories. This leaves many potential data sources ill-suited for decomposition applications. We develop two decomposition approaches that overcome these limitations: an extension of V, an ordinal gap statistic, and an extension of ordered probit models. Simulations show V decompositions have negligible bias with small within-school samples. Ordered probit decompositions have negligible bias with large within-school samples but more serious bias with small within-school samples. More broadly, our methods enable analysts to (1) decompose the difference between two groups on any ordinal outcome into portions within- and between some third categorical variable and (2) estimate scale-invariant between-group differences that adjust for a categorical covariate.

Commentary on “Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale”

2020 ◽  
pp. 107699862095666
Author(s):  
Alina A. von Davier
Keyword(s):  
School District ◽  
Test Score ◽  
Applied Research ◽  
Aggregate Level ◽  
Validation Methods ◽  
Common Scale ◽  
Scale Linking ◽  
Score Distributions

In this commentary, I share my perspective on the goals of assessments in general, on linking assessments that were developed according to different specifications and for different purposes, and I propose several considerations for the authors and the readers. This brief commentary is structured around three perspectives (1) the context of this research, (2) the methodology proposed here, and (3) the consequences for applied research.

No Evidence that Experiencing Physical Warmth Promotes Interpersonal Warmth: Two Failures to Replicate Williams and Bargh (2008)

2018 ◽  
Author(s):  
Christopher Chabris ◽  
Patrick Ryan Heck ◽  
Jaclyn Mandart ◽  
Daniel Jacob Benjamin ◽  
Daniel J. Simons
Keyword(s):  
Null Hypothesis ◽  
Small Sample ◽  
Sample Sizes ◽  
Double Blind ◽  
Bayesian Analyses ◽  
Physical Warmth ◽  
Small Sample Sizes ◽  
Interpersonal Warmth

Williams and Bargh (2008) reported that holding a hot cup of coffee caused participants to judge a person’s personality as warmer, and that holding a therapeutic heat pad caused participants to choose rewards for other people rather than for themselves. These experiments featured large effects (r = .28 and .31), small sample sizes (41 and 53 participants), and barely statistically significant results. We attempted to replicate both experiments in field settings with more than triple the sample sizes (128 and 177) and double-blind procedures, but found near-zero effects (r = –.03 and .02). In both cases, Bayesian analyses suggest there is substantially more evidence for the null hypothesis of no effect than for the original physical warmth priming hypothesis.

Sign in / Sign up

Export Citation Format

Share Document