The Average Response Method of Scaling
The average response method (ARM) of scaling nonbinary data was developed to scale the data from the assessments of writing conducted by the National Assessment of Educational Progress (NAEP). The ARM applies linear models and multiple imputations technologies to characterize the predictive distribution of the person-level average of ratings over a pool of exercises when each person has responded to only a few of the exercises. The derivations of “plausible values” from the individual-level distributions of potential scale scores are given. Conditions are provided for the unbiasedness of estimates based on the plausible values, and the potential magnitude of the bias when the conditions are not met is indicated. Also discussed is how the plausible values allow for an accounting of the uncertainties due to the sampling of individuals and to the incomplete information on each sampled individual. The technique is illustrated using data from the assessment of writing.