Prior research indicates mixed findings regarding the consistency of adjudicators’ ratings at large ensemble festivals, yet the results of these festivals have strong impacts on the perceived success of instrumental music programs and the perceived effectiveness of their directors. In this study, Rasch modeling was used to investigate the potential influence of adjudicators on performance ratings at a live large ensemble festival. Evaluation forms from a junior high school concert band festival adjudicated by a panel of three expert judges were analyzed using the Many-Facets Rasch Model. Analyses revealed several trends. First, the use of assigning “half points” between adjacent response options on the 5-point rating scale resulted in redundancy and measurement noise. Second, adjudicators provided relatively similar ratings for conceptually distinct criteria, which could be evidence of a halo effect. Third, although all judges demonstrated relatively lenient ratings overall, one judge provided more severe ratings as compared to peers. Finally, an exploratory interaction analysis among the facets of judges and bands indicated the presence of rater-mediated bias. Implications for music researchers and ensemble adjudicators are discussed in the context of ensemble performance evaluations, and a measurement framework that can be applied to other aspects of music performance evaluations is introduced.
An introduction to the many-facet Rasch model as a method to improve observational quality measures with an application to measuring the teaching of emotion skills
