Examining the Reliability of Scores From a Performance Assessment of Practice-Based Competencies

The purpose of this study was to examine the reliability and sources of score variation from a performance assessment of practice competencies within an occupational therapy program. Data from 99 students who participated in a practical exam were examined. A generalizability analysis of analytic, total, and overall holistic scores was completed accounting for the nested design. The results demonstrated that rater pairs produced highly reliable (i.e., G > 0.93, ϕ > 0.90) overall holistic, total, and analytic scores. In all scoring scenarios, student variation accounted for the largest percentage of total variance with less variation attributable to raters. However, the interaction-error term accounted for the second most variation within the system. Findings from this study suggest possible differences in rater behavior raising questions for future research around the processes raters use to generate holistic versus analytic scores.

Using analytic rating scales to assess English/Chinese bi-directional interpretation: A longitudinal Rasch analysis of scale utility and rater behavior

Descriptor-based analytic rating scales have been increasingly used to assess interpretation quality. However, little empirical evidence is available to unequivocally support the effectiveness of rating scales and rater reliability. This longitudinal study thus attempts to shed insight into scale utility and rater behavior in English/Chinese interpretation performance assessment, using multifaceted Rasch measurement. Specifically, the study focuses on criterion/scale difficulty, scale effectiveness, rater severity/leniency and rater self-consistency between English/Chinese interpreting and over three time points. Research results are discussed, highlighting the utility of analytic rating scales and the variability of rater behavior in interpretation assessment. The results also have implications for developing reliable, valid, and practical instruments to assess interpretation quality.