What the analytic versus holistic scoring of international teaching assistants can reveal: Lexical grammar matters

Second-language (L2) testing researchers have explored the relationship between speakers’ overall speaking ability, reflected by holistic scores, and the speakers’ performance on speaking subcomponents, reflected by analytic scores (e.g., McNamara, 1990; Sato, 2011). These research studies have advanced applied linguists’ understanding of how raters view the components of effective speaking skills, but the authors of the studies either used analytic composite scores, instead of true holistic ratings, or ran regression analyses with highly correlated subscores, which is problematic. To address these issues, 10 experienced ITA raters rated the speaking of 127 international teaching assistant (ITA) candidates using a four-component analytic rubric. In addition, holistic ratings were provided for the 127 test takers from a separate (earlier) scoring by two experienced ITA raters. The two types of scores differentiated examinees in similar ways. The variability observed in students’ holistic scores was reflected in their analytic scores. However, among the four analytic subscales, examinees’ scores on Lexical and Grammatical Competence had the greatest differentiating power. Its scores indicated with a high level of accuracy who passed the test and who did not. The paper discusses the components contributing to ITAs’ L2 oral speaking proficiency, and reviews pedagogical implications.

Holistic vs. Analytic Scoring between Expository and Narrative Genres: Does the Assessment Type Matter?

The unprecedented emphasis on EFL students' development of their writing skills has invited researchers and EFL instructors to look for the most effective methods of teaching writing and assessing it. Within this context, two main dominant assessment types have been used by EFL writing instructors: holistic and analytic scoring with ambiguous research findings about the use of each. Moreover, ambiguity has also been surrounding variability in raters' scoring across genre types. This study aims at uncovering the difference between using the two scoring schemes across two genres in EFL writing: expository and narrative. Two texts each representing a genre type from 10th grade EFL writers' compositions were presented to a sample of 90 in-service EFL teachers for holistic scoring. Two months later, the same texts were presented to the same teachers for analytic scoring in order to compare between the grades assigned in each round for arriving at results. Results suggest significant differences between the scores obtained according to the grading method. Also, narrative essays received higher scores in both assessment types. These results are discussed, and recommendations are derived.

Logistic calibrated items (LCI) method: does it solve subjectivity in translation evaluation and assessment?

<p>This research paper introduces a model of translation evaluation called Logistic Calibrated Items method. The aim of LCI method is to maximize a translators’ performance and to identify top competent translators through detecting all parsing items within a source text. Parsing items are extracted by the application of Brat software. The purpose of LCI was to identify parsing items having optimal item difficulty and item discrimination values. LCI method involves six stages: (1) holistic scoring; (2) the application of Brat software to extract all parsing items; (3) the calculation of item difficulty; (4) the calculation of item discrimination; (5) the identification of items with optimal item difficulty and item discrimination values; and (6) the recalculation of scores. 125 translation students and 4 professional translation evaluators took part in this research. The final results showed that LCI method was more consistent when compared to holistic method. Limitations and implications were also discussed.</p>