Large-Scale Writing Assessment: Teachers and their Students

The present study investigates integrated writing assessment performances with regard to the linguistic features of complexity, accuracy, and fluency (CAF). Given the increasing presence of integrated tasks in large-scale and classroom assessments, validity evidence is needed for the claim that their scores reflect targeted language abilities. Four hundred and eighty integrated writing essays from the Internet-based Test of English as a Foreign Language (TOEFL) were analyzed using CAF measures with correlation and regression to determine how well these linguistic features predict scores on reading–listening–writing tasks. The results indicate a cumulative impact on scores from these three features. Fluency was found to be the strongest predictor of integrated writing scores. Analysis of error type revealed that morphological errors contributed more to the regression statistic than syntactic or lexical errors. Complexity was significant but had the lowest correlation to score across all variables.

Download Full-text

Detecting Differential Rater Functioning in Severity and Centrality: The Dual DRF Facets Model

Educational and Psychological Measurement ◽

10.1177/00131644211043207 ◽

2021 ◽

pp. 001316442110432

Author(s):

Kuan-Yu Jin ◽

Thomas Eckes

Keyword(s):

Writing Assessment ◽

Large Scale ◽

Special Kind ◽

Performance Assessments ◽

Complex Case ◽

Sample Data ◽

Rater Severity ◽

And Performance ◽

Psychometric Quality

Performance assessments heavily rely on human ratings. These ratings are typically subject to various forms of error and bias, threatening the assessment outcomes’ validity and fairness. Differential rater functioning (DRF) is a special kind of threat to fairness manifesting itself in unwanted interactions between raters and performance- or construct-irrelevant factors (e.g., examinee gender, rater experience, or time of rating). Most DRF studies have focused on whether raters show differential severity toward known groups of examinees. This study expands the DRF framework and investigates the more complex case of dual DRF effects, where DRF is simultaneously present in rater severity and centrality. Adopting a facets modeling approach, we propose the dual DRF model (DDRFM) for detecting and measuring these effects. In two simulation studies, we found that dual DRF effects (a) negatively affected measurement quality and (b) can reliably be detected and compensated under the DDRFM. Using sample data from a large-scale writing assessment ( N = 1,323), we demonstrate the practical measurement consequences of the dual DRF effects. Findings have implications for researchers and practitioners assessing the psychometric quality of ratings.

Download Full-text

The impact of the scoring system of a large-scale standardized EFL writing assessment on its score variability and reliability: Implications for assessment policy makers

Studies In Educational Evaluation ◽

10.1016/j.stueduc.2020.100911 ◽

2020 ◽

Vol 67 ◽

pp. 100911 ◽

Cited By ~ 1

Author(s):

Changhan Zhao ◽

Jinyan Huang

Keyword(s):

Writing Assessment ◽

Scoring System ◽

Large Scale ◽

Policy Makers ◽

Efl Writing ◽

Assessment Policy ◽

The Impact

Download Full-text

Equating a large-scale writing assessment using pairwise comparisons of performances

The Australian Educational Researcher ◽

10.1007/s13384-014-0168-6 ◽

2015 ◽

Vol 42 (4) ◽

pp. 443-460 ◽

Cited By ~ 11

Author(s):

Stephen M. Humphry ◽

Joshua A. McGrane

Keyword(s):

Writing Assessment ◽

Large Scale ◽

Pairwise Comparisons

Download Full-text

A comparability study on differences between scores of handwritten and typed responses on a large-scale writing assessment

10.17077/etd.2j82ewmk ◽

2015 ◽

Author(s):

Angelica Desiree Rankin

Keyword(s):

Writing Assessment ◽

Large Scale

Download Full-text

A Methodologically Improved Study on Raters’ Personality and Rating Severity in Writing Assessment

SAGE Open ◽

10.1177/21582440211009476 ◽

2021 ◽

Vol 11 (2) ◽

pp. 215824402110094

Author(s):

Yu Zhu ◽

Andy Shui-Lung Fung ◽

Liuyan Yang

Keyword(s):

Writing Assessment ◽

Large Scale ◽

Measurement Model ◽

Order Effect ◽

Language Program ◽

Writing Assessments ◽

Relevant Variables ◽

Homogeneous Background ◽

The Relationship ◽

Practical Implications

Personality is an inherent rater’s characteristic influencing rating severity, but very few studies examined their relationship and the findings were inconclusive. This study aimed to re-investigate the relationship between raters’ personality and rating severity with more control on relevant variables and more reliable analysis of rating severity. Female novice raters ( n = 28) from a demographically homogeneous background were recruited to rate on two occasions essays written by 111 students in an intermediate-level Chinese as a foreign language program. Raters’ personality traits were measured using the complete version of NEO-PI-R. Many-faceted Rasch measurement model and repeated measurement were applied to yield more robust estimates of rating severity. In addition, rating order effect was carefully controlled. Extroversion was found to be positively correlated with severity, r(26) = .495, p = .010. Furthermore, Extroversion was found to be a valid predictor of severity, t(24) = 2.792, p = .010, R2 = .21, Cohen’s d = .77, Hattie’s r = .37. Practical implications for developing more individualized online rater calibration for large-scale writing assessments were discussed, followed by limitations of the present study.

Download Full-text