scholarly journals THE MÊTIS OF RELIABILITY: Using the Framework for Success to Aid the Performance of Neutrality Within Writing Assessment

Author(s):  
Tristan Abbott
Keyword(s):  
Author(s):  
Yeongmi Choi ◽  
Sumi Kim ◽  
So hyeon Lee ◽  
Youngmin Park

2005 ◽  
Vol 33 (1) ◽  
pp. 101-113 ◽  
Author(s):  
P. Adam Kelly

Powers, Burstein, Chodorow, Fowles, and Kukich (2002) suggested that automated essay scoring (AES) may benefit from the use of “general” scoring models designed to score essays irrespective of the prompt for which an essay was written. They reasoned that such models may enhance score credibility by signifying that an AES system measures the same writing characteristics across all essays. They reported empirical evidence that general scoring models performed nearly as well in agreeing with human readers as did prompt-specific models, the “status quo” for most AES systems. In this study, general and prompt-specific models were again compared, but this time, general models performed as well as or better than prompt-specific models. Moreover, general models measured the same writing characteristics across all essays, while prompt-specific models measured writing characteristics idiosyncratic to the prompt. Further comparison of model performance across two different writing tasks and writing assessment programs bolstered the case for general models.


Sign in / Sign up

Export Citation Format

Share Document