Identifying Guessing in English Language Tests via Rasch Fit Statistics: An Exploratory Study

This article explores the issue of identifying guessers – with a specific focus on multiple-choice tests. Guessing has long been considered a problem due to the fact that it compromises validity. A test taker scoring higher than they should through guessing does not provide a picture of their actual ability. After an initial description of issues associated with guessing, the article then outlines approaches which have been taken to either discourage test takers from guessing or which attempt statistically to handle the problem. From this, the article moves to a novel way of identifying potential guessers: from the post hoc use of Rasch fit statistics. Two datasets, each consisting of approximately 200 beginner level English language test takers were split into two. In each dataset, half the test takers’ answers were randomised – to approximate guessing. Results obtained via a Rasch analysis of the data was then passed to an analyst who used the Rasch fit statistics to identify possible guessers. On each dataset, 80% of guessers were identified.

Download Full-text

Measuring English language proficiency across subgroups: Using score equity assessment to evaluate test fairness

Language Testing ◽

10.1177/0265532218776040 ◽

2018 ◽

Vol 36 (2) ◽

pp. 289-309 ◽

Cited By ~ 1

Author(s):

Hanwook Yoo ◽

Venessa F. Manna ◽

Lora F. Monfils ◽

Hyeon-Joo Oh

Keyword(s):

Language Proficiency ◽

Test Scores ◽

English Language ◽

Educational Background ◽

Test Taker ◽

Background Characteristics ◽

Test Fairness ◽

Educational Backgrounds ◽

Language Tests ◽

Language Exposure

This study illustrates the use of score equity assessment (SEA) for evaluating the fairness of reported test scores from assessments intended for test takers from diverse cultural, linguistic, and educational backgrounds, using a workplace English proficiency test. Subgroups were defined by test-taker background characteristics that research has shown to be associated with performance on language tests. The characteristics studied included gender, age, educational background, language exposure, and previous experience with the assessment. Overall, the empirical results indicated that the statistical and psychometric methods used in producing test scores were not strongly influenced by the subgroups of test takers from which the scores were derived. This result provides evidence in support of the comparability and meaning of test scores across the various test-taker groups studied. This example may encourage language testing programs to incorporate SEA analyses to provide evidence to inform the validity and fairness of reported scores for all groups of test takers.

Download Full-text

VOCABULARY IN ESP APPLICATION AS REFLECTED TO THE LANGUAGE ACROSS CURRICULUM

10.31227/osf.io/k86p4 ◽

2018 ◽

Author(s):

Didik Rinan Sumekto

Keyword(s):

English Language Learners ◽

Language Learners ◽

Text Comprehension ◽

English Language ◽

Vocabulary Instruction ◽

Multiple Choice ◽

Multiple Choice Tests ◽

Choice Tests ◽

The One ◽

Mathematics Course

This paper discusses on the empirical experience of vocabulary instruction in English for specific purpose (ESP) as considered to the language across curriculum, particularly in mathematics course which stresses on syntax and semantics. ESP curriculum is relevantly designed to accommodate the specific needs of particular learners in groups, related to content, particular disciplines, occupations and activities, and centered on the language appropriateness to relevant activities, such as syntax, lexis, discourse, and semantics. The issue of vocabulary instruction for English language learners (ELLs) concerns with productive, receptive, and recognition vocabulary where the one main goals of the instruction is to assist ELLs improve their comprehension in the multiple-choice tests, text comprehension, and process of understanding new meaning of words or phrases.

Download Full-text

A Review of the Washback of English Language Tests on Classroom Teaching

English Language Teaching ◽

10.5539/elt.v13n9p10 ◽

2020 ◽

Vol 13 (9) ◽

pp. 10

Author(s):

Qi Kuang

Keyword(s):

English Language ◽

Pedagogical Practices ◽

Classroom Teaching ◽

Language Test ◽

Future Studies ◽

Course Content ◽

Language Tests ◽

Number Of Factors ◽

The One ◽

Complicated Mechanism

Scholars have long recognized the Washback effect of English language tests on English teaching inside the classroom. However, the lack of scholarly reports in this area is also nonnegligible. Therefore, the present study intends to review some empirical researches that focus on the washback of some English language tests on different aspects of classroom teaching, including the washback on course content, teaching materials, and teaching activities. Both positive and negative washback are found on these aspects and can be attributed to a number of factors, including differences in features of the test content, differences in tests’ coordination to course syllabus, differences in teachers’ adoption of teaching methods, etc. The final discussion recognizes the complicated mechanism of washback of the English language test on classroom teaching and serves to bring out some scholarly and pedagogical implications. On the one hand, future studies could focus more on how to bring out positive washback of English language tests on classroom teaching. On the other hand, pedagogical practices could take advantage of the latest scholarly findings to maximize the efficacy of the aforementioned positive washback.

Download Full-text

A theoretical formalization of the probability of solving multiple-choice tests and its application to different scoring rules

10.31234/osf.io/4pk6x ◽

2021 ◽

Author(s):

Rasmus Persson

Keyword(s):

Test Score ◽

Multiple Choice ◽

Choice Test ◽

Partial Knowledge ◽

Test Taker ◽

Test Error ◽

Multiple Choice Tests ◽

Item Level ◽

Choice Tests ◽

Latent Ability

In multiple-choice tests, guessing is a source of test error which can be suppressed if its expected score is made negative by either penalizing wrong answers or rewarding expressions of partial knowledge. We consider an arbitrarymultiple-choice test taken by a rational test-taker that knows an arbitrary fraction of its keys and distractors. For this model, we compare the relation between the obtained score for standard marking (where guessing is not penalized), marking where guessing is suppressed either by expensive score penalties for incorrect answers or by marking schemes that reward partial knowledge. While the “best” scoring system (in the sense that latent ability and test score are linearly related) will depend on the underlying ability distribution, we find a superiority of the scoring rule of Zapechelnyuk (Economics Letters, 132, 2015) but, except for item-level discrimination among test-takers, a single penalty for wrong answers seems to yield just as good or better results as more intricate schemes with partial credit.

Download Full-text

AUTOMATIC GENERATION OF ENGLISH LANGUAGE TEST QUESTIONS USING MATHEMATICA

CBU International Conference Proceedings ◽

10.12955/cbup.v4.794 ◽

2017 ◽

Vol 4 ◽

pp. 906-909

Author(s):

Anna Malinova ◽

Olga Rahneva

Keyword(s):

Computer Algebra ◽

English Language ◽

Automatic Generation ◽

Word Formation ◽

Large Set ◽

Language Test ◽

Wolfram Mathematica ◽

Language Tests ◽

Language Testing And Assessment ◽

Grammatical Structures

This paper describes a computer algebra-aided generation of two types of English language tests, which further develops our recent work in this domain. The computer algebra system Wolfram Mathematica significantly advances the process of English language testing and assessment. The automatic generation of questions allows us to create a large set of equivalent questions of a certain topic based on a small amount of input values. This reduces authoring time during test creation, avails application of equal criteria and a fair assessment, and decreases the influence of subjective factors. In our previous work, we proposed methods for automatic generation of English language test questions. These were aimed at evaluating the students’ knowledge of lexical and grammatical structures found in the text using test questions that involved matching words and their meaning, matching parts of the whole, and finding synonyms, antonyms, and generalizations or specializations of words. This paper provides new methods for the automatic generation of English language test questions. This includes generating questions for testing the students’ knowledge of adverbs and adjectives, as well as word formation, especially with negative forms of adjectives.

Download Full-text