Investigating the Comparability of Examination Difficulty Using Comparative Judgement and Rasch Modelling

2018 ◽  
Vol 18 (4) ◽  
pp. 366-391
Author(s):  
Stephen D. Holmes ◽  
Michelle Meadows ◽  
Ian Stockford ◽  
Qingping He
BMJ Open ◽  
2017 ◽  
Vol 7 (5) ◽  
pp. e013185 ◽  
Author(s):  
Astrid Austvoll-Dahlgren ◽  
Øystein Guttersrud ◽  
Allen Nsangi ◽  
Daniel Semakula ◽  
Andrew D Oxman

2019 ◽  
Author(s):  
Øystein Guttersrud ◽  
Christopher Le ◽  
Kjell Sverre Pettersen ◽  
Sølvi Helseth ◽  
Hanne Søberg Finbråten

Abstract Background The self-reported European Health Literacy Survey Questionnaire (HLS-EU-Q47) is a widely used measure for population health literacy. Based on confirmatory factor analyses and Rasch modelling, the short form HLS-Q12 was developed to meet the Rasch unidimensional measurement model expectations. After its publication, there was a worldwide call to identify HLS-Q12 cutoff scores and establish clearly delineated standards regarding the skills assessed. This study therefore aims to identify the HLS-Q12 scores associated with statistically distinct levels of proficiency and to construct a proficiency scale that may indicate what individuals typically know and can do at increasingly sophisticated levels of health literacy. Methods We applied the unidimensional Rasch measurement model for polytomous items to responses from 900 randomly sampled individuals and 388 individuals with type 2 diabetes. Using Rasch based item calibration, we constructed a proficiency scale by locating the ordered item thresholds along the scale. By applying Wright’s method for the maximum number of strata, we determined the cutoff scores for significantly different levels. By directly referring to item content that people who achieved the cutoff scores viewed as ‘easy’, we suggested what these gradually more advanced levels of health literacy might mean in terms of item content. Results Analysing the population sample, we identified statistically distinct levels of health literacy at the empirically identified cutoff scores 27, 33 and 39. We confirmed them by analysing the responses from individuals with diabetes. Using item calibration, the resulting HLS-Q12 proficiency scale expresses typical knowledge and skills at these three statistically distinct levels. The scale’s cumulative nature indicates what it may mean qualitatively to move from low to high health literacy. Conclusions By identifying levels of health literacy, we may initiate the improvement of current models of health literacy. Determining how to adapt information to patients’ health literacy level is a possible clinical outcome. A substantial methodological outcome is the inevitability of Rasch modelling in measurement. We found that Wright’s method identified rating scale cutoff scores consistently across independent samples. To reveal sources of potential biases, threats to validity and imprecision of benchmarks, replication of our study in other contexts is required


2010 ◽  
Vol 10 (2) ◽  
pp. 189-202 ◽  
Author(s):  
Helge Molde ◽  
Sigurd W. Hystad ◽  
Ståle Pallesen ◽  
Helga Myrseth ◽  
Ingeborg Lund
Keyword(s):  

2013 ◽  
Vol 40 (8) ◽  
pp. 1831-1851 ◽  
Author(s):  
Paola Annoni ◽  
Dorota Weziak-Bialowolska ◽  
Hania Farhan

Author(s):  
Melanie Farlie ◽  
Christina Johnson ◽  
Tim Wilkinson ◽  
Jennifer Keating

Educators want to assess learners using assessment processes that provide valid measures of learner ability. An ideal assessment tool would include items that are appropriate for assessing the target attributes. Ideal assessment results would accurately differentiate learners across the spectrum of ability, determine which learners satisfied the required standard and enable comparison between learner cohorts (e.g., across different years). Similar considerations are relevant to researchers who are designing or revising methods used to gather other kinds of assessment data, such as participant responses to surveys or clinical measurements of performance. Analysing assessment scores using Rasch analysis provides information about scores and the nature of each assessment item, and analysis output guides refinement of assessment. However, few health professional educators have published research that includes Rasch modelling methods. It may be that health professional educators find the language used to describe Rasch analysis to be somewhat impenetrable and that this has, to date, limited engagement in exploring applications for Rasch. In this paper, we lay out an overview of the potential benefits of Rasch analysis in health professional education and research.  


2021 ◽  
Vol 6 ◽  
Author(s):  
Tom Benton

This article describes an efficient way of using comparative judgement to calibrate scores from different educational assessments against one another (a task often referred to as test linking or equating). The context is distinct from other applications of comparative judgement as there is no need to create a new achievement scale using a Bradley-Terry model (or similar). The proposed method takes advantage of this fact to include evidence from the largest possible number of examples of students’ performances on the separate assessments whilst keeping the amount of time required from expert judges as low as possible. The paper describes the method and shows, via simulation, how it achieves greater accuracy than alternative approaches to the use of comparative judgement for test equating or linking.


Sign in / Sign up

Export Citation Format

Share Document