Comparative Judgement for Linking Two Existing Scales

This article describes an efficient way of using comparative judgement to calibrate scores from different educational assessments against one another (a task often referred to as test linking or equating). The context is distinct from other applications of comparative judgement as there is no need to create a new achievement scale using a Bradley-Terry model (or similar). The proposed method takes advantage of this fact to include evidence from the largest possible number of examples of students’ performances on the separate assessments whilst keeping the amount of time required from expert judges as low as possible. The paper describes the method and shows, via simulation, how it achieves greater accuracy than alternative approaches to the use of comparative judgement for test equating or linking.

PENGEMBANGAN IKM MAKANAN RINGAN OLAHAN DI KECAMATAN CIHAURBEUTI KABUPATEN CIAMIS

Perkembangan sektor industri dalam pembangunan di Indonesia tidak terlepas dari peranan Industri Kecil Menengah (IKM). Kabupaten Ciamis merupakan kabupaten yang berada diujung selatan provinsi jawa barat yang juga memiliki peranan potensi IKM sebagai daya tarik peluang pengembangan yang mampu menghasilkan Industri Kecil Menengah (IKM) tersebut terdiri atas usaha kecil menengah sebanyak 91 dan 39 jenis usaha didominasi oleh yang menguasai sektor IKM dan perekonomian masyarakat dengan olahan makanan ringan khas. Penelitian ini bertujuan untuk: (1) Mengidentifikasi sektor basis yang ada di Kecamatan Cihaurbeuti menggunakan metode Location Quotient (LQ). Serta (2) Menentukan Strategi pengembangan dari sektor prioritas IKM Makanan Ringan Olahan di Kecamatan Cihaurbeuti menggunakan metode Analitycal Hierarchy Process (AHP) Berdasarkan hasil pada analisis Location Quotient (LQ) diperoleh hasil yakni, terdapat tiga sektor IKM yang menjadi sektor basis di Kecamatan Cihaurbeuti. Dalam perhitungan LQ ini, IKM Makanan Ringan Olahan termasuk sektor basis dengan indeks LQ rata rata > 1yaitu 2,96. Berdasarkan hasil pada Analitical Hierarchy Process (AHP) dengan analisis menggunakan prinsip dekomposisi dan comparative judgement diperoleh strategi pengembangan IKM Makanan Ringan yaitu, pada Pertumbuhan Industri dengan tingkat kepentingan 51% dan untuk alternatif pengembangan industri yang menjadi fokus adalah Nilai Produksi dengan tingkat kepentingan 22%.

Assessing spoken-language interpreting

In this study, we applied and evaluated a scoring method known as comparative judgement to assess spoken-language interpreting. This methodological exploration represents an extension of previous efforts to optimise scoring methods for assessing interpreting. Essentially, comparative judgement requires judges to compare two similar objects and make a binary decision about their relative qualities. To evaluate its reliability, validity and usefulness in the assessment of interpreting, we recruited two groups of judges (novice and experienced) to assess 66 two-way English/Chinese interpretations based on a computerised comparative judgement system. Our data analysis shows that the new method produced reliable and valid results across judge types and interpreting directions. However, the judges held polarised opinions about the method’s usefulness: while some considered it convenient, efficient and reliable, the opposite view was expressed by others. We discuss the results by providing an integrated analysis of the data collected, outline the perceived drawbacks and propose possible solutions to the drawbacks. We call for more evidence-based, substantive investigation into comparative judgement as a potentially useful method for assessing spoken-language interpreting in certain settings.

What is an ecosystem?

In Content and Language Integrated Learning (CLIL) contexts, students are expected to express disciplinary knowledge in a second/foreign language. One construct that has proven useful for the identification and realization of language functions in disciplinary knowledge is Dalton-Puffer’s (2013) model of cognitive discourse functions (CDFs). Additionally, Systemic Functional Linguistics (SFL) has already been proven useful for distinguishing lexico-grammatical features that characterise different CDFs in CLIL students’ productions (e.g., Nashaat-Sobhy & Llinares, 2020; Evnitskaya & Dalton-Puffer, 2020). In this article, we use SFL to analyse the oral and written realisations of the CDF Define by 6th grade students participating in a CLIL program in Madrid, Spain. A total of 83 students responded to the same prompt (on science) in writing (in the form of a blog) as well as orally (in the form of an interview). In the oral interviews the co-construction of definitions by the students with the interviewer (researcher) and another peer are explored using the notion of Legitimation Code Theory and the concept of semantic waves (Maton, 2013). The analysis of students’ definitions is also related to primary CLIL teachers’ evaluations using comparative judgement.

Investigating insight and rigour as separate constructs in mathematical proof

In this paper we investigate undergraduate mathematics students' conceptions of rigour and insight. We conducted comparative judgement experiments in which students were asked to judge different proofs of the same theorem with five separate criteria: rigour, insight, understanding, simplicity and assessment marks. We predicted, and our experiment found, that rigour is a reliable construct. We predicted that insight is also a reliable construct but asking students to judge on the basis of ``which proof gives you more insight into why a theorem is true'' did not result in a reliable judging scale. Our analysis suggests two distinct dimensions: rigour and marks contribute to one factor whereas simplicity and personal understanding relate to a second factor. We suggest three reasons why insight was related almost equally to both factors. First, while comparative judgement was suitable for assessing some aesthetic criteria it may not be suited to investigating students conceptions of insight. Second, students may not have developed an aesthetic sense in which they appreciate insight in ways which are regularly discussed by mathematics educators. Lastly, insight may not be a coherent and well-defined construct after all.

Comparative judgement: assess student production without absolute judgements

What is it? Comparative Judgement (CJ) has emerged as a technique that typically makes use of holistic judgement to assess difficult-to-specify constructs such as production (speaking and writing) in Modern Foreign Languages (MFL). In traditional approaches, markers assess candidates’ work one-by-one in an absolute manner, assigning scores to different elements (analytic marking). In CJ, however, markers compare two pieces and consider the overall merits of each. They make one binary, holistic judgement as to which is better. This approach exploits humans’ natural ability to compare; we find it easy, for example, to say which of two people is taller, but struggle to give precise estimates of height.

Innovative language pedagogy report

The Innovative Language Pedagogy Report presents new and emerging approaches to language teaching, learning, and assessment in school, further education, and higher education settings. Researchers and practitioners provide 22 research-informed, short articles on their chosen pedagogy, with examples and resources. The report is jargon-free, written in a readable format, and covers, among others, gamification, open badges, comparative judgement, translanguaging, translation, learning without a teacher, and dialogue facilitation. It also includes technologies such as chatbots, augmented reality, automatic speech recognition, digital corpora, and LMOOCs, as well as pedagogical innovations around virtual exchange, digital storytelling, technology-facilitated oral homework, and TeachMeets.