Relating a concordance-based cloze test to the model of communicative language ability: a verbal protocol study

The literature of empirical studies on the concordance-based cloze test (ConCloze) is far and few between. This is despite the fact that it has a potential for item writing without the aid of native speakers and for making use of corpus-based technology in task design. This article explores the test-taking processes and strategies of a ConCloze item variant. The aim is to investigate the substantive aspect of the construct validity for the item type and increase the generalizability of the findings in the universe of admissible observations. The sample consists of 14 non-native English users who are in higher education and engage with 3 test tasks each, totaling 42 verbal reports. The sampling method is purposive sampling, in which their first language profiles are maximized in terms of heterogeneity for increased power of generalization. It is found that Reading concordance lines and recognizing clue words inside is a major process, and Assessing item components and testing a meaningful compatibility of a word in context a secondary one. A model of communicative language ability is used to provide a context for task use in this study, where strategic competence is represented in ConCloze substantive validity.

Analysis of Item Writing Flaws in a Communications Skills Test in a Ghanaian University

Adhering to the rules governing the writing of multiple-choice test items will ensure quality and validity. However, realizing this ideal could be challenging for non-native English language teachers and students. This is especially so for non-native English language teachers because developing test items in a language that neither they nor their students use as their mother tongue raises a multitude of issues related to quality and validity. A descriptive study on this problem was conducted at a Technical University in Ghana which focused on item writing flaws in a communication skills test. The use of multiple-choice test in Ghanaian universities has increased over the last decade due to increasing student intake. A 20-item multiple-choice test in communication skills was administered to 110 students. The test items were analyzed using a framework informed by standard item writing principles based on the revised taxonomy of multiple-choice item-writing guides by Haladyna, Downing and Rodriguez (2002). The facility and discrimination index (DI) was calculated for all the items. In total, 60% of the items were flawed based on standard items writing principles. The most violated guideline was wording stems negatively. Pearson correlation analysis indicated a weak relationship between the difficulty and discrimination indices. Using the discrimination indices of the flawed items showed that 84.6 % of them had discrimination indices below the optimal level of 0.40 and above. The lowest DI was recorded by an item with which was worded negatively. The mean facility of the test was 45%. It was observed that the flawed items were more difficult than the non-flawed items. The study suggested that test items must be properly reviewed before they are used to assess students’ knowledge.

Studiosta steissille – Multimodaaliset kuuntelutehtävät ylioppilastutkinnon pitkien oppimäärien kielikokeissa 2018

Tässä tutkimuksessa tarkasteltiin ylioppilastutkinnon kielikokeissa esiintyvien kuullunymmärtämistehtävien reiluutta ja tarkoituksenmukaisuutta. Keskityimme multimodaalisiin tehtävätyyppeihin, jotka tulivat mahdolliseksi digitaalisen ylioppilastutkinnon myötä. Aineisto käsitti englannin (n = 37086), ruotsin (n = 1626) ja saksan (n = 491) pitkien oppimäärien kokeisiin vuonna 2018 osallistuneiden kokelaiden osiokohtaiset pistemäärät, joita analysoitiin tilastollisin menetelmin. Lisäksi kyseisten kokeiden koetehtäviä ja niiden kuvauksia analysoitiin teorialähtöisellä sisällönanalyysillä. Multimodaaliset tehtävät edellyttävät kielitaidon lisäksi monilukutaitoa eli monimuotoisten tekstien ja niiden yhdistelmien tulkintaa. Eri kohderyhmien välisiä eroja tutkittaessa havaittiin joitakin eroja englannissa poikien ja ruotsinkielisten ja ruotsissa tyttöjen eduksi. Tasa-arvoisimmiksi osoittautuivat saksan kokeet. Tulokset hyödyttävät koetehtävien laatijoiden lisäksi kielten opetuksen parissa työskenteleviä. From Studio to Station – Multimodal Listening Comprehension Task Types in in Long Syllabus of Languages in Finnish Matriculation Examinations in 2018 Abstract This study addressed fairness and meaningfulness of listening comprehension tasks in the newly digitalized Finnish Matriculation Examination focusing on multimodal task types. The data comprised item scores of test takers in 2018 in long syllabus English (n = 37086), Swedish (n = 1626) and German (n = 491) and related test tasks together with task description sheets. Item scores were analyzed with statistical methods, and the task content with theory-driven content analysis. Multimodal items prerequisite multiple language-related and general competences and strategies and combinations of them. Group wise differences between investigated groups were detected in English in favour of boys and in Swedish in favor of girls, while all differences were smallest in the German tests. The results contribute to item writing and language education in general. Keywords: listening comprehension, language assessment, high-stakes tests, computer-assisted testing

Evaluating the impact of faculty development programme initiative: are we really improving skills in MCQ writing?

Objective: This study was done to evaluate the effectiveness of a 3-hour workshop in improving faculty competence in developing high quality test items. Methodology: The study took place in National University of Medical Sciences and its affiliated institutes in Pakistan, during a period of six months in year 2018. It was a descriptive, cross sectional study. A series of seven workshops on quality assurance in writing MCQs was conducted as part of faculty development program, emphasizing hand-on experience and prompt feedback. Participants’ satisfaction was evaluated with a post-workshop feedback questionnaire (Kirkpatrick 1). A self-made structured questionnaire was given as a pre-test and post-test, to check improvement in cognition, behaviour and item writing skills of faculty (Kirkpatrick 2). Paired t test was applied and difference in mean scores of responses was evaluated. Results: Total 141 faculty members were trained. The training session led to high satisfaction in all elements of workshop, significant improvements in boosting confidence in item writing skills(p=0.000), recognizing parts of MCQs (p=0.000), identifying item writing flaws (p=0.000) and levels of Millers pyramid and blooms taxonomy (p=0.000). Conclusion: Training sessions of short duration are effective in improving the competence of faculty in writing quality test items, provided hands-on experience is built-in and effective feedback is provided.

Assessment of Global Health Education: The Role of Multiple-Choice Questions

Introduction: The standardization of global health education and assessment remains a significant issue among global health educators. This paper explores the role of multiple choice questions (MCQs) in global health education: whether MCQs are appropriate in written assessment of what may be perceived to be a broad curriculum packed with fewer facts than biomedical science curricula; what form the MCQs might take; what we want to test; how to select the most appropriate question format; the challenge of quality item-writing; and, which aspects of the curriculum MCQs may be used to assess.Materials and Methods: The Medical School for International Health (MSIH) global health curriculum was blue-printed by content experts and course teachers. A 30-question, 1-h examination was produced after exhaustive item writing and revision by teachers of the course. Reliability, difficulty index and discrimination were calculated and examination results were analyzed using SPSS software.Results: Twenty-nine students sat the 1-h examination. All students passed (scores above 67% - in accordance with University criteria). Twenty-three (77%) questions were found to be easy, 4 (14%) of moderate difficulty, and 3 (9%) difficult (using examinations department difficulty index calculations). Eight questions (27%) were considered discriminatory and 20 (67%) were non-discriminatory according to examinations department calculations and criteria. The reliability score was 0.27.Discussion: Our experience shows that there may be a role for single-best-option (SBO) MCQ assessment in global health education. MCQs may be written that cover the majority of the curriculum. Aspects of the curriculum may be better addressed by non-SBO format MCQs. MCQ assessment might usefully complement other forms of assessment that assess skills, attitude and behavior. Preparation of effective MCQs is an exhaustive process, but high quality MCQs in global health may serve as an important driver of learning.

The Development of Preliminary HiTOP Internalizing Spectrum Scales

As part of a broader project to create a comprehensive self-report measure for the Hierarchical Taxonomy of Psychopathology consortium, we developed preliminary scales to assess internalizing symptoms. The item pool was created in four steps: (a) clarifying the range of content to be assessed, (b) identifying target constructs to guide item writing, (c) developing formal definitions for each construct, and (d) writing multiple items for each construct. This yielded 430 items assessing 57 target constructs. Responses from a heterogeneous scale development sample ( N = 1,870) were subjected to item-level factor analyses based on polychoric correlations. This resulted in 39 scales representing a total of 213 items. The psychometric properties of these scales replicated well across the development sample and an independent validation sample ( N = 496 adults). Internal consistency analyses established that most scales assess relatively narrow forms of psychopathology. Structural analyses demonstrated the presence of a strong general factor. Additional analyses of the 35 nonsexual dysfunction scales revealed a replicable four-factor structure with dimensions we labeled Distress, Fear, Body Dysmorphia, and Mania. A final set of analyses established that the internalizing scales varied widely—and consistently—in the strength of their associations with neuroticism and extraversion.

Item Analysis of Multiple Choice Questions in Pharmacology

Background: MCQs type assessment in medical education is replacing old theory style. There are concerns regarding the quality of the Multiple Choice Questions.Objectives: To determine the quality of Multiple Choice Questions by item analysis. Material and Methods: Study was a cross sectional descriptive .Fifty Multiple Choice Questions in the final internal evaluation exams in 2015 of Pharmacology at Bacha khan Medical College were analyzed. The quality of each Multiple Choice Questions item was assessed by the Difficulty index (Dif.I), Discriminative Index (D.I) and Distracter Efficiency (D.E).Results: Multiple Choice Questions that were of moderate difficulty were 66%. Easy were 4% and high difficulty were 30%.Reasons for high difficult Multiple Choice Questions were analyzed as Item Writing Flaws 41%, Irreverent Difficulty 36% and C2 level 23%. Discrimination Index shows that majority of MCQs were of Excellent Level (DI greater than 0.25) i.e 52 , Good 32% . (DI=2.15-0.25), Poor 16%. MCQs Distracter Effectiveness (DE)= 4, 3,2,1 were 52%, 34%, 14%, and 0% respectively. Conclusion: Item analysis gives us different parameters with reasons to recheck MCQ pool and teaching programme. High proportions of difficult and sizable amount of poor discriminative indices MCQs were the finding in this study and need to be resolved