COMPREHENSIVE ANALYSIS OF THE FORT INSTRUMENT: USING DISTRACTOR ANALYSIS TO EXPLORE STUDENTS’ SCIENTIFIC REASONING BASED ON ACADEMIC LEVEL AND GENDER DIFFERENCE

Scientific reasoning ability is essential to get developed in the current digital age, particularly in the process of judgement and decision-making in complex problems. Few studies have conducted an in-depth exploration of scientific reasoning ability, especially in relation to the confidence level and gender. The scientific reasoning ability of Indonesian upper-secondary school and university students were examined and compared with previous recorded data of US students. In this study, the data were collected from 372 university and 528 upper-secondary education students in Indonesia. Students’ scientific reasoning ability was measured using a scientific formal reasoning test (FORT). In addition, confidence level and metacognitive data was collected through self-reported measures. Two-way ANOVA was performed to compare mean differences between groups based on academic level and gender and to observe interaction between the variables. Students’ confidence level in selecting the correct answer and distractor answer was analyzed using an independent t-test. The results reveal that many Indonesian students selected specific distractors with relatively high confidence. Moreover, upper-secondary school students and female students selected more distractors than the groups’ counterparts. Finally, the factors related to Indonesian students’ responses to the scientific formal reasoning were discussed. Keywords: confidence level, distractor analysis, gender differences, scientific (formal) reasoning test, scientific reasoning ability, Indonesian student

The Development of Computer-Based Test (CBT) Based on Hypertext Preprocessor (PHP)

<p>This study aims to produce design, to know the profile and to determine the level of effectiveness of the Hypertext Preprocessor (PHP) based on Computer-Based Test (CBT) to Test of English For Academic Proficiency (TEAP) IKIP PGRI Pontianak. The development research method is the ADDIE development design model. The research subjects were students of the eighth semester English study program and the lecturers and staff who served at UPT Bahasa of IKIP PGRI Pontianak. Data collection tools used interviews and TEAP user questionnaires then analyzed with the formula from Khabibah. PHP-based CBT profile in the form of a web that can be accessed online and integrated with multiple-choice item analysis. This study found that TEAP requires a system that can automatically be correcting, storing test results, analysing test results and distributing test certificates. The results showed that the effectiveness level of the respondents' opinion of the test participants with a percentage score of 77% in good category and from the opinion of respondents who implement TEAP is very effective and can analyze the level of difficulty, difference power and distractor analysis properly and quickly. This illustrates that PHP-based CBT is suitable for the TEAP program.</p>

Nonfunctional distractor analysis: An indicator for quality of Multiple choice questions

Objectives: To analyze the low to medium distractor efficiency items in a multiple-choice question (MCQ) paper for item writing flaws. Methods: This qualitative study was conducted at Islamic International Medical College Rawalpindi, in October 2019. Archived item- analysis report from a midyear medium stakes MCQ paper of 2nd year MBBS class, was analyzed to determine the non-functional distractors (NFDs) and distractor efficiency (DE) of items, in a total of 181 MCQs. DE was categorized as low (3-4 NFDs), medium (1-2 NFDs) and high (0 NFD). Subsequently, qualitative document analysis of the MCQ paper whose item analysis report was assessed was conducted to investigate the item flaws in the low to medium DE items. The flaws identified were coded and grouped as, within option flaws, alignment flaws between options and stem/ lead-in and other flaws. Results: Distractor efficiency was high in 69 items (38%), moderate in 75 items (42%) and low in 37 items (20%). The item-writing flaws identified in low to moderate DE items within distractors included, non-homogenous length (1.8%), non-homogenous content (8%) and repeat in distractor (1.7%). Alignment flaws between distractors and stem/ lead-in identified were linguistic cues (10%), logic cues (12.5%) and irrelevant distractors (16%). Flaws unrelated to distractors were low cognitive level items (40%) and unnecessarily complicated stems (11.6%). Conclusions: Analyzing the low to medium DE items for item writing flaws, provides valuable information about item writing errors which negatively impact the distractor efficiency. doi: https://doi.org/10.12669/pjms.36.5.2439 How to cite this:Sajjad M, Iltaf S, Khan RA. Nonfunctional distractor analysis: An indicator for quality of Multiple choice questions. Pak J Med Sci. 2020;36(5):---------. doi: https://doi.org/10.12669/pjms.36.5.2439 This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

THE USE OF RASCH MEASUREMENT MODEL IN ENGLISH TESTING

This study aimed at determining the quality of the English Paper 1 (EP1) items of UPSR trial examination for six graders in terms of its reliability, validity and items characteristics. It also sought to determine the difficulty levels of 40 multiple-choice items consisting five constructs of vocabulary, language and social expression, grammar, cloze-comprehension and reading comprehension. A number of 525 primary schools students were randomly selected from 3876 students in Kuala Selangor, Malaysia. Using the Rasch measurement model, the validity evidences were shown through the results of Principle Component Analysis (PCA), fit statistics and item distractor analysis. The results from PCA analysis showed the absence of second dimension in the test, which met the assumption of modern testing theory. Fit statistics analyses have identified seven misfit items that are beyond the acceptable range (0.7 - 1.3 logit). Item distractor analysis has identified five problematic items whereby three of them are also misfit items. Summary statistics shows that the reliability indices of Cronbach’s Alpha were greater than 0.80 and separation indices were greater than 2. This study would benefit teachers in improving existing assessment practice by spreading out the importance of item analysis in schools, particularly in language testing.

Construction of Multiple Choice Questions Before and After An Educational Intervention

Introduction: Khesar Gyalpo University of Medical Sciences of Bhutan, established in 2014, has ushered in a new era in medical education in Bhutan. Multiple Choice Questions are a common means of written assessment in medical education. Methods: This was a quasi-experimental study conducted at the Faculty of Postgraduate Medicine, KGUMSB, Thimphu in December 2016. A total of 8 MCQs were prepared by four teaching faculties from different fields who had no prior training on construction of MCQs. It was delivered to a group of 16 randomly selected intern doctors. A 2 hours long workshop on construction of MCQs was conducted. After the workshop, the same MCQs were modified according to standard guidelines on developing MCQs and were tested in the same group of intern doctors. An analysis on the performance, difficulty factor, discrimination index and distractor analysis was done on the two sets of MCQs using Microsoft Excel and SPSS 20.0. Results: For the pre- and post-workshop questions respectively, the pass percentage was 69.8% (11) and 81.3% (13), difficulty factor was 0.51 and 0.53, discrimination index was 0.59 and 0.47, distractor effectiveness was 83.3% and 74.9%. Conclusions: The workshop on MCQ development apparently seemed highly valuable and effective in changing the learning and performances of medical educators in the development of MCQs. Keywords: difficulty factor; discrimination index; faculty development; medical education.