Defining and tracking medical student self-monitoring using multiple-choice question item certainty

Abstract Background Self-monitoring is an important component of clinical practice. It is underpinned by the framework of self-efficacy which is concerned with judgments of how well one believes one can achieve or perform a task. This research aimed to develop criteria for adequate self-monitoring, then to measure patterns of self-monitoring, and to explore how these patterns relate to a student’s year in a medical course and to patterns of knowledge. Methods Analysis of individuals’ levels of correctness in answering assessment items and their certainty in correctness may be used to inform assessments of ability to self-monitor. Two criteria were proposed and applied to define adequate self-monitoring. Firstly, increasing proportions correct with increasing levels of certainty. Secondly, having a proportion correct for high certainty responses that was not lower than cohort levels. Student responses in progress tests comprising multiple-choice questions (MCQs) and associated certainty were analysed. Criteria for the presence of adequate self-monitoring and for adequate knowledge were applied to the results of each of four tests conducted over 2 years, and used to categorise patterns of self-monitoring and knowledge. Results Data from 3 year group cohorts totalling 737 students were analysed. The majority (58%) of students demonstrated adequate knowledge and met both criteria for adequate self-monitoring across all four tests. The most advanced year group cohort had the highest rates of adequate knowledge and the highest rates of meeting both self-monitoring criteria. The patterns of self-monitoring were the same as the patterns of knowledge across the four tests for 454 students, but for the remaining 283 the patterns of self-monitoring and knowledge differed. Conclusion Analysis of responses to item level certainty has informed development of a definition of adequate self-monitoring that may be applied to individual student’s responses from a single test, and to track the adequacy of a student’s self-monitoring over time. Patterns of self-monitoring tend to match patterns of knowledge, but not in all cases, suggesting the self-monitoring measure could provide additional information about student ability.

Download Full-text

New Indices for Refining Multiple Choice Questions

Journal of Probability and Statistics ◽

10.1155/2014/240263 ◽

2014 ◽

Vol 2014 ◽

pp. 1-8 ◽

Cited By ~ 3

Author(s):

Mariano Amo-Salas ◽

María del Mar Arroyo-Jimenez ◽

David Bustos-Escribano ◽

Eva Fairén-Jiménez ◽

Jesús López-Fidalgo

Keyword(s):

Multiple Choice ◽

Reliability And Validity ◽

Large Database ◽

Multiple Choice Questions ◽

Chi Square ◽

Lack Of Fit ◽

Question Item ◽

Measure Reliability ◽

The Relationship

Multiple choice questions (MCQs) are one of the most popular tools to evaluate learning and knowledge in higher education. Nowadays, there are a few indices to measure reliability and validity of these questions, for instance, to check the difficulty of a particular question (item) or the ability to discriminate from less to more knowledge. In this work two new indices have been constructed: (i) the no answer index measures the relationship between the number of errors and the number of no answers; (ii) the homogeneity index measures homogeneity of the wrong responses (distractors). The indices are based on the lack-of-fit statistic, whose distribution is approximated by a chi-square distribution for a large number of errors. An algorithm combining several traditional and new indices has been developed to refine continuously a database of MCQs. The final objective of this work is the classification of MCQs from a large database of items in order to produce an automated-supervised system of generating tests with specific characteristics, such as more or less difficulty or capacity of discriminating knowledge of the topic.

Download Full-text

Enhancing Electronic Examinations through Advanced Multiple-Choice Questionnaires

Virtual Learning Environments ◽

10.4018/978-1-4666-0011-9.ch806 ◽

2012 ◽

pp. 1645-1664

Author(s):

Dimos Triantis ◽

Errikos Ventouras

Keyword(s):

Multiple Choice ◽

Content Management ◽

Management Systems ◽

Special Focus ◽

Multiple Choice Questions ◽

Constructed Response ◽

Adequate Knowledge ◽

Learning Content ◽

Training Activities ◽

And Training

The present chapter deals with the variants of grading schemes that are applied in current Multiple-Choice Questions (MCQs) tests. MCQs are ideally suited for electronic examinations, which, as assessment items, are typically developed in the framework of Learning Content Management Systems (LCMSs) and handled, in the cycle of educational and training activities, by Learning Management Systems (LMS). Special focus is placed in novel grading methodologies, that enable to surpass the limitations and drawbacks of the most commonly used grading schemes for MCQs in electronic examinations. The paired MCQs grading method, according to which a set of pairs of MCQs is composed, is presented. The MCQs in each pair are similar concerning the same topic, but this similarity is not evident for an examinee that does not possess adequate knowledge on the topic addressed in the questions of the pair. The adoption of the paired MCQs grading method might expand the use of electronic examinations, provided that the new method proves its equivalence to traditional methods that might be considered as standard, such as constructed response (CR) tests. Research efforts to that direction are presented.

Download Full-text

Enhancing Electronic Examinations through Advanced Multiple-Choice Questionnaires

Higher Education Institutions and Learning Management Systems ◽

10.4018/978-1-60960-884-2.ch009 ◽

2012 ◽

pp. 178-198 ◽

Cited By ~ 1

Author(s):

Dimos Triantis ◽

Errikos Ventouras

Keyword(s):

Multiple Choice ◽

Content Management ◽

Management Systems ◽

Special Focus ◽

Multiple Choice Questions ◽

Constructed Response ◽

Adequate Knowledge ◽

Learning Content ◽

Training Activities ◽

And Training

Download Full-text

Knowledge self-monitoring, efficiency, and determinants of self-confidence statement in multiple choice questions in medical students

BMC Medical Education ◽

10.1186/s12909-020-02352-6 ◽

2020 ◽

Vol 20 (1) ◽

Author(s):

Nahid Tabibzadeh ◽

Jimmy Mullaert ◽

Lara Zafrani ◽

Pauline Balagny ◽

Justine Frija-Masson ◽

...

Keyword(s):

Clinical Practice ◽

Medical Students ◽

Multiple Choice ◽

Cross Sectional Study ◽

Multiple Choice Questions ◽

Scoring Method ◽

Cross Sectional ◽

Self Monitoring ◽

Self Confidence ◽

Level Of Confidence

Abstract Background Multiple-choice question (MCQ) tests are commonly used to evaluate medical students, but they do not assess self-confidence nor penalize lucky guess or harmful behaviors. Based on a scoring method according to the appropriateness of confidence in answers, the study aimed at assessing knowledge self-monitoring and efficiency, and the determinants of self-confidence. Methods A cross-sectional study of 842 s- and third-year medical students who were asked to state their level of confidence (A: very confident, B: moderately confident and C: not confident) during 12 tests (106,806 events). A bonus was applied if the level of confidence matched with the correctness of the answer, and a penalty was applied in the case of inappropriate confidence. Results Level A was selected more appropriately by the top 20% students whereas level C was selected more appropriately by the lower 20% students. Efficiency of higher-performing students was higher when correct (among correct answers, rate of A statement), but worse when incorrect compared to the bottom 20% students (among incorrect answers, rate of C statement). B and C statements were independently associated with female and male gender, respectively (OR for male vs female = 0.89 [0.82–0.96], p = 0.004, for level B and 1.15 [1.01–1.32], p = 0.047, for level C). Conclusion While both addressing the gender confidence gap, knowledge self-monitoring might improve awareness of students’ knowledge whereas efficiency might evaluate appropriate behavior in clinical practice. These results suggest differential feedback during training in higher versus lower-performing students, and potentially harmful behavior in decision-making during clinical practice in higher-performing students.

Download Full-text

Institutions collaborating on an information literacy assessment tool

Library Management ◽

10.1108/lm-03-2014-0035 ◽

2014 ◽

Vol 35 (8/9) ◽

pp. 538-546 ◽

Cited By ~ 6

Author(s):

Sara Sharun ◽

Michelle Edwards Thomson ◽

Nancy Goebel ◽

Jessica Knoch

Keyword(s):

Student Learning ◽

Undergraduate Students ◽

Information Literacy ◽

Assessment Tool ◽

Multiple Choice ◽

Clear Understanding ◽

Multiple Choice Questions ◽

Student Responses ◽

Content Type ◽

Wide Range

Purpose – The purpose of this paper is to create an information literacy (IL) instruction assessment tool that responds to the unique needs of individual institutions and provides a strategic and relevant model for assessing IL skills among undergraduate students. Design/methodology/approach – The research team designed a post-test questionnaire comprised of two demographic questions, two open-ended questions and a pool of skill-based multiple-choice questions mapped to Association of College and Research Libraries Information Literacy (ACRL IL) Competency Standards for Higher Education. Participating librarians used a customized questionnaire to assess student learning at the end of their one-shot instruction sessions. Findings – In their responses to the multiple-choice questions, students demonstrated a clear understanding of ethical use of information and a strong ability to select appropriate tools for accessing information sources. Student responses to the open-ended questions revealed a wide range of confidence and ability levels, and provided insight into the frequency, depth and breadth with which various ACRL Standards are being addressed in library sessions. Research limitations/implications – This paper reports on student responses to questions that have subsequently been identified as problematic; therefore, strong inferences cannot be made about student learning from these responses. Questions have since been improved with further revision. In addition, the sample sizes for individual questions were too small to be generalizable. Practical implications – The intentional and strategic approach to the development of the assessment tool and its implementation is that it be practical and easy to implement for partner libraries. It is intended to make assessment of IL in the undergraduate context be assessable to all academic librarians who desire to participate. Originality/value – This paper describes a unique assessment tool that is designed to be responsive to local needs and provide a cost-free assessment option for academic libraries.

Download Full-text

Using Learning Analytics to evaluate the quality of multiple-choice questions

International Journal of Information and Learning Technology ◽

10.1108/ijilt-02-2019-0023 ◽

2019 ◽

Vol 36 (4) ◽

pp. 322-341 ◽

Cited By ~ 2

Author(s):

Jose Manuel Azevedo ◽

Ema P. Oliveira ◽

Patrícia Damas Beites

Keyword(s):

Learning Analytics ◽

Classical Test Theory ◽

Multiple Choice ◽

Assessment Method ◽

Test Theory ◽

Multiple Choice Questions ◽

Data Set ◽

Content Type ◽

Definition Of

Purpose The purpose of this paper is to find appropriate forms of analysis of multiple-choice questions (MCQ) to obtain an assessment method, as fair as possible, for the students. The authors intend to ascertain if it is possible to control the quality of the MCQ contained in a bank of questions, implemented in Moodle, presenting some evidence with Item Response Theory (IRT) and Classical Test Theory (CTT). The used techniques can be considered a type of Descriptive Learning Analytics since they allow the measurement, collection, analysis and reporting of data generated from students’ assessment. Design/methodology/approach A representative data set of students’ grades from tests, randomly generated with a bank of questions implemented in Moodle, was used for analysis. The data were extracted from a Moodle database using MySQL with an ODBC connector, and collected in MS ExcelTM worksheets, and appropriate macros programmed with VBA were used. The analysis with the CTT was done through appropriate MS ExcelTM formulas, and the analysis with the IRT was approached with an MS ExcelTM add-in. Findings The Difficulty and Discrimination Indexes were calculated for all the questions having enough answers. It was found that the majority of the questions presented values for these indexes, which leads to a conclusion that they have quality. The analysis also showed that the bank of questions presents some internal consistency and, consequently, some reliability. Groups of questions with similar features were obtained, which is very important for the teacher to develop tests as fair as possible. Originality/value The main contribution and originality that can be found in this research is the definition of groups of questions with similar features, regarding their difficulty and discrimination properties. These groups allow the identification of difficulty levels in the questions on the bank of questions, thus allowing teachers to build tests, randomly generated with Moodle, that include questions with several difficulty levels in the tests, as it should be done. As far as the authors’ knowledge, there are no similar results in the literature.

Download Full-text