scholarly journals Pass Rate Set by Borderline Regression Method but not by Modified Angoff is Independent on Difficulty of Content in Objective Structured Clinical Exams

2019 ◽  
Author(s):  
Petr Waldauf ◽  
Jerome Cockings ◽  
Christian Sitzwohl ◽  
Marco Maggiorini ◽  
Paul Elbers ◽  
...  
Keyword(s):  
Item Difficulty ◽  
Practical Implication ◽  
Standard Setting ◽  
Regression Method ◽  
Pass Rate ◽  
Pass Rates ◽  
Angoff Method ◽  
Pass Mark ◽  
Computer Based ◽  
Modified Angoff Method

Abstract Background Standard setting is a method of determining the cut-off point on the scoring scale that separates the competent from the non-competent. This is a crucial feature of each exam. Pass rate should ideally be independent on the difficulty of exam content. Methods We compared the modified Angoff method (MAM) with the borderline regression method (BRM) of standard setting in 185 candidates examined by 137 examiners in the oral part of the European Diploma in Intensive Care exam, June 2018. We then compared the effect of removal of the hardest questions on the performance of the two techniques. The exam comprised 299 items in total across 6 OSCE stations. OSCE stations were of two types; short computer based OSCE stations (3 x 12 minutes), and longer structured discussion stations based on a clinical case (3 x 25 minutes). Our focus was the effect of item difficulty on the performance of the two standard setting techniques in determining the pass mark. Results MAM and BRM both led to similar pass rates overall for the shorter computer based 12 min OSCE stations. In the longer structured discussion 25 min stations MAM set a pass mark much higher than BRM, failing more of the candidates whose performance during the examination was judged by examiners on their global assessment as above the standard required to pass. Further analysis showed the exam items most affecting this were the more difficult items with lower discrimination; Angoff judges over-estimated the borderline candidates ability for these items. Elimination of these items led to convergence of pass marks achieved by the two methods. Conclusion Pass mark setting by Modified Angoff Method, but not by Borderline Regression Method, is influenced by the difficulty of exam content. This has practical implication for evaluating the results of OSCE exams.

scholarly journals Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School

2020 ◽  
Vol 7 ◽  
pp. 238212052098199
Author(s):  
Neelam Rekha Dwivedi ◽  
Narasimha Prasad Vijayashankar ◽  
Manisha Hansda ◽  
Arun Kumar Dubey ◽  
Fidelis Nwachukwu ◽  
...  
Keyword(s):  
Clinical Skills ◽  
Skills Training ◽  
Standard Setting ◽  
Regression Method ◽  
Organ System ◽  
Cutoff Score ◽  
Failure Rates ◽  
High Stake ◽  
Angoff Method ◽  
Modified Angoff Method

Background: OSCE are widely used for assessing clinical skills training in medical schools. Use of traditional pass fail cut off yields wide variations in the results of different cohorts of students. This has led to a growing emphasis on the application of standard setting procedures in OSCEs. Purpose/aim: The purpose of the study was comparing the utility, feasibility and appropriateness of 4 different standard setting methods with OSCEs at XUSOM. Methods: A 15-station OSCE was administered to 173 students over 6 months. Five stations were conducted for each organ system (Respiratory, Gastrointestinal and Cardiovascular). Students were assessed for their clinical skills in 15 stations. Four different standard setting methods were applied and compared with a control (Traditional method) to establish cut off scores for pass/fail decisions. Results: OSCE checklist scores revealed a Cronbach’s alpha of 0.711, demonstrating acceptable level of internal consistency. About 13 of 15 OSCE stations performed well with “Alpha if deleted values” lower that 0.711 emphasizing the reliability of OSCE stations. The traditional standard setting method (cut off score of 70) resulted in highest failure rate. The Modified Angoff Method and Relative methods yielded the lowest failure rates, which were typically less than 10% for each system. Failure rates for the Borderline methods ranged from 28% to 57% across systems. Conclusions: In our study, Modified Angoff method and Borderline regression method have shown to be consistently reliable and practically suitable to provide acceptable cut-off score across different organ system. Therefore, an average of Modified Angoff Method and Borderline Regression Method appeared to provide an acceptable cutoff score in OSCE. Further studies, in high-stake clinical examinations, utilizing larger number of judges and OSCE stations are recommended to reinforce the validity of combining multiple methods for standard setting.

scholarly journals Variation in performance on common content items at UK medical schools

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
David Hope ◽  
David Kluth ◽  
Matthew Homer ◽  
Avril Dewar ◽  
Richard Fuller ◽  
...  
Keyword(s):  
Undergraduate Students ◽  
Standard Setting ◽  
Medical Schools ◽  
Cross Sectional Study ◽  
Pass Rates ◽  
Cross Sectional ◽  
Study Results ◽  
Institutional Differences ◽  
Common Content ◽  
The Impact

Abstract Background Due to differing assessment systems across UK medical schools, making meaningful cross-school comparisons on undergraduate students’ performance in knowledge tests is difficult. Ahead of the introduction of a national licensing assessment in the UK, we evaluate schools’ performances on a shared pool of “common content” knowledge test items to compare candidates at different schools and evaluate whether they would pass under different standard setting regimes. Such information can then help develop a cross-school consensus on standard setting shared content. Methods We undertook a cross-sectional study in the academic sessions 2016-17 and 2017-18. Sixty “best of five” multiple choice ‘common content’ items were delivered each year, with five used in both years. In 2016-17 30 (of 31 eligible) medical schools undertook a mean of 52.6 items with 7,177 participants. In 2017-18 the same 30 medical schools undertook a mean of 52.8 items with 7,165 participants, creating a full sample of 14,342 medical students sitting common content prior to graduation. Using mean scores, we compared performance across items and carried out a “like-for-like” comparison of schools who used the same set of items then modelled the impact of different passing standards on these schools. Results Schools varied substantially on candidate total score. Schools differed in their performance with large (Cohen’s d around 1) effects. A passing standard that would see 5 % of candidates at high scoring schools fail left low-scoring schools with fail rates of up to 40 %, whereas a passing standard that would see 5 % of candidates at low scoring schools fail would see virtually no candidates from high scoring schools fail. Conclusions Candidates at different schools exhibited significant differences in scores in two separate sittings. Performance varied by enough that standards that produce realistic fail rates in one medical school may produce substantially different pass rates in other medical schools – despite identical content and the candidates being governed by the same regulator. Regardless of which hypothetical standards are “correct” as judged by experts, large institutional differences in pass rates must be explored and understood by medical educators before shared standards are applied. The study results can assist cross-school groups in developing a consensus on standard setting future licensing assessment.

scholarly journals Evaluation of a course to prepare international students for the United States Medical Licensing Examination step 2 clinical skills exam

2017 ◽  
Vol 14 ◽  
pp. 25 ◽  
Author(s):  
Rachel B. Levine ◽  
Andrew P. Levy ◽  
Robert Lubin ◽  
Sarah Halevi ◽  
Rebeca Rios ◽  
...  
Keyword(s):  
United States ◽  
Medical School ◽  
Clinical Skills ◽  
The United States ◽  
International Medical ◽  
Pass Rate ◽  
Pass Rates ◽  
Medical Licensing ◽  
Licensing Examination ◽  
Usmle Step

Purpose: United States (US) and Canadian citizens attending medical school abroad often desire to return to the US for residency, and therefore must pass US licensing exams. We describe a 2-day United States Medical Licensing Examination (USMLE) step 2 clinical skills (CS) preparation course for students in the Technion American Medical School program (Haifa, Israel) between 2012 and 2016.Methods: Students completed pre- and post-course questionnaires. The paired t-test was used to measure students’ perceptions of knowledge, preparation, confidence, and competence in CS pre- and post-course. To test for differences by gender or country of birth, analysis of variance was used. We compared USMLE step 2 CS pass rates between the 5 years prior to the course and the 5 years during which the course was offered.Results: Ninety students took the course between 2012 and 2016. Course evaluations began in 2013. Seventy-three students agreed to participate in the evaluation, and 64 completed the pre- and post-course surveys. Of the 64 students, 58% were US-born and 53% were male. Students reported statistically significant improvements in confidence and competence in all areas. No differences were found by gender or country of origin. The average pass rate for the 5 years prior to the course was 82%, and the average pass rate for the 5 years of the course was 89%.Conclusion: A CS course delivered at an international medical school may help to close the gap between the pass rates of US and international medical graduates on a high-stakes licensing exam. More experience is needed to determine if this model is replicable.

A COMPARATIVE STUDY OF TWO STANDARD-SETTING TECHNIQUES

2007 ◽  
Vol 35 (4) ◽  
pp. 479-486 ◽  
Author(s):  
Abdalla A. Alsmadi
Keyword(s):  
Comparative Study ◽  
Strong Evidence ◽  
Standard Setting ◽  
Introductory Psychology ◽  
Angoff Method ◽  
Significant Difference ◽  
Passing Score

This study compares the Angoff method (1971) for setting standards with the Ebel method (1972), using a test in introductory psychology. Although results showed a significant difference in the minimum passing score obtained through these methods, results also showed strong evidence of agreement between both methods about the number of students classified as pass/fail. The Angoff method tended to be more stringent and produce a higher minimum passing score than the Ebel method.

Assessing Curriculum Efficiency Through Monte Carlo Simulation

2018 ◽  
Vol 22 (4) ◽  
pp. 597-610
Author(s):  
David Torres ◽  
Jorge Crichigno ◽  
Carmella Sanchez
Keyword(s):  
Monte Carlo Simulation ◽  
Monte Carlo ◽  
Frequency Distribution ◽  
Relative Frequency ◽  
Monte Carlo Algorithm ◽  
Pass Rate ◽  
Random Numbers ◽  
Pass Rates ◽  
Low Pass ◽  
Relative Frequency Distribution

A Monte Carlo algorithm is designed to predict the average time to graduate by enrolling virtual students in a degree plan. The algorithm can be used to improve graduation rates by identifying bottlenecks in a degree plan (e.g., low pass rate courses and prerequisites). Random numbers are used to determine whether students pass or fail classes by comparing them to institutional pass rates. Courses cannot be taken unless prerequisites and corequisites are satisfied. The output of the algorithm generates a relative frequency distribution which plots the number of students who graduate by semester. Pass rates of courses can be changed to determine the courses that have the greatest impact on the time to graduate. Prerequisites can also be removed to determine whether certain prerequisites significantly affect the time to graduate.

Benefits of Mock Oral Examinations in a Multi-institutional Consortium for Board Certification in General Surgery Training

2009 ◽  
Vol 75 (9) ◽  
pp. 817-821 ◽  
Author(s):  
Gokulakkrishna Subhas ◽  
Stephen Yoo ◽  
Yeon-Jeen Chang ◽  
David Peiper ◽  
Mark J. Frikker ◽  
...  
Keyword(s):  
Real Life ◽  
Oral Examination ◽  
Pass Rate ◽  
Teaching Hospitals ◽  
Pass Rates ◽  
Oral Examinations ◽  
The Many ◽  
Certifying Examination ◽  
The Cost ◽  
American Board Of Surgery

The Southeast Michigan Center for Medical Education (SEMCME) is a consortium of teaching hospitals in the Greater Detroit metropolitan area. SEMCME pools its resources for several educational means, including mock oral board examinations. The educational and cost benefits to mock oral examinations on a multi-institutional basis in preparation for the American Board of Surgery (ABS) certifying examination were analyzed. Ten-year multi-institution data from the mock oral examinations were correlated with ABS certifying examination pass rates. Mock oral examination scores were available for 107 of 147 graduates, which included 12 candidates who failed their certifying examination on the first attempt (pass rate = 89%). Four of 31 examinees who had a low score (4.9 or less) in their mock oral exams failed their certifying examination in their first attempt. The cost of running the mock examination was low (approximately $35/resident for 50 residents). When graduates from the last 10 years were surveyed, the majority of respondents believed that the mock oral examination helped in their success and with their preparation for the certifying examination. Thus, the many benefits of administering the examination with the resources of a consortium of hospitals result in the accurate reproduction of real-life testing conditions with reasonable overall costs per resident.

Sign in / Sign up

Export Citation Format

Share Document