scholarly journals Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

2021 ◽  
Vol 18 ◽  
pp. 25
Author(s):  
Song Yi Park ◽  
Sang-Hwa Lee ◽  
Min-Jeong Kim ◽  
Ki-Hwan Ji ◽  
Ji Ho Ryu
Keyword(s):  
Clinical Examination ◽  
Standard Setting ◽  
Regression Method ◽  
Objective Structured Clinical Examination ◽  
Group Method ◽  
Global Rating ◽  
Cut Scores ◽  
Cut Score ◽  
Significant Difference ◽  
Fourth Year Medical Students

Purpose: Setting standards is critical in health professions. However, appropriate standard setting methods do not always apply to the set cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed from the norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.Methods: This was an explorative study to model of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at seven stations with encountering standardized patients (SPs) and one station with performing skills on a manikin on 15 July 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.Results: The cut score of the norm-referenced method was lower than that of the BGM (p<0.01) and BRM (p<0.02). There was no significant difference in the cut score between the BGM and BRM (p=0.40). The station with the highest standard deviation and the highest proportion of the borderline group showed the largest cut score difference in standard setting methods.Conclusion: Prefixed cut scores by the norm-referenced method without considering station contents or examinee performance can vary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensus on the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

scholarly journals Standard setting in objective structured clinical examination: Assigning a pass/fail cut score

2017 ◽  
Vol 5 (1) ◽  
pp. 8
Author(s):  
Abdullah Alnemari ◽  
Fadi Munshi ◽  
Hatim Al-Jifree ◽  
Abdulaziz Alshehri
Keyword(s):  
Clinical Examination ◽  
Standard Setting ◽  
Objective Structured Clinical Examination ◽  
Cut Score

Reconciling Experts’ Differences in Setting Cut Scores for Pass-Fail Decisions

1996 ◽  
Vol 21 (3) ◽  
pp. 203-213 ◽  
Author(s):  
Nicholas T. Longford
Keyword(s):  
Logistic Regression ◽  
Random Effects ◽  
Standard Error ◽  
Standard Setting ◽  
Regression Method ◽  
Educational Testing ◽  
Cut Scores ◽  
Routine Procedure ◽  
Cut Score ◽  
Logistic Regression Method

Standard setting is a routine procedure in educational testing. Each examinee in the administration of a test is assigned a score, and the responses of a sample of the examinees are reviewed by a panel of experts. Each expert rates every selected examinee as pass or fail. Based on these ratings a cut score is to be established. A random effects logistic regression method is applied to set the cut score and to estimate the associated standard error. Two examples illustrate that taking account of between-rater differences is essential.

scholarly journals Traditional Versus Objective Structured Clinical Examination on Pediatric Nursing Students’ Performance

2020 ◽  
Vol 2 (1) ◽  
pp. 11
Author(s):  
Khadiga M. Said ◽  
Safaa F. Draz
Keyword(s):  
Nursing Students ◽  
Clinical Examination ◽  
Clinical Skills ◽  
Objective Structured Clinical Examination ◽  
Simple Random Sample ◽  
Control Group ◽  
Pediatric Nursing ◽  
Clinical Scenarios ◽  
Significant Difference ◽  
Nursing Skills

Context: The Objective Structured Clinical Examination (OSCE) method is an effective tool for evaluating the clinical nursing skills of nursing students. OSCE is an assessment technique in which the student demonstrates their competence under a variety of simulated conditions Aim: This study aimed to compare objective structured clinical examinations versus traditional clinical examination on pediatric nursing students' performance. Methods: A quasi-experimental research design (study and control group) was used to conduct this study. The study was conducted in the laboratory of pediatric nursing skills at the faculty of nursing, Benha University. All male and female pediatric nursing students in the 3rd year (n. =228), who were studying in the academic year 2018-2019, first semester, faculty of nursing, Benha University, was recruited. A simple random sample chose to achieve the aim of this study. The odd number was for the OSCE group and the even number for the traditional clinical examination (TCE) group. Four tools were utilized to collect data for the current study. A Structured Questionnaire Sheet, A modified Self-administered Questionnaire, Pediatric Nursing Students' Practice Observational Checklists, and Clinical Scenarios were designed to assess the pediatric nursing student OSCE exam and compare between the traditional method and OSCE method of exams. Results: The current study discovered statistically significant differences were found with a high percentage of agreement responses among pediatric nursing students for related items of OSCE method compared to those in TCE. Additionally, the results clarify a highly statistically significant difference between the studied pediatric nursing students' in TCE and OSCE total performance scores. Conclusion: Objective Structured Clinical Examination (OSCE) was opinioned as a tool for clinical evaluation. This finding appeared in pediatric nursing students' responses, which confirmed their acceptance of OSCE. The OSCE subsequently remains a more objective method of assessment than the traditional clinical forms of the exam that was previously used. OSCE can be used most effectively in undergraduate nursing curricula to assess fair practice. This type of exam provided an accurate measure of clinical skill competencies. Therefore, OSCE should be adopted as a strategy for examining clinical skills for students in all academic years. The current study recommended that Objective Structured Clinical Examination can be used as effective and meaningful assistance to fitness for practice, and OSCE should be adopted as a strategy for examining clinical skills for students in all academic years.

scholarly journals The effect of a stress and anxiety coping program on objective structured clinical examination performance among nursing students in shiraz, Iran

2020 ◽  
Vol 20 (1) ◽  
Author(s):  
Sadaf Mojarrab ◽  
Leila Bazrafkan ◽  
Azita Jaberi
Keyword(s):  
Nursing Students ◽  
Clinical Examination ◽  
Relaxation Training ◽  
Substantial Reduction ◽  
Intervention Group ◽  
Demographic Characteristics ◽  
Objective Structured Clinical Examination ◽  
Control Group ◽  
Significant Difference ◽  
Exam Anxiety

Abstract Background Evaluation of the competence and practical skills of nursing students, using the objective structured clinical examination (OSCE), is an integral part of the nursing education program. However, their performance could be negatively influenced by a significant level of stress and anxiety prior to the test. The present study aimed to evaluate the effect of an anxiety coping program on the OSCE performance level of first-year nursing students in Shiraz, Iran. Methods The present quasi-experimental study was conducted among 76 nursing students; control group (n = 35) and intervention group (n = 41). To attain the study purpose, the intervention group received a pre-exam anxiety coping program that included relaxation and soothing techniques, diaphragmatic breathing training, and progressive muscle relaxation training accompanied by light instrumental music, while the control group received no intervention before the exam. Data collection instruments included a demographic characteristics form and State-Trait Anxiety Inventory (STAI) questionnaire. Demographic characteristics of the participants indicated an overall homogeneity within the study population. The STAI questionnaire was filled in before and after the OSCE and the results were compared with those of the control group. The data were analysed using SPSS software (version 22.0). P < 0.05 was considered statistically significant. Results As a result of the anxiety coping program, a substantial reduction in the anxiety score (by 11.61 units) in the intervention group was observed. There was a significant difference in the pre- and post-exam anxiety scores between the control and intervention groups (P < 0.001). The anxiety coping program improved the examination results of nursing students in the final exam compared to the midterm results (an increase of 0.9487 units, P < 0.001). Conclusion The anxiety coping program reduced the anxiety level among nursing students and improved their OSCE results. Our findings can be utilized to better evaluate clinical activities in different medical and paramedical groups. Moreover, educators can implement such coping programs prior to evaluations in order to effectively assess the knowledge, attitude, and performance of the students.

scholarly journals Estimasi kesalahan pengukuran standard setting dalam penilaian kompetensi matematika tingkat SMP di Kabupaten Sumbawa

2018 ◽  
Vol 22 (1) ◽  
pp. 24-34
Author(s):  
Weni Wendari ◽  
Samsul Hadi
Keyword(s):  
Measurement Error ◽  
Focus Group ◽  
Error Estimation ◽  
Focus Group Discussion ◽  
Generalizability Theory ◽  
Group Discussion ◽  
Standard Setting ◽  
Group Method ◽  
Theory Approach ◽  
Cut Score

Penelitian ini bertujuan untuk mengetahui metode yang lebih akurat dalam mengestimasi kesalahan pengukuran standard setting pada metode Ebel, Bookmark, dan Contrasting group. Data penelitian ini merupakan dokumen Dinas Pendidikan dan Kebudayaan Kabupaten Sumbawa berupa respon peserta Ujian Nasional Matematika Paket P0C5520 tahun ajaran 2015/2016 yang berjumlah 352 siswa. Guru juga dilibatkan dalam penelitian sebagai panelis dalam Focus Group Discussion  (FGD). Data yang terkumpul kemudian dianalisis melaui tiga tahap.. Tahap pertama yaitu persiapan, kegiatan pada tahap ini mencakup penyiapan data, penggolongan SMP, dan penentuan karakteristik butir. Tahap kedua yaitu FGD dilakukan dalam dua putaran. Tahap ketiga yaitu mengestimasi kesalahan pengukuran dengan menggunakan pendekatan Generalizability Theory dengan bantuan program eduG. Hasil penelitian menunjukkan bahwa metode Contrasting group memiliki estimasi kesalahan pengukuran paling kecil dibandingkan metode Ebel dan Bookmark, oleh karena itu, metode Contrasting grouplebih akurat dibandingkan dengan dua metode lainnya.Kata kunci: cut score, standard setting, generalizability theory MEASUREMENT ERROR ESTIMATION OF STANDARD SETTING IN MATHEMATICS COMPETENCY ASSESSMENT FOR JUNIOR HIGH SCHOOL IN SUMBAWA REGENCYAbstractThis research aims to find the most accurate methods in estimating measurement error of standard setting among Ebel, Bookmark, and Contrasting group methods. The data used in this study were 352 students’ responses on Mathematics National Exam Package P0C5520 in the academic year of 2015/2016. The document was collected from the Department of Education and Culture in Sumbawa Regency. Teachers were also involved in this research as panelists in the Focus Group Discussion (FGD). The data collected were then analyzed through three stages. The first stage was preparation stage, including the activities of data preparation, school classification, and item characteristics analysis. The second stage was two-round FGD. The third stage was estimating the measurement error using Generalizability Theory approach assisted by eduG program. The research result shows that Contrasting Group method produces the smallest measurement error estimation compared to Ebel and Bookmark methods, therefore, Contrasting group method is considered as the most accurate method.Keywords: cut score, standard setting, generalizability theory

scholarly journals Standardsetting av læringsstøttende prøver i engelsk for Vg1

2018 ◽  
Vol 12 (4) ◽  
pp. 15
Author(s):  
Eli Moe ◽  
Hildegunn Lahlum Helness ◽  
Craig Grocott ◽  
Norman Verhelst
Keyword(s):  
Standard Setting ◽  
Good Knowledge ◽  
Cut Scores ◽  
Listening Tests ◽  
Cut Score ◽  
Margin Of Error ◽  
The Common ◽  
11Th Grade ◽  
Different Levels ◽  
Level Of Agreement

Formålet med denne artikkelen er å beskrive framgangsmåten som ble brukt for å bestemme kuttskårer (grenser) mellom tre nivåer i Det europeiske ramme-verket for språk (A2, B1 og B2) på to læringsstøttende lytteprøver i engelsk for Vg1-elever. Målet har vært å undersøke om det er mulig å etablere enighet om kuttskårene, og om standardsetterne som deltok i arbeidet fikk tilstrekkelig opp-læring på forhånd. Videre var det et mål å se på hvilke konsekvenser kuttskårene vil få for fordeling av elever på de ulike rammeverksnivåene. Standardsettingen ble gjennomført med utgangspunkt i pilotdata fra 3199 elever på Vg1, Cito-metoden og 16 panelmedlemmer med god kjennskap til Rammeverkets nivåer. Flere av panelmedlemmene var eller hadde vært lærere i engelsk for elever på 10. trinn eller Vg1. Cito-metoden fungerte bra for å etablere kuttskårer som standardsetterne var forholdsvis enige om. Sluttresultatene viser at målefeilen var relativt liten. Resultatene viser større enighet om kuttskåren mellom nivåene B1 og B2 enn mellom A2 og B1, og dette kan ha en sammenheng med at det ble brukt mer tid på forberedelsesarbeid for B1 og B2. Lærere i panelet som kjenner elevgruppa godt, mener at konsekvensen kutt-skåren har for fordeling av elever på de ulike rammeverksnivåene, stemmer med deres egen vurdering av elevenes lytteferdigheter.Nøkkelord: standardsetting, testsentrert metode, Cito-metoden, standard, kutt-skår, vippekandidatStandard setting for English tests for 11th grade students in NorwayAbstractThis article presents the process used to determine the cut scores between three levels of the Common European Framework of Reference for languages (A2, B1 and B2) for two English listening tests, taken by Norwegian pupils at the 11th grade. The aim was to establish whether agreement can be reached on cut scores and whether the standard setters received enough preparation before the event. Another aim was to examine the potential consequences the cut scores would have for the distribution of pupils across the different levels. The standard setting took place using pilot data from 3199 pupils, the Cito method and 16 panel members with a good knowledge of the framework levels. Some panel members were or had been 10th or 11th grade English teachers. The Cito method worked well for establishing cut scores with which the panel members mostly agreed. The results indicated a small margin of error. The results showed a higher level of agreement for the cut score between B1 and B2 than between A2 and B1, possibly connected to the longer preparation time dedicated to B1 and B2. Teachers on the panel with good knowledge of the pupil base believe that the consequences these cut scores have for the distribution of pupils, correlate with their own experiences of pupils' ability.Keywords: standard setting, test-centered method, the Cito method, standard, cut score, borderline person / minimally competent user

scholarly journals Setting a standard for low reading proficiency: A comparison of the bookmark procedure and constrained mixture Rasch model

PLoS ONE ◽  
2021 ◽  
Vol 16 (11) ◽  
pp. e0257871
Author(s):  
Tabea Feseker ◽  
Timo Gnambs ◽  
Cordula Artelt
Keyword(s):  
Rasch Model ◽  
Panel Study ◽  
External Validation ◽  
Reading Proficiency ◽  
Standard Setting ◽  
Cut Scores ◽  
Model Based ◽  
Constrained Mixture ◽  
Cut Score ◽  
Efficient Alternative

In order to draw pertinent conclusions about persons with low reading skills, it is essential to use validated standard-setting procedures by which they can be assigned to their appropriate level of proficiency. Since there is no standard-setting procedure without weaknesses, external validity studies are essential. Traditionally, studies have assessed validity by comparing different judgement-based standard-setting procedures. Only a few studies have used model-based approaches for validating judgement-based procedures. The present study addressed this shortcoming and compared agreement of the cut score placement between a judgement-based approach (i.e., Bookmark procedure) and a model-based one (i.e., constrained mixture Rasch model). This was performed by differentiating between individuals with low reading proficiency and those with a functional level of reading proficiency in three independent samples of the German National Educational Panel Study that included students from the ninth grade (N = 13,897) as well as adults (Ns = 5,335 and 3,145). The analyses showed quite similar mean cut scores for the two standard-setting procedures in two of the samples, whereas the third sample showed more pronounced differences. Importantly, these findings demonstrate that model-based approaches provide a valid and resource-efficient alternative for external validation, although they can be sensitive to the ability distribution within a sample.

Sign in / Sign up

Export Citation Format

Share Document