Development and evaluation of a simulation-based resuscitation scenario assessment tool for emergency medicine residents

ABSTRACT Objective: We sought to develop and validate a three-station simulation-based Objective Structured Clinical Examination (OSCE) tool to assess emergency medicine resident competency in resuscitation scenarios. Methods: An expert panel of emergency physicians developed three scenarios for use with high-fidelity mannequins. For each scenario, a corresponding assessment tool was developed with an essential actions (EA) checklist and a global assessment score (GAS). The scenarios were (1) unstable ventricular tachycardia, (2) respiratory failure, and (3) ST elevation myocardial infarction. Emergency medicine residents were videotaped completing the OSCE, and three clinician experts independently evaluated the videotapes using the assessment tool. Results: Twenty-one residents completed the OSCE (nine residents in the College of Family Physicians of Canada– Emergency Medicine [CCFP-EM] program, six junior residents in the Fellow of the Royal College of Physicians of Canada–Emergency Medicine [FRCP-EM] program, six senior residents in the FRCP-EM). Interrater reliability for the EA scores was good but varied between scenarios (Spearman rho 5 [1] 0.68, [2] 0.81, [3] 0.41). Interrater reliability for the GAS was also good, with less variability (rho 5 [1] 0.64, [2] 0.56, [3] 0.62). When comparing GAS scores, senior FRCP residents outperformed CCFP-EM residents in all scenarios and junior residents in two of three scenarios (p , 0.001 to 0.01). Based on EA scores, senior FRCP residents outperformed CCFP-EM residents, but junior residents outperformed senior FRCP residents in scenario 1 and CCFPEM residents in all scenarios (p 5 0.006 to 0.04). Conclusions: This study outlines the creation of a high-fidelity simulation assessment tool for trainees in emergency medicine. A single-point GAS demonstrated stronger relational validity and more consistent reliability in comparison with an EA checklist. This preliminary work will provide a foundation for ongoing future development of simulationbased assessment tools.

Download Full-text

Simulation-based Emergency Medicine Resident Self-Assessment

Journal of Emergency Medicine ◽

10.1016/j.jemermed.2011.05.041 ◽

2011 ◽

Vol 41 (6) ◽

pp. 679-685 ◽

Cited By ~ 13

Author(s):

Annie T. Sadosty ◽

M. Fernanda Bellolio ◽

Torrey A. Laack ◽

Anuradha Luke ◽

Amy Weaver ◽

...

Keyword(s):

Emergency Medicine ◽

Self Assessment ◽

Emergency Medicine Resident ◽

Medicine Resident ◽

Simulation Based

Download Full-text

Competency-based simulation assessment of resuscitation skills in emergency medicine postgraduate trainees – a Canadian multi-centred study

Canadian Medical Education Journal ◽

10.36834/cmej.36682 ◽

2016 ◽

Vol 7 (1) ◽

pp. e57-e67 ◽

Cited By ~ 11

Author(s):

J. Damon Dagnone ◽

Andrew K. Hall ◽

Stefanie Sebok-Syer ◽

Don Klinger ◽

Karen Woolfrey ◽

...

Keyword(s):

Emergency Medicine ◽

Assessment Tool ◽

Postgraduate Medical Education ◽

Intraclass Correlation ◽

Correlation Coefficients ◽

High Fidelity ◽

Rater Reliability ◽

Simulation Based ◽

Competency Based ◽

Generalizability Study

Background: The use of high-fidelity simulation is emerging as a desirable method for competency-based assessment in postgraduate medical education. We aimed to demonstrate the feasibility and validity of a multi-centre simulation-based Objective Structured Clinical Examination (OSCE) of resuscitation competence with Canadian Emergency Medicine (EM) trainees.Method: EM postgraduate trainees (n=98) from five Canadian academic centres participated in a high fidelity, 3-station simulation-based OSCE. Expert panels of three emergency physicians evaluated trainee performances at each centre using the Queen’s Simulation Assessment Tool (QSAT). Intraclass correlation coefficients were used to measure the inter-rater reliability, and analysis of variance was used to measure the discriminatory validity of each scenario. A fully crossed generalizability study was also conducted for each examination centre. Results: Inter-rater reliability in four of the five centres was strong with a median absolute intraclass correlation coefficient (ICC) across centres and scenarios of 0.89 [0.65-0.97]. Discriminatory validity was also strong (p < 0.001 for scenarios 1 and 3; p < 0.05 for scenario 2). Generalizability studies found significant variations at two of the study centres.Conclusions: This study demonstrates the successful pilot administration of a multi-centre, 3-station simulation-based OSCE for the assessment of resuscitation competence in post-graduate Emergency Medicine trainees.

Download Full-text

Oral Board vs. High-fidelity Simulation for Competency Assessment: Senior Emergency Medicine Resident Management of an Acute Coronary Syndrome

Academic Emergency Medicine ◽

10.1197/j.aem.2004.02.473 ◽

2004 ◽

Vol 11 (5) ◽

pp. 500-500 ◽

Cited By ~ 3

Author(s):

B. Clyne

Keyword(s):

Emergency Medicine ◽

Acute Coronary Syndrome ◽

Competency Assessment ◽

High Fidelity ◽

High Fidelity Simulation ◽

Emergency Medicine Resident ◽

Medicine Resident ◽

Coronary Syndrome

Download Full-text

Computer-Based Simulation by Emergency Medicine Resident-Educator for Medical Students during the COVID-19 Pandemic

Education in Medicine Journal ◽

10.21315/eimj2021.13.2.4 ◽

2021 ◽

Vol 13 (2) ◽

pp. 41-53

Author(s):

Shu Fang Ho ◽

Ming Jing Elizabeth Tan ◽

Fatimah Lateef

Keyword(s):

Emergency Medicine ◽

Medical Students ◽

Medical School ◽

Wide Spectrum ◽

Academic Medical Center ◽

Role Playing ◽

Emergency Medicine Resident ◽

Medicine Resident ◽

Simulation Based ◽

Computer Based

Today, residents in all disciplines are expected to be involved in not just educating themselves but in the education of others and peers as well. They are involved in a wide spectrum of teaching and instruction techniques such as case presentations, lectures, practical hands-on teaching, bedside clinical tutorials, informal discussions and simulation-based training. Simulation-based teaching has been playing an increasingly important role in both residency training as well as medical school curricula. In particular, it appeals to adult learners as it very task-driven and task-oriented, it allows for constant active engagement during role-playing in simulated scenarios and enables repetitive practice until a certain level of mastery or competency is achieved. The SingHealth residents training in emergency medicine have been collaborating with and engaging medical students from the Duke-NUS Graduate Medical School, as the two entities for a common Academic Medical Center. They share many collaborative projects and activities, research as well as educational training programmes. However, with the recent COVID-19 pandemic, both face-to-face medical teaching as well as simulation-based teaching proved to be challenging. One alternative is to move these teaching collaborations and programmes onto the online platform. This study describes the experience of emergency medicine resident-educators who conducted emergency medicine computer-based simulations (CBS) in collaboration with a group of medical students from the Duke-NUS Emergency Medicine Student Interest Group during the COVID-19 pandemic.

Download Full-text

LO69: Evaluating the impact of night shifts on emergency medicine resident competence in simulated resuscitations

CJEM ◽

10.1017/cem.2017.131 ◽

2017 ◽

Vol 19 (S1) ◽

pp. S51-S52

Author(s):

S. Edgerley ◽

C. McKaigney ◽

D. Boyne ◽

D. Dagnone ◽

A.K. Hall

Keyword(s):

Emergency Medicine ◽

Assessment Tool ◽

Night Shift ◽

Emergency Medicine Resident ◽

Medicine Resident ◽

The Difference ◽

One Year ◽

Primary Assessment ◽

The Impact ◽

Night Shifts

Introduction: Sleep deprivation negatively affects cognitive and behavioural performance. Emergency Medicine (EM) residents commonly work night shifts and are then expected to perform with competence. This study examines the impact of night shifts on EM resident performance in simulated resuscitation scenarios. Methods: A retrospective cohort study was completed at a single Canadian academic centre where residents participate in twice-annual simulation-based resuscitation objective structured clinical examinations (OSCEs). OSCE scores for all EM residents between 2010-2016 were collected, as well as post-graduate year (PGY1-5), gender, and shift schedules. OSCEs were scored using the Queen’s Simulation Assessment Tool (QSAT) evaluating four domains: primary assessment, diagnostic actions, therapeutic actions and communication, and an overall global assessment score (GAS). A night shift was defined as a late evening (beyond 23:00) or overnight shift within the three days before an OSCE. A mixed effects linear regression model was used to model the association between night shifts and OSCE scores while adjusting for gender and PGY. Results: A total of 136 OSCE scores were collected from 56 residents. PGY-5 residents had 37.1% (31.3 to 34.0%; p<0.01) higher OSCE scores than those in PGY-1 with an average increase of 8.8% (7.5 to 10.1%; p<0.01) per year. Working one or more night shifts in the three days before an OSCE reduced the total and communication scores by an average of 3.8% (p=0.04) and 4.5% (p=0.04) respectively. We observed a significant gender difference in the effects of acute shift work (p=0.03). Working a night shift one night prior to an OSCE was not associated with total score among male residents (p=0.33) but was associated with a 6.1% (-11.9 to -0.2; p=0.04) decrease in total score among female residents. This difference was consistent across PGY and was primarily due to an 8.5% (-15.5 to -1.6%; p=0.02) decrease in communication scores and a 6.7% (-13.1 to -0.3%; p=0.04) reduction in GAS. Conclusion: Proximity to night shifts significantly impaired the performance of EM trainees in simulated resuscitation scenarios, particularly in the domain of communication. For female residents, the magnitude of difference in total scores after working such shifts one night prior to a resuscitation OSCE was approximately equal to the difference seen between residents one year apart in training.

Download Full-text