scholarly journals E-ASSESS: Creating an EPA Assessment Tool for Structured Simulated Emergency Scenarios

2020 ◽  
Vol 12 (2) ◽  
pp. 153-158 ◽  
Author(s):  
Caroline Andler ◽  
Sneha Daya ◽  
Katie Kowalek ◽  
Christy Boscardin ◽  
Sandrijn M. van Schaik
Keyword(s):  
Clinical Practice ◽  
Direct Observation ◽  
Assessment Tool ◽  
Pearson Correlation ◽  
Intraclass Correlation ◽  
Professional Activity ◽  
Clinical Tool ◽  
Validity Data ◽  
Simulation Based ◽  
Emergency Scenarios

ABSTRACT Background The entrustable professional activity (EPA) assessment framework allows supervisors to assign entrustment levels to physician trainees for specific activities. Limited opportunity for direct observation of trainees hampers entrustment decisions, in particular for infrequently performed activities. Simulation allows for direct observation, so tools to assess performance of EPAs in simulation could potentially provide additional data to complement clinical assessments. Objective We developed and collected validity evidence for a simulation-based tool grounded in the EPA framework. Methods We developed E-ASSESS (EPA Assessment for Structured Simulated Emergency ScenarioS) to assess performance in 2 EPAs among pediatric residents participating in simulation-based team training in 2017–2018. We collected validity data, applying Messick's unitary view. Three raters used E-ASSESS to assign entrustment levels based on performance in simulation. We compared those ratings to entrustment levels assigned by clinical supervisors (different from the study raters) for the same residents on a separate tool designed for clinical practice. We calculated intraclass correlation (ICC) for each tool and Pearson correlation coefficients to compare ratings between tools. Results Twenty-eight residents participated in the study. The ICC between the 3 raters for entrustment ratings on E-ASSESS ranged from 0.65 to 0.77, while ICC among raters of the clinical tool were 0.59 and 0.57. We found no significant correlations between E-ASSESS ratings and clinical practice ratings for either EPA (r = -0.35 and 0.38, P > .05). Conclusions Assessment following an EPA framework in the simulation context may be useful to provide data points to inform entrustment decisions as part of resident assessment.

scholarly journals Evaluation of a new clinical performance assessment tool : A reliability study

2012 ◽  
Vol 68 (3) ◽  
Author(s):  
C. Joseph ◽  
J. Frantz ◽  
C. Hendricks ◽  
M. Smith
Keyword(s):  
Clinical Practice ◽  
Performance Assessment ◽  
Internal Consistency ◽  
Quantitative Research ◽  
Assessment Tool ◽  
Clinical Performance ◽  
Intraclass Correlation ◽  
Cross Sectional ◽  
Clinical Educators ◽  
Clinical Performance Assessment

Clinical practice is an essential requirement of any graduatephysiotherapy programme. For this purpose, valid and reliable assessment toolsare paramount for the measurement of key competencies in the real-worldsetting. This study aims to determine the internal consistency and inter-raterreliability of a newly developed and validated clinical performance assessmentform. A cross-sectional quantitative research design was used, which includedpaired evaluations of 32 (17 treatment and 15 assessment) student examinationsperformed by two independent clinical educators. Chronbachs alpha was computedto assess internal consistency and intraclass correlation coefficient (ICC’s) withconfidence intervals of 95% were computed to determine the percentage agreement between paired examiners. Thedegree of internal consistency was substantial for all key performance areas of both examinations, except for timeand organisational management (0.21) and professionalism (0.42) in the treatment and evaluation examinationsrespectively. The overall internal consistency was 0.89 and 0.73 for both treatment and assessment examinations,indicating substantial agreement. With regard to agreement between raters, the ICC’s for the overall marks were0.90 and 0.97 for both treatment and assessment examinations. Clinical educators demonstrated a high level ofreliability in the assessment of students’ competence using the newly developed clinical performance assessment form.These findings greatly underscore the reliability of results obtained through observation of student examinations, andadd another tool to the basket of ensuring quality assurance in physiotherapy clinical practice assessment.

Validation evidence of the paediatric Objective Structured Assessment of Debriefing (OSAD) tool

2016 ◽  
Vol 2 (3) ◽  
pp. 61-67 ◽  
Author(s):  
Jane Runnacles ◽  
Libby Thomas ◽  
James Korndorffer ◽  
Sonal Arora ◽  
Nick Sevdalis
Keyword(s):  
Assessment Tool ◽  
Learning Experience ◽  
Pearson Correlation ◽  
Phase 1 ◽  
Evidence Based ◽  
Validity Evidence ◽  
Multiple Sources ◽  
Simulation Based ◽  
Expert Input

IntroductionDebriefing is essential to maximise the simulation-based learning experience, but until recently, there was little guidance on an effective paediatric debriefing. A debriefing assessment tool, Objective Structured Assessment of Debriefing (OSAD), has been developed to measure the quality of feedback in paediatric simulation debriefings. This study gathers and evaluates the validity evidence of OSAD with reference to the contemporary hypothesis-driven approach to validity.MethodsExpert input on the paediatric OSAD tool from 10 paediatric simulation facilitators provided validity evidence based on content and feasibility (phase 1). Evidence for internal structure validity was sought by examining reliability of scores from video ratings of 35 postsimulation debriefings; and evidence for validity based on relationship to other variables was sought by comparing results with trainee ratings of the same debriefings (phase 2).ResultsSimulation experts’ scores were significantly positive regarding the content of OSAD and its instructions. OSAD's feasibility was demonstrated with positive comments regarding clarity and application. Inter-rater reliability was demonstrated with intraclass correlations above 0.45 for 6 of the 7 dimensions of OSAD. The internal consistency of OSAD (Cronbach α) was 0.78. Pearson correlation of trainee total score with OSAD total score was 0.82 (p<0.001) demonstrating validity evidence based on relationships to other variables.ConclusionThe paediatric OSAD tool provides a structured approach to debriefing, which is evidence-based, has multiple sources of validity evidence and is relevant to end-users. OSAD may be used to improve the quality of debriefing after paediatric simulations.

scholarly journals Exploring undergraduate students achievement emotions during ward round simulation: A mixed method study

2019 ◽  
Author(s):  
Claudia Behrens ◽  
DIANA H. DOLMANS ◽  
GERARD J. GORMLEY ◽  
Erik Driessen
Keyword(s):  
Clinical Practice ◽  
Undergraduate Students ◽  
Assessment Tool ◽  
Ward Round ◽  
Learning Activity ◽  
Clear Understanding ◽  
Emotional States ◽  
Achievement Emotions ◽  
Complex Simulation ◽  
Simulation Based

Abstract Background Simulation based learning (SBL) has increased in its use to best equip students for clinical practice. Simulations that mirror the complex realities of clinical practice have the potential to induce a range of emotions, without a clear understanding of their impact on learning and the learner. Students’ emotional states have important effects on their learning process that can be either positive or negative, and are often difficult to predict. We aimed to determine: (1) To what extent achievement emotions are experienced by medical students during a complex simulation based learning activity, i.e. a ward round simulation (WRS). (2) what their performance scores are and too which extent performance scores do correlate with emotions and 3) how these emotions are perceived to impact learning. Methods A mixed methods approach was used in this study. Using an Achievement Emotion Questionnaire, we explored undergraduate medical student’s emotions as they participated in a complex ward round-based simulation. Their performance was rated using an observational ward round assessment tool and correlated with emotions scores. Six focus groups were conducted to provide a deeper understanding of their emotional and learning experiences. Results Students experienced a range of emotions during the simulation, they felt proud, enjoyed the simulation and performed well. Students felt proud because they could show in the complex simulation what they had learned so far. Students reported moderate levels of anxiety and low levels of frustration and shame. We found non-significant correlations between achievement emotions and performance during ward round simulation. Conclusions Placing undergraduate students in high complex simulations that they can handle raises positive academic achievement emotions which seem to support students’ learning and motivation.

scholarly journals Competency-based simulation assessment of resuscitation skills in emergency medicine postgraduate trainees – a Canadian multi-centred study

2016 ◽  
Vol 7 (1) ◽  
pp. e57-e67 ◽  
Author(s):  
J. Damon Dagnone ◽  
Andrew K. Hall ◽  
Stefanie Sebok-Syer ◽  
Don Klinger ◽  
Karen Woolfrey ◽  
...  
Keyword(s):  
Emergency Medicine ◽  
Assessment Tool ◽  
Postgraduate Medical Education ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
High Fidelity ◽  
Rater Reliability ◽  
Simulation Based ◽  
Competency Based ◽  
Generalizability Study

Background: The use of high-fidelity simulation is emerging as a desirable method for competency-based assessment in postgraduate medical education. We aimed to demonstrate the feasibility and validity of a multi-centre simulation-based Objective Structured Clinical Examination (OSCE) of resuscitation competence with Canadian Emergency Medicine (EM) trainees.Method: EM postgraduate trainees (n=98) from five Canadian academic centres participated in a high fidelity, 3-station simulation-based OSCE.  Expert panels of three emergency physicians evaluated trainee performances at each centre using the Queen’s Simulation Assessment Tool (QSAT).  Intraclass correlation coefficients were used to measure the inter-rater reliability, and analysis of variance was used to measure the discriminatory validity of each scenario.  A fully crossed generalizability study was also conducted for each examination centre.   Results: Inter-rater reliability in four of the five centres was strong with a median absolute intraclass correlation coefficient (ICC) across centres and scenarios of 0.89 [0.65-0.97]. Discriminatory validity was also strong (p < 0.001 for scenarios 1 and 3; p < 0.05 for scenario 2). Generalizability studies found significant variations at two of the study centres.Conclusions: This study demonstrates the successful pilot administration of a multi-centre, 3-station simulation-based OSCE for the assessment of resuscitation competence in post-graduate Emergency Medicine trainees.

scholarly journals Exploring undergraduate students achievement emotions during ward round simulation: A mixed-method study

2019 ◽  
Author(s):  
Claudia Behrens ◽  
DIANA H. DOLMANS ◽  
GERARD J. GORMLEY ◽  
Erik Driessen
Keyword(s):  
Clinical Practice ◽  
Undergraduate Students ◽  
Assessment Tool ◽  
Ward Round ◽  
Learning Activity ◽  
Clear Understanding ◽  
Emotional States ◽  
Achievement Emotions ◽  
Complex Simulation ◽  
Simulation Based

Abstract Background Simulation based learning (SBL) has increased in its use to best equip students for clinical practice. Simulations that mirror the complex realities of clinical practice have the potential to induce a range of emotions, without a clear understanding of their impact on learning and the learner. Students’ emotional states have important effects on their learning process that can be either positive or negative, and are often difficult to predict. We aimed to determine: (1) To what extent achievement emotions are experienced by medical students during a complex simulation based learning activity, i.e. a ward round simulation (WRS). (2) What their performance scores are and too which extent performance scores do correlate with emotions and 3) how these emotions are perceived to impact learning. Methods A mixed methods approach was used in this study. Using an Achievement Emotion Questionnaire, we explored undergraduate medical student’s emotions as they participated in a complex ward round-based simulation. Their performance was rated using an observational ward round assessment tool and correlated with emotions scores. Six focus groups were conducted to provide a deeper understanding of their emotional and learning experiences. Results Students experienced a range of emotions during the simulation, they felt proud, enjoyed the simulation and performed well. Students felt proud because they could show in the complex simulation what they had learned so far. Students reported moderate levels of anxiety and low levels of frustration and shame. We found non-significant correlations between achievement emotions and performance during ward round simulation. Conclusions Placing undergraduate students in high complex simulations that they can handle raises positive academic achievement emotions which seem to support students’ learning and motivation.

The Youth Throwing Score: Validating Injury Assessment in Young Baseball Players

2016 ◽  
Vol 45 (2) ◽  
pp. 317-324 ◽  
Author(s):  
Christopher S. Ahmad ◽  
Ajay S. Padaki ◽  
Manish S. Noticewala ◽  
Eric C. Makhni ◽  
Charles A. Popkin
Keyword(s):  
Upper Extremity ◽  
Assessment Tool ◽  
Pearson Correlation ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Importance Rating ◽  
Extremity Injury ◽  
Level Of Evidence ◽  
Baseball Players ◽  
Upper Extremity Injury

Background: Epidemic levels of shoulder and elbow injuries have been reported recently in youth and adolescent baseball players. Despite the concerning frequency of these injuries, no instrument has been validated to assess upper extremity injury in this patient population. Purpose/Hypothesis: The purpose of this study was to validate an upper extremity assessment tool specifically designed for young baseball players. We hypothesized that this tool will be both reliable and valid. Study Design: Cohort study (diagnosis); Level of evidence, 2. Methods: The Youth Throwing Score (YTS) was constructed by an interdisciplinary team of providers and coaches as a tool to assess upper extremity injury in youth and adolescent baseball players (age range, 10-18 years). The psychometric properties of the test were then determined. Results: A total of 223 players completed the final survey. The players’ mean age was 14.3 ± 2.7 years. Pilot analysis showed that none of the 14 questions received a mean athlete importance rating less than 3 of 5, and the final survey read at a Flesch-Kincaid level of 4.1, which is appropriate for patients aged 9 years and older. The players self-assigned their injury status, resulting in a mean instrument score of 59.7 ± 8.4 for the 148 players “playing without pain,” 42.0 ± 11.5 for the 60 players “playing with pain,” and 40.4 ± 10.5 for the 15 players “not playing due to pain.” Players playing without pain scored significantly higher than those playing with pain and those not playing due to pain ( P < .001). Psychometric analysis showed a test-retest intraclass correlation coefficient of 0.90 and a Cronbach alpha intra-item reliability coefficient of 0.93, indicating excellent reliability and internal consistency. Pearson correlation coefficients of 0.65, 0.62, and 0.31 were calculated between the YTS and the Pediatric Outcomes Data Collection Instrument sports/physical functioning module, the Kerlan-Jobe Orthopaedic Clinic Shoulder and Elbow score, and the Quick Disabilities of the Arm, Shoulder, and Hand (QuickDASH) score, respectively. Injured players scored a mean of 9.4 points higher after treatment ( P < .001), and players who improved in their self-assigned pain categorization scored 16.5 points higher ( P < .001). Conclusion: The YTS is the first valid and reliable instrument for assessing young baseball players’ upper extremity health.

scholarly journals Evaluating Professionalism and Interpersonal and Communication Skills: Implementing a 360-Degree Evaluation Instrument in an Anesthesiology Residency Program

2009 ◽  
Vol 1 (2) ◽  
pp. 216-220 ◽  
Author(s):  
Li Meng ◽  
David G. Metro ◽  
Rita M. Patel
Keyword(s):  
Communication Skills ◽  
Assessment Tool ◽  
Pearson Correlation ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Postanesthesia Care Unit ◽  
Staff Nurse ◽  
Evaluation Instrument ◽  
Intraclass Correlation Coefficients ◽  
Resident Assessment

Abstract Objectives To implement a 360-degree resident evaluation instrument on the postanesthesia care unit (PACU) rotation and to determine the reliability, feasibility, and validity of this tool for assessing residents' professionalism and interpersonal and communication skills. Methods Thirteen areas of evaluation were selected to assess the professionalism and interpersonal and communication skills of residents during their PACU rotation. Each area was measured on a 9-point Likert scale (1, unsatisfactory performance, to 9, outstanding performance). Rating forms were distributed to raters after the completion of the PACU rotation. Raters included PACU nurses, secretarial staff, nurse aides, and medical technicians. Residents were aware of the 360-degree assessment and participated voluntarily. The multiple raters' evaluations were then compared with those of the traditional faculty. Intraclass correlation coefficients were calculated to measure the reliability of ratings within each category of raters by the Pearson correlation coefficient. Results Four hundred twenty-nine rating forms were returned during the study period. Fifteen residents were evaluated. The response rate was 88%. Residents were ranked highest on availability and lowest on management skill. The average rating across all areas was high (8.23). The average mean rating across all items from PACU nurses was higher (8.34) than from secretarial staff (7.99, P &gt; .08). The highest ranked resident ranked high with all raters and the lowest ranked was low with most raters. The intraclass coefficients of correlations were 0.8719, 0.7860, 0.8268, and 0.8575. Conclusions This type of resident assessment tool may be useful for PACU rotations. It appears to correlate with traditional faculty ratings, is feasible to use, and provides formative feedback to residents regarding their professionalism and interpersonal and communication skills.

scholarly journals Inter-rater reliability of a newly developed gait analysis and motion score

2020 ◽  
pp. 221049172096736
Author(s):  
Christina Dürregger ◽  
Klemens A Adamer ◽  
Michael Pirchl ◽  
Michael J Fischer
Keyword(s):  
Clinical Practice ◽  
Gait Analysis ◽  
Clinical Training ◽  
Assessment Tool ◽  
Intraclass Correlation ◽  
Rater Reliability ◽  
Gait Patterns ◽  
Orthopedic Rehabilitation ◽  
The Mean ◽  
Rating Agreement

Background: Gait analysis constitutes an essential part of orthopedic rehabilitation assessment. Previous studies indicate that observational-based gait analysis lacks reliability and requires extensive clinical training. Therefore, gait analysis in the clinical practice heavily relies on technical aids. The aim of the present study is to develop a reliable gait analysis assessment tool that can accurately assess clinically relevant gait cycle parameters in daily clinical practice. Methods: In this pilot study, a new gait analysis and motion score (GAMS), comprising 10 observational and 5 technically measured (e.g. pressure plate) gait parameters, was developed. The parameters were dichotomously operationalized, reflecting pathological versus physiological manifestations of the parameters. The rating algorithm was administered by 12 raters using videotaped treadmill sessions of 10 orthopedic subjects ( n = 120 ratings). Inter-rater reliability was calculated using the intraclass correlation coefficient (ICC) and the percentage of rating agreement. Results: The mean (standard deviation (SD)) GAMS ratings ranged from 10.0 (1.1) to 21.5 (1.3) points. The overall GAMS ICC was 0.98 (95% confidence interval (CI) 0.96–1.00), whereas the ICC of observational parameters alone was 0.97 (95% CI 0.93–0.99). The mean (SD) percentage of rating agreement was 86.1% (3.3%). For the observational parameters, the mean (SD) rating agreement was 82.5% (4.5%). Conclusion: This new GAMS shows excellent overall inter-rater reliability for a continuum of functional gait statuses. The new score may be an appropriate clinical tool to objectively evaluate patients’ gait patterns. Furthermore, the GAMS may find application as a clinician-reported outcome measure in orthopedic rehabilitation. Further studies are required to verify the validity and accuracy of the new GAMS and its functionality in assessing clinical changes in gait patterns.

scholarly journals LO097: A novel curriculum for assessing competency in resuscitation at the foundations of discipline level of training

CJEM ◽  
2016 ◽  
Vol 18 (S1) ◽  
pp. S63-S63
Author(s):  
T. Chaplin ◽  
L. McMurray ◽  
A.K. Hall
Keyword(s):  
Assessment Tool ◽  
Professional Activity ◽  
First Year ◽  
Residency Programs ◽  
Modified Delphi ◽  
Simulation Based ◽  
Level Of Consciousness ◽  
Objective Structured Clinical Exam ◽  
Competency Based ◽  
Level Of Training

Introduction / Innovation Concept: Junior residents are often the first physicians who attend to the acutely unwell floor patient, especially at night and on weekends. The ‘Nightmares Course’ at Queen’s University was designed to address an Entrustable Professional Activity (EPA) relevant to several residency programs at the ‘Foundations of Discipline’ level of training: “to manage the acutely unwell floor patient for the first 5-10 minutes until help arrives”. In keeping with competency based medical education principles, this course offers longitudinal and repetitive practice and assessment. We have also designed a summative objective structured clinical exam (OSCE) in order to identify trainees who require additional remedial practice of this EPA. Methods: We developed simulated cases that reflect common but “scary” calls to the floor. We then, using a modified Delphi process with experts in resuscitation, defined relevant milestones applicable to the Foundations of Discipline level of training in order to inform our formative assessment. We also modified the Queen’s Simulated Assessment Tool (QSAT) to adopt CBME terminology and this will be used to provide a summative assessment during a four-scenario OSCE in the spring. Residents with QSAT scores below the competency threshold will be enrolled in a remediation course. Curriculum, Tool, or Material: Weekly sessions were led by staff physicians and were offered to first-year residents from internal medicine, core surgery, obstetrics and gynecology, and anesthesiology over the academic year. Each resident participated in one session every 4-week block. Sessions were organized into themes such as “shortness of breath” or “decreased level of consciousness” and involved three high-fidelity simulated cases with a structured debrief following each case. Formative feedback was given following each case. Conclusion: The Nightmares Course is a novel simulation-based, multidisciplinary curriculum in resuscitation medicine. It includes longitudinal practice and repetitive assessment, as well as summative testing and remediation of an EPA common to several residency programs.

scholarly journals Finnish Version of the Eating Assessment Tool (F-EAT-10): A Valid and Reliable Patient-reported Outcome Measure for Dysphagia Evaluation

Dysphagia ◽  
2021 ◽  
Author(s):  
Pia Järvenpää ◽  
Jonna Kuuskoski ◽  
Petra Pietarinen ◽  
Mari Markkanen-Leppänen ◽  
Hanna Freiberg ◽  
...  
Keyword(s):  
Rating Scale ◽  
Outcome Measure ◽  
Assessment Tool ◽  
Pearson Correlation ◽  
Intraclass Correlation ◽  
Reliability And Validity ◽  
Patient Reported Outcome ◽  
Specific Patient ◽  
Patient Reported Outcome Measure ◽  
Patient Reported

AbstractOur aim was to validate a Finnish version of the Eating Assessment Tool (F-EAT-10) for clinical use and to test its reliability and validity in a multicenter nationwide study. Normative data were acquired from 180 non-dysphagic participants (median age 57.0 years, 62.2% female). Dysphagia patients (n = 117, median age 69.7 years, 53.0% female) referred to fiberoptic endoscopic evaluation of swallowing (FEES) completed F-EAT-10 before the examination and after 2 weeks. Patients underwent the 100-ml water swallow test (WST) and FEES was evaluated using the following three scales: the Yale Pharyngeal Residue Severity Rating Scale, Penetration-Aspiration Scale, and the Dysphagia Outcome Severity Scale. An operative cohort of 19 patients (median age 75.8 years, 57.9% female) underwent an endoscopic operation on Zenker’s diverticulum, tight cricopharyngeal muscle diagnosed in videofluorography, or both. Patients completed the F-EAT-10 preoperatively and 3 months postoperatively. The cut-off score for controls was < 3 (sensitivity 94.0%, specificity 96.1%) suggesting that ≥ 3 is abnormal. Re-questionnaires for test–retest reliability analysis were available from 92 FEES patients and 123 controls. The intraclass correlation coefficient was excellent for the total F-EAT-10 score (0.93, 95% confidence interval 0.91–0.95). Pearson correlation coefficients were strong (p < 0.001) for each of the questions and the total score. Internal consistency as assessed by Cronbach’s alpha was excellent (0.95). Some correlations between findings in FEES and 100-ml WST with F-EAT-10 were observed. The change in subjective symptoms of operative patients paralleled the change in F-EAT-10. F-EAT-10 is a reliable, valid, and symptom-specific patient-reported outcome measure for assessing dysphagia among Finnish speakers.

Sign in / Sign up

Export Citation Format

Share Document