scholarly journals Influence of video-based feedback on self-assessment accuracy of endoscopic skills: a randomized controlled trial

2019 ◽  
Vol 07 (05) ◽  
pp. E678-E684 ◽  
Author(s):  
Michael Scaffidi ◽  
Catharine Walsh ◽  
Rishad Khan ◽  
Colleen Parker ◽  
Ahmed Al-Mazroui ◽  
...  
Keyword(s):  
Rating Scale ◽  
Assessment Tool ◽  
Controlled Trial ◽  
Global Rating Scale ◽  
Competency Assessment ◽  
Global Rating ◽  
Self Assessment ◽  
Video Review ◽  
Group Assignment ◽  
Self Assessments

Abstract Background and study aims Novice endoscopists are inaccurate in self-assessment of procedures. One means of improving self-assessment accuracy is through video-based feedback. We aimed to determine the comparative effectiveness of three video-based interventions on novice endoscopists’ self-assessment accuracy of endoscopic competence. Materials and methods Novice endoscopists (performed < 20 previous procedures) were recruited. Participants completed a simulated esophagogastroduodenoscopy (EGD) on a virtual reality simulator. They were then randomized to one of three groups: self-video review (SVR), which involved watching a recorded video of their own performance; benchmark review (BVR), which involved watching a video of a simulated EGD completed by an expert; and self- and benchmark video (SBVR), which involved both videos. Participants then completed two additional simulated EGD cases. Self-assessments were conducted immediately after the first procedure, after the video intervention and after the additional two procedures. External assessments were conducted by two experienced endoscopists, who were blinded to participant identity and group assignment through video recordings. External and self-assessments were completed using the global rating scale component of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT GRS). Results Fifty-one participants completed the study. The BVR group had significantly improved self-assessment accuracy in the short-term, compared to the SBVR group (P = .005). The SBVR group demonstrated significantly improved self-assessment accuracy over time (P = .016). There were no significant effects of group or of time for the SVR group. Conclusions Video-based interventions, particularly combined use of self- and benchmark video review, can improve accuracy of self-assessment of endoscopic competence among novices.

Objective structured assessment of technical skill in temporal bone dissection: validation of a novel tool

2021 ◽  
pp. 1-11
Author(s):  
M Stavrakas ◽  
G Menexes ◽  
S Triaridis ◽  
P Bamidis ◽  
J Constantinidis ◽  
...  
Keyword(s):  
Temporal Bone ◽  
Rating Scale ◽  
Assessment Tool ◽  
Three Dimensional ◽  
Technical Skills ◽  
Technical Skill ◽  
Global Rating Scale ◽  
Global Rating ◽  
Validity And Reliability ◽  
Cortical Mastoidectomy

Abstract Objective This study developed an assessment tool that was based on the objective structured assessment for technical skills principles, to be used for evaluation of surgical skills in cortical mastoidectomy. The objective structured assessment of technical skill is a well-established tool for evaluation of surgical ability. This study also aimed to identify the best material and printing method to make a three-dimensional printed temporal bone model. Methods Twenty-four otolaryngologists in training were asked to perform a cortical mastoidectomy on a three-dimensional printed temporal bone (selective laser sintering resin). They were scored according to the objective structured assessment of technical skill in temporal bone dissection tool developed in this study and an already validated global rating scale. Results Two external assessors scored the candidates, and it was concluded that the objective structured assessment of technical skill in temporal bone dissection tool demonstrated some main aspects of validity and reliability that can be used in training and performance evaluation of technical skills in mastoid surgery. Conclusion Apart from validating the new tool for temporal bone dissection training, the study showed that evolving three-dimensional printing technologies is of high value in simulation training with several advantages over traditional teaching methods.

scholarly journals A Tablet-Computer-Based Tool to Facilitate Accurate Self-Assessments in Third- and Fourth-Graders

2018 ◽  
Vol 13 (10) ◽  
pp. 225
Author(s):  
Denise Villanyi ◽  
Romain Martin ◽  
Philipp Sonnleitner ◽  
Christina Siry ◽  
Antoine Fischbach
Keyword(s):  
Reading Comprehension ◽  
Rating Scale ◽  
Assessment Tool ◽  
Standardized Test ◽  
Fourth Graders ◽  
Tablet Computer ◽  
School Students ◽  
Self Assessment ◽  
Computer Based ◽  
Self Assessments

Although student self-assessment is positively related to achievement, skepticism about the accuracy of students’ self-assessments remains. A few studies have shown that even elementary school students are able to provide accurate self-assessments when certain conditions are met. We developed an innovative tablet-computer-based tool for capturing self-assessments of mathematics and reading comprehension. This tool integrates the conditions required for accurate self-assessment: (1) a non-competitive setting, (2) items formulated on the task level, and (3) limited reading and no verbalization required. The innovation consists of using illustrations and a language-reduced rating scale. The correlations between students’ self-assessment scores and their standardized test scores were moderate to large. Independent of their proficiency level, students’ confidence in completing a task decreased as task difficulty increased, but these findings were more consistent in mathematics than in reading comprehension. We conclude that third- and fourth-graders have the ability to provide accurate self-assessments of their competencies, particularly in mathematics, when provided with an adequate self-assessment tool.

Do we need a confederate in full-scale medical simulation? A randomized controlled trial.

2021 ◽  
Author(s):  
Jeremie Traoré ◽  
Frédéric Balen ◽  
Thomas Geeraerts ◽  
Sandrine Charpentier ◽  
Xavier Dubucs ◽  
...  
Keyword(s):  
Emergency Medicine ◽  
Rating Scale ◽  
Controlled Trial ◽  
Full Scale ◽  
Global Rating Scale ◽  
Controlled Study ◽  
Control Group ◽  
Global Rating ◽  
Performance Score ◽  
Overall Performance

Abstract Background: During simulation training, the confederate is a member of the pedagogical team. Its role is to facilitate the interaction between participants and the environment, and is thought to increase realism and immersion. Its influence on participants' performance in full-scale simulation remains however unknown. The purpose of this study was to explore the effect of the presence of a confederate on the participants’ performance during full-scale simulation of crisis medical situations. Methods: This was a prospective, randomized study comparing 2 parallel groups. Participants were emergency medicine residents engaging in a simulation session, with or without confederates. Participants were then evaluated on their Crisis Resource Management performance (CRM). The overall performance score on the Ottawa Global Rating Scale was assessed as primary outcome and the 5 non-technical CRM skills as secondary outcomes.Results: A total of 63 simulation sessions, including 63 residents, were included for statistical analysis (n= 32 for Control group and 31 for Confederate group). The mean Overall Performance score was 3.9± 0.8 in the Control group and 4.0± 1.1 in the Confederate group, 95% confidence interval of the difference [-0.6; 0.4], p=0.60. No significant differences between the two groups were observed on each CRM items (leadership, situational awareness, communication, problem solving, resource utilization)Conclusion: In this randomized and controlled study, the presence of confederates during full-scale simulated practice of crisis medical situations does not seem to improve the CRM skills performance of Emergency medicine residents. Trial registration: this study does not need to be registered on Clintrial as it doesn’t report a health care intervention on human participants.

scholarly journals Evaluating Teamwork in a Simulated Obstetric Environment

2007 ◽  
Vol 106 (5) ◽  
pp. 907-915 ◽  
Author(s):  
Pamela J. Morgan ◽  
Richard Pittini ◽  
Glenn Regehr ◽  
Carol Marrs ◽  
Michèle F. Haley
Keyword(s):  
Rating Scale ◽  
Assessment Tool ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Obstetric Care ◽  
Global Rating Scale ◽  
Global Rating ◽  
Good Reliability ◽  
Domain Specific ◽  
Team Assessment

Background The National Confidential Enquiry into Maternal Deaths identified "lack of communication and teamwork" as a leading cause of substandard obstetric care. The authors used high-fidelity simulation to present obstetric scenarios for team assessment. Methods Obstetric nurses, physicians, and resident physicians were repeatedly assigned to teams of five or six, each team managing one of four scenarios. Each person participated in two or three scenarios with differently constructed teams. Participants and nine external raters rated the teams' performances using a Human Factors Rating Scale (HFRS) and a Global Rating Scale (GRS). Interrater reliability was determined using intraclass correlations and the Cronbach alpha. Analyses of variance were used to determine the reliability of the two measures, and effects of both scenario and rater profession (R.N. vs. M.D.) on scores. Pearson product-moment correlations were used to compare external with self-generated assessments. Results The average of nine external rater scores showed good reliability for both HFRS and GRS; however, the intraclass correlation coefficients for a single rater was low. There was some effect of rater profession on self-generated HFRS but not on GRS. An analysis of profession-specific subscores on the HFRS revealed no interaction between profession of rater and profession being rated. There was low correlation between externally and self-generated team assessments. Conclusions This study does not support the use of the HFRS for assessment of obstetric teams. The GRS shows promise as a summative but not a formative assessment tool. It is necessary to develop a domain specific behavioral marking system for obstetric teams.

scholarly journals Resident self-assessment versus faculty assessment of laparoscopic technical skills using a global rating scale

2010 ◽  
Vol 1 ◽  
pp. 37-41 ◽  
Author(s):  
Sarah E. Peyre ◽  
Heather MacDonald ◽  
Laila Al-Marayati ◽  
Claire Templeman ◽  
Laila I. Muderspach
Keyword(s):  
Rating Scale ◽  
Technical Skills ◽  
Global Rating Scale ◽  
Global Rating ◽  
Self Assessment ◽  
Faculty Assessment

Validity Evidence for the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT)

2016 ◽  
Vol 13 (1) ◽  
pp. 60-68 ◽  
Author(s):  
Gerben E. Breimer ◽  
Faizal A. Haji ◽  
Giuseppe Cinalli ◽  
Eelco W. Hoving ◽  
James M. Drake
Keyword(s):  
Internal Structure ◽  
Internal Consistency ◽  
Interrater Reliability ◽  
Rating Scale ◽  
Assessment Tool ◽  
Global Rating Scale ◽  
Global Rating ◽  
Validity Evidence ◽  
Simulation Based ◽  
Endoscopic Ventriculostomy

Abstract BACKGROUND: Growing demand for transparent and standardized methods for evaluating surgical competence prompted the construction of the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT). OBJECTIVE: To provide validity evidence of the NEVAT by reporting on the tool's internal structure and its relationship with surgical expertise during simulation-based training. METHODS: The NEVAT was used to assess performance of trainees and faculty at an international neuroendoscopy workshop. All participants performed an endoscopic third ventriculostomy (ETV) on a synthetic simulator. Participants were simultaneously scored by 2 raters using the NEVAT procedural checklist and global rating scale (GRS). Evidence of internal structure was collected by calculating interrater reliability and internal consistency of raters' scores. Evidence of relationships with other variables was collected by comparing the ETV performance of experts, experienced trainees, and novices using Jonckheere's test (evidence of construct validity). RESULTS: Thirteen experts, 11 experienced trainees, and 10 novices participated. The interrater reliability by the intraclass correlation coefficient for the checklist and GRS was 0.82 and 0.94, respectively. Internal consistency (Cronbach's α) for the checklist and the GRS was 0.74 and 0.97, respectively. Median scores with interquartile range on the checklist and GRS for novices, experienced trainees, and experts were 0.69 (0.58-0.86), 0.85 (0.63-0.89), and 0.85 (0.81-0.91) and 3.1 (2.5-3.8), 3.7 (2.2-4.3) and 4.6 (4.4-4.9), respectively. Jonckheere's test showed that the median checklist and GRS score increased with performer expertise (P = .04 and .002, respectively). CONCLUSION: This study provides validity evidence for the NEVAT to support its use as a standardized method of evaluating neuroendoscopic competence during simulation-based training.

scholarly journals A novel approach: Simulating multiple simultaneous encounters to assess multitasking ability in emergency medicine

PLoS ONE ◽  
2021 ◽  
Vol 16 (9) ◽  
pp. e0257887
Author(s):  
Wen-Cheng Huang ◽  
Shih-Chang Hsu ◽  
Chih-Hao Yang ◽  
Che-Wei Lin ◽  
Fat-Moon Suk ◽  
...  
Keyword(s):  
Rating Scale ◽  
Assessment Tool ◽  
Global Rating Scale ◽  
Global Rating ◽  
Study Objective ◽  
Priority Score ◽  
Quality Feedback ◽  
Novel Approach ◽  
Simulation Scenario ◽  
Total Performance Scores

Study objective The purpose of this feasibility study is to develop and validate a new assessment tool and scoring system for multitasking competency for physicians in-training in a timed simulated setting. The multitasking competency includes ability to appropriately prioritize and implement tasks for different patients who present simultaneously. Methods We designed three single task stations with different levels of difficulty and priority. These skill stations were then combined to create a multitasking simulation scenario. Skill checklists and the global rating scale were utilized to assess the participants’ performance. A multitasking score, multitasking index, and priority score were developed to measure the multitasking ability of participants. Results Thirty-three first-year postgraduate physicians were recruited for this prospective study. The total performance scores were significantly higher for the single-tasking stations than for the multitasking scenario. In terms of the time needed to complete the tasks, the participants spent more time on the multitasking scenario than on the single-tasking scenario. There were significant correlations between the global rating scale and the multitasking score (rho = 0.693, p < 0.001) and between the global rating scale and the multitasking index (rho = 0.515, p < 0.001). The multitasking score, multitasking index, and priority score did not have any significant correlations with the total single-tasking score. Conclusion We demonstrated that the use of a simulated multitasking scenario could be an effective method of assessing multitasking ability and allow assessors to offer better quality feedback.

scholarly journals The effect of microsurgical training on novice medical students’ basic surgical skills—a randomized controlled trial

2019 ◽  
Vol 43 (4) ◽  
pp. 459-466
Author(s):  
Stian Kreken Almeland ◽  
Andrew Lindford ◽  
Henriette Pisani Sundhagen ◽  
Karl Ove Hufthammer ◽  
Eivind Strandenes ◽  
...  
Keyword(s):  
Medical Students ◽  
Plastic Surgery ◽  
Rating Scale ◽  
Assessment Tool ◽  
Intervention Group ◽  
Global Rating Scale ◽  
Control Group ◽  
Diagnostic Study ◽  
Global Rating ◽  
Suturing Skills

Abstract Background It has been demonstrated that medical students are capable of learning microsurgical techniques. We hypothesize that microsurgical training might give insight into the importance of delicate tissue handling and correct knot tying that could have a positive influence on macrosurgical skills. The primary aim of this study was to evaluate the effect of microsurgical training on macrosurgical suturing skills in novice medical students. Subjects and methods In 2018, 46 novice medical students were enrolled and randomized into two groups. The intervention group received both macro- and microsurgical training and the control group received only microsurgical training. Both groups underwent an assessment test that consisted of macrosurgical tasks of three simple interrupted sutures with a square knot and continuous three-stitch long over-and-over sutures. These tests were individually filmed and assessed using the University of Bergen suturing skills assessment tool (UBAT) and the Objective Structured Assessment of Technical Skill global rating scale (OSATS). Questionnaires regarding future career ambitions and attitudes towards plastic surgery were also completed both prior to and following the tests. Results The intervention group needed a longer time to complete the tasks than the control group (12.2 min vs. 9.6 min, p > 0.001), and scored lower on both the UBAT (5.6 vs. 9.0, p > 0.001) and the OSATS (11.1 vs. 13.1, p > 0.001) assessments. The microsurgery course tended to positively influence the students’ attitudes towards a career in plastic surgery (p = 0.002). This study demonstrates poorer macrosurgical skills in the medical students group exposed to microsurgical training. The true effect of microsurgical training warrants further investigation. Level of evidence: Level I, diagnostic study.

scholarly journals A CanMEDS Competency-Based Assessment Tool for High-Fidelity Simulation in Internal Medicine: The Montreal Internal Medicine Evaluation Scale (MIMES)

2018 ◽  
Vol 13 (4) ◽  
pp. e10-e16
Author(s):  
Patrice Chrétien Raymer ◽  
Jean-Paul Makhzoum ◽  
Robert Gagnon ◽  
Arielle Levy ◽  
Jean-Pascal Costa
Keyword(s):  
Internal Medicine ◽  
Rating Scale ◽  
Assessment Tool ◽  
Teaching Method ◽  
Global Rating Scale ◽  
High Fidelity ◽  
Global Rating ◽  
High Fidelity Simulation ◽  
Holistic Teaching ◽  
Competency Based

Introduction: High-fidelity simulation is an efficient and holistic teaching method. However, assessing simulation performances remains a challenge. We aimed to develop a CanMEDS competency-based global rating scale for internal medicine trainees during simulated acute care scenarios. Methods: Our scale was developed using a formal Delphi process. Validity was tested using six videotaped scenarios of two residents managing unstable atrial fibrillation, rated by 6 experts. Psychometric properties were determined using a G-study and a satisfaction questionnaire.Results: Most evaluators favorably rated the usability of our scale, and attested that the tool fully covered CanMEDS competencies. The scale showed low to intermediate generalization validity.Conclusions: This study demonstrated some validity arguments for our scale. The best assessed aspect of performance was communication; further studies are planned to gather further validity arguments for our scale and to compare assessment of teamwork and communication during scenarios with multiple versus single residents.

scholarly journals Otoskills training during covid-19 pandemic: a before-after study

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Maxime Fieux ◽  
Antoine Gavoille ◽  
Fabien Subtil ◽  
Sophie Bartier ◽  
Stéphane Tringali
Keyword(s):  
Surgical Training ◽  
Rating Scale ◽  
Surgical Simulation ◽  
Global Rating Scale ◽  
Middle Ear Surgery ◽  
Global Rating ◽  
Training Requirements ◽  
Significant Difference ◽  
Before And After ◽  
Total Performance

Abstract Background The ongoing COVID-19 pandemic has disrupted the surgical training of residents. There is a real concern that trainees will not be able to meet their training requirements. Low-fidelity surgical simulation appears to be an alternative for surgical training. The educational benefits of repeating ossiculoplasty simulations under a microscope have never been evaluated. With this study we aimed to evaluate the differences in performance scores and on a global rating scale before and after training on an ossiculoplasty simulator. Methods In this quasi-experimental, prospective, single-centre, before-after study with blinded rater evaluation, residents performed five microscopic ossiculoplasty tasks with a difficulty gradient (sliding beads onto rods, the insertion of a partial prosthesis, the insertion of a total prosthesis, and the insertion of a stapedotomy piston under microscopic or endoscopic surgery) before and after training on the same simulator. Performance scores were defined for each task, and total performance scores (score/min) were calculated. All data were collected prospectively. Results Six out of seven intermediate residents and 8/9 novices strongly agreed that the simulator was an effective training device and should be included in the ENT residency program. The mean effect of training was a significant increase in the total performance score (+ 0.52 points/min, [95 % CI, 0.40–0.64], p < 0.001), without a significant difference between novice and intermediate residents. Conclusions This preliminary study shows that techniques for middle-ear surgery can be acquired using a simulator, avoiding any risk for patients, even under lockdown measures.

Sign in / Sign up

Export Citation Format

Share Document