Assessing Performance in Simulated Cleft Palate Repair Using a Novel Video Recording Setup

2020 ◽  
Vol 57 (6) ◽  
pp. 687-693
Author(s):  
Dale J. Podolsky ◽  
David M. Fisher ◽  
Karen W. Wong Riff ◽  
Ronald M. Zuker ◽  
James M. Drake ◽  
...  
Keyword(s):  
Cleft Palate ◽  
Rating Scale ◽  
Intraclass Correlation ◽  
Video Recording ◽  
Global Rating Scale ◽  
Global Rating ◽  
Global Score ◽  
Technical Performance ◽  
Cleft Palate Repair ◽  
Palate Repair

Objective: To test the feasibility of implementing a high-fidelity cleft palate simulator during a workshop in Santiago, Chile, using a novel video endoscope to assess technical performance. Design: Sixteen cleft surgeons from South America participated in a 2-day cleft training workshop. All 16 participants performed a simulated repair, and 13 of them performed a second simulated repair. The repairs were recorded using a low-cost video camera and a newly designed camera mouth retractor attachment. Twenty-nine videos were assessed by 3 cleft surgeons using a previously developed cleft palate objective structured assessment of technical skill (CLOSATS with embedded overall score assessment) and global rating scale. The reliability of the ratings and technical performance in relation to minimum acceptable scores and previous experience was assessed. Results: The video setup provided acceptable recording quality for the purpose of assessment. Average intraclass correlation coefficient for the CLOSATS, global, and overall performance score was 0.69, 0.75, and 0.82, respectively. None of the novice surgeons passed the CLOSATS and global score for both sessions. One participant in the intermediate group, and 2 participants in the advanced group passed the CLOSATS and global score for both sessions. There were highly experienced participants who failed to pass the CLOSATS and global score for both sessions. Conclusions: The cleft palate simulator can be practically implemented with video-recording capability to assess performance in cleft palate repair. This technology may be of assistance in assessing surgical competence in cleft palate repair.

Validation of a Dry Model for Assessing the Performance of Arthroscopic Hip Labral Repair

2017 ◽  
Vol 45 (9) ◽  
pp. 2125-2130 ◽  
Author(s):  
Lisa Phillips ◽  
Jeffrey J.H. Cheung ◽  
Daniel B. Whelan ◽  
Michael Lucas Murnaghan ◽  
Jas Chahal ◽  
...  
Keyword(s):  
Rating Scale ◽  
Intraclass Correlation ◽  
Total Asset ◽  
Labral Repair ◽  
Global Rating Scale ◽  
Task Completion ◽  
Global Rating ◽  
Evaluation Tool ◽  
Level Of Evidence ◽  
Level Of Training

Background: Arthroscopic hip labral repair is a technically challenging and demanding surgical technique with a steep learning curve. Arthroscopic simulation allows trainees to develop these skills in a safe environment. Purpose: The purpose of this study was to evaluate the use of a combination of assessment ratings for the performance of arthroscopic hip labral repair on a dry model. Study Design: Cross-sectional study; Level of evidence, 3. Methods: A total of 47 participants including orthopaedic surgery residents (n = 37), sports medicine fellows (n = 5), and staff surgeons (n = 5) performed arthroscopic hip labral repair on a dry model. Prior arthroscopic experience was noted. Participants were evaluated by 2 orthopaedic surgeons using a task-specific checklist, the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task completion time, and a final global rating scale. All procedures were video-recorded and scored by an orthopaedic fellow blinded to the level of training of each participant. Results: The internal consistency/reliability (Cronbach alpha) using the total ASSET score for the procedure was high (intraclass correlation coefficient > 0.9). One-way analysis of variance for the total ASSET score demonstrated a difference between participants based on the level of training ( F3,43 = 27.8, P < .001). A good correlation was seen between the ASSET score and previous exposure to arthroscopic procedures ( r = 0.52-0.73, P < .001). The interrater reliability for the ASSET score was excellent (>0.9). Conclusion: The results of this study demonstrate that the use of dry models to assess the performance of arthroscopic hip labral repair by trainees is both valid and reliable. Further research will be required to demonstrate a correlation with performance on cadaveric specimens or in the operating room.

Evaluation of Patient Simulator Performance as an Adjunct to the Oral Examination for Senior Anesthesia Residents

2006 ◽  
Vol 104 (3) ◽  
pp. 475-481 ◽  
Author(s):  
Georges L. Savoldelli ◽  
Viren N. Naik ◽  
Hwan S. Joo ◽  
Patricia L. Houston ◽  
Marianne Graham ◽  
...  
Keyword(s):  
Concurrent Validity ◽  
Rating Scale ◽  
Intraclass Correlation ◽  
Oral Examination ◽  
Added Value ◽  
Laboratory Simulation ◽  
Global Rating Scale ◽  
Average Score ◽  
Global Rating ◽  
Anesthesia Residents

Background Patient simulators possess features for performance assessment. However, the concurrent validity and the "added value" of simulator-based examinations over traditional examinations have not been adequately addressed. The current study compared a simulator-based examination with an oral examination for assessing the management skills of senior anesthesia residents. Methods Twenty senior anesthesia residents were assessed sequentially in resuscitation and trauma scenarios using two assessment modalities: an oral examination, followed by a simulator-based examination. Two independent examiners scored the performances with a previously validated global rating scale developed by the Anesthesia Oral Examination Board of the Royal College of Physicians and Surgeons of Canada. Different examiners were used to rate the oral and simulation performances. Results Interrater reliability was good to excellent across scenarios and modalities: intraclass correlation coefficients ranged from 0.77 to 0.87. The within-scenario between-modality score correlations (concurrent validity) were moderate: r = 0.52 (resuscitation) and r = 0.53 (trauma) (P &lt; 0.05). Forty percent of the average score variance was accounted for by the participants, and 30% was accounted for by the participant-by-modality interaction. Conclusions Variance in participant scores suggests that the examination is able to perform as expected in terms of discriminating among test takers. The rather large participant-by-modality interaction, along with the pattern of correlations, suggests that an examinee's performance varies based on the testing modality and a trainee who "knows how" in an oral examination may not necessarily be able to "show how" in a simulation laboratory. Simulation may therefore be considered a useful adjunct to the oral examination.

Advanced closed-loop communication training: the blindfolded resuscitation

2019 ◽  
Vol 6 (4) ◽  
pp. 235-238
Author(s):  
Kate E Hughes ◽  
Patrick G Hughes ◽  
Thomas Cahir ◽  
Jennifer Plitt ◽  
Vivienne Ng ◽  
...  
Keyword(s):  
Closed Loop ◽  
Rating Scale ◽  
Intraclass Correlation ◽  
Calculated Data ◽  
Global Rating Scale ◽  
Global Rating ◽  
Control Groups ◽  
Advanced Technique ◽  
Task Load ◽  
And Control

Closed-loop communication (CLC) improves task efficiency and decreases medical errors; however, limited literature on strategies to improve real-time use exist. The primary objective was whether blindfolding a resuscitation leader was effective to improve crisis resource management (CRM) skills, as measured by increased frequency of CLC. Secondary objectives included whether blindfolding affected overall CRM performance or perceived task load. Participants included emergency medicine (EM) or EM/paediatric dual resident physicians. Participants completed presurveys, were block randomised into intervention (blindfolded) or control groups, lead both adult and paediatric resuscitations and completed postsurveys before debriefing. Video recordings of the simulations were reviewed by simulation fellowship-trained EM physicians and rated using the Ottawa CRM Global Rating Scale (GRS). Frequency of CLC was assessed by one rater via video review. Summary statistics were performed. Intraclass correlation coefficient was calculated. Data were analysed using R program for analysis of variance and regression analysis. There were no significant differences between intervention and control groups in any Ottawa CRM GRS category. Postgraduate year (PGY) significantly impacts all Ottawa GRS categories. Frequency of CLC use significantly increased in the blindfolded group (31.7, 95% CI 29.34 to 34.1) vs the non-blindfolded group (24.6, 95% CI 21.5 to 27.7). Participant’s self-rated perceived NASA Task Load Index scores demonstrated no difference between intervention and control groups via a Wilcoxon rank sum test. Blindfolding the resuscitation leader significantly increases frequency of CLC. The blindfold code training exercise is an advanced technique that may increase the use of CLC.

scholarly journals Evaluating Teamwork in a Simulated Obstetric Environment

2007 ◽  
Vol 106 (5) ◽  
pp. 907-915 ◽  
Author(s):  
Pamela J. Morgan ◽  
Richard Pittini ◽  
Glenn Regehr ◽  
Carol Marrs ◽  
Michèle F. Haley
Keyword(s):  
Rating Scale ◽  
Assessment Tool ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Obstetric Care ◽  
Global Rating Scale ◽  
Global Rating ◽  
Good Reliability ◽  
Domain Specific ◽  
Team Assessment

Background The National Confidential Enquiry into Maternal Deaths identified "lack of communication and teamwork" as a leading cause of substandard obstetric care. The authors used high-fidelity simulation to present obstetric scenarios for team assessment. Methods Obstetric nurses, physicians, and resident physicians were repeatedly assigned to teams of five or six, each team managing one of four scenarios. Each person participated in two or three scenarios with differently constructed teams. Participants and nine external raters rated the teams' performances using a Human Factors Rating Scale (HFRS) and a Global Rating Scale (GRS). Interrater reliability was determined using intraclass correlations and the Cronbach alpha. Analyses of variance were used to determine the reliability of the two measures, and effects of both scenario and rater profession (R.N. vs. M.D.) on scores. Pearson product-moment correlations were used to compare external with self-generated assessments. Results The average of nine external rater scores showed good reliability for both HFRS and GRS; however, the intraclass correlation coefficients for a single rater was low. There was some effect of rater profession on self-generated HFRS but not on GRS. An analysis of profession-specific subscores on the HFRS revealed no interaction between profession of rater and profession being rated. There was low correlation between externally and self-generated team assessments. Conclusions This study does not support the use of the HFRS for assessment of obstetric teams. The GRS shows promise as a summative but not a formative assessment tool. It is necessary to develop a domain specific behavioral marking system for obstetric teams.

scholarly journals Development and Testing of Screen-Based and Psychometric Instruments for Assessing Resident Performance in an Operating Room Simulator

2016 ◽  
Vol 2016 ◽  
pp. 1-13 ◽  
Author(s):  
Richard R. McNeer ◽  
Roman Dudaryk ◽  
Nicholas B. Nedeff ◽  
Christopher L. Bennett
Keyword(s):  
Operating Room ◽  
Internal Consistency ◽  
Rating Scale ◽  
Clinical Skills ◽  
Intraclass Correlation ◽  
Rank Correlation ◽  
Correlation Coefficients ◽  
Global Rating Scale ◽  
Global Rating ◽  
Anesthesia Residents

Introduction. Medical simulators are used for assessing clinical skills and increasingly for testing hypotheses. We developed and tested an approach for assessing performance in anesthesia residents using screen-based simulation that ensures expert raters remain blinded to subject identity and experimental condition.Methods. Twenty anesthesia residents managed emergencies in an operating room simulator by logging actions through a custom graphical user interface. Two expert raters rated performance based on these entries using custom Global Rating Scale (GRS) and Crisis Management Checklist (CMC) instruments. Interrater reliability was measured by calculating intraclass correlation coefficients (ICC), and internal consistency of the instruments was assessed with Cronbach’s alpha. Agreement between GRS and CMC was measured using Spearman rank correlation (SRC).Results. Interrater agreement (GRS: ICC = 0.825, CMC: ICC = 0.878) and internal consistency (GRS: alpha = 0.838, CMC: alpha = 0.886) were good for both instruments. Subscale analysis indicated that several instrument items can be discarded. GRS and CMC scores were highly correlated (SRC = 0.948).Conclusions. In this pilot study, we demonstrated that screen-based simulation can allow blinded assessment of performance. GRS and CMC instruments demonstrated good rater agreement and internal consistency. We plan to further test construct validity of our instruments by measuring performance in our simulator as a function of training level.

scholarly journals Impact of video games on ultrasound-guided regional anesthesia skills

2020 ◽  
Vol 45 (11) ◽  
pp. 860-865
Author(s):  
Atif Shafqat ◽  
Shumaila Mukarram ◽  
Nigel M Bedforth ◽  
Jonathan G Hardman ◽  
Robert A McCahon
Keyword(s):  
Mental Rotation ◽  
Regional Anesthesia ◽  
Video Game ◽  
Rating Scale ◽  
Global Rating Scale ◽  
Global Rating ◽  
Ultrasound Guided ◽  
Technical Performance ◽  
Visuospatial Abilities ◽  
Game Practice

BackgroundThere is an association between video game practice and laparoscopic expertize in trainee surgeons. Ultrasound-guided regional anesthesia has many parallels with laparoscopic surgery. The aim of this study was to explore whether video game experience is associated with enhanced performance in a simulated ultrasound-guided task in novice operators.MethodsIn this prospective observational study, 60 medical student volunteers were recruited. Following characterization of video game experience, they underwent an assessment of visuospatial abilities. Following standardized teaching, the recruits’ technical performance of an ultrasound-guided needle task was assessed for overall quality by global rating scale (GRS).ResultsOut of a total possible GRS score of 35, gamers compared with non-gamers demonstrated 5.2 (95% CI 1.9 to 8.4) units of better performance. Gamers also performed better in mental rotation test scores (difference 4.1, 95% CI 1.2 to 7.0) .ConclusionVideo game practice is associated with increased mental rotation ability and enhanced technical performance in a simulated ultrasound-guided task.

scholarly journals Otoskills training during covid-19 pandemic: a before-after study

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Maxime Fieux ◽  
Antoine Gavoille ◽  
Fabien Subtil ◽  
Sophie Bartier ◽  
Stéphane Tringali
Keyword(s):  
Surgical Training ◽  
Rating Scale ◽  
Surgical Simulation ◽  
Global Rating Scale ◽  
Middle Ear Surgery ◽  
Global Rating ◽  
Training Requirements ◽  
Significant Difference ◽  
Before And After ◽  
Total Performance

Abstract Background The ongoing COVID-19 pandemic has disrupted the surgical training of residents. There is a real concern that trainees will not be able to meet their training requirements. Low-fidelity surgical simulation appears to be an alternative for surgical training. The educational benefits of repeating ossiculoplasty simulations under a microscope have never been evaluated. With this study we aimed to evaluate the differences in performance scores and on a global rating scale before and after training on an ossiculoplasty simulator. Methods In this quasi-experimental, prospective, single-centre, before-after study with blinded rater evaluation, residents performed five microscopic ossiculoplasty tasks with a difficulty gradient (sliding beads onto rods, the insertion of a partial prosthesis, the insertion of a total prosthesis, and the insertion of a stapedotomy piston under microscopic or endoscopic surgery) before and after training on the same simulator. Performance scores were defined for each task, and total performance scores (score/min) were calculated. All data were collected prospectively. Results Six out of seven intermediate residents and 8/9 novices strongly agreed that the simulator was an effective training device and should be included in the ENT residency program. The mean effect of training was a significant increase in the total performance score (+ 0.52 points/min, [95 % CI, 0.40–0.64], p < 0.001), without a significant difference between novice and intermediate residents. Conclusions This preliminary study shows that techniques for middle-ear surgery can be acquired using a simulator, avoiding any risk for patients, even under lockdown measures.

Sign in / Sign up

Export Citation Format

Share Document