scholarly journals Reliability of Simulation-Based Assessment for Practicing Physicians: Performance is Context-Specific

2020 ◽  
Author(s):  
Elizabeth Sinz ◽  
Arna Banerjee ◽  
Randolph Steadman ◽  
Matthew S Shotwell ◽  
Jason Slagle ◽  
...  
Keyword(s):  
Reliability And Validity ◽  
Critical Events ◽  
Assessment Scores ◽  
Technical Performance ◽  
Complex Simulation ◽  
Simulation Based ◽  
Study Population ◽  
Behavioral Score ◽  
Context Specific ◽  
Practicing Physicians

Abstract Introduction: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.Methods: Standardized complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioral and technical performance measures. Analysis using Generalizability and Decision studies were completed for the two scenarios with two raters.Results: The technical score was not more reliable than the behavioral score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters would have little effect on reliability.Discussion: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. Moreover, aggregate data from a population of participants may yield even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and continuing education improvements across the profession.

Reliability of Simulation-based Assessment for Practicing Physicians: Performance is Context-Specific

2021 ◽  
Author(s):  
Elizabeth Sinz ◽  
Arna Banerjee ◽  
Randolph Steadman ◽  
Matthew S Shotwell ◽  
Jason Slagle ◽  
...  
Keyword(s):  
Reliability And Validity ◽  
High Fidelity Simulation ◽  
Critical Events ◽  
Assessment Scores ◽  
Technical Performance ◽  
Complex Simulation ◽  
Simulation Based ◽  
Study Population ◽  
Context Specific ◽  
Practicing Physicians

Abstract Introduction: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.Methods: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.Results: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.Conclusions: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

scholarly journals Reliability of simulation-based assessment for practicing physicians: performance is context-specific

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Elizabeth Sinz ◽  
Arna Banerjee ◽  
Randolph Steadman ◽  
Matthew S. Shotwell ◽  
Jason Slagle ◽  
...  
Keyword(s):  
Reliability And Validity ◽  
High Fidelity Simulation ◽  
Critical Events ◽  
Assessment Scores ◽  
Technical Performance ◽  
Complex Simulation ◽  
Simulation Based ◽  
Study Population ◽  
Context Specific ◽  
Practicing Physicians

Abstract Introduction Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. Methods Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. Results The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. Conclusions The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Reliability and Validity of a Simulation-based Acute Care Skills Assessment for Medical Students and Residents

2003 ◽  
Vol 99 (6) ◽  
pp. 1270-1280 ◽  
Author(s):  
John R. Boulet ◽  
David Murray ◽  
Joe Kras ◽  
Julie Woodhouse ◽  
John McAllister ◽  
...  
Keyword(s):  
Medical Students ◽  
Acute Care ◽  
Reliability And Validity ◽  
Faculty Members ◽  
First Year ◽  
Clinical Experiences ◽  
Critical Events ◽  
Simulation Based ◽  
Broad Domain ◽  
Simulation Exercises

Background Medical students and residents are expected to be able to manage a variety of critical events after training, but many of these individuals have limited clinical experiences in the diagnosis and treatment of these conditions. Life-sized mannequins that model critical events can be used to evaluate the skills required to manage and treat acute medical conditions. The purpose of this study was to develop and test simulation exercises and associated scoring methods that could be used to evaluate the acute care skills of final-year medical students and first-year residents. Methods The authors developed and tested 10 simulated acute care situations that clinical faculty at a major medical school expects graduating physicians to be able to recognize and treat at the conclusion of training. Forty medical students and residents participated in the evaluation of the exercises. Four faculty members scored the students/residents. Results The reliability of the simulation scores was moderate and was most strongly influenced by the choice and number of simulated encounters. The validity of the simulation scores was supported through comparisons of students'/residents' performances in relation to their clinical backgrounds and experience. Conclusion Acute care skills can be validly and reliably measured using a simulation technology. However, multiple simulated encounters, covering a broad domain, are needed to effectively and accurately estimate student/resident abilities in acute care settings.

scholarly journals Simulation-based Assessment to Identify Critical Gaps in Safe Anesthesia Resident Performance

2014 ◽  
Vol 120 (1) ◽  
pp. 129-141 ◽  
Author(s):  
Richard H. Blum ◽  
John R. Boulet ◽  
Jeffrey B. Cooper ◽  
Sharon L. Muret-Wagstaff
Keyword(s):  
Rating Scale ◽  
Reliability And Validity ◽  
Assessment System ◽  
Assessment Tools ◽  
First Year ◽  
Anesthesia Resident ◽  
Resident Performance ◽  
Assessment Scores ◽  
Simulation Based ◽  
Safe Anesthesia

Abstract Background: Valid methods are needed to identify anesthesia resident performance gaps early in training. However, many assessment tools in medicine have not been properly validated. The authors designed and tested use of a behaviorally anchored scale, as part of a multiscenario simulation-based assessment system, to identify high- and low-performing residents with regard to domains of greatest concern to expert anesthesiology faculty. Methods: An expert faculty panel derived five key behavioral domains of interest by using a Delphi process (1) Synthesizes information to formulate a clear anesthetic plan; (2) Implements a plan based on changing conditions; (3) Demonstrates effective interpersonal and communication skills with patients and staff; (4) Identifies ways to improve performance; and (5) Recognizes own limits. Seven simulation scenarios spanning pre-to-postoperative encounters were used to assess performances of 22 first-year residents and 8 fellows from two institutions. Two of 10 trained faculty raters blinded to trainee program and training level scored each performance independently by using a behaviorally anchored rating scale. Residents, fellows, facilitators, and raters completed surveys. Results: Evidence supporting the reliability and validity of the assessment scores was procured, including a high generalizability coefficient (ρ2 = 0.81) and expected performance differences between first-year resident and fellow participants. A majority of trainees, facilitators, and raters judged the assessment to be useful, realistic, and representative of critical skills required for safe practice. Conclusion: The study provides initial evidence to support the validity of a simulation-based performance assessment system for identifying critical gaps in safe anesthesia resident performance early in training.

scholarly journals Implementing the transvaginal ultrasound simulation training (TRUSST) programme for obstetric registrars

2021 ◽  
Vol 6 (1) ◽  
Author(s):  
Sally Byford ◽  
Sarah Janssens ◽  
Rachel Cook
Keyword(s):  
Early Pregnancy ◽  
Confidence Level ◽  
Simulation Training ◽  
Systematic Approach ◽  
Transvaginal Ultrasound ◽  
Skills Training ◽  
Ultrasound Simulation ◽  
Learning Gap ◽  
Assessment Scores ◽  
Simulation Based

Abstract Background Transvaginal ultrasound (TVUS) training opportunities are limited due to its intimate nature; however, TVUS is an important component of early pregnancy assessment. Simulation can bridge this learning gap. Aim To describe and measure the effect of a transvaginal ultrasound simulation programme for obstetric registrars. Materials and methods The transvaginal ultrasound simulation training (TRUSST) curriculum consisted of supported practice using virtual reality transvaginal simulators (ScanTrainer, Medaphor) and communication skills training to assist obstetric registrars in obtaining required competencies to accurately and holistically care for women with early pregnancy complications. Trainee experience of live transvaginal scanning was evaluated with a questionnaire. Programme evaluation was by pre-post self-reported confidence level and objective pre-post training assessment using Objective Structured Assessment of Ultrasound Skills (OSAUS) and modified Royal Australian and New Zealand College of Obstetrics and Gynaecology assessment scores. Quantitative data was compared using paired t tests. Results Fifteen obstetric registrars completed the programme. Numbers of performed live transvaginal ultrasound by trainees were low. Participants reported an increase in confidence level in performing a TVUS following training: mean pre score 1.6/5, mean post score 3/5. Objective assessments improved significantly across both OSAUS and RANZCOG scores following training; mean improvement scores 7.6 points (95% CI 6.2–8.9, p < 0.05) and 32.5 (95% CI 26.4–38.6, p < 0.05) respectively. It was noted that scores for a systematic approach and documentation were most improved: 1.9 (95% CI 1.4–2.5, p < 0.05) and 2.1 (95% CI 1.5–2.7, p < 0.05) respectively. Conclusion The implementation of a simulation-based training curriculum resulted in improved confidence and ability in TVUS scanning, especially with regard to a systematic approach and documentation.

scholarly journals The Issue of Alternative Assessment on Students' Achievement from the Viewpoints of Teachers’ at Elementary Schools in Gaza

2019 ◽  
Vol 8 (2S3) ◽  
pp. 414-418
Keyword(s):  
Academic Achievement ◽  
Elementary Schools ◽  
Alternative Assessment ◽  
Gaza Strip ◽  
Reliability And Validity ◽  
Study Population ◽  
The Gaza Strip ◽  
Level Of Significance ◽  
High Level ◽  
High Degree

This study aimed at identifying the effect of alternative assessment on academic achievement, from the viewpoint of teachers in elementary schools in Gaza. To achieve the study objectives, a questionnaire developed consisting of 21 items then reliability and validity were calculated, the study population consisted of 120 teachers. The study sample consisted of 92 teachers, selected in a stratified random method, where the total of 87 valid statistical analyzes retrieved by 94.56%. The results of the study showed: The perceptions of the members of the study sample to the alternative assessment came to a high degree, the academic achievement from teachers in the basic schools in the Gaza Strip came at the macro level at a high level, and there is a statistically significant effect at the level of significance (α<=0.05) of alternative assessment on academic achievement. In light of the results, a set of recommendations were proposed, including The Directorate of Education in the Gaza Strip should be concerned with the adoption of alternative assessment methods in its affiliated schools, because of the importance of alternative assessment in improving the level of academic achievement of students

Development and Standardization of an Alienation Scale for Visually Impaired Students

2017 ◽  
Vol 111 (5) ◽  
pp. 427-439 ◽  
Author(s):  
Poonam Punia ◽  
Sandeep Berwal
Keyword(s):  
Visually Impaired ◽  
Low Vision ◽  
Item Analysis ◽  
Visual Impairments ◽  
Reliability And Validity ◽  
Future Research ◽  
Rehabilitation Professionals ◽  
Cultural Estrangement ◽  
Visually Impaired Students ◽  
Context Specific

Introduction The present study was undertaken to develop a valid and reliable scale for measuring a feeling of alienation in students with visual impairments (that is, those who are blind or have low vision). Methods In this study, a pool of 60 items was generated to develop an Alienation Scale for Visually Impaired Students (AL-VI) based on a review of the literature and discussions with colleagues and experts in the field. The items were organized into six dimensions of alienation, namely powerlessness, meaninglessness, normlessness, social isolation, self-estrangement, and cultural estrangement, and were rated on the five-point Likert scale. The standardization of the scale was completed with 118 students with visual impairments in the age group of 10 to 25 years, selected randomly from specialized and inclusive schools in the state of Haryana, India. Results The item analysis was done by calculating t- and r-values; seven items were deleted, and a final 45 items were retained. The calculated value of Cronbach's alpha and split-half correlation came out to be 0.87 and 0.86, respectively. The construct validity was determined by computing the coefficient of correlation between scores of this scale and the scores obtained by using the Student Alienation scale (SAS) of R. R. Sharma (Sharma, 2012). The percentile norm for the scale was determined after verifying normality of the scores by using a Q-Q plot. Discussion The findings of the present study suggest that the AL-VI may serve as a useful tool in future research to assess alienation in persons with visual impairments in India. The findings further demonstrate that the AL-VI produced scores that are reliable and valid. The AL-VI can be used outside India after determining its reliability and validity in context-specific conditions. Implications for practitioners The AL-VI scale is applicable to students with visual impairments, teachers of students with visual impairments, principals, social workers, psychologists, and rehabilitation professionals. Practitioners could use this tool for assessing and understanding the level of alienation among students with visual impairments, thereby helping them in planning and executing strategies for remediating alienation.

scholarly journals P100: Iterative prototype development of a mobile tele-simulation unit for remote training: an update

CJEM ◽  
2017 ◽  
Vol 19 (S1) ◽  
pp. S112 ◽  
Author(s):  
M.H. Parsons ◽  
A. Smith ◽  
K.J. Hoover ◽  
J. Jewer ◽  
S. Noseworthy ◽  
...  
Keyword(s):  
Emergency Medicine ◽  
Mixed Methods ◽  
Learning Experience ◽  
Rural And Remote ◽  
Mixed Methods Approach ◽  
Technical Performance ◽  
Prototype Development ◽  
Simulation Based ◽  
Technical Issues ◽  
Simulation Unit

Introduction/Innovation Concept: Rural and remote practice of emergency medicine presents unique challenges, particularly when faced with infrequently encountered cases and procedures. Simulation-based training is a valuable tool in the acquisition and maintenance of knowledge and skills; however, simulators are often located in larger centers and they are not widely outside these centers due to geographic, cost and time constraints. Mobile tele-simulation has the potential to overcome barriers but challenges such as comfort, technical issues and ability to teach desired content via tele-simulation must be addressed. We are developing a mobile-tele-simulation unit (MTU) prototype that will enable emergency medicine practitioners and trainees to access simulation-based instruction in rural and remote settings. Methods: Through application of a mixed-methods approach with input of a multidisciplinary team we are iteratively developing an MTU prototype to assess key factors in design and function, including: technical issues, environmental features, and human factors. The Delphi method is being used to collect input from experts on key design components and feedback is also being collected from trainees after participating in trial deployments of the MTU in different educational and environmental settings. Curriculum, Tool, or Material: The effective application of the MTU in a variety of learning settings will be optimized through ongoing evaluation in the iterative design cycle. Feedback to ensure a quality learning experience in the MTU will direct features of physical design and technical performance that can be applied in deployment of the unit. In addition, challenges to the delivery of module content and instructional modality/ features of lessons to be executed will be important considerations as we move toward developing content that can effectively be taught using the MTU. Conclusion: To ensure effective use of tele-simulation in the delivery of a meaningful simulation experience to rural and remote trainees a number of important challenges must be overcome. We describe our evolving multidisciplinary mixed-methods approach to develop an effective mobile tele-simulation unit.

scholarly journals Adaptación y validación del cuestionario Epod 2.1 a usuarios libres de centros deportivos (Adaptation and validation of the Epod 2.1 questionnaire in free customers of sport centers)

Retos ◽  
2016 ◽  
pp. 40-45
Author(s):  
Alberto Nuviala ◽  
Javier Antonio Tamayo-Fajardo ◽  
Cesar Ruiz-Alejos Gómez ◽  
Román Nuviala Nuviala ◽  
Josep María Dalmau Torres
Keyword(s):  
Factor Analysis ◽  
Reliability And Validity ◽  
Validity And Reliability ◽  
Sports Organizations ◽  
Perceived Values ◽  
Sport Activities ◽  
Instrument Measuring ◽  
Study Population ◽  
Sport Centers ◽  
Invariance Test

Los servicios deportivos han evolucionado de forma considerable en los últimos años. Se ha visto incrementado el tipo, número de actividades y servicios ofrecidos a los clientes. Al mismo tiempo han aparecido en las organizaciones deportivas usuarios que realizan sus prácticas físico deportivas de forma libre. Los instrumentos creados para conocer las valoraciones del servicio recibido por parte de los usuarios han sido diversos. Sin embargo, ninguno de ellos ha sido diseñado para medir específicamente los juicios de valor de estos clientes que realizan sus actividades sin la presencia ni dirección de un técnico deportivo. El objetivo de esta investigación ha sido validar un instrumento que permita conocer la calidad percibida, valor percibido y satisfacción de usuarios que realizan actividades de forma libre. La población del estudio está conformada por 591 usuarios que realizan este tipo de actividad física en organizaciones deportivas públicas. Se realizó un análisis estadístico de los ítems, un análisis factorial exploratorio, un análisis factorial confirmatorio, pruebas de invarianza factorial, así como pruebas de fiabilidad y validez. Los resultados garantizan la validez y fiabilidad del instrumento para medir los juicios de valor de usuarios libres referentes a los tres constructos que conforman el cuestionario de análisis del servicio recibido, compuesto finalmente por 22 ítems agrupados en 8 dimensiones.Abstract. Sport services have considerably evolved in recent years. There has been an increase in the type and number of activities and services offered to customers. At the same time, an increased number of clients of sports organizations do physical-sport activities on their own. Several instruments were created in order to assess customers’ evaluation of services received. However, none of them has been designed to specifically measure services appreciation of those customers who exercise without any support or presence of a sport specialist. The objective of this research was to validate an instrument that allows to analyze perceived quality, perceived values, and satisfaction from customers who workout on their own. The study population was composed by 591 customers of sport organizations who exercise on their own. Statistical analysis of items, Exploratory Factor Analysis, Confirmatory Factor Analysis, and factor invariance test were run, as well as reliability and validity were tested. Results demonstrated the validity and reliability of the instrument measuring free-use customers’ perception of the three constructs that constitute the questionnaire, whose final version was composed by 22 items grouped in 8 dimensions.

Sign in / Sign up

Export Citation Format

Share Document