A Multirater Instrument for the Assessment of Simulated Pediatric Crises

Abstract Background Few validated instruments exist to measure pediatric code team skills. The goal of this study was to develop an instrument for the assessment of resuscitation competency and self-appraisal using multirater and gap analysis methodologies. Methods Multirater assessment with gap analysis is a robust methodology that enables the measurement of self-appraisal as well as competency, offering faculty the ability to provide enhanced feedback. The Team Performance during Simulated Crises Instrument (TPDSCI) was grounded in the Accreditation Council for Graduate Medical Education competencies. The instrument contains 5 competencies, each assessed by a series of descriptive rubrics. It was piloted during a series of simulation-based interdisciplinary pediatric crisis resource management education sessions. Course faculty assessed participants, who also did self-assessments. Internal consistency and interrater reliability were analyzed using Cronbach α and intraclass correlation (ICC) statistics. Gap analysis results were examined descriptively. Results Cronbach α for the instrument was between 0.72 and 0.69. The overall ICC was 0.82. ICC values for the medical knowledge, clinical skills, communication skills, and systems-based practice were between 0.87 and 0.72. The ICC for the professionalism domain was 0.22. Further examination of the professionalism competency revealed a positive skew, 43 simulated sessions (98%) had significant gaps for at least one of the competencies, 38 sessions (86%) had gaps indicating self-overappraisal, and 15 sessions (34%) had gaps indicating self-underappraisal. Conclusions The TPDSCI possesses good measures of internal consistency and interrater reliability with respect to medical knowledge, clinical skills, communication skills, systems-based practice, and overall competence in the context of simulated interdisciplinary pediatric medical crises. Professionalism remains difficult to assess. These results provide an encouraging first step toward instrument validation. Gap analysis reveals disparities between faculty and self-assessments that indicate inadequate participant self-reflection. Identifying self-overappraisal can facilitate focused interventions.

Download Full-text

Self-Assessment Scores Improve After Simulation-Based Palliative Care Communication Skill Workshops

American Journal of Hospice and Palliative Medicine® ◽

10.1177/1049909116681972 ◽

2016 ◽

Vol 35 (1) ◽

pp. 45-51 ◽

Cited By ~ 14

Author(s):

Crystal E. Brown ◽

Anthony L. Back ◽

Dee W. Ford ◽

Erin K. Kross ◽

Lois Downey ◽

...

Keyword(s):

Palliative Care ◽

South Carolina ◽

Randomized Trial ◽

Communication Skills ◽

Communication Skill ◽

Linear Regression Models ◽

Care Needs ◽

Self Assessment ◽

Simulation Based ◽

Self Assessments

Background: We conducted a randomized trial of a simulation-based multisession workshop to improve palliative care communication skills (Codetalk). Standardized patient assessments demonstrated improved communication skills for trainees receiving the intervention; however, patient and family assessments failed to demonstrate improvement. This article reports findings from trainees’ self-assessments. Aim: To examine whether Codetalk resulted in improved self-assessed communication competence by trainees. Design: Trainees were recruited from the University of Washington and the Medical University of South Carolina. Internal medicine residents, medicine subspecialty fellows, nurse practitioner students, or community-based advanced practice nurses were randomized to Codetalk, a simulation-based workshop, or usual education. The outcome measure was self-assessed competence discussing palliative care needs with patients and was assessed at the start and end of the academic year. We used robust linear regression models to predict self-assessed competency, both as a latent construct and as individual indicators, including randomization status and baseline self-assessed competency. Results: We randomized 472 trainees to the intervention (n = 232) or usual education (n = 240). The intervention was associated with an improvement in trainee’s overall self-assessment of competence in communication skills ( P < .001). The intervention was also associated with an improvement in trainee self-assessments of 3 of the 4 skill-specific indicators—expressing empathy, discussing spiritual issues, and eliciting goals of care. Conclusion: Simulation-based communication training was associated with improved self-assessed competency in overall and specific communication skills in this randomized trial. Further research is needed to fully understand the importance and limitations of self-assessed competence in relation to other outcomes of improved communication skill.

Download Full-text

Reliability of Physiological Responses Induced by Basic Emotions: A Pilot Study

Journal of PHYSIOLOGICAL ANTHROPOLOGY ◽

10.1186/s40101-019-0209-y ◽

2019 ◽

Vol 38 (1) ◽

Author(s):

Eun-Hye Jang ◽

Sangwon Byun ◽

Mi-Sook Park ◽

Jin-Hun Sohn

Keyword(s):

Internal Consistency ◽

Interrater Reliability ◽

Physiological Responses ◽

Skin Conductance Level ◽

Intraclass Correlation ◽

Repeated Measurements ◽

Physiological Signals ◽

Basic Emotions ◽

Reliability Indices ◽

Film Clips

Abstract Background Although emotion-specific autonomic responses based on the discrete theory of emotion have been widely studied, studies on the reliability of physiological responses to emotional stimuli are limited. In this study, we aimed to assess the reliability of physiological changes induced by the six basic emotions (happiness, sadness, anger, fear, disgust, and surprise) that were measured during 10 weekly repeated experiments. Methods Twelve college students participated, and in each experiment, physiological signals were collected before and while participants were watching emotion-provoking film clips. Additionally, the participants self-evaluated the emotions that they experienced during the film presentation at the end of each emotional stimulus. To avoid adaptation of participants to identical stimuli during repeated measurements, we used 10 different film clips for each emotion, and thus a total of 60 film clips over 10 weeks were used. Physiological features, such as skin conductance level (SCL), fingertip temperature (FT), heart rate (HR), and blood volume pulse (BVP), were extracted from the physiological signals. Two reliability indices, Cronbach’s alpha and intraclass correlation coefficient, were calculated from the physiological features to assess internal consistency and interrater reliability, respectively. Results We found that SCL, HR, and BVP measured during the emotion-provoking phase over the 10 weekly sessions were more reliable than those assessed at baseline. Furthermore, SCL, HR, and BVP from the emotion-provoking phase exhibited excellent internal consistency and interrater reliability. Conclusions Our findings suggest that these features can be used as reliable physiological indices in emotion studies. The results also support the significance of physiological signals as meaningful indicators for emotion recognition in HCI (human computer interface) area.

Download Full-text

Validation of an Evidence-Based Medicine Critically Appraised Topic Presentation Evaluation Tool (EBM C-PET)

Journal of Graduate Medical Education ◽

10.4300/jgme-d-12-00049.1 ◽

2013 ◽

Vol 5 (2) ◽

pp. 252-256 ◽

Cited By ~ 2

Author(s):

Hans B. Kersten ◽

John G. Frohna ◽

Erin L. Giudice

Keyword(s):

Internal Consistency ◽

Evidence Based Medicine ◽

Interrater Reliability ◽

Intraclass Correlation ◽

Correlation Coefficients ◽

Evidence Based ◽

Evaluation Tool ◽

Intraclass Correlation Coefficients ◽

Excellent Internal Consistency ◽

Based Medicine

Abstract Background Competence in evidence-based medicine (EBM) is an important clinical skill. Pediatrics residents are expected to acquire competence in EBM during their education, yet few validated tools exist to assess residents' EBM skills. Objective We sought to develop a reliable tool to evaluate residents' EBM skills in the critical appraisal of a research article, the development of a written EBM critically appraised topic (CAT) synopsis, and a presentation of the findings to colleagues. Methods Instrument development used a modified Delphi technique. We defined the skills to be assessed while reviewing (1) a written CAT synopsis and (2) a resident's EBM presentation. We defined skill levels for each item using the Dreyfus and Dreyfus model of skill development and created behavioral anchors using a frame-of-reference training technique to describe performance for each skill level. We evaluated the assessment instrument's psychometric properties, including internal consistency and interrater reliability. Results The EBM Critically Appraised Topic Presentation Evaluation Tool (EBM C-PET) is composed of 14 items that assess residents' EBM and global presentation skills. Resident presentations (N = 27) and the corresponding written CAT synopses were evaluated using the EBM C-PET. The EBM C-PET had excellent internal consistency (Cronbach α = 0.94). Intraclass correlation coefficients were used to assess interrater reliability. Intraclass correlation coefficients for individual items ranged from 0.31 to 0.74; the average intraclass correlation coefficients for the 14 items was 0.67. Conclusions We identified essential components of an assessment tool for an EBM CAT synopsis and presentation with excellent internal consistency and a good level of interrater reliability across 3 different institutions. The EBM C-PET is a reliable tool to document resident competence in higher-level EBM skills.

Download Full-text

Evaluation of an Ophthalmology Session for Third-Year Medical Students at the University of Colorado School of Medicine

Journal of Academic Ophthalmology ◽

10.1055/s-0038-1631880 ◽

2018 ◽

Vol 10 (01) ◽

pp. e23-e31

Author(s):

Judas Kelley ◽

Jasleen Singh

Keyword(s):

Medical Students ◽

Test Score ◽

Clinical Skills ◽

Medical Knowledge ◽

Colorado School ◽

Self Reflection ◽

University Of Colorado ◽

School Of Medicine ◽

Post Test ◽

The University

Background Medical school ophthalmology education continues to be marginalized, creating the challenge of teaching students how to adequately diagnose and manage common ocular diseases in a limited time. Objective This study aimed to improve the ophthalmic medical knowledge and clinical skills of third-year medical students. Methods This curriculum was provided to 76 out of 124 third-year medical students in the 2016 class at the University of Colorado-Denver School of Medicine program. It was a half-day session that incorporated lectures, problem-based cases, and clinical skills. Participating students were given a self-reflection survey on their knowledge at the beginning and end of the session. Participating students were given a pre-test and post-test 6 months later via SurveyMonkey. Another quiz was given to all students within the class. Comparisons of pre- and post-tests, pre- and post-surveys, and quiz results in the participating and nonparticipating groups were made. Results Participating students' mean pre-test score was 61.8% (n = 76) and post-test score was 72.3% (n = 47), p < 0.01, indicating a significant increase in knowledge. There was an increase in 2 points on the Likert scale of understanding in ophthalmology between surveys. Participating students received a quiz mean score of 61.2%, which was significantly higher than those in the nonparticipating group (54.3%), p < 0.05. Conclusion This session improved medical students' confidence and medical knowledge in ophthalmology.

Download Full-text

Assessing Balance Function in Patients With Total Knee Arthroplasty

Physical Therapy ◽

10.2522/ptj.20140486 ◽

2015 ◽

Vol 95 (10) ◽

pp. 1397-1407 ◽

Cited By ~ 21

Author(s):

Andy C.M. Chan ◽

Marco Y.C. Pang

Keyword(s):

Total Knee Arthroplasty ◽

Internal Consistency ◽

Knee Arthroplasty ◽

Convergent Validity ◽

Interrater Reliability ◽

Intraclass Correlation ◽

Cronbach Alpha ◽

Ceiling Effects ◽

Total Knee ◽

Abc Scale

BackgroundThe Balance Evaluation Systems Test (BESTest) is a relatively new balance assessment tool. Recently, the Mini-BESTest and the Brief-BESTest, which are shortened versions of the BESTest, were developed.ObjectiveThe purpose of this study was to estimate interrater and intrarater-interoccasion reliability, internal consistency, concurrent and convergent validity, and floor and ceiling effects of the 3 BESTests and other related measures, namely, the Berg Balance Scale (BBS), Functional Gait Assessment (FGA), and Activities-specific Balance Confidence (ABC) Scale, among patients with total knee arthroplasty (TKA).DesignThis was an observational measurement study.MethodsTo establish interrater reliability, the 3 BESTests were administered by 3 independent raters to 25 participants with TKA. Intrarater-interoccasion reliability was evaluated in 46 participants with TKA (including the 25 individuals who participated in the interrater reliability experiments) by repeating the 3 BESTests, BBS, and FGA within 1 week by the same rater. Internal consistency of each test also was assessed with Cronbach alpha. Validity was assessed in another 46 patients with TKA by correlating the 3 BESTests with BBS, FGA, and ABC. The floor and ceiling effects also were examined.ResultsThe 3 BESTests demonstrated excellent interrater reliability (intraclass correlation coefficient [ICC] [2,1]=.96–.99), intrarater-interoccasion reliability (ICC [2,1]=.92–.96), and internal consistency (Cronbach alpha=.96–.98). These values were comparable to those for the BBS and FGA. The 3 BESTests also showed moderate-to-strong correlations with the BBS, FGA, and ABC (r=.35–.81), thus demonstrating good concurrent and convergent validity. No significant floor and ceiling effects were observed, except for the BBS.LimitationsThe results are generalizable only to patients with TKA due to end-stage knee osteoarthritis.ConclusionsThe 3 BESTests have good reliability and validity for evaluating balance in people with TKA. The Brief-BESTest is the least time-consuming and may be more useful clinically.

Download Full-text

Application of a Simulation-Based Interprofessional Teamwork Assessment Tool (SITAT) to Individual Student Performance in a Team-Based Simulation

Journal of Medical Education and Curricular Development ◽

10.1177/23821205211042436 ◽

2021 ◽

Vol 8 ◽

pp. 238212052110424

Author(s):

Brittany J Daulton ◽

Laura Romito ◽

Zach Weber ◽

Jennifer Burba ◽

Rami A Ahmed

Keyword(s):

Student Performance ◽

Internal Consistency ◽

Interprofessional Education ◽

Interrater Reliability ◽

Assessment Tool ◽

Individual Performance ◽

Individual Student ◽

Interprofessional Teamwork ◽

Significant Difference ◽

Simulation Based

There are a very limited number of instruments to assess individual performance in simulation-based interprofessional education (IPE). The purpose of this study was to apply the Simulation-Based Interprofessional Teamwork Assessment Tool (SITAT) to the individualized assessment of medicine, pharmacy, and nursing students (N = 94) in a team-based IPE simulation, as well as to explore potential differences between disciplines, and calculate reliability estimates for utilization of the tool. Results of an analysis of variance provided evidence that there was no statistically significant difference among professions on overall competency ( F(2, 91) = 0.756, P = .472). The competency reports for nursing ( M = 3.06, SD = 0.45), medicine ( M = 3.19, SD = 0.42), and pharmacy ( M = 3.08, SD = 0.49) students were comparable across professions. Cronbach's alpha provided a reliability estimate of the tool, with evidence of high internal consistency ( α = .92). The interrater reliability of the SITAT was also investigated. There was moderate absolute agreement across the 3 faculty raters using the 2-way mixed model design and “average” unit (kappa = 0.536, P = .000, 95% CI [0.34, 0.68]). The novel SITAT demonstrates internal consistency and interrater reliability when used for evaluation of individual performance during IPE simulation. The SITAT provides value in the education and evaluation of individual students engaged in IPE curriculum.

Download Full-text

An Innovative Clinical Skills “Boot Camp” for Dental Medicine Residents

Journal of Medical Education and Curricular Development ◽

10.1177/2382120517710018 ◽

2017 ◽

Vol 4 ◽

pp. 238212051771001 ◽

Cited By ~ 3

Author(s):

Jenny Castillo ◽

Jared Kutzin ◽

Kathleen Agoglia ◽

Patricia Janicke ◽

Zachary Milligan ◽

...

Keyword(s):

Emergency Medicine ◽

Clinical Skills ◽

Clinical Skill ◽

Medical Knowledge ◽

Boot Camp ◽

School Curriculum ◽

Dental School ◽

Knowledge And Skills ◽

Inpatient Hospital ◽

Simulation Based

During a 1-year hospital-based residency, dental residents are required to rotate through many departments including surgery, medicine, and emergency medicine. It became apparent that there was a gap between clinical skills knowledge taught in dental school curriculum and skills required for hospital-based patient care. In response, a simulation-based intensive clinical skill “boot camp” was created. The boot camp provided an intensive, interactive 3-day session for the dental residents. During the 3 days, residents were introduced to medical knowledge and skills that were necessary for their inpatient hospital rotations but were lacking in traditional dental school curriculum. Effectiveness of the boot camp was assessed in terms of knowledge base and comfort through presession and postsession surveys. According to resident feedback, this intensive introduction for the dental residents improved their readiness for their inpatient hospital-based residency.

Download Full-text

Medical Simulation as a Competency-Based Assessment within Physician Assistant Education

10.18122/td.1811.boisestate ◽

2021 ◽

Author(s):

Michele Toussaint

Keyword(s):

Patient Care ◽

Communication Skills ◽

Internal Consistency ◽

Clinical Training ◽

Rating Scales ◽

Rating Scale ◽

Physician Assistant ◽

Global Rating ◽

Simulation Based ◽

Competency Based

Simulation-based practices are widely utilized in medical education and are known to be a safe and effective way to train and assess learners, improve provider confidence and competency, and improve patient safety. Competency-based initiatives are being more broadly utilized to assess learner proficiency in health professions education. Recent publication of competencies expected of new graduate physician assistants, and updated accreditation requirements which include assessment of learner competencies in non-knowledge based domains, have led to the creation of this simulation-based summative assessment of learner competency in communication and patient care skills for Physician Assistant students. The purpose of this quantitative study was to identify if this simulation assessment had appropriate construct validity and rater consistency, and to identify if correlation existed between learner performance on the simulation exam and in required Supervised Clinical Training Experiences for measures of communication skills and patient care skills. While raters for the simulation assessment had minimal variability, measures of internal consistency did not achieve suitable thresholds for patient care skills. Communication skills assessment was able to achieve the minimum suitable threshold for internal consistency with minor revisions. No correlation was noted between exam performance for communication skills or patient care skills and clinical practice ratings. Several key areas exist which may explain these results including the rating scale for the simulation exam which utilized checklists and not global rating scales, faculty raters with broad and diverse clinical backgrounds, observation-related factors on the part of the student, and the high-complexity and multidimensional nature of provider-patient interactions.

Download Full-text

Extended Version of a Test Battery for Visual Assessment of Postural Orientation Errors: Face Validity, Internal Consistency, and Reliability

Physical Therapy ◽

10.1093/ptj/pzaa092 ◽

2020 ◽

Vol 100 (9) ◽

pp. 1542-1556 ◽

Cited By ~ 1

Author(s):

Jenny Nae ◽

Mark W Creaby ◽

Eva Ageberg

Keyword(s):

Internal Consistency ◽

Interrater Reliability ◽

Late Phase ◽

Intraclass Correlation ◽

Acl Injury ◽

Visual Assessment ◽

Test Battery ◽

Face Validity ◽

Extended Version ◽

Postural Orientation

Abstract Objective Undesirable postural orientation may be a risk factor for a second anterior cruciate ligament (ACL) injury. The purpose of this study was to evaluate face validity, internal consistency, and interrater reliability of an extended version of a previous test battery for visual assessment of postural orientation errors (POEs) in patients during the late phase of rehabilitation following ACL reconstruction (ACLR) (ie, when they have initiated jumping exercises). Methods This study used a cross-sectional design. Fifty-three patients (45% women) in the late phase of ACLR rehabilitation performed 5 functional tasks of varying difficulty. POEs of the lower extremity and trunk were visually assessed from video and scored on a scale from 0 (good) to 2 (poor). Results The side-hop and 2 new POEs (femur medial to shank, femoral valgus) were added to the test battery after expert focus group discussions. Internal consistency was calculated for all tasks (α = .712–.823). Interrater reliability showed fair to substantial agreement for femur medial to shank and femoral valgus during all tasks (K = 0.31–0.815) and almost perfect agreement for side-hop (intraclass correlation coefficient = 0.88). Conclusions The good internal consistency and reliability after adding side-hop, femur medial to shank, and femoral valgus suggests that this test battery is a suitable tool to quantify postural orientation throughout ACLR rehabilitation. Impact This test battery for visual assessment of POEs was evaluated in a heterogeneous group of patients in different phases of ACLR battery and can be used in clinical practice to measure POEs in patients with ACLR, including in the late phase of rehabilitation to return to sport. This study encourages research on more demanding tasks and additional POEs to cover the entire rehabilitation period after ACL injury or reconstruction.

Download Full-text

A Mobile App to Rapidly Appraise the In-Store Food Environment: Reliability, Utility, and Construct Validity Study (Preprint)

10.2196/preprints.16971 ◽

2019 ◽

Author(s):

Emma Joy McMahon ◽

Rachael Jaenke ◽

Julie Brimblecombe

Keyword(s):

Construct Validity ◽

Internal Consistency ◽

Food Environment ◽

Interrater Reliability ◽

Intraclass Correlation ◽

End Users ◽

Mobile App ◽

Convenience Stores ◽

Significant Difference ◽

Store Environment

BACKGROUND Consumer food environments are increasingly being recognized as influential determinants of food purchasing and subsequent intake and health. We developed a tool to enable efficient, but relatively comprehensive, appraisal of the in-store food environment. The Store Scout mobile app facilitates the evaluation of product (availability and range), placement (visibility, accessibility, proximity to high-traffic areas, and location relative to other products), price (price promotion), and promotion (displays and advertising) across 7 categories of food products, with appraisal given immediately as scores (0-100, where a higher score is more in line with best practice). Primary end users are public health nutritionists and nutritionists employed by store organizations; however, store managers and staff are also potential end users. OBJECTIVE This study aims to evaluate the reliability (interrater reliability and internal consistency), utility (distribution of scores), and construct validity (score by store type) of measurements using the Store Scout mobile app. METHODS The Store Scout mobile app was used independently by 2 surveyors to evaluate the store environment in 54 stores: 34 metropolitan stores (9 small and 11 large supermarkets, 10 convenience stores, and 4 petrol stations) in Brisbane, Australia, and 20 remote stores (19 small supermarkets and 1 petrol station) in Indigenous Australian communities in Northern Australia. The agreement between surveyors in the overall and category scores was evaluated using intraclass correlation coefficients (ICCs). Interrater reliability of measurement items was assessed using percentage agreement and the Gwet agreement coefficient (AC). Internal consistency was assessed by comparing the responses of items measuring similar aspects of the store environment. We examined the distribution of score values using boxplots and differences by store type using the Kruskal-Wallis test. RESULTS The median difference in the overall score between surveyors was 4.4 (range 0.0-11.1), with an ICC of 0.954 (95% CI 0.914-0.975). Most measurement items had very good (n=74/196, 37.8%) or good (n=81/196, 41.3%) interrater reliability using the Gwet AC. A minimal inconsistency of measurement was found. Overall scores ranged from 19.2 to 81.6. There was a significant difference in score by store type (<i>P</i><.001). Large Brisbane supermarkets scored highest (median 77.4, range 53.2-81.6), whereas small Brisbane supermarkets (median 63.9, range 41.0-71.3) and small remote supermarkets (median 63.8, range 56.5-74.9) scored significantly higher than Brisbane petrol stations (median 33.1, range 19.2-37.8) and convenience stores (median 39.0, range 22.4-63.8). CONCLUSIONS These findings suggest good reliability and internal consistency of food environment measurements using the Store Scout mobile app. We identified specific aspects that can be improved to further increase the reliability of this tool. We found a good distribution of score values and evidence that scoring could capture differences by store type in line with previous evidence, which gives an indication of construct validity. The Store Scout mobile app shows promise in its capability to measure and track the health-enabling characteristics of store environments.

Download Full-text