Teaching and assessing intra-operative consultations in competency-based medical education: development of a workplace-based assessment instrument

AbstractCompetency-based medical education (CBME) is being implemented worldwide. In CMBE, residency training is designed around competencies required for unsupervised practice and use entrustable professional activities (EPAs) as workplace “units of assessment”. Well-designed workplace-based assessment (WBA) tools are required to document competence of trainees in authentic clinical environments. In this study, we developed a WBA instrument to assess residents’ performance of intra-operative pathology consultations and conducted a validity investigation. The entrustment-aligned pathology assessment instrument for intra-operative consultations (EPA-IC) was developed through a national iterative consultation and used clinical supervisors to assess residents’ performance at an anatomical pathology program. Psychometric analyses and focus groups were conducted to explore the sources of evidence using modern validity theory: content, response process, internal structure, relations to other variables, and consequences of assessment. The content was considered appropriate, the assessment was feasible and acceptable by residents and supervisors, and it had a positive educational impact by improving performance of intra-operative consultations and feedback to learners. The results had low reliability, which seemed to be related to assessment biases, and supervisors were reluctant to fully entrust trainees due to cultural issues. With CBME implementation, new workplace-based assessment tools are needed in pathology. In this study, we showcased the development of the first instrument for assessing resident’s performance of a prototypical entrustable professional activity in pathology using modern education principles and validity theory.

Download Full-text

Building a Shared Mental Model of Competence Across the Continuum: Trainee Perceptions of Subinternships for Residency Preparation

Journal of Medical Education and Curricular Development ◽

10.1177/23821205211063350 ◽

2021 ◽

Vol 8 ◽

pp. 238212052110633

Author(s):

Johannah M. Scheurer ◽

Cynthia Davey ◽

Anne G. Pereira ◽

Andrew P. J. Olson

Keyword(s):

Medical Education ◽

Response Rate ◽

Professional Activity ◽

University Of Minnesota ◽

Competency Based Medical Education ◽

Chi Squared ◽

Competency Based ◽

Kappa Agreement ◽

Physician Competency ◽

The Continuum

INTRODUCTION Toward a vision of competency-based medical education (CBME) spanning the undergraduate to graduate medical education (GME) continuum, University of Minnesota Medical School (UMMS) developed the Subinternship in Critical Care (SICC) offered across specialties and sites. Explicit course objectives and assessments focus on internship preparedness, emphasizing direct observation of handovers (Core Entrustable Professional Activity, “EPA,” 8) and cross-cover duties (EPA 10). METHODS To evaluate students’ perceptions of the SICC's and other clerkships’ effectiveness toward internship preparedness, all 2016 and 2017 UMMS graduates in GME training ( n = 440) were surveyed regarding skill development and assessment among Core EPAs 1, 4, 6, 8, 9, 10. Analysis included descriptive statistics plus chi-squared and Kappa agreement tests. RESULTS Respondents ( n = 147, response rate 33%) rated the SICC as a rotation during which they gained most competence among EPAs both more (#4, 57% rated important; #8, 75%; #10, 70%) and less explicit (#6, 53%; #9, 69%) per rotation objectives. Assessments of EPA 8 (80% rated important) and 10 (76%) were frequently perceived as important toward residency preparedness. Agreement between importance of EPA development and assessment was moderate (Kappa = 0.40-0.59, all surveyed EPAs). CONCLUSIONS Graduates’ perceptions support the SICC's educational utility and assessments. Based on this and other insight from the SICC, the authors propose implications toward collectively envisioning the continuum of physician competency.

Download Full-text

Adapting Feedback to Individual Residents: An Examination of Preceptor Challenges and Approaches

Journal of Graduate Medical Education ◽

10.4300/jgme-d-17-00590.1 ◽

2018 ◽

Vol 10 (2) ◽

pp. 168-175 ◽

Cited By ~ 1

Author(s):

Amanda Roze des Ordons ◽

Adam Cheng ◽

Jonathan Gaudet ◽

James Downar ◽

Jocelyn Lockyer

Keyword(s):

Medical Education ◽

Professional Activity ◽

Framework Analysis ◽

Competency Based Medical Education ◽

Performance Levels ◽

Skill Sets ◽

Entrustable Professional Activity ◽

Competency Based ◽

Individualize Feedback ◽

Insight Into

ABSTRACT Background Feedback conversations between preceptors and residents usually occur in closed settings. Little is known about how preceptors address the challenges posed by residents with different skill sets, performance levels, and personal contexts. Objective This study explored the challenges that preceptors experienced and approaches taken in adapting feedback conversations to individual residents. Methods In 2015, 18 preceptors participated in feedback simulations portraying residents with variations in skill, insight, confidence, and distress, followed by debriefing of the feedback conversation with a facilitator. These interactions were recorded, transcribed, and analyzed using thematic and framework analysis. Results The preceptors encountered common challenges with feedback conversations, including uncertainty in how to individualize feedback to residents and how to navigate tensions between resident- and preceptor-identified goals. Preceptors questioned their ability to enhance skills for highly performing residents, whether they could be directive when residents had insight gaps, how they could reframe the perceptions of the overly confident resident, and whether they should offer support to emotionally distressed residents or provide feedback about performance. Preceptors adapted their approach to feedback, drawing on techniques of coaching for highly performing residents, directing for residents with insight gaps, mediation with overly confident residents, and mentoring with emotionally distressed residents. Conclusions Examining the feedback challenges preceptors encounter and the approaches taken to adapt feedback to individual residents can provide insight into how preceptors meet the challenges of competency-based medical education, in which frequent, focused feedback is essential for residents to achieve educational milestones and entrustable professional activity expectations.

Download Full-text

Sustainability of physical exam skills in a resident-led curriculum in a large internal medicine program with competency based medical education

Canadian Medical Education Journal ◽

10.36834/cmej.43273 ◽

2018 ◽

Vol 9 (4) ◽

pp. e78-92 ◽

Cited By ~ 2

Author(s):

Don Thiwanka Wijeratne ◽

Siddhartha Srivastava ◽

Barry Chan ◽

Wilma Hopman ◽

Benjamin Thomson

Keyword(s):

Internal Medicine ◽

Medical Education ◽

Physical Examination ◽

Peer Assessment ◽

Physical Exam ◽

Professional Activity ◽

Faculty Assessment ◽

Competency Based Medical Education ◽

Competency Based ◽

Faculty Assessments

Background: Competency Based Medical Education (CBME) designates physical examination competency as an Entrustable Professional Activity (EPA). Considerable concern persists regarding the increased time burden CBME may place on educators. We developed a novel physical examination curriculum that shifted the burden of physical examination case preparation and performance assessment from faculty to residents. Our first objective was to determine if participation led to sustainable improvements in physical examination skills. The second objective was to determine if resident peer assessment was comparable to faculty assessment. Methods: We selected physical exam case topics based on the Objectives of Training in the Specialty of Internal Medicine as prescribed by the Royal College of Physicians and Surgeons of Canada. Internal Medicine residents compiled evidence-based physical exam checklists that faculty reviewed before distribution to all learners. Physical exam practice sessions with whole-group demonstration followed by small-group practice sessions were performed weekly. We evaluated this pilot curriculum with a formative OSCE, during which a resident peer and a faculty member simultaneously observed and assessed examinee performance by .Results: Participation in the novel curriculum practice sessions improved OSCE performance (faculty score mean 78.96 vs. 62.50, p<0.05). Peer assessment overestimated faculty scores (76.2 vs. 65.7, p<0.001), but peer and faculty assessments were highly correlated (R2 = 0.73 (95% CI 0.50-0.87).Conclusion: This novel physical examination curriculum leads to sustainable improvement of physical examination skills. Peer assessment correlated well with the gold standard faculty assessment. This resident-led physical examination curriculum enhanced physical examination skills in a CBME environment, with minimal time commitment from faculty members.

Download Full-text

Assessment of emergency medicine residents: a systematic review

Canadian Medical Education Journal ◽

10.36834/cmej.36770 ◽

2017 ◽

Vol 8 (1) ◽

pp. e106-122 ◽

Cited By ~ 4

Author(s):

Isabelle N Colmers-Gray ◽

Kieran Walsh ◽

Teresa M Chan

Keyword(s):

Emergency Medicine ◽

Medical Education ◽

Direct Observation ◽

Full Text ◽

Assessment Tools ◽

Competency Based Medical Education ◽

Mesh Terms ◽

Competency Based ◽

Resident Assessment ◽

Assessment Frequency

Background: Competency-based medical education is becoming the new standard for residency programs, including Emergency Medicine (EM). To inform programmatic restructuring, guide resources and identify gaps in publication, we reviewed the published literature on types and frequency of resident assessment.Methods: We searched MEDLINE, EMBASE, PsycInfo and ERIC from Jan 2005 - June 2014. MeSH terms included “assessment,” “residency,” and “emergency medicine.” We included studies on EM residents reporting either of two primary outcomes: 1) assessment type and 2) assessment frequency per resident. Two reviewers screened abstracts, reviewed full text studies, and abstracted data. Reporting of assessment-related costs was a secondary outcome.Results: The search returned 879 articles; 137 articles were full-text reviewed; 73 met inclusion criteria. Half of the studies (54.8%) were pilot projects and one-quarter (26.0%) described fully implemented assessment tools/programs. Assessment tools (n=111) comprised 12 categories, most commonly: simulation-based assessments (28.8%), written exams (28.8%), and direct observation (26.0%). Median assessment frequency (n=39 studies) was twice per month/rotation (range: daily to once in residency). No studies thoroughly reported costs.Conclusion: EM resident assessment commonly uses simulation or direct observation, done once-per-rotation. Implemented assessment systems and assessment-associated costs are poorly reported. Moving forward, routine publication will facilitate transitioning to competency-based medical education.

Download Full-text

An Entrustable Professional Activity Addressing Racism and Pediatric Health Inequities

PEDIATRICS ◽

10.1542/peds.2021-054604 ◽

2022 ◽

Author(s):

Ndidi I. Unaka ◽

Ariel Winn ◽

Adiaha Spinks-Franklin ◽

Patricia Poitevien ◽

Franklin Trimm ◽

...

Keyword(s):

Medical Education ◽

Health Care Systems ◽

Health Inequities ◽

Professional Activity ◽

Competency Based Medical Education ◽

Pediatric Health ◽

Professional Activities ◽

Care Systems ◽

Competency Based ◽

Pediatric Populations

Racism and discrimination are the root of many pediatric health inequities and are well described in the literature. Despite the pervasiveness of pediatric health inequities, we have failed to adequately educate and prepare general pediatricians and pediatric subspecialists to address them. Deficiencies within education across the entire continuum and in our health care systems as a whole contribute to health inequities in unacceptable ways. To address these deficiencies, the field of pediatrics, along with other specialties, has been on a journey toward a more competency-based approach to education and assessment, and the framework created for the future is built on entrustable professional activities (EPAs). Competency-based medical education is one approach to addressing the deficiencies within graduate medical education and across the continuum by allowing educators to focus on the desired equitable patient outcomes and then develop an approach to teaching and assessing the tasks, knowledge, skills, and attitudes needed to achieve the goal of optimal, equitable patient care. To that end, we describe the development and content of a revised EPA entitled: Use of Population Health Strategies and Quality Improvement Methods to Promote Health and Address Racism, Discrimination, and Other Contributors to Inequities Among Pediatric Populations. We also highlight the ways in which this EPA can be used to inform curricula, assessments, professional development, organizational systems, and culture change.

Download Full-text

Resident evaluations in the age of competency-based medical education: faculty perspectives on minimizing burdens

Journal of Neurosurgery ◽

10.3171/2020.7.jns201688 ◽

2020 ◽

pp. 1-6

Author(s):

Jessica E. Rabski ◽

Ashirbani Saha ◽

Michael D. Cusimano

Keyword(s):

Medical Education ◽

Mobile Application ◽

Postgraduate Medical Education ◽

Professional Activity ◽

Successful Implementation ◽

Neurosurgical Training ◽

Complete Evaluation ◽

Competency Based Medical Education ◽

Main Challenge ◽

Competency Based

OBJECTIVECompetency-based medical education (CBME), an outcomes-based approach to medical education, continues to be implemented across many postgraduate medical education programs worldwide, including a recent introduction into Canadian neurosurgical training programs (July 2019). The success of this educational paradigm shift requires frequent faculty observation and evaluation of residents performing defined tasks of the specialty. A main challenge involves providing residents with frequent performance evaluations and feedback that are feasible for faculty to complete. This study aims to define what is currently happening and what changes are needed to make CBME successful for the certification of neurosurgeons’ competence.METHODSA 55-item questionnaire was emailed nationwide to survey Canadian neurosurgical faculty.RESULTSFifty-two complete responses were received and achieved a distribution highly correlated with the number of faculty neurosurgeons practicing in each Canadian province (Pearson’s r = 0.94). Two-thirds (35/52) of faculty reported currently taking a median of 10 minutes to complete evaluation forms at the end of a resident’s rotation block. Regardless of the faculty’s province of practice (p = 0.50) or years of experience (p = 0.06), they reported 3 minutes (minimum 1 minute, maximum 10 minutes, interquartile range [IQR] 3 minutes) as a feasible amount of time to spend completing an evaluation form following an observation of a resident’s performance of an entrustable professional activity (EPA). If evaluation forms took 3 minutes to complete, 85% of respondents (44/52) would complete EPA evaluations weekly or daily. The faculty recommended 5 minutes as a feasible amount of time to provide oral feedback (minimum 1 minute, maximum 20 minutes, IQR 3.25 minutes), which was significantly higher (p = 0.00099) than their recommended amount of time for completing evaluation forms. The majority of faculty (71%) stated they would prefer to access resident evaluation forms through a mobile application compared to a paper form (12%), an evaluation website (8%), or through a URL link sent via email (10%; p = 0.0032).CONCLUSIONSTo facilitate the successful implementation of CBME into a neurosurgical training curriculum, resident EPA assessment forms should take 3 minutes or less to complete and be accessible through a mobile application.

Download Full-text

WEB-BASED 5-DIMENSIONAL ELECTRONIC PORTFOLIO (5DEP) AS A COMPETENCY-BASED ASSESSMENT TOOL IN POST-GRADUATE MEDICAL TRAINING

International Journal of Medicine and Medical Research ◽

10.11603/ijmmr.2413-6077.2020.1.11504 ◽

2020 ◽

Vol 6 (1) ◽

pp. 81-87

Author(s):

Z. Bokhua ◽

K. Chelidze ◽

K. Ebralidze

Keyword(s):

Internal Medicine ◽

Professional Development ◽

Medical Education ◽

Assessment Tool ◽

Electronic Portfolio ◽

Assessment Tools ◽

Assessment Instrument ◽

First Year ◽

Web Based ◽

Competency Based

Background. New challenges of permanently changing context of healthcare system requires new methods of medical education and new assessment tools, as well. Competency-based Medical Education (CBME), framework which has been adopted as a new approach in medical education, needs appropriate assessment tool such as portfolio. Portfolio is learner-centered assessment instrument which evaluates learner’s progression towards outcomes and enables both residents and teachers to engage in a process of learning through assessment. Objective. In this paper we aim to share our successful experience of an effective use of web-based 5-Dimensional Electronic Portfolio (5DeP) as an assessment tool in the Pilot Group. Methods. Pilot Group of sixteen residents (six first year residents of Obstetrics/Gynecology and ten first year residents of Internal Medicine. Tbilisi State Medical University Institute of Postgraduate Medical Education and Continuous Professional Development) and twelve mentors (four Obstetrics/Gynecology mentor and eight Internal Medicine mentors) reported some feedback about 5-Dimensional Electronic Portfolio (5DeP) as a new assessment tool. Results. Feedback about 5-Dimensional Electronic Portfolio (5DeP) as a new assessment tool from mentors and students demonstrated efficiency of the program. It enables assessment within a framework of transparent and declared criteria and learning objectives; provides a model for lifelong learning and continuing professional development; increases competence in a wider context with benefits to both professional and personal roles; improves organizing skills. Conclusions. 5DeP have been recognized as an extremely effective assessment tool.

Download Full-text

Exploring resident perceptions of initial competency based medical education implementation

Canadian Medical Education Journal ◽

10.36834/cmej.70943 ◽

2021 ◽

Author(s):

Shivani Upadhyaya ◽

Marghalara Rashid ◽

Andrea Davila Cervantes ◽

Anna Oswald

Keyword(s):

Medical Education ◽

Professional Activity ◽

Competency Based Medical Education ◽

Programmatic Assessment ◽

Resident Perceptions ◽

Societal Needs ◽

Core Components ◽

Competency Based ◽

Curricular Model ◽

Outcome Competencies

Background: Competence by design (CBD) is a nationally developed hybrid competency based medical education (CBME) curricular model that focuses on residents’ abilities to promote successful practice and better meet societal needs. CBD is based on a commonly used framework of five core components of CBME: outcome competencies, sequenced progression, tailored learning experiences, competency-focused instruction and programmatic assessment. There is limited literature concerning residents’ perceptions of implementation of CBME. Objective: We explored resident perceptions of this transformation and their views as they relate to the intended framework. Methods: We recruited residents enrolled in current CBME implementation between August 2018 and January 2019. We interviewed residents representing eight disciplines from the initial two CBME implementation cohorts. Inductive thematic analysis was used to analyse the data through iterative consensus building until saturation. Results: We identified five themes: 1) Value of feedback for residents; 2) Resident strategies for successful Entrustable Professional Activity observation completion; 3) Residents experience challenges; 4) Resident concerns regarding CBME; and 5) Resident recommendations to improve existing challenges. We found that while there was clear alignment with residents’ perceptions of the programmatic assessment core CBME component, alignment was not as clear for other components. Conclusions: Residents perceived aspects of this transformation as helpful but overall had mixed perceptions and variable understanding of the intended underlying framework. Understanding and disseminating successes and challenges from the resident lens may assist programs at different stages of CBME implementation.

Download Full-text