Stages of Milestones Implementation: A Template Analysis of 16 Programs Across 4 Specialties

ABSTRACT Background Since 2013, US residency programs have used the competency-based framework of the Milestones to report resident progress and to provide feedback to residents. The implementation of Milestones-based assessments, clinical competency committee (CCC) meetings, and processes for providing feedback varies among programs and warrants systematic examination across specialties. Objective We sought to determine how varying assessment, CCC, and feedback implementation strategies result in different outcomes in resource expenditure and stakeholder engagement, and to explore the contextual forces that moderate these outcomes. Methods From 2017 to 2018, interviews were conducted of program directors, CCC chairs, and residents in emergency medicine (EM), internal medicine (IM), pediatrics, and family medicine (FM), querying their experiences with Milestone processes in their respective programs. Interview transcripts were coded using template analysis, with the initial template derived from previous research. The research team conducted iterative consensus meetings to ensure that the evolving template accurately represented phenomena described by interviewees. Results Forty-four individuals were interviewed across 16 programs (5 EM, 4 IM, 5 pediatrics, 3 FM). We identified 3 stages of Milestone-process implementation, including a resource-intensive early stage, an increasingly efficient transition stage, and a final stage for fine-tuning. Conclusions Residency program leaders can use these findings to place their programs along an implementation continuum and gain an understanding of the strategies that have enabled their peers to progress to improved efficiency and increased resident and faculty engagement.

Better Decision-Making: Shared Mental Models and the Clinical Competency Committee

ABSTRACT Background Shared mental models (SMMs) help groups make better decisions. Clinical competency committees (CCCs) can benefit from the development and use of SMMs in their decision-making as a way to optimize the quality and consistency of their decisions. Objective We reviewed the use of SMMs for decision making in graduate medical education, particularly their use in CCCs. Methods In May 2020, the authors conducted a narrative review of the literature related to SMMs. This review included the SMM related to teams, team functioning, CCCs, and graduate medical education. Results The literature identified the general use of SMMs, SMMs in graduate medical education, and strategies for building SMMs into the work of the CCC. Through the use of clear communication and guidelines, and a shared understanding of goals and expectations, CCCs can make better decisions. SMMs can be applied to Milestones, resident performance, assessment, and feedback. Conclusions To ensure fair and robust decision-making, the CCC must develop and maintain SMMs through excellent communication and understanding of expectations among members.

Discriminatory ability of milestones: An analysis of milestone variability by obstetrics and gynecology subspecialty

Background: Little evidence exists regarding Accreditation Council for Graduate Medical Education (ACGME) milestone discriminatory ability. This short report describes variability in milestone scores by category to determine their utility in discerning high and low performers in a single Obstetrics and Gynecology residency. Methods: A Clinical Competency Committee (CCC) design was implemented with four subcommittees, each responsible for a predetermined milestones subset: Obstetrics, Gynecology, Ambulatory Practice, and Professional Activities. Milestone scores for 44 residents per year over four biannual evaluation cycles (2014-2016) were evaluated, for a total of 176 independent evaluations. Results: Findings indicate that discriminatory ability, assessed by variability between resident scores, differed by subcommittee. Subcommittees that were primarily tasked with evaluating clinical- and procedural-based milestones demonstrated lower discriminatory ability among trainees. Conclusion: Greater Professional Activity milestone variability indicates better differentiation; future research should determine correlation of these findings with other professionalism performance metrics and novel intervention strategies.

Feasibility of an Entrustable Professional Activity for Pathology Resident Frozen Section Training

Entrustable professional activities are an intuitive form of workplace-based assessment that can support competency-based medical education. Many entrustable professional activities have been written and published, but few studies describe the feasibility or implementation of entrustable professional activities in graduate medical education. The frozen section entrustable professional activit was introduced into the pathology residency training at the University of Vermont for postgraduate year 1 at the start of their training in frozen section. The feasibility of the entrustable professional activit was evaluated based on 3 criteria: (a) utilization, (b) support of frozen section training, and (c) generating data to support entrustment decision about residents’ readiness to take call. The entrustable professional activit was well utilized and satisfactory to residents, faculty, pathologists’ assistants, and Clinical Competency Committee members. Most members of the Clinical Competency Committee agreed they had sufficient data and noted higher confidence in assessing resident readiness to take call with the addition of entrustable professional activit to the residents’ assessment portfolio. Residents did not endorse it helped them prepare for call; however, the interruption to frozen section training due to the COVID-19 pandemic was a significant contributing factor. The frozen section entrustable professional activit is a feasible addition to pathology resident training based on utilization, support of training, and generation of data to support entrustment decisions for graduated responsibilities. The implementation and integration of the entrustable professional activit into pathology training at our institution is described with discussion of adjustments for future use.

Faculty Versus Resident Self-Assessment Using Pathology Milestones: How Aligned Are We?

Competent physicians must be able to self-assess skill level; however, previous studies suggest that medical trainees may not accurately self-assess. We utilized Pathology Milestones (PM) data to determine whether there were discrepancies in self- versus Clinical Competency Committee (CCC) ratings by sex, program year (PGY), time of evaluation, and question category (Patient Care, Medical Knowledge, Systems-Based Practice [SBP], Practice-Based Learning and Improvement [PBL], Professionalism [PRO], and Interpersonal and Communication Skills) and Residency In-Service Examination (RISE) score. We completed retrospective analyses of PM evaluation scores from 2016 to 2019 (n = 23 residents) 2 times per year. Discrepancies in evaluation scores were calculated by subtracting CCC scores from resident self-evaluation scores. There was no significant difference in discrepancy scores between male versus female residents (P = .94). Discrepancy scores among all PGYs were significantly different (P < .0001), with PGY1 tending to overrate the most, followed by PGY2. PGY3 and PGY4 underrated themselves on average compared to CCC ratings, with PGY4 having significantly lower self-ratings than CCC compared to any other PGY. In January, residents underscored themselves and in July residents overscored themselves compared to CCC (P < .0001 for both). Question types resulted in variable discrepancy scores, with SBP significantly lower than and PRO significantly higher than all other categories (P < .05 for both). Increases in RISE score correlated to increases in self- and CCC-scoring. These discrepancies can help trainees improve self-assessment. Discrepancies indicate potential areas for amelioration, such as curriculum adjustments or Milestone’s verbiage.

Professionalism Milestones Assessments Used by Emergency Medicine Residency Programs: A Cross-sectional Survey

Introduction: Professionalism is a vital component of quality patient care. While competency in professionalism is Accreditation Council for Graduate Medical Education (ACGME)-mandated, the methods used to evaluate professionalism are not standardized, calling into question the validity of reported measurements. We aimed to determine the type and frequency of methods used by United States (US) -based emergency medicine (EM) residencies to assess accountability (Acc) and professional values (PV), as well as how often graduating residents achieve competency in these areas. Methods: We created a cross-sectional survey exploring assessment and perceived competency in Acc and PV, and then modified the survey for content and clarity through feedback from emergency physicians not involved in the study. The final survey was sent to the clinical competency committee (CCC) chair or program director (PD) of the 185 US-based ACGME-accredited EM residencies. We summarized results using descriptive statistics and Fisher’s exact testing. Results: A total of 121 programs (65.4%) completed the survey. The most frequently used methods of assessment were faculty shift evaluation (89.7%), CCC opinion (86.8%), and faculty summative evaluation (76.4%). Overall, 37% and 42% of residency programs stated that nearly all (greater than 95%) of their graduating residents achieve mastery of Acc and PV non-technical skills, respectively. Only 11.2% of respondents felt their programs were very effective at determining mastery of non-technical skills. Conclusion: EM residency programs relied heavily on faculty shift evaluations and summative opinions to determine resident competency in professionalism, with feedback from peers, administrators, and other staff less frequently incorporated. Few residency programs felt their current methods of evaluating professionalism were very effective.