Multi-source feedback following simulated resuscitation scenarios: a qualitative study

Background: The direct observation and assessment of learners’ resuscitation skills by an attending physician is challenging due to the unpredictable and time-sensitive nature of these events. Multisource feedback (MSF) may address this challenge and improve the quality of assessments provided to learners. We aimed to describe the similarities and differences in the assessment rationale of attending physicians, registered nurses, and resident peers in the context of a simulation-based resuscitation curriculum. Methods: We conducted a qualitative content analysis of narrative MSF of medical residents in their first postgraduate year of training who were participating in a simulation-based resuscitation course at two Canadian institutions. Assessments included an entrustment score and narrative comments from attending physicians, registered nurses, and resident peers in addition to self-assessment. Narrative comments were transcribed and analyzed thematically using a constant comparative method. Results: All 87 residents (100%) participating in the 2017-2018 course provided consent. A total of 223 assessments were included in our analysis. Four themes emerged from the narrative data: 1) Communication, 2) Leadership, 3) Demeanor, and 4) Medical Expert. Relative to other assessor groups, feedback from nurses focused on patient-centred care and communication while attending physicians focused on the medical expert theme. Peer feedback was the most positive. Self-assessments included comments within each of the four themes. Conclusions: In the context of a simulation-based resuscitation curriculum, MSF provided learners with different perspectives in their narrative assessment rationale and may offer a more holistic assessment of resuscitation skills within a competency-based medical education (CBME) program of assessment.

Relating Cultural Distance to Self-Other Agreement of Leader–Observer Dyads: The Role of Hierarchical Position

Multisource feedback is important for leadership development and effectiveness. An important asset of such feedback is that it provides information about the self-other agreement between leaders and observers. Self-other agreement relates to several positive individual, dyadic, and organizational outcomes. Given the increasingly intercultural context in organizations, it is imperative to understand whether and how cultural distance between leaders and observers relates to self-other agreement. We hypothesize that cultural distance within leader-observer dyads is negatively associated with self-other agreement. Moreover, we expect that this relationship is stronger for leader-superior than leader-subordinate dyads. We use a unique multi-cultural dataset of 7,778 leaders (52 nationalities) rated by 22,997 subordinates (56 nationalities) and 10,132 superiors (54 nationalities) to test our hypotheses. Results confirm that cultural distance is negatively associated with self-other agreement; we show that this relationship is driven by increased self-ratings and by reduced other-ratings. In addition, we find that these results are more pronounced for leader–superior than for leader–subordinate dyads. Implications for the theory and practice of self-other agreement and multisource feedback are discussed.

The Impact of implementing multisource feedback on behaviors of young doctors

Objectives: Multisource feedback (MSF) is a workplace-based assessment tool that offers 360-degree evaluation of the trainee doctor. Little is known about its receptiveness among stakeholders in Pakistan. This study explores house officers’ perceptions regarding MSF since its implementation in Eye Unit-II, Institute of Ophthalmology, King Edward Medical University/ Mayo Hospital, Lahore. Methods: A qualitative case study was conducted from July 2019 to February 2020 in Eye Unit II. A purposive (maximum variation) sample of 12 house surgeons was taken. Two focus group discussions were conducted. Data were transcribed and analyzed thematically. Results: The study identified the impact of MSF on house surgeons. Most participants reported positive experiences. The feedback they received increased their motivation, management skills and team working. A number of factors affecting the receptiveness of MSF were also identified which mainly included characteristics of raters and emotional response to MSF. Conclusion: Multisource Feedback is a useful tool for feedback that impacts the young doctors in many ways. It contributes to increasing their sense of responsibility, management skills and self-directed learning. The improvement in individual abilities and teamwork also helped in improving patient care. doi: https://doi.org/10.12669/pjms.37.7.4155 How to cite this:Ahmad CN, Sethi A, Khan RA. Impact of implementing multisource feedback on behaviors of young doctors. Pak J Med Sci. 2021;37(7):---------. doi: https://doi.org/10.12669/pjms.37.7.4155 This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

A Training Intervention through a 360° Multisource Feedback Model

Physicians and other health sciences professionals need continuous training, not only in technical aspects of their activity but also in nontechnical, transversal competencies with a cost-efficient impact on the proper functioning of healthcare. The objective of this paper is to analyze the behavioral change among health professionals at a large public hospital following a training intervention on a set of core nontechnical competencies: Teamwork, Adaptability-Flexibility, Commitment-Engagement, Results Orientation, and Leadership Skills for Supervisors. The 360° Multisource Feedback (MSF) model was applied using three sources of information: supervisors, co-workers, and the workers themselves (self-assessment). A quasi-experimental pretest–post-test single-group design with two points in time was utilized. The training intervention improved the scores of only one of the trained competencies—the “Results Orientation” competency—although the scores were slightly inflated. Moreover, significant discrepancies were detected between the three sources, with supervisors awarding the highest scores. The magnitude of behavioral change was related to certain sociodemographic and organizational variables. The study was not immune to the ceiling effect, despite control measures aimed at avoiding it. The empirical evidence suggests that the 360° MSF model must be maintained over time to enhance and reinforce an evaluation culture for better patient care.

Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study

ObjectivesTo examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.DesignQualitative text analysis supplemented by quantitative descriptive analysis.SettingInternal Medicine Departments in Zealand, Denmark.Participants48 postgraduate trainees in internal medicine specialties, 1 clinical supervisor for each trainee and 376 feedback givers (respondents).InterventionThis study examines the use of an electronic, purely narrative MSF instrument. After the MSF process, the trainee and the supervisor answered a postquestionnaire concerning their perception of the process. The authors coded the comments in the MSF reports for valence (positive or negative), specificity, relation to behaviour and whether the comment suggested a strategy for improvement. Four of the authors independently classified the MSF reports as either ‘no reasons for concern’ or ‘possibly some concern’, thereby examining discriminating capacity. Through iterative readings, the authors furthermore tried to identify how many respondents were needed in order to get a reliable impression of a trainee.ResultsOut of all comments coded for valence (n=1935), 89% were positive and 11% negative. Out of all coded comments (n=4684), 3.8% were suggesting ways to improve. 92% of trainees and supervisors preferred a narrative MSF to a numerical MSF, and 82% of the trainees discovered performance in need of development, but only 53% had made a specific plan for development. Kappa coefficients for inter-rater correlations between four authors were 0.7–1. There was a significant association (p<0.001) between the number of negative comments and the qualitative judgement by the four authors. It was not possible to define a specific number of respondents needed.ConclusionsA purely narrative MSF contributes with educational value and experienced supervisors can discriminate between trainees’ performances based on the MSF reports.

EULAR portfolio for Rheumatology training: a EULAR School of Rheumatology initiative

ObjectiveAbout half of the rheumatology trainees do not use a portfolio. This project was established to reach consensus about the content of a EULAR portfolio for Rheumatology training and subsequently develop portfolio assessment forms.MethodsAfter establishing a portfolio working group (WG), including nine rheumatologists and one educationalist, a systematic literature review (SLR) on the content and structure of portfolios for postgraduate learning was conducted (November 2018). This was followed by a survey among WG members and members of the EMerging EUlar NETwork, inquiring about the content and structure of existing national portfolios. The portfolio WG selected the key components of the portfolio, taking previous experience and feasibility into account. Assessment forms (eg, case-based discussion) were developed and pilot-tested.Results13/2034 articles were included in the SLR (12 high/1 moderate risk of bias). Information on procedural skills, personal reflections, learning goals and multisource feedback was most often included a portfolio. Twenty-five respondents completed the survey (response≈50%). Feedback from assessors, reflective writing and formulation of learning goals were considered important dimensions to be covered in a portfolio. Six key components of the portfolio were established: curriculum vitae, personal development plan, clinical work, professional behaviours, education and research activities. Suggested minimal content for each component was formulated. Four assessment forms were successfully pilot-tested by 11 rheumatologists and their trainees.ConclusionA EULAR portfolio for Rheumatology training and assessment forms were developed. Portfolio implementation, particularly in countries without an existing portfolio, may promote a higher standard of rheumatology training across Europe.

Multisource feedback for the physician associate role

Introduction Physician associates (PAs) are dependent medical practitioners who work alongside doctors and other healthcare professionals. There are approximately 350 PAs working throughout the UK, with another 550 in training. Methods An online questionnaire was sent to healthcare professionals throughout the North West region with a total of 119 responding. Participants were asked nine questions on clinical skills and three on communication skills. These were to be rated as ‘excellent’, ‘good’, ‘satisfactory’, ‘needs to improve’ or ‘unacceptable’. In addition, respondents were asked to indicate their agreement with 14 statements about the role of PAs. There was also a field for free text comments. Results Of the 1,071 individual responses to the survey questions on clinical skills, 859 (80%) were ‘excellent’ or ‘good’. Similarly, among the 357 answers to the questions on communication skills, 317 (89%) were ‘excellent’ or ‘good’. Comments in the free text field included both positive and negative observations, with all negative comments coming from junior doctors. Conclusions Overall feedback for the PA role was overwhelmingly positive. However, the most commonly mentioned problem was that they are not being utilised effectively and in turn, that they are taking learning opportunities from junior doctors. This is an important concern to note and warrants further investigation. PAs are clearly useful assets and are set to increase in number in the NHS over the coming years. It is therefore imperative that this new role works well with the existing roles and training structure of junior doctors.

Entrustable Professional Activities for Undergraduate Nutrition Student Clinical Training: Development and Implementation

Objectives Entrustable professional activities (EPAs) have been applied to postgraduate and undergraduate medical training, but the application of nutrition intern students is rare. The clinical training of nutrition intern students must be connected with the nutrition care ability after graduation. Outpatient diabetes nutrition care is an important job of hospital dietitian. Therefore, this is the good point for the development of EPAs evaluation. The objective of this study was to describe the development of EPAs for nutrition intern training in the nutrition department of a medical center. Methods We chose the topic “Diet and health education for diabetic patients in outpatient clinics” and set the EPAs framework. EPAs was developed through an iterative consensus process involving the advanced dietitians. The content includes task description, capabilities (knowledge, skills, attitudes and behaviors), evaluation information, setting the trust level and validity period. The evaluation was established through expert validity. Final EPAs revisions followed from the multisource feedback. The Clinical Competency Committees (CCC) was established for final evaluation and approval. Results The EPAs was piloted for intern students in the Nutrition Department of the Medical Center from 2019. The assessment methods include test, mini-CEX, ad-hoc EPA, OSCE. The assessment content includes the knowledge of diabetes medical, diet, drug, clinical consultation, education, empathy and attitude, etc. Students were evaluated during each period of training. Finally, all the evaluation results were summarized and the CCC gave each student a credit rating. Conclusions Through multiple evaluations and timely feedback, the ability of intern students was defined in more precision. It allowed clinical teachers to achieve a teaching consensus. Combine competence with actual work, and integrate evaluation with authorization. We need to construct a complete information system to make evaluation more efficient in the future. Funding Sources None.

Self-Reported Assessment of Outcome-Based Education in Philippine Otolaryngology - Head and Neck Surgery Residency Training Programs by Consultants and Residents

ABSTRACT Objective: To determine the self-reported assessment of initial implementation of the 3 domains of Outcome-Based Education in accredited Otolaryngology - Head and Neck Surgery residency training programs in the Philippines by consultants and residents and explore any associations between their demographic profiles and assessments. Methods: Design: Mixed Method Research Design Setting: Multicenter - 30 accredited ORL-HNS residency training institutions in the Philippines- National Capital Region (NCR) 19, Luzon 7, Visayas 2, and Mindanao 2. Participants: A total of 129 consultants and 82 second to fourth year residents in training were included in the study by convenience sampling. First-year residents who started their residency training in January 2020 were excluded. Respondents answered self-reported questionnaires to assess implementation of the 3 domains of OBE: intended learning outcomes (ILO), teaching and learning activities (TLA) and assessment tasks (AT) using the 4-point scale score from “fully implemented” (4) to “not implemented” (1). Results of questionnaires were confirmed using open-ended questions on the challenges of OBE with a focused group discussion among 4 consultants and 1 resident. Results: The self-reported assessment of respondents on OBE implementation was “fully implemented” in the 3 domains. However, low numerical scores were seen for “managing community health and social need” in the ILO and “laboratory activities and workshops” in the TLA for both consultants and residents, in the assessment task (AT “multisource feedback by nurses and administrative staff” for the consultants, and “direct observation of performance skills for patient encounter” for residents. Among the 7 modules, “research methodology” had the lowest score for both consultants’ and residents’ self-perception. Challenges of OBE revealed included “mastery,” “time” and “data keeping.” Consultants younger than 60 years of age who had been in the department longer than 3 years and residents who attended an OBE workshop / lecture tended to give higher scores Conclusion: Two years after distribution of the manual on OBE to ORL-HNS residency training institutions, the consultants’ and residents’ self-reported assessment on implementation in all the 3 domains of OBE was “fully implemented.”