Data-driven Exploration of Engagement with Workplace-based Assessment in the Clinical Skills Domain

The paper presents a multi-faceted data-driven computational approach to analyse workplace-based assessment (WBA) of clinical skills in medical education. Unlike formal university-based part of the degree, the setting of WBA can be informal and only loosely regulated, as students are encouraged to take every opportunity to learn from the clinical setting. For clinical educators and placement coordinators it is vital to follow and analyse students’ engagement with WBA while on placements, in order to understand how students are participating in the assessment, and what improvements can be made. We analyse digital data capturing the students’ WBA attempts and comments on how the assessments went, using process mining and text analytics. We compare Year 1 cohorts across three years, focusing on differences between primary vs. secondary care placements. The main contribution of the work presented in this paper is the exploration of computational approaches for multi-faceted, data-driven assessment analytics for workplace learning which includes:(i) a set of features for analysing clinical skills WBA data, (ii) analysis of the temporal aspects ofthat data using process mining, and (iii) utilising text analytics to compare student reflections on WBA. We show how assessment data captured during clinical placements can provide insights about the student engagement and inform the medical education practice. Our work is inspired by Jim Greer’s vision that intelligent methods and techniques should be adopted to address key challenges faced by educational practitioners in order to foster improvement of learning and teaching. In the broader AI in Education context, the paper shows the application of AI methods to address educational challenges in a new informal learning domain - practical healthcare placements in higher education medical training.

Empirical analysis comparing the tele-objective structured clinical examination (teleOSCE) and the in-person assessment in Australia

Purpose: It aimed to compare the use of the tele objective structured clinical examination (teleOSCE) with in-person assessment in high-stakes clinical examination so as to determine the impact of the teleOSCE on the assessment undertaken. Discussion follows regarding what skills and domains can effectively be assessed in a teleOSCE. Methods: This study is a retrospective observational analysis. It compares the results achieved by final year medical students in their clinical examination, assessed using the teleOSCE in 2020 (n=285), with those who were examined using the traditional in-person format in 2019 (n=280). The study was undertaken at the University of New South Wales, Australia.Results: In the domain of physical examination, students in 2020 scored 0.277 points higher than those in 2019 (mean difference –0.277, P<0.001, effect size 0.332). Across all other domains, there was no significant difference in mean scores between 2019 and 2020.Conclusion: The teleOSCE does not negatively impact assessment in clinical examination in all domains except physical examination. If the teleOSCE is the future of clinical skills examination, assessment of physical examination will require concomitant workplace-based assessment.

Pre-Clerkship EPA Assessments: A Thematic Analysis of Rater Cognition

Background. Entrustable Professional Activities (EPAs) assessments measure learners’ competence with an entrustment or supervisory scale. Designed for workplace-based assessment EPA assessments have also been proposed for undergraduate medical education (UME), where assessments frequently occur outside the workplace and may be less intuitive, raising validity concerns. This study explored how assessors make entrustment determinations in UME, to include the impact of longitudinal student-assessor relationships.Methods. A qualitative approach using think-alouds was employed. Assessors assessed two students (familiar and unfamiliar) completing a history and physical exam using a supervisory scale and then thought-aloud after each assessment. We conducted a thematic analysis of assessors’ response processes and compared them based on their familiarity with a student.Results. Four themes and fifteen subthemes were identified. The most prevalent theme related to “student performance.” The other three themes included “frame of reference,” “assessor uncertainty,” and “the patient.” “Previous student performance” and “affective reactions” were subthemes more likely to inform scoring when faculty were familiar with a student, while unfamiliar faculty were more likely to reference “self” and “lack confidence in their ability to assess.”Conclusions. Student performance appears to be assessors’ main consideration for all students, providing some validity evidence for the response process in EPA assessments. Several problematic themes could be addressed with faculty development while others appear to be inherent to entrustment and may be more challenging to mitigate. Differences based on assessor familiarity with student merits further research on how trust develops over time.

Leadership in primary care

Leadership is an important concept for GPs and has its own topic guide in the MRCGP curriculum. Simply put, leadership is about influencing others to take action for change. You may like to start this article by thinking of a few public figures you know, and what makes you think they are ‘good leaders’ – or not. This often highlights the fact that ‘good’ is defined both by how effective leaders are (…‘ he really changed the practice’…); and whether they lead people into actions that help or harm (… ‘ pay went up, but care got worse…’). As GPs, we need to recognise and develop our own leadership skills and be able to help others to do what is needed for effective patient care. This starts with having some definitions and building up through ‘knowing about’ and ‘knowing how’, towards understanding and applying our skills in practice. Leadership can also need action, beyond our own practice and that of our team, to the wider setting of the community, other parts of the workforce, and in the wider ‘macro’ sphere of changing the systems of care. This article will give you a framework for this learning. The article highlights some of the core capabilities of GPs, why you need them, and how you can develop them, linking this in with MRCGP Workplace-Based Assessment (WPBA) requirements. It also aims to show the links between ‘knowing yourself’ and ‘working in organisations’ - as GP leadership is needed at many levels.