EP.TU.4Performance at Medical School predicts success in the Intercollegiate Membership of the Royal College of Surgery (MRCS) examination

Aims Identifying predictors of success in post-graduate examinations can help guide the career choices of medical students and may aid early identification of trainees requiring extra support to progress in specialty training. We assessed whether performance at medical school as quantified by the Educational Performance Measurement (EPM) and scores from the Situational Judgement Test (SJT) used for selection into Foundation Training predicted success at the Membership of the Royal College of Surgeons (MRCS) examination. Methods We analysed data from the UKMED Database for UK graduates who had attempted MRCS Part A (n = 1,975) and Part B n = 630) between 2013-2017. Univariate analysis examined the relationship between performance and the likelihood of passing MRCS at first-attempt. Logistic regression identified independent predictors of MRCS success. Results For every additional EPM decile point gained the chances of passing MRCS at first attempt increased by 52% for Part A (odds ratio 1.52 [95% confidence interval (CI) 1.46-1.60]) and 27% for Part B (1.27 [1.18-1.38]). For every point awarded for additional degrees in the EPM, candidates were 29% more likely to pass MRCS Part A first time (1.29 [1.12-1.48]). SJT score was not a statistically significant independent predictor of MRCS Part A or Part B success after adjusting for sociodemographic factors (P = 0.182 and P = 0.125 respectively). Conclusions This, the first study to investigate the relationship between medical school and success at a high stakes UK postgraduate surgical examination found that medical school performance deciles are the most significant measure of predicting later success in the MRCS.

Does performance at medical school predict success at the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination? A retrospective cohort study

BackgroundIdentifying predictors of success in postgraduate examinations can help guide the career choices of medical students and may aid early identification of trainees requiring extra support to progress in specialty training. We assessed whether performance on the educational performance measurement (EPM) and situational judgement test (SJT) used for selection into foundation training predicted success at the Membership of the Royal College of Surgeons (MRCS) examination.MethodsThis was a longitudinal, cohort study using data from the UK Medical Education Database (https://www.ukmed.ac.uk). UK medical graduates who had attempted Part A (n=2585) and Part B (n=755) of the MRCS between 2014 and 2017 were included. χ2 and independent t-tests were used to examine the relationship between medical school performance and sociodemographic factors with first-attempt success at MRCS Part A and B. Multivariate logistic regression was employed to identify independent predictors of MRCS performance.ResultsThe odds of passing MRCS increased by 55% for Part A (OR 1.55 (95% CI 1.48 to 1.61)) and 23% for Part B (1.23 (1.14 to 1.32)) for every additional EPM decile point gained. For every point awarded for additional degrees in the EPM, candidates were 20% more likely to pass MRCS Part A (1.20 (1.13 to 1.29)) and 17% more likely to pass Part B (1.17 (1.04 to 1.33)). For every point awarded for publications in the EPM, candidates were 14% more likely to pass MRCS Part A (1.14 (1.01 to 1.28)). SJT score was not a statistically significant independent predictor of MRCS success.ConclusionThis study has demonstrated the EPM’s independent predictive power and found that medical school performance deciles are the most significant measure of predicting later success in the MRCS. These findings can be used by medical schools, training boards and workforce planners to inform evidence-based and contemporary selection and assessment strategies.

Measuring hygiene competence: the picture-based situational judgement test HygiKo

Background With the onset of the COVID-19 pandemic at the beginning of 2020, the crucial role of hygiene in healthcare settings has once again become very clear. For diagnostic and for didactic purposes, standardized and reliable tests suitable to assess the competencies involved in “working hygienically” are required. However, existing tests usually use self-report questionnaires, which are suboptimal for this purpose. In the present study, we introduce the newly developed, competence-oriented HygiKo test instrument focusing health-care professionals’ hygiene competence and report empirical evidence regarding its psychometric properties. Methods HygiKo is a Situational Judgement Test (SJT) to assess hygiene competence. The HygiKo-test consists of twenty pictures (items), each item presents only one unambiguous hygiene lapse. For each item, test respondents are asked (1) whether they recognize a problem in the picture with respect to hygiene guidelines and, (2) if yes, to describe the problem in a short verbal response. Our sample comprised n = 149 health care professionals (79.1 % female; age: M = 26.7 years, SD = 7.3 years) working as clinicians or nurses. The written responses were rated by two independent raters with high agreement (α > 0.80), indicating high reliability of the measurement. We used Item Response Theory (IRT) for further data analysis. Results We report IRT analyses that show that the HygiKo-test is suitable to assess hygiene competence and that it allows to distinguish between persons demonstrating different levels of ability for seventeen of the twenty items), especially for the range of low to medium person abilities. Hence, the HygiKo-SJT is suitable to get a reliable and competence-oriented measure for hygiene-competence. Conclusions In its present form, the HygiKo-test can be used to assess the hygiene competence of medical students, medical doctors, nurses and trainee nurses in cross-sectional measurements. In order to broaden the difficulty spectrum of the current test, additional test items with higher difficulty should be developed. The Situational Judgement Test designed to assess hygiene competence can be helpful in testing and teaching the ability of working hygienically. Further research for validity is needed.

Indirect skill assessment using AI technology

In the new context of transitional, mobilised and globalised labour markets, an urgent need has emerged for meaningful assessment tools, methods and techniques to measure and recognize the workers’ skills. This paper aims to present a novel approach in skill assessment developed under NADINE H2020 project, the indirect skill assessment, as an alternative or supplement to the traditional selection methods. AI technology is utilised to facilitate the indirect skill assessment via two agnostic content serious games (Tetris and 2048), based on the evaluation of an individual’s performance in playing a game outside a situational judgement test (SJT) context. Novel datasets have been developed, comprised of game sessions and the corresponding skill assessments of the players through validated psychometric questionnaires, which were the basis for the algorithm training that would provide the estimation of a player’s skillset. The trained neural models for both games proved to have strong skill assessment capabilities, indicating that there is indeed a correlation between a person’s action sequence and his/her different skills.

191 Performance at Medical School and Foundation Programme Selection Can Predict Success in The MRCS Examination

Background Identifying factors that predict success in the Membership of the Royal College of Surgeons (MRCS) examination can aid trainees’ exam preparation and help deaneries identify candidates likely to require additional support. We assessed whether performance at Medical School and UK Foundation Programme selection (Educational Performance Measurement (EPM) and Situational Judgement Test (SJT)) predicted MRCS success. Method We analysed data from the UKMED Database for UK graduates who had attempted MRCS Part A (n = 1,975) and Part B (n = 630) between 2013-2017. Univariate analysis examined the relationship between performance and the likelihood of passing MRCS at first attempt. Logistic regression identified independent predictors of MRCS success. Results Each additional EPM decile increased the chances of passing MRCS by 52% for Part A (odds ratio 1.52 [95% confidence interval 1.46-1.60]) and 27% for Part B (1.27 [1.18-1.38]). Each EPM point awarded for degrees increased the likelihood of passing Part A by 29% (1.29 [1.12-1.48]). SJT score was not a statistically significant independent predictor of MRCS success (P>0.05). Conclusions This is the first study to investigate the relationship between UK medical school performance and postgraduate surgical exam success. Higher ranked students are more likely to succeed in the MRCS compared to their lower ranked peers.