The application of predictive analytics to identify at-risk students in health professions education

Introduction: When a learner fails to reach a milestone, educators often wonder if there had been any warning signs that could have allowed them to intervene sooner. Machine learning is used to predict which students are at risk of failing a national certifying exam. Predictions are made well in advance of the exam, such that educators can meaningfully intervene before students take the exam.Methods: Using already-collected, first-year student assessment data from four cohorts in a Master of Physician Assistant Studies program, the authors implement an "adaptive minimum match" version of the k-nearest neighbors algorithm (AMMKNN), using changing numbers of neighbors to predict each student's future exam scores on the Physician Assistant National Certifying Examination (PANCE). Leave-one-out cross validation (LOOCV) was used to evaluate the practical capabilities of this model, before making predictions for new students. Results: The best predictive model has an accuracy of 93%, sensitivity of 69%, and specificity of 94%. It generates a predicted PANCE score for each student, one year before they are scheduled to take the exam. Students can then be prospectively categorized into groups that need extra support, optional extra support, or no extra support. The educator then has one year to provide the appropriate customized support to each type of student. Conclusions: Predictive analytics can help health professions educators allocate scarce time and resources across their students. Interprofessional educators can use the included methods and code to generate predicted test outcomes for students. The authors recommend that educators using this or similar predictive methods act responsibly and transparently.

Correlating Transfusion Medicine In-Service Examination Scores With Outcomes of the American Board of Pathology Transfusion Medicine Subspecialty Certifying Examination

Objectives The Transfusion Medicine In-Service Examination (TMISE) is offered twice a year to transfusion medicine (TM) fellows. We examined the relationship between TMISE scores and outcomes of the American Board of Pathology (ABP) TM subspecialty certifying examination (TM boards). Methods TM fellowship programs were contacted to provide anonymous data about TM fellows, their scores on TMISE, and outcome of TM boards. Results Of 48 TM fellowship programs contacted, 24 (50%) responded with data for 170 fellows. Average TMISE score of fellows who passed their first TM boards attempt was 71.3, while the average TMISE score of fellows who failed their first TM boards attempt was 64.3 (P = .009). Conclusions TMISE scores correlated with passing TM boards on the first attempt. Fellows who took the TM boards the same year that they graduated from TM fellowship had a significantly higher first-time pass rate than fellows who delayed taking TM boards.