Examining the Factors Influencing Applicants’ Placement on One General Surgery Program’s Rank Order List

Purpose Surgery residency applications include variables that determine an individual’s rank on a program’s match list. We performed this study to determine which residency application variables are the most impactful in creating our program’s rank order list. Methods We completed a retrospective examination of all interviewed applicants for the 2019 match. We recorded United States Medical Licensing Examinations (USMLE) step I and II scores, class quartile rank from the Medical Student Performance Evaluation (MSPE), Alpha Omega Alpha (AOA) membership, geographic region, surgery clerkship grade, and grades on other clerkships. The MSPE and letters of recommendation were reviewed by two of the authors and assigned a score of 1 to 3, where 1 was weak and 3 was strong. The same two authors reviewed the assessments from each applicant’s interview and assigned a score from 1-5, where 1 was poor and 5 was excellent. Univariate analysis was performed, and the significant variables were used to construct an adjusted multivariate model with significance measured at P < .05. Results Univariate analysis for all 92 interviewed applicants demonstrated that USMLE step 2 scores ( P = .002), class quartile rank ( P = .004), AOA status ( P = .014), geographic location ( P < .001), letters of recommendation ( P < .001), and interview rating ( P < .001) were significant in predicting an applicant’s position on the rank list. On multivariate analysis only USMLE step 2 ( P = .018) and interview ( P < .001) remained significant. Conclusion USMLE step 2 and an excellent interview were the most important factors in constructing our rank order list. Applicants with a demonstrated strong clinical fund of knowledge that develop a rapport with our faculty and residents receive the highest level of consideration for our program.

Is there Bias in Alternatives to Standardized Tests? An Investigation into Content Differences in Letters of Recommendation

Individuals concerned with subgroup differences on standardized tests suggest replacing these tests with holistic evaluations of unstructured application materials, such as letters of recommendation (LORs), which they posit show less bias. We empirically investigate this proposition that LORs are bias-free, and argue that LORs might actually invite systematic, race and gender subgroup differences in the content and evaluation of LORs. We text analyzed over 37,000 LORs submitted on behalf of over 10,000 graduate school applicants. Results showed that LOR content does differ across applicants. Furthermore, we see some systematic gender, race, and gender-race intersection differences in LOR content. Content of LORs also systematically differed between degree programs (S.T.E.M. vs. non-S.T.E.M.) and degree sought (doctoral vs. masters). Finally, LOR content alone did not predict an appreciable amount of variance in offers of admission (the first barrier to increasing diversity and inclusion in graduate programs). Our results, combined with past research on LOR content bias, highlight concerns that LORs can be biased against marginalized groups. We conclude with suggestions for reducing potential bias in LOR and for increasing diversity in graduate programs.