Longitudinal Case-Based Evaluation of Diagnostic Competency Among Pathology Residents: A Statistical Approach

Context.—Accreditation Council for Graduate Medical Education guidelines require the evaluation of residents in 6 competencies. Pathology residents demonstrate medical knowledge, patient care, communication, and practice-based learning competencies in their attainment of competency in surgical pathology diagnosis. Objective.—To implement a prospective case-based approach to longitudinally evaluate the acquisition of competency in surgical pathology diagnosis by trainees. Design.—Each resident made his or her surgical pathology diagnosis on cases before faculty review of the cases. Faculty members scored each resident diagnosis as to whether they agree, partially agree, or disagree with the diagnosis. Forty-three months of surgical pathology reports (August 2001 through January 2005) and 22 252 surgical pathology cases were analyzed. Setting.—Pathology residency program. Participants.—Thirteen faculty members and 21 trainees. Main Outcome Measure.—Time and training year trends for the number of cases reviewed and the percent agreement between faculty and trainees on the diagnoses. Results.—A mean of 146 cases (range, 12–327 cases) was reviewed during each month-long rotation. The number of cases reviewed increased through postgraduate year 4. The percent agreement on the diagnoses was 78% (range, 56%– 99%) for all trainees, with improvement by postgraduate year, although the improvement attenuated by postgraduate year 3. Residents were less likely to preview the most complex cases. Faculty rank and sex and resident sex did not significantly affect outcomes. The overall agreement on the diagnoses increased over time. Residents experiencing difficulty could be identified clearly and early. Conclusions.—Individual resident performance was easily tracked over time. The review of hundreds of reports increases systems accountability and allows more objectivity than traditional evaluations. The use of case-based evaluation fosters earlier identification and remediation of deficiencies.