Physician inter-annotator agreement in the Quality Oncology Practice Initiative (QOPI) manual abstraction task.
226 Background: The Quality Oncology Practice Initiative (QOPI) relies upon the accuracy of manual abstraction of clinical data from paper-based and electronic medical records (EMR’s). While there is no “gold standard” to measure manual abstraction accuracy, measurement of inter-annotator agreement (IAA) is a commonly agreed-upon surrogate. We quantified the IAA of QOPI abstractions on a cohort of cancer patients treated at Beth Israel Deaconess Medical Center. Methods: The EMR charts of 49 patients (20 colorectal cancer; 18 breast cancer; 11 non-Hodgkin lymphoma) were abstracted by separate physician abstractors in the Fall 2010 and Fall 2011 QOPI abstraction rounds. The Fall 2011 abstractors were unaware that the charts had been previously abstracted. We analyzed data elements that were common to both rounds, had data entry for both rounds, and did not contain information expected to change after the Fall 2010 round. Cohen’s kappa (κ) was calculated for encoded data; raw levels of agreement and magnitude of discrepancies were calculated for numeric and dated data. IRB approval was obtained and all investigators completed appropriate human subjects research training. Results: 109 data elements with 2,035 paired entries met the above criteria; four narrative elements were not analyzed. Overall IAA for 1,496 coded entries was κ=0.75; median IAA for n=85 individual coded elements was κ=0.84 (interquartile range 0.30-1.00). Overall IAA for 421 dated entries was 73%; median IAA for n=17 individual dated elements was 67% (interquartile range 61-86%). The median discrepancy for the 113 discrepant dated entries was +6 days (range -217 to +391 days). Conclusions: This study establishes a baseline level of accuracy for a complex medical abstraction task with clear relevance for the oncology community. Despite the fact that all information was available in the EMR, there was disagreement for objective elements (e.g. cancer stage) as well as subjective elements (e.g. patient wellbeing). Given that the observed κ is considered only fair IAA, and that the rate of date discrepancy is high, caution is necessary in interpreting the results of QOPI and other manual abstractions of clinical oncology data.