Beyond kappa: an informational index for diagnostic agreement in dichotomous and multivalue ordered-categorical ratings

AbstractAgreement measures are useful tools to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. Cohen’s kappa (κ) certainly is the most popular agreement method between two raters, and proved its effectiveness in the last sixty years. In spite of that, this method suffers from some alleged issues, which have been highlighted since the 1970s; moreover, its value is strongly dependent on the prevalence of the disease in the considered sample. This work introduces a new agreement index, the informational agreement (IA), which seems to avoid some of Cohen’s kappa’s flaws, and separates the contribution of the prevalence from the nucleus of agreement. These goals are achieved by modelling the agreement—in both dichotomous and multivalue ordered-categorical cases—as the information shared between two raters through the virtual diagnostic channel connecting them: the more information exchanged between the raters, the higher their agreement. In order to test its fair behaviour and the effectiveness of the method, IA has been tested on some cases known to be problematic for κ, in the machine learning context and in a clinical scenario to compare ultrasound (US) and automated breast volume scanner (ABVS) in the setting of breast cancer imaging.

Download Full-text

Comparison of handheld ultrasound and automated breast ultrasound in women recalled after mammography screening

Acta Radiologica ◽

10.1177/0284185116665421 ◽

2016 ◽

Vol 58 (5) ◽

pp. 515-520 ◽

Cited By ~ 7

Author(s):

Roxanna Hellgren ◽

Paul Dickman ◽

Karin Leifland ◽

Ariel Saracco ◽

Per Hall ◽

...

Keyword(s):

Breast Cancer ◽

Mammography Screening ◽

Breast Ultrasound ◽

Breast Volume ◽

Large Area ◽

Mammographic Finding ◽

Automated Breast Volume Scanner ◽

Handheld Ultrasound ◽

Study Population ◽

Automated Breast Ultrasound

Background Automated breast volume scanner (ABVS) is an ultrasound (US) device with a wide scanner that sweeps over a large area of the breast and the acquired transverse images are sent to a workstation for reconstruction and review. Whether ABVS is as reliable as handheld US is, however, still not established. Purpose To compare the sensitivity and specificity of ABVS to handheld breast US for detection of breast cancer, in the situation of recall after mammography screening. Material and Methods A total of 113 women, five with bilateral suspicious findings, undergoing handheld breast US due to a suspicious mammographic finding in screening, underwent additional ABVS. The methods were assessed for each breast and each detected lesion separately and classified into two categories: breasts with mammographic suspicion of malignancy and breasts with a negative mammogram. Results Twenty-six cancers were found in 25 women. In the category of breasts with a suspicious mammographic finding (n = 118), the sensitivity of both handheld US and ABVS was 88% (22/25). The specificity of handheld US was 93.5% (87/93) and ABVS was 89.2% (83/93). In the category of breasts with a negative mammography (n = 103), the sensitivity of handheld US and ABVS was 100% (1/1). The specificity of handheld US was 100% (102/102) and ABVS was 94.1% (96/102). Conclusion ABVS can potentially replace handheld US in the investigation of women recalled from mammography screening due to a suspicious finding. Due to the small size of our study population, further investigation with larger study populations is necessary before the implementation of such practice.

Download Full-text