The Establishment of Transformer-Based Computer-Aided Diagnosis Model to Improve the Classification Consistency of BI-RADS-US3–5 Nodules Among Radiologists: A Multi-Center Study

Background: Significant differences exist in classification outcomes for radiologists using ultrasonography-based breast imaging-reporting and data systems for diagnosing category 3–5 (BI-RADS-US 3–5) breast nodules, due to a lack of clear and distinguishing image features. As such, this study investigates the use of a transformer-based computer-aided diagnosis (CAD) model for improved BI-RADS-US 3–5 classification consistency.Methods: Five radiologists independently performed BI-RADS-US annotations on a breast ultrasonography image set collected from 20 hospitals in China. The data were divided into training, validation, testing, and sampling sets. The trained transformer-based CAD model was then used to classify test images, for which sensitivity, specificity, and accuracy were calculated. Variations in these metrics among the 5 radiologists were analyzed by referencing BI-RADS-US classification results for the sampling test set, provided by CAD, to determine whether classification consistency (the kappa value)，sensitivity, specificity, and accuracy had improved.Results: Classification accuracy for the CAD model applied to the test set was 95.7% for category 3 nodules, 97.6% for category 4A nodules, 95.60% for category 4B nodules, 94.2% for category 4C nodules, and 97.5% for category 5 nodules. Adjustments were made to 1,583 nodules, as 905 were classified to a higher category and 678 to a lower category in the sampling test set. As a result, the accuracy, sensitivity, and specificity of classification by each radiologist improved, with the consistency (kappa values) for all radiologists increasing to >0.60.Conclusions: The proposed transformer-based CAD model improved BI-RADS-US 3–5 nodule classification by individual radiologists and increased diagnostic consistency.

Evaluating Classification Consistency of Oral Lesion Images for Use in an Image Classification Teaching Tool

A web-based image classification tool (DiLearn) was developed to facilitate active learning in the oral health profession. Students engage with oral lesion images using swipe gestures to classify each image into pre-determined categories (e.g., left for refer and right for no intervention). To assemble the training modules and to provide feedback to students, DiLearn requires each oral lesion image to be classified, with various features displayed in the image. The collection of accurate meta-information is a crucial step for enabling the self-directed active learning approach taken in DiLearn. The purpose of this study is to evaluate the classification consistency of features in oral lesion images by experts and students for use in the learning tool. Twenty oral lesion images from DiLearn’s image bank were classified by three oral lesion experts and two senior dental hygiene students using the same rubric containing eight features. Classification agreement among and between raters were evaluated using Fleiss’ and Cohen’s Kappa. Classification agreement among the three experts ranged from identical (Fleiss’ Kappa = 1) for “clinical action”, to slight agreement for “border regularity” (Fleiss’ Kappa = 0.136), with the majority of categories having fair to moderate agreement (Fleiss’ Kappa = 0.332–0.545). Inclusion of the two student raters with the experts yielded fair to moderate overall classification agreement (Fleiss’ Kappa = 0.224–0.554), with the exception of “morphology”. The feature of clinical action could be accurately classified, while other anatomical features indirectly related to diagnosis had a lower classification consistency. The findings suggest that one oral lesion expert or two student raters can provide fairly consistent meta-information for selected categories of features implicated in the creation of image classification tasks in DiLearn.

Hierarchical Diagnostic Classification Modeling of Reading Comprehension

The Hierarchical Diagnostic Classification Model (HDCM) reflects on the sequences of the presentation of the essential materials and attributes to answer the items of a test correctly. In this study, a foreign language reading comprehension test was analyzed employing HDCM and the generalized deterministic-input, noisy and gate (G-DINA) model to determine and compare respondents’ mastery profiles in the test’s predefined skills and to illustrate the relationships among the attributes involved in the test to capture the influence of sequential teaching of materials on increasing the probability of getting an item a correct answer. Furthermore, Differential Item Functioning (DIF) analysis was applied to detect whether the test functions as a reason for the gender gap in participants’ achievement. Finally, classification consistency and accuracy indices are studied. The results showed that the G-DINA and one of the HDCMs fit the data well. However, although the results of HDCM showed the existence of attribute dependencies in the reading comprehension test, the relative fit indices highlight a significant difference between the G-DINA and HDCM, favoring G-DINA. Moreover, results indicate that there is a significant difference between males and females in six items in favor of females. Besides, classification consistency and accuracy indices specify that the Iranian University Entrance Examination holds a 71% chance of categorizing a randomly selected test taker consistently on two distinct test settings and a 78% likelihood of accurately classifying any randomly selected student into the true latent classes. As a result, it can be concluded that the Iranian University Entrance Examination can be considered as a valid and reliable test.