Comparison of Object Detection Approaches Applied to Field Images of Papanicolaou Stained Cytology Slides

Papanicolaou is an inexpensive and non-invasive method, generally applied to detect cervical cancer, that can also be useful to detect cancer on oral cavities. Although oral cancer is considered a global health issue with 350.000 people diagnosed over a year it can successfully be treated if diagnosed at early stages. The manual process of analyzing cells to detect abnormalities is time-consuming and subject to variations in perceptions from different professionals. To evaluate a possible solution to the automation of this process, in this paper we employ the object detection deep learning approach in the analysis of this type of image using 3 models: RetinaNet, Faster R-CNN, and Mask R-CNN. We trained and tested the models using images from 6 cytology slides (4 cancer cases and 2 healthy samples) and our results show that Mask R-CNN was the best model for localization and classification of nuclei with an IoU of 0.51 and recall of abnormal nuclei of 0.67.