A deep learning based graph-transformer for whole slide image classification

Deep learning is a powerful tool for assessing pathology data obtained from digitized biopsy slides. In the context of supervised learning, most methods typically divide a whole slide image (WSI) into patches, aggregate convolutional neural network outcomes on them and estimate overall disease grade. However, patch-based methods introduce label noise in training by assuming that each patch is independent with the same label as the WSI and neglect the important contextual information that is significant in disease grading. Here we present a Graph-Transformer (GT) based framework for processing pathology data, called GTP, that interprets morphological and spatial information at the WSI-level to predict disease grade. To demonstrate the applicability of our approach, we selected 3,024 hematoxylin and eosin WSIs of lung tumors and with normal histology from the Clinical Proteomic Tumor Analysis Consortium, the National Lung Screening Trial, and The Cancer Genome Atlas, and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from those that have normal histology. Our model achieved consistently high performance on binary (tumor versus normal: mean overall accuracy = 0.975+/-0.013) as well as three-label (normal versus LUAD versus LSCC: mean accuracy = 0.932+/-0.019) classification on held-out test data, underscoring the power of GT-based deep learning for WSI-level classification. We also introduced a graph-based saliency mapping technique, called GraphCAM, that captures regional as well as contextual information and allows our model to highlight WSI regions that are highly associated with the class label. Taken together, our findings demonstrate GTP as a novel interpretable and effective deep learning framework for WSI-level classification.

Download Full-text

Deep learning systems detect dysplasia with human-like accuracy using histopathology and probe-based confocal laser endomicroscopy

Scientific Reports ◽

10.1038/s41598-021-84510-4 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Shan Guleria ◽

Tilak U. Shah ◽

J. Vincent Pulido ◽

Matthew Fasullo ◽

Lubaina Ehsan ◽

...

Keyword(s):

Deep Learning ◽

Diagnostic Accuracy ◽

High Sensitivity ◽

Confocal Laser Endomicroscopy ◽

Confocal Laser ◽

Learning Approaches ◽

Learning Models ◽

Whole Slide Image ◽

Slide Image ◽

Level Model

AbstractProbe-based confocal laser endomicroscopy (pCLE) allows for real-time diagnosis of dysplasia and cancer in Barrett’s esophagus (BE) but is limited by low sensitivity. Even the gold standard of histopathology is hindered by poor agreement between pathologists. We deployed deep-learning-based image and video analysis in order to improve diagnostic accuracy of pCLE videos and biopsy images. Blinded experts categorized biopsies and pCLE videos as squamous, non-dysplastic BE, or dysplasia/cancer, and deep learning models were trained to classify the data into these three categories. Biopsy classification was conducted using two distinct approaches—a patch-level model and a whole-slide-image-level model. Gradient-weighted class activation maps (Grad-CAMs) were extracted from pCLE and biopsy models in order to determine tissue structures deemed relevant by the models. 1970 pCLE videos, 897,931 biopsy patches, and 387 whole-slide images were used to train, test, and validate the models. In pCLE analysis, models achieved a high sensitivity for dysplasia (71%) and an overall accuracy of 90% for all classes. For biopsies at the patch level, the model achieved a sensitivity of 72% for dysplasia and an overall accuracy of 90%. The whole-slide-image-level model achieved a sensitivity of 90% for dysplasia and 94% overall accuracy. Grad-CAMs for all models showed activation in medically relevant tissue regions. Our deep learning models achieved high diagnostic accuracy for both pCLE-based and histopathologic diagnosis of esophageal dysplasia and its precursors, similar to human accuracy in prior studies. These machine learning approaches may improve accuracy and efficiency of current screening protocols.

Download Full-text

Land Cover Mapping with Higher Order Graph-Based Co-Occurrence Model

Remote Sensing ◽

10.3390/rs10111713 ◽

2018 ◽

Vol 10 (11) ◽

pp. 1713 ◽

Cited By ~ 5

Author(s):

Wenzhi Zhao ◽

William Emery ◽

Yanchen Bo ◽

Jiage Chen

Keyword(s):

Deep Learning ◽

Land Cover ◽

Spatial Information ◽

Conditional Random Field ◽

Contextual Information ◽

Learning Algorithms ◽

Classification Performance ◽

Higher Order ◽

Land Cover Mapping ◽

Initial Classification

Deep learning has become a standard processing procedure in land cover mapping for remote sensing images. Instead of relying on hand-crafted features, deep learning algorithms, such as Convolutional Neural Networks (CNN) can automatically generate effective feature representations, in order to recognize objects with complex image patterns. However, the rich spatial information still remains unexploited, since most of the deep learning algorithms only focus on small image patches that overlook the contextual information at larger scales. To utilize these contextual information and improve the classification performance for high-resolution imagery, we propose a graph-based model in order to capture the contextual information over semantic segments of the image. First, we explore semantic segments which build on the top of deep features and obtain the initial classification result. Then, we further improve the initial classification results with a higher-order co-occurrence model by extending the existing conditional random field (HCO-CRF) algorithm. Compared to the pixel- and object-based CNN methods, the proposed model achieved better performance in terms of classification accuracy.

Download Full-text

Klasifikasi Teks Hadis Bukhari Terjemahan Indonesia Menggunakan Recurrent Convolutional Neural Network (CRNN)

Jurnal Teknologi Informasi dan Ilmu Komputer ◽

10.25126/jtiik.2021853750 ◽

2021 ◽

Vol 8 (5) ◽

pp. 907

Author(s):

Muhammad Yuslan Abu Bakar ◽

Adiwijaya Adiwijaya

Keyword(s):

Neural Network ◽

Deep Learning ◽

Convolutional Neural Network ◽

Recurrent Neural Network ◽

Recurrent Neural Networks ◽

Spatial Information ◽

Contextual Information ◽

Computing Time ◽

Sequential Data ◽

Text Data

Hadis merupakan sumber hukum dan pedoman kedua bagi umat Islam setelah Al-Qur’an dan banyak sekali hadis yang telah diriwayatkan oleh para ahli hadis selama ini. Penelitian ini membangun sebuah sistem yang dapat melakukan klasifikasi teks hadis Bukhari terjemahan berbahasa Indonesia. Topik ini diangkat untuk memenuhi kebutuhan umat Islam dalam mengetahui apa saja informasi mengenai anjuran dan larangan yang terdapat dalam suatu hadis. Klasifikasi teks memiliki tantangannya tersendiri terkait dengan jumlah fitur yang sangat banyak (dimensi sangat besar) sehingga waktu komputasi menjadi besar dan mengakibatkan sulitnya mendapatkan hasil yang optimal. Pada penelitian ini, digunakan salah satu metode hibrid dalam dunia deep learning dengan menggabungkan Convolutional Neural Network dan Recurrent Neural Network, yaitu Convolutional Recurrent Neural Network (CRNN). Convolutional Neural Network dipilih sebagai metode seleksi dan reduksi data dikarenakan dapat menangkap informasi spasial yang saling berhubungan dan berkorelasi. Sementara Recurrent Neural Network digunakan sebagai metode klasifikasi dengan mengusung kemampuan utamanya yaitu dapat menangkap informasi kontekstual yang sangat panjang khususnya pada data sekuens seperti data teks dengan mengandalkan ‘memori’ yang dimilikinya. Hasil penelitian menyajikan beberapa hasil klasifikasi menggunakan deep learning, dimana hasil akurasi terbaik diberikan oleh Convolutional Recurrent Neural Network (CRNN), yakni sebesar 80.79%. Abstract Hadith is a source of law and guidance for Muslims after the Qur'an and many hadith have been narrated by hadith experts so far. This research builds a system that can classify Bukhari hadith in Indonesian translations. This topic was raised to meet the needs of Muslims in knowing what information about the suggestions and prohibitions that exist in a hadith. Text classification has its own challenges related to several features whose dimensions are very large so that it increases computing time and causes difficulties in getting optimal results. This research uses a hybrid method in deep learning by combining a Convolutional Neural Network and a Recurrent Neural Network, namely Convolutional Recurrent Neural Network (CRNN). Convolutional Neural Network was chosen as a method of selecting and reducing data that can be determined as spatial information that is interrelated and correlated. While Recurrent Neural Networks are used as a classification method by carrying out capabilities that can be used as very long contextual information specifically on sequential data such as text data by relying on the ‘memory’ it has. This research presents several classification results using deep learning, where the best accuracy results are given by the Convolutional Recurrent Neural Network (CRNN), which is equal to 80.79%.

Download Full-text

Whole Slide Image Classification and Segmentation using Deep Learning

10.1145/3440943.3444357 ◽

2020 ◽

Author(s):

Sahadev Poudel ◽

Sang-Woong Lee

Keyword(s):

Deep Learning ◽

Image Classification ◽

Whole Slide Image ◽

Slide Image

Download Full-text

Automatic classification of sentinel lymph node (SLN) metastases in breast carcinoma whole slide image (WSI) through densenet deep learning network

IP Journal of Diagnostic Pathology and Oncology ◽

10.18231/j.jdpo.2020.046 ◽

2020 ◽

Vol 5 (2) ◽

pp. 144-150

Author(s):

Rajasekaran Subramanian ◽

Swathi Guptha ◽

R Devika Rubi ◽

Abhay Krishna Kasavaraju ◽

Samayk Jain

Keyword(s):

Deep Learning ◽

Lymph Node ◽

Sentinel Lymph Node ◽

Breast Carcinoma ◽

Automatic Classification ◽

Whole Slide Image ◽

Learning Network ◽

Slide Image ◽

Deep Learning Network

Download Full-text

Deep Learning for Whole Slide Image Analysis: An Overview

Frontiers in Medicine ◽

10.3389/fmed.2019.00264 ◽

2019 ◽

Vol 6 ◽

Cited By ~ 20

Author(s):

Neofytos Dimitriou ◽

Ognjen Arandjelović ◽

Peter D. Caie

Keyword(s):

Image Analysis ◽

Deep Learning ◽

Whole Slide Image ◽

Slide Image

Download Full-text

Evaluation of Deep Learning Segmentation Models for Detection of Pine Wilt Disease in Unmanned Aerial Vehicle Images

Remote Sensing ◽

10.3390/rs13183594 ◽

2021 ◽

Vol 13 (18) ◽

pp. 3594

Author(s):

Lang Xia ◽

Ruirui Zhang ◽

Liping Chen ◽

Longlong Li ◽

Tongchuan Yi ◽

...

Keyword(s):

Deep Learning ◽

Unmanned Aerial Vehicle ◽

High Performance ◽

Spatial Information ◽

Pine Wilt Disease ◽

Wilt Disease ◽

Large Area ◽

Pine Wilt ◽

Aerial Vehicle ◽

Segmentation Models

Pine wilt disease (PWD) is a serious threat to pine forests. Combining unmanned aerial vehicle (UAV) images and deep learning (DL) techniques to identify infected pines is the most efficient method to determine the potential spread of PWD over a large area. In particular, image segmentation using DL obtains the detailed shape and size of infected pines to assess the disease’s degree of damage. However, the performance of such segmentation models has not been thoroughly studied. We used a fixed-wing UAV to collect images from a pine forest in Laoshan, Qingdao, China, and conducted a ground survey to collect samples of infected pines and construct prior knowledge to interpret the images. Then, training and test sets were annotated on selected images, and we obtained 2352 samples of infected pines annotated over different backgrounds. Finally, high-performance DL models (e.g., fully convolutional networks for semantic segmentation, DeepLabv3+, and PSPNet) were trained and evaluated. The results demonstrated that focal loss provided a higher accuracy and a finer boundary than Dice loss, with the average intersection over union (IoU) for all models increasing from 0.656 to 0.701. From the evaluated models, DeepLLabv3+ achieved the highest IoU and an F1 score of 0.720 and 0.832, respectively. Also, an atrous spatial pyramid pooling module encoded multiscale context information, and the encoder–decoder architecture recovered location/spatial information, being the best architecture for segmenting trees infected by the PWD. Furthermore, segmentation accuracy did not improve as the depth of the backbone network increased, and neither ResNet34 nor ResNet50 was the appropriate backbone for most segmentation models.

Download Full-text

STATISTICAL LOCAL BINARY PATTERNS (SLBP): APPLICATION TO PROSTATE CANCER GLEASON SCORE PREDICTION FROM WHOLE SLIDE PATHOLOGY IMAGES

10.1101/503094 ◽

2018 ◽

Author(s):

Hongming Xu ◽

Tae Hyun Hwang

Keyword(s):

Prostate Cancer ◽

Gleason Score ◽

Local Binary Patterns ◽

Cellular Level ◽

The Cancer Genome Atlas ◽

Whole Slide Image ◽

Structure Information ◽

Slide Image ◽

Cancer Genome Atlas ◽

Tcga Dataset

Computerized whole slide image analysis is important for assisting pathologists in cancer grading and predicting patient clinical outcomes. However, it is challenging to analyze whole slide image (WSI) at cellular level due to its huge size and nuclear variations. For efficient WSI analysis, this paper presents a general texture descriptor, statistical local binary patterns (SLBP), which is applied to prostate cancer Gleason score prediction from WSI. Unlike traditional local binary patterns (LBP) and many its variants, the presented SLBP encodes local texture patterns via analyzing both median and standard deviation over a regional sampling scheme, so that it can capture more micro- and macro-structure information in the image. Experiments on Gleason score prediction have been performed on 317 different patient cases selected from the cancer genome atlas (TCGA) dataset. The presented SLBP descriptor provides over 80% accuracy on two class (grade <=7 vs grade >=8) distinction, which is superior to traditional texture descriptors such as histogram, Haralick and other state-of-art LBP variants.

Download Full-text

Development of a Deep Learning Model to Assist With Diagnosis of Hepatocellular Carcinoma

Frontiers in Oncology ◽

10.3389/fonc.2021.762733 ◽

2021 ◽

Vol 11 ◽

Author(s):

Shi Feng ◽

Xiaotian Yu ◽

Wenjie Liang ◽

Xuejie Li ◽

Weixiang Zhong ◽

...

Keyword(s):

Hepatocellular Carcinoma ◽

Deep Learning ◽

Malignant Tumors ◽

Learning Model ◽

The Cancer Genome Atlas ◽

Classification Model ◽

Step Method ◽

Histopathological Classification ◽

Slide Image ◽

Deep Learning Model

BackgroundAn accurate pathological diagnosis of hepatocellular carcinoma (HCC), one of the malignant tumors with the highest mortality rate, is time-consuming and heavily reliant on the experience of a pathologist. In this report, we proposed a deep learning model that required minimal noise reduction or manual annotation by an experienced pathologist for HCC diagnosis and classification.MethodsWe collected a whole-slide image of hematoxylin and eosin-stained pathological slides from 592 HCC patients at the First Affiliated Hospital, College of Medicine, Zhejiang University between 2015 and 2020. We propose a noise-specific deep learning model. The model was trained initially with 137 cases cropped into multiple-scaled datasets. Patch screening and dynamic label smoothing strategies are adopted to handle the histopathological liver image with noise annotation from the perspective of input and output. The model was then tested in an independent cohort of 455 cases with comparable tumor types and differentiations.ResultsExhaustive experiments demonstrated that our two-step method achieved 87.81% pixel-level accuracy and 98.77% slide-level accuracy in the test dataset. Furthermore, the generalization performance of our model was also verified using The Cancer Genome Atlas dataset, which contains 157 HCC pathological slides, and achieved an accuracy of 87.90%.ConclusionsThe noise-specific histopathological classification model of HCC based on deep learning is effective for the dataset with noisy annotation, and it significantly improved the pixel-level accuracy of the regular convolutional neural network (CNN) model. Moreover, the model also has an advantage in detecting well-differentiated HCC and microvascular invasion.

Download Full-text