Generalizability of deep learning models for dental image analysis

AbstractWe assessed the generalizability of deep learning models and how to improve it. Our exemplary use-case was the detection of apical lesions on panoramic radiographs. We employed two datasets of panoramic radiographs from two centers, one in Germany (Charité, Berlin, n = 650) and one in India (KGMU, Lucknow, n = 650): First, U-Net type models were trained on images from Charité (n = 500) and assessed on test sets from Charité and KGMU (each n = 150). Second, the relevance of image characteristics was explored using pixel-value transformations, aligning the image characteristics in the datasets. Third, cross-center training effects on generalizability were evaluated by stepwise replacing Charite with KGMU images. Last, we assessed the impact of the dental status (presence of root-canal fillings or restorations). Models trained only on Charité images showed a (mean ± SD) F1-score of 54.1 ± 0.8% on Charité and 32.7 ± 0.8% on KGMU data (p < 0.001/t-test). Alignment of image data characteristics between the centers did not improve generalizability. However, by gradually increasing the fraction of KGMU images in the training set (from 0 to 100%) the F1-score on KGMU images improved (46.1 ± 0.9%) at a moderate decrease on Charité images (50.9 ± 0.9%, p < 0.01). Model performance was good on KGMU images showing root-canal fillings and/or restorations, but much lower on KGMU images without root-canal fillings and/or restorations. Our deep learning models were not generalizable across centers. Cross-center training improved generalizability. Noteworthy, the dental status, but not image characteristics were relevant. Understanding the reasons behind limits in generalizability helps to mitigate generalizability problems.

Download Full-text

Effect of Sequence Padding on the Performance of Protein-Based Deep Learning Models

10.21203/rs.2.21336/v1 ◽

2020 ◽

Author(s):

Angela Lopez-del Rio ◽

Maria Martin ◽

Alexandre Perera-Lluna ◽

Rabie Saidi

Keyword(s):

Deep Learning ◽

Amino Acid ◽

Enzyme Commission ◽

Model Performance ◽

Enzyme Commission Number ◽

Amino Acid Sequences ◽

Learning Models ◽

Zero Padding ◽

The One ◽

The Impact

Abstract Background The use of raw amino acid sequences as input for protein-based deep learning models has gained popularity in recent years. This scheme obliges to manage proteins with different lengths, while deep learning models require same-shape input. To accomplish this, zeros are usually added to each sequence up to a established common length in a process called zero-padding. However, the effect of different padding strategies on model performance and data structure is yet unknown. Results We analysed the impact of different ways of padding the amino acid sequences in a hierarchical Enzyme Commission number prediction problem. Our results show that padding has an effect on model performance even when there are convolutional layers implied. We propose and implement four novel types of padding the amino acid sequences. Conclusions The present study highlights the relevance of the step of padding the one-hot encoded amino acid sequences when building deep learning-based models for Enzyme Commission number prediction. The fact that this has an effect on model performance should raise awareness on the need of justifying the details of this step on future works. The code of this analysis is available at https://github.com/b2slab/padding_benchmark.

Download Full-text

U-Infuse: Democratization of Customizable Deep Learning for Object Detection

Sensors ◽

10.3390/s21082611 ◽

2021 ◽

Vol 21 (8) ◽

pp. 2611

Author(s):

Andrew Shepley ◽

Greg Falzon ◽

Christopher Lawson ◽

Paul Meek ◽

Paul Kwan

Keyword(s):

Deep Learning ◽

Intellectual Property ◽

Object Detection ◽

Image Data ◽

Learning Technologies ◽

Training Data ◽

Learning Models ◽

Ecological Data ◽

Single Class ◽

Large Numbers

Image data is one of the primary sources of ecological data used in biodiversity conservation and management worldwide. However, classifying and interpreting large numbers of images is time and resource expensive, particularly in the context of camera trapping. Deep learning models have been used to achieve this task but are often not suited to specific applications due to their inability to generalise to new environments and inconsistent performance. Models need to be developed for specific species cohorts and environments, but the technical skills required to achieve this are a key barrier to the accessibility of this technology to ecologists. Thus, there is a strong need to democratize access to deep learning technologies by providing an easy-to-use software application allowing non-technical users to train custom object detectors. U-Infuse addresses this issue by providing ecologists with the ability to train customised models using publicly available images and/or their own images without specific technical expertise. Auto-annotation and annotation editing functionalities minimize the constraints of manually annotating and pre-processing large numbers of images. U-Infuse is a free and open-source software solution that supports both multiclass and single class training and object detection, allowing ecologists to access deep learning technologies usually only available to computer scientists, on their own device, customised for their application, without sharing intellectual property or sensitive data. It provides ecological practitioners with the ability to (i) easily achieve object detection within a user-friendly GUI, generating a species distribution report, and other useful statistics, (ii) custom train deep learning models using publicly available and custom training data, (iii) achieve supervised auto-annotation of images for further training, with the benefit of editing annotations to ensure quality datasets. Broad adoption of U-Infuse by ecological practitioners will improve ecological image analysis and processing by allowing significantly more image data to be processed with minimal expenditure of time and resources, particularly for camera trap images. Ease of training and use of transfer learning means domain-specific models can be trained rapidly, and frequently updated without the need for computer science expertise, or data sharing, protecting intellectual property and privacy.

Download Full-text

Pansformers: Transformer-Based Self-Attention Network for Pansharpening

10.36227/techrxiv.17153228.v1 ◽

2021 ◽

Author(s):

Nithin G R ◽

Nitish Kumar M ◽

Venkateswaran Narasimhan ◽

Rajanikanth Kakani ◽

Ujjwal Gupta ◽

...

Keyword(s):

Deep Learning ◽

High Resolution ◽

Performance Metrics ◽

Satellite Image ◽

Image Data ◽

Landsat 8 ◽

Learning Models ◽

Attention Network ◽

Panchromatic Image ◽

Recent Success

Pansharpening is the task of creating a High-Resolution Multi-Spectral Image (HRMS) by extracting and infusing pixel details from the High-Resolution Panchromatic Image into the Low-Resolution Multi-Spectral (LRMS). With the boom in the amount of satellite image data, researchers have replaced traditional approaches with deep learning models. However, existing deep learning models are not built to capture intricate pixel-level relationships. Motivated by the recent success of self-attention mechanisms in computer vision tasks, we propose Pansformers, a transformer-based self-attention architecture, that computes band-wise attention. A further improvement is proposed in the attention network by introducing a Multi-Patch Attention mechanism, which operates on non-overlapping, local patches of the image. Our model is successful in infusing relevant local details from the Panchromatic image while preserving the spectral integrity of the MS image. We show that our Pansformer model significantly improves the performance metrics and the output image quality on imagery from two satellite distributions IKONOS and LANDSAT-8.

Download Full-text

The Impact of Digital Histopathology Batch Effect on Deep Learning Model Accuracy and Bias

10.1101/2020.12.03.410845 ◽

2020 ◽

Author(s):

Frederick M. Howard ◽

James Dolezal ◽

Sara Kochanny ◽

Jefree Schulte ◽

Heather Chen ◽

...

Keyword(s):

Deep Learning ◽

Expression Patterns ◽

Model Performance ◽

Batch Effect ◽

The Cancer Genome Atlas ◽

Driver Mutations ◽

Cancer Subtypes ◽

Breast Cancer Cohort ◽

Quadratic Programming Method ◽

The Impact

AbstractThe Cancer Genome Atlas (TCGA) is one of the largest biorepositories of digital histology. Deep learning (DL) models have been trained on TCGA to predict numerous features directly from histology, including survival, gene expression patterns, and driver mutations. However, we demonstrate that these features vary substantially across tissue submitting sites in TCGA for over 3,000 patients with six cancer subtypes. Additionally, we show that histologic image differences between submitting sites can easily be identified with DL. This site detection remains possible despite commonly used color normalization and augmentation methods, and we quantify the digital image characteristics constituting this histologic batch effect. As an example, we show that patient ethnicity within the TCGA breast cancer cohort can be inferred from histology due to site-level batch effect, which must be accounted for to ensure equitable application of DL. Batch effect also leads to overoptimistic estimates of model performance, and we propose a quadratic programming method to guide validation that abrogates this bias.

Download Full-text

Improving generalization of deep learning models for diagnostic pathology by increasing variability in training data: experiments on osteosarcoma subtypes

10.1101/2020.09.10.20192294 ◽

2020 ◽

Author(s):

Haiming Tang ◽

Nanfei Sun ◽

Steven Shen

Keyword(s):

Deep Learning ◽

Model Performance ◽

High Variability ◽

Training Data ◽

Classification Model ◽

Training Dataset ◽

Learning Models ◽

Diagnostic Pathology ◽

Model Generalization ◽

Histopathological Images

Artificial intelligence (AI) has an emerging progress in diagnostic pathology. A large number of studies of applying deep learning models to histopathological images have been published in recent years. While many studies claim high accuracies, they may fall into the pitfalls of overfitting and lack of generalization due to the high variability of the histopathological images. We use the example of Osteosarcoma to illustrate the pitfalls and how the addition of model input variability can help improve model performance. We use the publicly available osteosarcoma dataset to retrain a previously published classification model for osteosarcoma. We partition the same set of images into the training and testing datasets differently than the original study: the test dataset consists of images from one patient while the training dataset consists images of all other patients. The performance of the model on the test set using the new partition schema declines dramatically, indicating a lack of model generalization and overfitting.We also show the influence of training data variability on model performance by collecting a minimal dataset of 10 osteosarcoma subtypes as well as benign tissues and benign bone tumors of differentiation. We show the additions of more and more subtypes into the training data step by step under the same model schema yield a series of coherent models with increasing performances. In conclusion, we bring forward data preprocessing and collection tactics for histopathological images of high variability to avoid the pitfalls of overfitting and build deep learning models of higher generalization abilities.

Download Full-text

Automated Analysis of Three-dimensional CBCT Images Taken in Natural Head Position That Combines Facial Profile Processing and Multiple Deep-learning Models

10.21203/rs.3.rs-795041/v1 ◽

2021 ◽

Author(s):

Janghoon Ahn ◽

Thong Phi Nguyen ◽

Yoon-Ji Kim ◽

Taeyong Kim ◽

Jonghun Yoon

Keyword(s):

Deep Learning ◽

Profile Analysis ◽

Traditional Approach ◽

Three Dimensional ◽

Image Data ◽

Automated Analysis ◽

Head Position ◽

X Rays ◽

Learning Models ◽

Facial Profile

Abstract Analysing cephalometric X-rays, which is mostly performed by orthodontists or dentists, is an indispensable procedure for diagnosis and treatment planning with orthodontic patients. Artificial intelligence, especially deep-learning techniques for analysing image data, shows great potential for medical and dental image analysis and diagnosis. To explore the feasibility of automating measurement of 13 geometric parameters from three-dimensional cone beam computed tomography (CBCT) images taken in a natural head position, we here describe a smart system that combines a facial profile analysis algorithm with deep-learning models. Using multiple views extracted from the CBCT data as the dataset, our proposed method partitions and detects regions of interest by extracting the facial profile and applying Mask-RCNN, a trained decentralized convolutional neural network (CNN) that positions the key parameters. All the techniques are integrated into a software application with a graphical user interface designed for user convenience. To demonstrate the system’s ability to replace human experts, we validated the performance of the proposed method by comparing it with measurements made by two orthodontists and one advanced general dentist using a commercial dental program. The time savings compared with the traditional approach was remarkable, reducing the processing time from about 30 minutes to about 30 seconds.

Download Full-text

Explaining Deep Learning Models for Tabular Data Using Layer-Wise Relevance Propagation

Applied Sciences ◽

10.3390/app12010136 ◽

2021 ◽

Vol 12 (1) ◽

pp. 136

Author(s):

Ihsan Ullah ◽

Andre Rios ◽

Vaibhav Gala ◽

Susan Mckeever

Keyword(s):

Deep Learning ◽

Credit Card ◽

Model Performance ◽

Feature Subset Selection ◽

Mixed Data ◽

Computational Time ◽

Feature Subset ◽

Learning Models ◽

Customer Churn ◽

Interpretable Model

Trust and credibility in machine learning models are bolstered by the ability of a model to explain its decisions. While explainability of deep learning models is a well-known challenge, a further challenge is clarity of the explanation itself for relevant stakeholders of the model. Layer-wise Relevance Propagation (LRP), an established explainability technique developed for deep models in computer vision, provides intuitive human-readable heat maps of input images. We present the novel application of LRP with tabular datasets containing mixed data (categorical and numerical) using a deep neural network (1D-CNN), for Credit Card Fraud detection and Telecom Customer Churn prediction use cases. We show how LRP is more effective than traditional explainability concepts of Local Interpretable Model-agnostic Explanations (LIME) and Shapley Additive Explanations (SHAP) for explainability. This effectiveness is both local to a sample level and holistic over the whole testing set. We also discuss the significant computational time advantage of LRP (1–2 s) over LIME (22 s) and SHAP (108 s) on the same laptop, and thus its potential for real time application scenarios. In addition, our validation of LRP has highlighted features for enhancing model performance, thus opening up a new area of research of using XAI as an approach for feature subset selection.

Download Full-text

Training confounder-free deep learning models for medical applications

Nature Communications ◽

10.1038/s41467-020-19784-9 ◽

2020 ◽

Vol 11 (1) ◽

Author(s):

Qingyu Zhao ◽

Ehsan Adeli ◽

Kilian M. Pohl

Keyword(s):

Machine Learning ◽

Deep Learning ◽

Bone Age ◽

Magnetic Resonance Images ◽

Data Sets ◽

Large Set ◽

Learning Models ◽

End To End ◽

The Impact ◽

The Relationship

AbstractThe presence of confounding effects (or biases) is one of the most critical challenges in using deep learning to advance discovery in medical imaging studies. Confounders affect the relationship between input data (e.g., brain MRIs) and output variables (e.g., diagnosis). Improper modeling of those relationships often results in spurious and biased associations. Traditional machine learning and statistical models minimize the impact of confounders by, for example, matching data sets, stratifying data, or residualizing imaging measurements. Alternative strategies are needed for state-of-the-art deep learning models that use end-to-end training to automatically extract informative features from large set of images. In this article, we introduce an end-to-end approach for deriving features invariant to confounding factors while accounting for intrinsic correlations between the confounder(s) and prediction outcome. The method does so by exploiting concepts from traditional statistical methods and recent fair machine learning schemes. We evaluate the method on predicting the diagnosis of HIV solely from Magnetic Resonance Images (MRIs), identifying morphological sex differences in adolescence from those of the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA), and determining the bone age from X-ray images of children. The results show that our method can accurately predict while reducing biases associated with confounders. The code is available at https://github.com/qingyuzhao/br-net.

Download Full-text

Optimising Deep Learning at the Edge for Accurate Hourly Air Quality Prediction

Sensors ◽

10.3390/s21041064 ◽

2021 ◽

Vol 21 (4) ◽

pp. 1064

Author(s):

I Nyoman Kusuma Wardana ◽

Julian W. Gardner ◽

Suhaib A. Fahmy

Keyword(s):

Deep Learning ◽

Air Quality ◽

Short Term Memory ◽

Model Performance ◽

Learning Model ◽

Sensor Data ◽

Raspberry Pi ◽

Learning Models ◽

Proposed Model ◽

Deep Learning Model

Accurate air quality monitoring requires processing of multi-dimensional, multi-location sensor data, which has previously been considered in centralised machine learning models. These are often unsuitable for resource-constrained edge devices. In this article, we address this challenge by: (1) designing a novel hybrid deep learning model for hourly PM2.5 pollutant prediction; (2) optimising the obtained model for edge devices; and (3) examining model performance running on the edge devices in terms of both accuracy and latency. The hybrid deep learning model in this work comprises a 1D Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) to predict hourly PM2.5 concentration. The results show that our proposed model outperforms other deep learning models, evaluated by calculating RMSE and MAE errors. The proposed model was optimised for edge devices, the Raspberry Pi 3 Model B+ (RPi3B+) and Raspberry Pi 4 Model B (RPi4B). This optimised model reduced file size to a quarter of the original, with further size reduction achieved by implementing different post-training quantisation. In total, 8272 hourly samples were continuously fed to the edge device, with the RPi4B executing the model twice as fast as the RPi3B+ in all quantisation modes. Full-integer quantisation produced the lowest execution time, with latencies of 2.19 s and 4.73 s for RPi4B and RPi3B+, respectively.

Download Full-text

Novel Transfer Learning Approach for Medical Imaging with Limited Labeled Data

Cancers ◽

10.3390/cancers13071590 ◽

2021 ◽

Vol 13 (7) ◽

pp. 1590

Author(s):

Laith Alzubaidi ◽

Muthana Al-Amidie ◽

Ahmed Al-Asadi ◽

Amjad J. Humaidi ◽

Omran Al-Shamma ◽

...

Keyword(s):

Deep Learning ◽

Medical Imaging ◽

Transfer Learning ◽

Medical Image ◽

Medical Images ◽

Image Data ◽

Learning Approach ◽

Learning Models ◽

Annotation Process ◽

Deep Learning Model

Deep learning requires a large amount of data to perform well. However, the field of medical image analysis suffers from a lack of sufficient data for training deep learning models. Moreover, medical images require manual labeling, usually provided by human annotators coming from various backgrounds. More importantly, the annotation process is time-consuming, expensive, and prone to errors. Transfer learning was introduced to reduce the need for the annotation process by transferring the deep learning models with knowledge from a previous task and then by fine-tuning them on a relatively small dataset of the current task. Most of the methods of medical image classification employ transfer learning from pretrained models, e.g., ImageNet, which has been proven to be ineffective. This is due to the mismatch in learned features between the natural image, e.g., ImageNet, and medical images. Additionally, it results in the utilization of deeply elaborated models. In this paper, we propose a novel transfer learning approach to overcome the previous drawbacks by means of training the deep learning model on large unlabeled medical image datasets and by next transferring the knowledge to train the deep learning model on the small amount of labeled medical images. Additionally, we propose a new deep convolutional neural network (DCNN) model that combines recent advancements in the field. We conducted several experiments on two challenging medical imaging scenarios dealing with skin and breast cancer classification tasks. According to the reported results, it has been empirically proven that the proposed approach can significantly improve the performance of both classification scenarios. In terms of skin cancer, the proposed model achieved an F1-score value of 89.09% when trained from scratch and 98.53% with the proposed approach. Secondly, it achieved an accuracy value of 85.29% and 97.51%, respectively, when trained from scratch and using the proposed approach in the case of the breast cancer scenario. Finally, we concluded that our method can possibly be applied to many medical imaging problems in which a substantial amount of unlabeled image data is available and the labeled image data is limited. Moreover, it can be utilized to improve the performance of medical imaging tasks in the same domain. To do so, we used the pretrained skin cancer model to train on feet skin to classify them into two classes—either normal or abnormal (diabetic foot ulcer (DFU)). It achieved an F1-score value of 86.0% when trained from scratch, 96.25% using transfer learning, and 99.25% using double-transfer learning.

Download Full-text