Deep Metric Learning with Online Hard Mining for Hyperspectral Classification

Recently, deep learning has developed rapidly, while it has also been quite successfully applied in the field of hyperspectral classification. Generally, training the parameters of a deep neural network to the best is the core step of a deep learning-based method, which usually requires a large number of labeled samples. However, in remote sensing analysis tasks, we only have limited labeled data because of the high cost of their collection. Therefore, in this paper, we propose a deep metric learning with online hard mining (DMLOHM) method for hyperspectral classification, which can maximize the inter-class distance and minimize the intra-class distance, utilizing a convolutional neural network (CNN) as an embedded network. First of all, we utilized the triplet network to learn better representations of raw data so that raw data were capable of having their dimensionality reduced. Afterward, an online hard mining method was used to mine the most valuable information from the limited hyperspectral data. To verify the performance of the proposed DMLOHM, we utilized three well-known hyperspectral datasets: Salinas Scene, Pavia University, and HyRANK for verification. Compared with CNN and DMLTN, the experimental results showed that the proposed method improved the classification accuracy from 0.13% to 4.03% with 85 labeled samples per class.

Download Full-text

Predicting TCR-Epitope Binding Specificity Using Deep Metric Learning and Multimodal Learning

Genes ◽

10.3390/genes12040572 ◽

2021 ◽

Vol 12 (4) ◽

pp. 572

Author(s):

Alan M. Luu ◽

Jacob R. Leistico ◽

Tim Miller ◽

Somang Kim ◽

Jun S. Song

Keyword(s):

Neural Network ◽

Amino Acid ◽

Cytotoxic T Cells ◽

Metric Learning ◽

Binding Specificity ◽

Class I ◽

Multimodal Learning ◽

Binding Prediction ◽

Deep Metric Learning ◽

Epitope Binding

Understanding the recognition of specific epitopes by cytotoxic T cells is a central problem in immunology. Although predicting binding between peptides and the class I Major Histocompatibility Complex (MHC) has had success, predicting interactions between T cell receptors (TCRs) and MHC class I-peptide complexes (pMHC) remains elusive. This paper utilizes a convolutional neural network model employing deep metric learning and multimodal learning to perform two critical tasks in TCR-epitope binding prediction: identifying the TCRs that bind a given epitope from a TCR repertoire, and identifying the binding epitope of a given TCR from a list of candidate epitopes. Our model can perform both tasks simultaneously and reveals that inconsistent preprocessing of TCR sequences can confound binding prediction. Applying a neural network interpretation method identifies key amino acid sequence patterns and positions within the TCR, important for binding specificity. Contrary to common assumption, known crystal structures of TCR-pMHC complexes show that the predicted salient amino acid positions are not necessarily the closest to the epitopes, implying that physical proximity may not be a good proxy for importance in determining TCR-epitope specificity. Our work thus provides an insight into the learned predictive features of TCR-epitope binding specificity and advances the associated classification tasks.

Download Full-text

Unsupervised Multi-Level Feature Extraction for Improvement of Hyperspectral Classification

Remote Sensing ◽

10.3390/rs13081602 ◽

2021 ◽

Vol 13 (8) ◽

pp. 1602

Author(s):

Qiaoqiao Sun ◽

Xuefeng Liu ◽

Salah Bourennane

Keyword(s):

Feature Extraction ◽

Deep Learning ◽

Spatial Information ◽

Hyperspectral Data ◽

Great Promise ◽

Learning Models ◽

Single Level ◽

Multiple Networks ◽

Multi Level ◽

Hyperspectral Classification

Deep learning models have strong abilities in learning features and they have been successfully applied in hyperspectral images (HSIs). However, the training of most deep learning models requires labeled samples and the collection of labeled samples are labor-consuming in HSI. In addition, single-level features from a single layer are usually considered, which may result in the loss of some important information. Using multiple networks to obtain multi-level features is a solution, but at the cost of longer training time and computational complexity. To solve these problems, a novel unsupervised multi-level feature extraction framework that is based on a three dimensional convolutional autoencoder (3D-CAE) is proposed in this paper. The designed 3D-CAE is stacked by fully 3D convolutional layers and 3D deconvolutional layers, which allows for the spectral-spatial information of targets to be mined simultaneously. Besides, the 3D-CAE can be trained in an unsupervised way without involving labeled samples. Moreover, the multi-level features are directly obtained from the encoded layers with different scales and resolutions, which is more efficient than using multiple networks to get them. The effectiveness of the proposed multi-level features is verified on two hyperspectral data sets. The results demonstrate that the proposed method has great promise in unsupervised feature learning and can help us to further improve the hyperspectral classification when compared with single-level features.

Download Full-text

Detection and Severity Evaluation of Combined Rail Defects Using Deep Learning

Vibration ◽

10.3390/vibration4020022 ◽

2021 ◽

Vol 4 (2) ◽

pp. 341-356

Author(s):

Jessada Sresakoolchai ◽

Sakdirat Kaewunruen

Keyword(s):

Neural Network ◽

Machine Learning ◽

Deep Learning ◽

Mean Absolute Error ◽

Absolute Error ◽

Machine Learning Techniques ◽

Rolling Stock ◽

Raw Data ◽

Learning Techniques ◽

Combined Defects

Various techniques have been developed to detect railway defects. One of the popular techniques is machine learning. This unprecedented study applies deep learning, which is a branch of machine learning techniques, to detect and evaluate the severity of rail combined defects. The combined defects in the study are settlement and dipped joint. Features used to detect and evaluate the severity of combined defects are axle box accelerations simulated using a verified rolling stock dynamic behavior simulation called D-Track. A total of 1650 simulations are run to generate numerical data. Deep learning techniques used in the study are deep neural network (DNN), convolutional neural network (CNN), and recurrent neural network (RNN). Simulated data are used in two ways: simplified data and raw data. Simplified data are used to develop the DNN model, while raw data are used to develop the CNN and RNN model. For simplified data, features are extracted from raw data, which are the weight of rolling stock, the speed of rolling stock, and three peak and bottom accelerations from two wheels of rolling stock. In total, there are 14 features used as simplified data for developing the DNN model. For raw data, time-domain accelerations are used directly to develop the CNN and RNN models without processing and data extraction. Hyperparameter tuning is performed to ensure that the performance of each model is optimized. Grid search is used for performing hyperparameter tuning. To detect the combined defects, the study proposes two approaches. The first approach uses one model to detect settlement and dipped joint, and the second approach uses two models to detect settlement and dipped joint separately. The results show that the CNN models of both approaches provide the same accuracy of 99%, so one model is good enough to detect settlement and dipped joint. To evaluate the severity of the combined defects, the study applies classification and regression concepts. Classification is used to evaluate the severity by categorizing defects into light, medium, and severe classes, and regression is used to estimate the size of defects. From the study, the CNN model is suitable for evaluating dipped joint severity with an accuracy of 84% and mean absolute error (MAE) of 1.25 mm, and the RNN model is suitable for evaluating settlement severity with an accuracy of 99% and mean absolute error (MAE) of 1.58 mm.

Download Full-text

Early Detection of Plant Viral Disease Using Hyperspectral Imaging and Deep Learning

Sensors ◽

10.3390/s21030742 ◽

2021 ◽

Vol 21 (3) ◽

pp. 742

Author(s):

Canh Nguyen ◽

Vasit Sagan ◽

Matthew Maimaitiyiming ◽

Maitiniyazi Maimaitijiang ◽

Sourav Bhadra ◽

...

Keyword(s):

Neural Network ◽

Deep Learning ◽

Early Detection ◽

Convolutional Neural Network ◽

Near Infrared ◽

Hyperspectral Data ◽

Viral Diseases ◽

Support Vector ◽

Spectral Features ◽

Feature Spaces

Early detection of grapevine viral diseases is critical for early interventions in order to prevent the disease from spreading to the entire vineyard. Hyperspectral remote sensing can potentially detect and quantify viral diseases in a nondestructive manner. This study utilized hyperspectral imagery at the plant level to identify and classify grapevines inoculated with the newly discovered DNA virus grapevine vein-clearing virus (GVCV) at the early asymptomatic stages. An experiment was set up at a test site at South Farm Research Center, Columbia, MO, USA (38.92 N, −92.28 W), with two grapevine groups, namely healthy and GVCV-infected, while other conditions were controlled. Images of each vine were captured by a SPECIM IQ 400–1000 nm hyperspectral sensor (Oulu, Finland). Hyperspectral images were calibrated and preprocessed to retain only grapevine pixels. A statistical approach was employed to discriminate two reflectance spectra patterns between healthy and GVCV vines. Disease-centric vegetation indices (VIs) were established and explored in terms of their importance to the classification power. Pixel-wise (spectral features) classification was performed in parallel with image-wise (joint spatial–spectral features) classification within a framework involving deep learning architectures and traditional machine learning. The results showed that: (1) the discriminative wavelength regions included the 900–940 nm range in the near-infrared (NIR) region in vines 30 days after sowing (DAS) and the entire visual (VIS) region of 400–700 nm in vines 90 DAS; (2) the normalized pheophytization index (NPQI), fluorescence ratio index 1 (FRI1), plant senescence reflectance index (PSRI), anthocyanin index (AntGitelson), and water stress and canopy temperature (WSCT) measures were the most discriminative indices; (3) the support vector machine (SVM) was effective in VI-wise classification with smaller feature spaces, while the RF classifier performed better in pixel-wise and image-wise classification with larger feature spaces; and (4) the automated 3D convolutional neural network (3D-CNN) feature extractor provided promising results over the 2D convolutional neural network (2D-CNN) in learning features from hyperspectral data cubes with a limited number of samples.

Download Full-text

Comparative Analysis on Machine Learning and Deep Learning to Predict Post-Induction Hypotension

Sensors ◽

10.3390/s20164575 ◽

2020 ◽

Vol 20 (16) ◽

pp. 4575 ◽

Cited By ~ 1

Author(s):

Jihyun Lee ◽

Jiyoung Woo ◽

Ah Reum Kang ◽

Young-Seob Jeong ◽

Woohyun Jung ◽

...

Keyword(s):

Neural Network ◽

Machine Learning ◽

Feature Selection ◽

Deep Learning ◽

Random Forest ◽

Tracheal Intubation ◽

Feature Engineering ◽

Learning Models ◽

Raw Data ◽

Vital Records

Hypotensive events in the initial stage of anesthesia can cause serious complications in the patients after surgery, which could be fatal. In this study, we intended to predict hypotension after tracheal intubation using machine learning and deep learning techniques after intubation one minute in advance. Meta learning models, such as random forest, extreme gradient boosting (Xgboost), and deep learning models, especially the convolutional neural network (CNN) model and the deep neural network (DNN), were trained to predict hypotension occurring between tracheal intubation and incision, using data from four minutes to one minute before tracheal intubation. Vital records and electronic health records (EHR) for 282 of 319 patients who underwent laparoscopic cholecystectomy from October 2018 to July 2019 were collected. Among the 282 patients, 151 developed post-induction hypotension. Our experiments had two scenarios: using raw vital records and feature engineering on vital records. The experiments on raw data showed that CNN had the best accuracy of 72.63%, followed by random forest (70.32%) and Xgboost (64.6%). The experiments on feature engineering showed that random forest combined with feature selection had the best accuracy of 74.89%, while CNN had a lower accuracy of 68.95% than that of the experiment on raw data. Our study is an extension of previous studies to detect hypotension before intubation with a one-minute advance. To improve accuracy, we built a model using state-of-art algorithms. We found that CNN had a good performance, but that random forest had a better performance when combined with feature selection. In addition, we found that the examination period (data period) is also important.

Download Full-text

Blood Stain Classification with Hyperspectral Imaging and Deep Neural Networks

Sensors ◽

10.3390/s20226666 ◽

2020 ◽

Vol 20 (22) ◽

pp. 6666

Author(s):

Kamil Książek ◽

Michał Romaszewski ◽

Przemysław Głomb ◽

Bartosz Grabowski ◽

Michał Cholewa

Keyword(s):

Neural Network ◽

Neural Networks ◽

Deep Learning ◽

Hyperspectral Imaging ◽

Network Architecture ◽

Confusion Matrix ◽

Hyperspectral Data ◽

Matrix Analysis ◽

Support Vector ◽

Test Set

In recent years, growing interest in deep learning neural networks has raised a question on how they can be used for effective processing of high-dimensional datasets produced by hyperspectral imaging (HSI). HSI, traditionally viewed as being within the scope of remote sensing, is used in non-invasive substance classification. One of the areas of potential application is forensic science, where substance classification on the scenes is important. An example problem from that area—blood stain classification—is a case study for the evaluation of methods that process hyperspectral data. To investigate the deep learning classification performance for this problem we have performed experiments on a dataset which has not been previously tested using this kind of model. This dataset consists of several images with blood and blood-like substances like ketchup, tomato concentrate, artificial blood, etc. To test both the classic approach to hyperspectral classification and a more realistic application-oriented scenario, we have prepared two different sets of experiments. In the first one, Hyperspectral Transductive Classification (HTC), both a training and a test set come from the same image. In the second one, Hyperspectral Inductive Classification (HIC), a test set is derived from a different image, which is more challenging for classifiers but more useful from the point of view of forensic investigators. We conducted the study using several architectures like 1D, 2D and 3D convolutional neural networks (CNN), a recurrent neural network (RNN) and a multilayer perceptron (MLP). The performance of the models was compared with baseline results of Support Vector Machine (SVM). We have also presented a model evaluation method based on t-SNE and confusion matrix analysis that allows us to detect and eliminate some cases of model undertraining. Our results show that in the transductive case, all models, including the MLP and the SVM, have comparative performance, with no clear advantage of deep learning models. The Overall Accuracy range across all models is 98–100% for the easier image set, and 74–94% for the more difficult one. However, in a more challenging inductive case, selected deep learning architectures offer a significant advantage; their best Overall Accuracy is in the range of 57–71%, improving the baseline set by the non-deep models by up to 9 percentage points. We have presented a detailed analysis of results and a discussion, including a summary of conclusions for each tested architecture. An analysis of per-class errors shows that the score for each class is highly model-dependent. Considering this and the fact that the best performing models come from two different architecture families (3D CNN and RNN), our results suggest that tailoring the deep neural network architecture to hyperspectral data is still an open problem.

Download Full-text

Enhancing Multi-tissue and Multi-scale Cell Nuclei Segmentation with Deep Metric Learning

Applied Sciences ◽

10.3390/app10020615 ◽

2020 ◽

Vol 10 (2) ◽

pp. 615 ◽

Cited By ~ 2

Author(s):

Tomas Iesmantas ◽

Agne Paulauskaite-Taraseviciene ◽

Kristina Sutiene

Keyword(s):

Deep Learning ◽

Large Scale ◽

Metric Learning ◽

Cell Nuclei ◽

Similarity Coefficients ◽

Clinical Practices ◽

Nuclei Segmentation ◽

Wide Range ◽

Triplet Loss ◽

Deep Metric Learning

(1) Background: The segmentation of cell nuclei is an essential task in a wide range of biomedical studies and clinical practices. The full automation of this process remains a challenge due to intra- and internuclear variations across a wide range of tissue morphologies, differences in staining protocols and imaging procedures. (2) Methods: A deep learning model with metric embeddings such as contrastive loss and triplet loss with semi-hard negative mining is proposed in order to accurately segment cell nuclei in a diverse set of microscopy images. The effectiveness of the proposed model was tested on a large-scale multi-tissue collection of microscopy image sets. (3) Results: The use of deep metric learning increased the overall segmentation prediction by 3.12% in the average value of Dice similarity coefficients as compared to no metric learning. In particular, the largest gain was observed for segmenting cell nuclei in H&E -stained images when deep learning network and triplet loss with semi-hard negative mining were considered for the task. (4) Conclusion: We conclude that deep metric learning gives an additional boost to the overall learning process and consequently improves the segmentation performance. Notably, the improvement ranges approximately between 0.13% and 22.31% for different types of images in the terms of Dice coefficients when compared to no metric deep learning.

Download Full-text

Feature Learning Approach for Facial Recognition Using Deep Metric Learning

Journal of Computational and Theoretical Nanoscience ◽

10.1166/jctn.2020.9031 ◽

2020 ◽

Vol 17 (9) ◽

pp. 4125-4130

Author(s):

Gaurav Karkal ◽

K. Dhanush Reddy ◽

Kaushik Singh ◽

Nikith Hosangadi ◽

Annapurna P. Patil

Keyword(s):

Neural Network ◽

Facial Recognition ◽

Metric Learning ◽

Feature Learning ◽

Human Interaction ◽

Single Image ◽

Whole Process ◽

Knn Classifier ◽

Deep Metric Learning ◽

Trained Neural Network

Standard deep learning in the context of facial recognition involves inputting a single image and outputting a label for that image. Deep metric learning distinguishes itself by outputting a real valued feature vector instead of a single label. The usage of deep metric learning has revolutionised facial recognition, making it very accurate and reliable. This paper exhibits the accuracy and reliability of the facial recognition model using deep metric learning in the application of an automated attendance system. The paper presents a non-intrusive attendance system which uses the described neural network to recognize faces and record attendance. The system uses the pre-trained neural network to generate embeddings for faces, using a method known as the triple training step, which is described in the paper. These embeddings are generated from a collection of photos per person. After the embeddings are generated, the system is ready to perform facial recognition on sample photos. CNN is used for facial detection in the sample group photos. Once the faces are detected, a KNN classifier is used for recognizing faces. Finally after the faces are recognized, the attendance for each recognized student is marked in the database. Thus, the whole process of attendance was automated without the requirement of human interaction.

Download Full-text

Investigation of optimal configurations of a convolutional neural network for the identification of objects in real-time

Information Technology and Nanotechnology ◽

10.18287/1613-0073-2019-2416-417-423 ◽

2019 ◽

pp. 417-423

Author(s):

M A Isayev ◽

D A Savelyev

Keyword(s):

Neural Network ◽

Neural Networks ◽

Deep Learning ◽

Convolutional Neural Network ◽

Real Time ◽

State Of The Art ◽

Average Precision ◽

The Core ◽

Particular Solution ◽

Optimal Configurations

The comparison of different convolutional neural networks which are the core of the most actual solutions in the computer vision area is considers in hhe paper. The study includes benchmarks of this state-of-the-art solutions by some criteria, such as mAP (mean average precision), FPS (frames per seconds), for the possibility of real-time usability. It is concluded on the best convolutional neural network model and deep learning methods that were used at particular solution.

Download Full-text

Hyperspectral imaging and artificial intelligence to detect oral malignancy – part 1 - automated tissue classification of oral muscle, fat and mucosa using a light-weight 6-layer deep neural network

Head & Face Medicine ◽

10.1186/s13005-021-00292-0 ◽

2021 ◽

Vol 17 (1) ◽

Author(s):

Daniel G. E. Thiem ◽

Paul Römer ◽

Matthias Gielisch ◽

Bilal Al-Nawas ◽

Martin Schlüter ◽

...

Keyword(s):

Neural Network ◽

Computer Vision ◽

Deep Learning ◽

Hyperspectral Imaging ◽

Oral Mucosa ◽

Deep Neural Network ◽

Light Weight ◽

Tissue Classification ◽

Raw Data

Abstract Background Hyperspectral imaging (HSI) is a promising non-contact approach to tissue diagnostics, generating large amounts of raw data for whose processing computer vision (i.e. deep learning) is particularly suitable. Aim of this proof of principle study was the classification of hyperspectral (HS)-reflectance values into the human-oral tissue types fat, muscle and mucosa using deep learning methods. Furthermore, the tissue-specific hyperspectral signatures collected will serve as a representative reference for the future assessment of oral pathological changes in the sense of a HS-library. Methods A total of about 316 samples of healthy human-oral fat, muscle and oral mucosa was collected from 174 different patients and imaged using a HS-camera, covering the wavelength range from 500 nm to 1000 nm. HS-raw data were further labelled and processed for tissue classification using a light-weight 6-layer deep neural network (DNN). Results The reflectance values differed significantly (p < .001) for fat, muscle and oral mucosa at almost all wavelengths, with the signature of muscle differing the most. The deep neural network distinguished tissue types with an accuracy of > 80% each. Conclusion Oral fat, muscle and mucosa can be classified sufficiently and automatically by their specific HS-signature using a deep learning approach. Early detection of premalignant-mucosal-lesions using hyperspectral imaging and deep learning is so far represented rarely in in medical and computer vision research domain but has a high potential and is part of subsequent studies.

Download Full-text