Layer-Level Knowledge Distillation for Deep Neural Network Learning

Motivated by the recently developed distillation approaches that aim to obtain small and fast-to-execute models, in this paper a novel Layer Selectivity Learning (LSL) framework is proposed for learning deep models. We firstly use an asymmetric dual-model learning framework, called Auxiliary Structure Learning (ASL), to train a small model with the help of a larger and well-trained model. Then, the intermediate layer selection scheme, called the Layer Selectivity Procedure (LSP), is exploited to determine the corresponding intermediate layers of source and target models. The LSP is achieved by two novel matrices, the layered inter-class Gram matrix and the inter-layered Gram matrix, to evaluate the diversity and discrimination of feature maps. The experimental results, demonstrated using three publicly available datasets, present the superior performance of model training using the LSL deep model learning framework.

Download Full-text

Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i07.6963 ◽

2020 ◽

Vol 34 (07) ◽

pp. 12701-12708

Author(s):

Yingruo Fan ◽

Jacqueline Lam ◽

Victor Li

Keyword(s):

Probabilistic Models ◽

Superior Performance ◽

Feature Maps ◽

Dynamic Graph ◽

Intensity Estimation ◽

Facial Action ◽

Learning Framework ◽

Benchmark Datasets ◽

Relationship Of ◽

Semantic Correspondence

The intensity estimation of facial action units (AUs) is challenging due to subtle changes in the person's facial appearance. Previous approaches mainly rely on probabilistic models or predefined rules for modeling co-occurrence relationships among AUs, leading to limited generalization. In contrast, we present a new learning framework that automatically learns the latent relationships of AUs via establishing semantic correspondences between feature maps. In the heatmap regression-based network, feature maps preserve rich semantic information associated with AU intensities and locations. Moreover, the AU co-occurring pattern can be reflected by activating a set of feature channels, where each channel encodes a specific visual pattern of AU. This motivates us to model the correlation among feature channels, which implicitly represents the co-occurrence relationship of AU intensity levels. Specifically, we introduce a semantic correspondence convolution (SCC) module to dynamically compute the correspondences from deep and low resolution feature maps, and thus enhancing the discriminability of features. The experimental results demonstrate the effectiveness and the superior performance of our method on two benchmark datasets.

Download Full-text

A Lightweight Fusion Distillation Network for Image Deblurring and Deraining

Sensors ◽

10.3390/s21165312 ◽

2021 ◽

Vol 21 (16) ◽

pp. 5312

Author(s):

Yanni Zhang ◽

Yiming Liu ◽

Qiang Li ◽

Jianzhong Wang ◽

Miao Qi ◽

...

Keyword(s):

Deep Learning ◽

Image Deblurring ◽

Image Features ◽

Image Feature ◽

Model Complexity ◽

Small Scale ◽

Feature Maps ◽

Learning Framework ◽

Channel Information ◽

Scale Spaces

Recently, deep learning-based image deblurring and deraining have been well developed. However, most of these methods fail to distill the useful features. What is more, exploiting the detailed image features in a deep learning framework always requires a mass of parameters, which inevitably makes the network suffer from a high computational burden. We propose a lightweight fusion distillation network (LFDN) for image deblurring and deraining to solve the above problems. The proposed LFDN is designed as an encoder–decoder architecture. In the encoding stage, the image feature is reduced to various small-scale spaces for multi-scale information extraction and fusion without much information loss. Then, a feature distillation normalization block is designed at the beginning of the decoding stage, which enables the network to distill and screen valuable channel information of feature maps continuously. Besides, an information fusion strategy between distillation modules and feature channels is also carried out by the attention mechanism. By fusing different information in the proposed approach, our network can achieve state-of-the-art image deblurring and deraining results with a smaller number of parameters and outperform the existing methods in model complexity.

Download Full-text

HMD-ARG: hierarchical multi-task deep learning for annotating antibiotic resistance genes

Microbiome ◽

10.1186/s40168-021-01002-3 ◽

2021 ◽

Vol 9 (1) ◽

Author(s):

Yu Li ◽

Zeling Xu ◽

Wenkai Han ◽

Huiluo Cao ◽

Ramzan Umarov ◽

...

Keyword(s):

Antibiotic Resistance ◽

Deep Learning ◽

Resistance Genes ◽

Antibiotic Resistance Genes ◽

Third Party ◽

Superior Performance ◽

Beta Lactamase ◽

Learning Framework ◽

Sequence Encoding ◽

Global Threat

Abstract Background The spread of antibiotic resistance has become one of the most urgent threats to global health, which is estimated to cause 700,000 deaths each year globally. Its surrogates, antibiotic resistance genes (ARGs), are highly transmittable between food, water, animal, and human to mitigate the efficacy of antibiotics. Accurately identifying ARGs is thus an indispensable step to understanding the ecology, and transmission of ARGs between environmental and human-associated reservoirs. Unfortunately, the previous computational methods for identifying ARGs are mostly based on sequence alignment, which cannot identify novel ARGs, and their applications are limited by currently incomplete knowledge about ARGs. Results Here, we propose an end-to-end Hierarchical Multi-task Deep learning framework for ARG annotation (HMD-ARG). Taking raw sequence encoding as input, HMD-ARG can identify, without querying against existing sequence databases, multiple ARG properties simultaneously, including if the input protein sequence is an ARG, and if so, what antibiotic family it is resistant to, what resistant mechanism the ARG takes, and if the ARG is an intrinsic one or acquired one. In addition, if the predicted antibiotic family is beta-lactamase, HMD-ARG further predicts the subclass of beta-lactamase that the ARG is resistant to. Comprehensive experiments, including cross-fold validation, third-party dataset validation in human gut microbiota, wet-experimental functional validation, and structural investigation of predicted conserved sites, demonstrate not only the superior performance of our method over the state-of-art methods, but also the effectiveness and robustness of the proposed method. Conclusions We propose a hierarchical multi-task method, HMD-ARG, which is based on deep learning and can provide detailed annotations of ARGs from three important aspects: resistant antibiotic class, resistant mechanism, and gene mobility. We believe that HMD-ARG can serve as a powerful tool to identify antibiotic resistance genes and, therefore mitigate their global threat. Our method and the constructed database are available at http://www.cbrc.kaust.edu.sa/HMDARG/.

Download Full-text

ResBCDU-Net: A Deep Learning Framework for Lung CT Image Segmentation

Sensors ◽

10.3390/s21010268 ◽

2021 ◽

Vol 21 (1) ◽

pp. 268

Author(s):

Yeganeh Jalali ◽

Mansoor Fateh ◽

Mohsen Rezvani ◽

Vahid Abolghasemi ◽

Mohammad Hossein Anisi

Keyword(s):

Image Segmentation ◽

Deep Learning ◽

Ct Images ◽

Medical Image Segmentation ◽

Great Success ◽

Morphological Operations ◽

Ct Image ◽

Feature Maps ◽

Learning Framework ◽

Lung Ct

Lung CT image segmentation is a key process in many applications such as lung cancer detection. It is considered a challenging problem due to existing similar image densities in the pulmonary structures, different types of scanners, and scanning protocols. Most of the current semi-automatic segmentation methods rely on human factors therefore it might suffer from lack of accuracy. Another shortcoming of these methods is their high false-positive rate. In recent years, several approaches, based on a deep learning framework, have been effectively applied in medical image segmentation. Among existing deep neural networks, the U-Net has provided great success in this field. In this paper, we propose a deep neural network architecture to perform an automatic lung CT image segmentation process. In the proposed method, several extensive preprocessing techniques are applied to raw CT images. Then, ground truths corresponding to these images are extracted via some morphological operations and manual reforms. Finally, all the prepared images with the corresponding ground truth are fed into a modified U-Net in which the encoder is replaced with a pre-trained ResNet-34 network (referred to as Res BCDU-Net). In the architecture, we employ BConvLSTM (Bidirectional Convolutional Long Short-term Memory)as an advanced integrator module instead of simple traditional concatenators. This is to merge the extracted feature maps of the corresponding contracting path into the previous expansion of the up-convolutional layer. Finally, a densely connected convolutional layer is utilized for the contracting path. The results of our extensive experiments on lung CT images (LIDC-IDRI database) confirm the effectiveness of the proposed method where a dice coefficient index of 97.31% is achieved.

Download Full-text

Learning to Incorporate Structure Knowledge for Image Inpainting

10.20944/preprints202002.0125.v1 ◽

2020 ◽

Cited By ~ 1

Author(s):

Jie Yang ◽

Zhiquan Qi ◽

Yong Shi

Keyword(s):

Structure Learning ◽

State Of The Art ◽

Image Inpainting ◽

Image Completion ◽

Image Structure ◽

Learning Framework ◽

Task Learning ◽

Pyramid Structure ◽

Benchmark Datasets ◽

Structure Knowledge

This paper develops a multi-task learning framework that attempts to incorporate the image structure knowledge to assist image inpainting, which is not well explored in previous works. The primary idea is to train a shared generator to simultaneously complete the corrupted image and corresponding structures --- edge and gradient, thus implicitly encouraging the generator to exploit relevant structure knowledge while inpainting. In the meantime, we also introduce a structure embedding scheme to explicitly embed the learned structure features into the inpainting process, thus to provide possible preconditions for image completion. Specifically, a novel pyramid structure loss is proposed to supervise structure learning and embedding. Moreover, an attention mechanism is developed to further exploit the recurrent structures and patterns in the image to refine the generated structures and contents. Through multi-task learning, structure embedding besides with attention, our framework takes advantage of the structure knowledge and outperforms several state-of-the-art methods on benchmark datasets quantitatively and qualitatively.

Download Full-text

Neural networks to learn protein sequence-function relationships from deep mutational scanning data

10.1101/2020.10.25.353946 ◽

2020 ◽

Author(s):

Sam Gelman ◽

Philip A. Romero ◽

Anthony Gitter

Keyword(s):

Protein Structure ◽

Protein Sequence ◽

Internal Representation ◽

Superior Performance ◽

Network Architectures ◽

Convolutional Network ◽

Learning Framework ◽

And Function ◽

Multiple Neural Network ◽

Function Mapping

ABSTRACTThe mapping from protein sequence to function is highly complex, making it challenging to predict how sequence changes will affect a protein’s behavior and properties. We present a supervised deep learning framework to learn the sequence-function mapping from deep mutational scanning data and make predictions for new, uncharacterized sequence variants. We test multiple neural network architectures, including a graph convolutional network that incorporates protein structure, to explore how a network’s internal representation affects its ability to learn the sequence-function mapping. Our supervised learning approach displays superior performance over physics-based and unsupervised prediction methods. We find networks that capture nonlinear interactions and share parameters across sequence positions are important for learning the relationship between sequence and function. Further analysis of the trained models reveals the networks’ ability to learn biologically meaningful information about protein structure and mechanism. Our software is available from https://github.com/gitter-lab/nn4dms.

Download Full-text

Better Performance with Transformer: CPPFormer in precise prediction of cell-Penetrating Peptides

Current Medicinal Chemistry ◽

10.2174/0929867328666210920103140 ◽

2021 ◽

Vol 28 ◽

Author(s):

Yuyang Xue ◽

Xiucai Ye ◽

Lesong Wei ◽

Xin Zhang ◽

Tetsuya Sakurai ◽

...

Keyword(s):

Machine Learning ◽

Language Processing ◽

Cell Penetrating Peptides ◽

Machine Learning Techniques ◽

Superior Performance ◽

Practical Applications ◽

Deep Model ◽

Cell Penetrating ◽

Precise Prediction ◽

Transformer Model

: With its superior performance, the Transformer model, which is based on the 'Encoder-Decoder' paradigm, has become the mainstream in natural language processing. On the other hand, bioinformatics has embraced machine learning and made great progress in drug design and protein property prediction. Cell-penetrating peptides (CPPs) are one kind of permeable protein that is convenient as a kind of 'postman' in drug penetration tasks. However, a small number of CPPs have been discovered by research, let alone practical applications in drug permeability. Therefore, correctly identifying the CPPs has opened up a new way to take macromolecules into cells without other potentially harmful materials in the drug. Most of the previous work only uses trivial machine learning techniques and hand-crafted features to construct a simple classifier. In CPPFormer, we learn from the idea of implementing the attention structure of Transformer, rebuilding the network based on the characteristics of CPPs according to its short length, and using an automatic feature extractor with a few manual engineered features to co-direct the predicted results. Compared to all previous methods and other classic text classification models, the empirical result has shown that our proposed deep model-based method has achieved the best performance of 92.16% accuracy in the CPP924 dataset and has passed various index tests.

Download Full-text

Multimodal deep representation learning for protein interaction identification and protein family classification

BMC Bioinformatics ◽

10.1186/s12859-019-3084-y ◽

2019 ◽

Vol 20 (S16) ◽

Cited By ~ 4

Author(s):

Da Zhang ◽

Mansur Kabuka

Keyword(s):

Protein Interactions ◽

Protein Sequence ◽

Representation Learning ◽

Superior Performance ◽

Sequence Information ◽

Protein Protein Interactions ◽

Learning Framework ◽

Topological Features ◽

Ppi Networks ◽

Ppi Prediction

Abstract Background Protein-protein interactions(PPIs) engage in dynamic pathological and biological procedures constantly in our life. Thus, it is crucial to comprehend the PPIs thoroughly such that we are able to illuminate the disease occurrence, achieve the optimal drug-target therapeutic effect and describe the protein complex structures. However, compared to the protein sequences obtainable from various species and organisms, the number of revealed protein-protein interactions is relatively limited. To address this dilemma, lots of research endeavor have investigated in it to facilitate the discovery of novel PPIs. Among these methods, PPI prediction techniques that merely rely on protein sequence data are more widespread than other methods which require extensive biological domain knowledge. Results In this paper, we propose a multi-modal deep representation learning structure by incorporating protein physicochemical features with the graph topological features from the PPI networks. Specifically, our method not only bears in mind the protein sequence information but also discerns the topological representations for each protein node in the PPI networks. In our paper, we construct a stacked auto-encoder architecture together with a continuous bag-of-words (CBOW) model based on generated metapaths to study the PPI predictions. Following by that, we utilize the supervised deep neural networks to identify the PPIs and classify the protein families. The PPI prediction accuracy for eight species ranged from 96.76% to 99.77%, which signifies that our multi-modal deep representation learning framework achieves superior performance compared to other computational methods. Conclusion To the best of our knowledge, this is the first multi-modal deep representation learning framework for examining the PPI networks.

Download Full-text

Zebrafish Embryo Vessel Segmentation Using a Novel Dual ResUNet Model

Computational Intelligence and Neuroscience ◽

10.1155/2019/8214975 ◽

2019 ◽

Vol 2019 ◽

pp. 1-14 ◽

Cited By ~ 3

Author(s):

Kun Zhang ◽

Hongbin Zhang ◽

Huiyu Zhou ◽

Danny Crookes ◽

Ling Li ◽

...

Keyword(s):

Deep Learning ◽

Domain Knowledge ◽

Zebrafish Embryo ◽

Fluorescent Protein ◽

Vessel Segmentation ◽

Dual Model ◽

Shape Constraint ◽

Learning Framework ◽

Robust Segmentation ◽

Characteristic Features

Zebrafish embryo fluorescent vessel analysis, which aims to automatically investigate the pathogenesis of diseases, has attracted much attention in medical imaging. Zebrafish vessel segmentation is a fairly challenging task, which requires distinguishing foreground and background vessels from the 3D projection images. Recently, there has been a trend to introduce domain knowledge to deep learning algorithms for handling complex environment segmentation problems with accurate achievements. In this paper, a novel dual deep learning framework called Dual ResUNet is developed to conduct zebrafish embryo fluorescent vessel segmentation. To avoid the loss of spatial and identity information, the U-Net model is extended to a dual model with a new residual unit. To achieve stable and robust segmentation performance, our proposed approach merges domain knowledge with a novel contour term and shape constraint. We compare our method qualitatively and quantitatively with several standard segmentation models. Our experimental results show that the proposed method achieves better results than the state-of-art segmentation methods. By investigating the quality of the vessel segmentation, we come to the conclusion that our Dual ResUNet model can learn the characteristic features in those cases where fluorescent protein is deficient or blood vessels are overlapped and achieves robust performance in complicated environments.

Download Full-text

Learning to Incorporate Structure Knowledge for Image Inpainting

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i07.6951 ◽

2020 ◽

Vol 34 (07) ◽

pp. 12605-12612 ◽

Cited By ~ 1

Author(s):

Jie Yang ◽

Zhiquan Qi ◽

Yong Shi

Keyword(s):

Structure Learning ◽

State Of The Art ◽

Image Inpainting ◽

Image Completion ◽

Image Structure ◽

Learning Framework ◽

Task Learning ◽

Pyramid Structure ◽

Benchmark Datasets ◽

Structure Knowledge

This paper develops a multi-task learning framework that attempts to incorporate the image structure knowledge to assist image inpainting, which is not well explored in previous works. The primary idea is to train a shared generator to simultaneously complete the corrupted image and corresponding structures — edge and gradient, thus implicitly encouraging the generator to exploit relevant structure knowledge while inpainting. In the meantime, we also introduce a structure embedding scheme to explicitly embed the learned structure features into the inpainting process, thus to provide possible preconditions for image completion. Specifically, a novel pyramid structure loss is proposed to supervise structure learning and embedding. Moreover, an attention mechanism is developed to further exploit the recurrent structures and patterns in the image to refine the generated structures and contents. Through multi-task learning, structure embedding besides with attention, our framework takes advantage of the structure knowledge and outperforms several state-of-the-art methods on benchmark datasets quantitatively and qualitatively.

Download Full-text