scholarly journals Characteristic Examples: High-Robustness, Low-Transferability Fingerprinting of Neural Networks

2021 ◽  
Author(s):  
Siyue Wang ◽  
Xiao Wang ◽  
Pin-Yu Chen ◽  
Pu Zhao ◽  
Xue Lin
Keyword(s):  
Neural Networks ◽  
False Alarm ◽  
False Positive ◽  
Deep Neural Networks ◽  
False Positives ◽  
Superior Performance ◽  
Baseline Method ◽  
Trade Off ◽  
The Difference ◽  
Fingerprinting Methods

This paper proposes Characteristic Examples for effectively fingerprinting deep neural networks, featuring high-robustness to the base model against model pruning as well as low-transferability to unassociated models. This is the first work taking both robustness and transferability into consideration for generating realistic fingerprints, whereas current methods lack practical assumptions and may incur large false positive rates. To achieve better trade-off between robustness and transferability, we propose three kinds of characteristic examples: vanilla C-examples, RC-examples, and LTRC-example, to derive fingerprints from the original base model. To fairly characterize the trade-off between robustness and transferability, we propose Uniqueness Score, a comprehensive metric that measures the difference between robustness and transferability, which also serves as an indicator to the false alarm problem. Extensive experiments demonstrate that the proposed characteristic examples can achieve superior performance when compared with existing fingerprinting methods. In particular, for VGG ImageNet models, using LTRC-examples gives 4X higher uniqueness score than the baseline method and does not incur any false positives.

scholarly journals An efficient pruning scheme of deep neural networks for Internet of Things applications

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Chen Qi ◽  
Shibo Shen ◽  
Rongpeng Li ◽  
Zhifeng Zhao ◽  
Qing Liu ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Internet Of Things ◽  
Deep Neural Networks ◽  
Computational Cost ◽  
Superior Performance ◽  
Compact Structure ◽  
Resource Limited ◽  
Benchmark Datasets ◽  
Iot Devices

AbstractNowadays, deep neural networks (DNNs) have been rapidly deployed to realize a number of functionalities like sensing, imaging, classification, recognition, etc. However, the computational-intensive requirement of DNNs makes it difficult to be applicable for resource-limited Internet of Things (IoT) devices. In this paper, we propose a novel pruning-based paradigm that aims to reduce the computational cost of DNNs, by uncovering a more compact structure and learning the effective weights therein, on the basis of not compromising the expressive capability of DNNs. In particular, our algorithm can achieve efficient end-to-end training that transfers a redundant neural network to a compact one with a specifically targeted compression rate directly. We comprehensively evaluate our approach on various representative benchmark datasets and compared with typical advanced convolutional neural network (CNN) architectures. The experimental results verify the superior performance and robust effectiveness of our scheme. For example, when pruning VGG on CIFAR-10, our proposed scheme is able to significantly reduce its FLOPs (floating-point operations) and number of parameters with a proportion of 76.2% and 94.1%, respectively, while still maintaining a satisfactory accuracy. To sum up, our scheme could facilitate the integration of DNNs into the common machine-learning-based IoT framework and establish distributed training of neural networks in both cloud and edge.

scholarly journals Critical Assessment of Artificial Intelligence Methods for Prediction of hERG Channel Inhibition in the ‘Big Data’ Era

2020 ◽  
Author(s):  
Vishal Babu Siramshetty ◽  
Dac-Trung Nguyen ◽  
Natalia J. Martinez ◽  
Anton Simeonov ◽  
Noel T. Southall ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Neural Networks ◽  
Big Data ◽  
Recurrent Neural Networks ◽  
Deep Neural Networks ◽  
Prediction Models ◽  
Chemical Space ◽  
Superior Performance ◽  
Gradient Boosting ◽  
Artificial Intelligence Methods

The rise of novel artificial intelligence methods necessitates a comparison of this wave of new approaches with classical machine learning for a typical drug discovery project. Inhibition of the potassium ion channel, whose alpha subunit is encoded by human Ether-à-go-go-Related Gene (hERG), leads to prolonged QT interval of the cardiac action potential and is a significant safety pharmacology target for the development of new medicines. Several computational approaches have been employed to develop prediction models for assessment of hERG liabilities of small molecules including recent work using deep learning methods. Here we perform a comprehensive comparison of prediction models based on classical (random forests and gradient boosting) and modern (deep neural networks and recurrent neural networks) artificial intelligence methods. The training set (~9000 compounds) was compiled by integrating hERG bioactivity data from ChEMBL database with experimental data generated from an in-house, high-throughput thallium flux assay. We utilized different molecular descriptors including the latent descriptors, which are real-valued continuous vectors derived from chemical autoencoders trained on a large chemical space (> 1.5 million compounds). The models were prospectively validated on ~840 in-house compounds screened in the same thallium flux assay. The deep neural networks performed significantly better than the classical methods with the latent descriptors. The recurrent neural networks that operate on SMILES provided highest model sensitivity. The best models were merged into a consensus model that offered superior performance compared to reference models from academic and commercial domains. Further, we shed light on the potential of artificial intelligence methods to exploit the chemistry big data and generate novel chemical representations useful in predictive modeling and tailoring new chemical space.<br>

scholarly journals Inter-Class Angular Loss for Convolutional Neural Networks

2019 ◽  
Vol 33 ◽  
pp. 3894-3901 ◽  
Author(s):  
Le Hui ◽  
Xiang Li ◽  
Chen Gong ◽  
Meng Fang ◽  
Joey Tianyi Zhou ◽  
...  
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Deep Neural Networks ◽  
Learning Difficulties ◽  
Feature Space ◽  
Superior Performance ◽  
Strongly Correlated ◽  
Discriminative Ability ◽  
Practical Applications ◽  
Classification Tasks

Convolutional Neural Networks (CNNs) have shown great power in various classification tasks and have achieved remarkable results in practical applications. However, the distinct learning difficulties in discriminating different pairs of classes are largely ignored by the existing networks. For instance, in CIFAR-10 dataset, distinguishing cats from dogs is usually harder than distinguishing horses from ships. By carefully studying the behavior of CNN models in the training process, we observe that the confusion level of two classes is strongly correlated with their angular separability in the feature space. That is, the larger the inter-class angle is, the lower the confusion will be. Based on this observation, we propose a novel loss function dubbed “Inter-Class Angular Loss” (ICAL), which explicitly models the class correlation and can be directly applied to many existing deep networks. By minimizing the proposed ICAL, the networks can effectively discriminate the examples in similar classes by enlarging the angle between their corresponding class vectors. Thorough experimental results on a series of vision and nonvision datasets confirm that ICAL critically improves the discriminative ability of various representative deep neural networks and generates superior performance to the original networks with conventional softmax loss.

scholarly journals Integrating Pathway Knowledge with Deep Neural Networks to Reduce the Dimensionality in Single-Cell RNA-Seq Data

2021 ◽  
Author(s):  
Pelin Gundogdu ◽  
Carlos Loucera ◽  
Inmaculada Alamo-Alvarez ◽  
Joaquin Dopazo ◽  
Isabel Nepomuceno
Keyword(s):  
Neural Networks ◽  
Single Cell ◽  
Deep Neural Networks ◽  
Cell Types ◽  
Cellular Heterogeneity ◽  
Biological Information ◽  
Superior Performance ◽  
Biological Knowledge ◽  
Additional Advantage ◽  
Functional Relationships

Abstract BackgroundSingle-cell RNA sequencing (scRNA-seq) data provides valuable insights into cellular heterogeneity which is significantly improving the current knowledge on biology and human disease. One of the main applications of scRNA-seq data analysis is the identification of new cell types and cell states. Deep neural networks (DNNs) are among the best methods to address this problem. However, this performance comes with the trade-off for a lack of interpretability in the results. In this work we propose an intelligible pathway-driven neural network to correctly solve cell-type related problems at single-cell resolution while providing a biologically meaningful representation of the data.ResultsIn this study, we explored the deep neural networks constrained by several types of prior biological information, e.g. signaling pathway information, as a way to reduce the dimensionality of the scRNA-seq data. We have tested the proposed biologically-based architectures on thousands of cells of human and mouse origin across a collection of public datasets in order to check the performance of the model. Specifically, we tested the architecture across different validation scenarios that try to mimic how unknown cell types are clustered by the DNN and how it correctly annotates cell types by querying a database in a retrieval problem. Moreover, our approach demonstrated to be comparable to other less interpretable DNN approaches constrained by using protein-protein interactions gene regulation data. Finally, we show how the latent structure learned by the network could be used to visualize and to interpret the composition of human single cell datasets.ConclusionsHere we demonstrate how the integration of pathways, which convey fundamental information on functional relationships between genes, with DNNs, that provide an excellent classification framework, results in an excellent alternative to learn a biologically meaningful representation of scRNA-seq data. In addition, the introduction of prior biological knowledge in the DNN reduces the size of the network architecture. Comparative results demonstrate a superior performance of this approach with respect to other similar approaches. As an additional advantage, the use of pathways within the DNN structure enables easy interpretability of the results by connecting features to cell functionalities by means of the pathway nodes, as demonstrated with an example with human melanoma tumor cells.

scholarly journals Integrating pathway knowledge with deep neural networks to reduce the dimensionality in single-cell RNA-seq data

2022 ◽  
Vol 15 (1) ◽  
Author(s):  
Pelin Gundogdu ◽  
Carlos Loucera ◽  
Inmaculada Alamo-Alvarez ◽  
Joaquin Dopazo ◽  
Isabel Nepomuceno
Keyword(s):  
Neural Networks ◽  
Single Cell ◽  
Deep Neural Networks ◽  
Cell Types ◽  
Cellular Heterogeneity ◽  
Biological Information ◽  
Superior Performance ◽  
Biological Knowledge ◽  
Additional Advantage ◽  
Functional Relationships

Abstract Background Single-cell RNA sequencing (scRNA-seq) data provide valuable insights into cellular heterogeneity which is significantly improving the current knowledge on biology and human disease. One of the main applications of scRNA-seq data analysis is the identification of new cell types and cell states. Deep neural networks (DNNs) are among the best methods to address this problem. However, this performance comes with the trade-off for a lack of interpretability in the results. In this work we propose an intelligible pathway-driven neural network to correctly solve cell-type related problems at single-cell resolution while providing a biologically meaningful representation of the data. Results In this study, we explored the deep neural networks constrained by several types of prior biological information, e.g. signaling pathway information, as a way to reduce the dimensionality of the scRNA-seq data. We have tested the proposed biologically-based architectures on thousands of cells of human and mouse origin across a collection of public datasets in order to check the performance of the model. Specifically, we tested the architecture across different validation scenarios that try to mimic how unknown cell types are clustered by the DNN and how it correctly annotates cell types by querying a database in a retrieval problem. Moreover, our approach demonstrated to be comparable to other less interpretable DNN approaches constrained by using protein-protein interactions gene regulation data. Finally, we show how the latent structure learned by the network could be used to visualize and to interpret the composition of human single cell datasets. Conclusions Here we demonstrate how the integration of pathways, which convey fundamental information on functional relationships between genes, with DNNs, that provide an excellent classification framework, results in an excellent alternative to learn a biologically meaningful representation of scRNA-seq data. In addition, the introduction of prior biological knowledge in the DNN reduces the size of the network architecture. Comparative results demonstrate a superior performance of this approach with respect to other similar approaches. As an additional advantage, the use of pathways within the DNN structure enables easy interpretability of the results by connecting features to cell functionalities by means of the pathway nodes, as demonstrated with an example with human melanoma tumor cells.

scholarly journals Differential Equation Units: Learning Functional Forms of Activation Functions from Data

2020 ◽  
Vol 34 (04) ◽  
pp. 6030-6037
Author(s):  
MohamadAli Torkamani ◽  
Shiv Shankar ◽  
Amirmohammad Rooshenas ◽  
Phillip Wallis
Keyword(s):  
Differential Equation ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Functional Form ◽  
Activation Function ◽  
The Other ◽  
Superior Performance ◽  
Activation Functions ◽  
Functional Forms ◽  
Nonlinear Activation Function

Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural networks, which enables each neuron to learn a particular nonlinear activation function from a family of solutions to an ordinary differential equation. Specifically, each neuron may change its functional form during training based on the behavior of the other parts of the network. We show that using neurons with DEU activation functions results in a more compact network capable of achieving comparable, if not superior, performance when compared to much larger networks.

scholarly journals Energy-efficient Amortized Inference with Cascaded Deep Classifiers

2018 ◽  
Author(s):  
Jiaqi Guan ◽  
Yang Liu ◽  
Qiang Liu ◽  
Jian Peng
Keyword(s):  
Neural Networks ◽  
Energy Cost ◽  
Deep Neural Networks ◽  
Predictive Accuracy ◽  
Computational Cost ◽  
Mobile Sensing ◽  
Test Time ◽  
Trade Off ◽  
Effective Cost ◽  
Energy Constrained

Deep neural networks have been remarkable successful in various AI tasks but often cast high computation and energy cost for energy-constrained applications such as mobile sensing. We address this problem by proposing a novel framework that optimizes the prediction accuracy and energy cost simultaneously, thus enabling effective cost-accuracy trade-off at test time. In our framework, each data instance is pushed into a cascade of deep neural networks with increasing sizes, and a selection module is used to sequentially determine when a sufficiently accurate classifier can be used for this data instance. The cascade of neural networks and the selection module are jointly trained in an end-to-end fashion by the REINFORCE algorithm to optimize a trade-off between the computational cost and the predictive accuracy. Our method is able to simultaneously improve the accuracy and efficiency by learning to assign easy instances to fast yet sufficiently accurate classifiers to save computation and energy cost, while assigning harder instances to deeper and more powerful classifiers to ensure satisfiable accuracy. Moreover, we demonstrate our method's effectiveness with extensive experiments on CIFAR-10/100, ImageNet32x32 and original ImageNet dataset.

scholarly journals Heterogeneous Gaussian Mechanism: Preserving Differential Privacy in Deep Learning with Provable Robustness

2019 ◽  
Author(s):  
NhatHai Phan ◽  
Minh N. Vu ◽  
Yang Liu ◽  
Ruoming Jin ◽  
Dejing Dou ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Theoretical Analysis ◽  
Gaussian Noise ◽  
Deep Neural Networks ◽  
Differential Privacy ◽  
Trade Off ◽  
Adversarial Examples ◽  
Hidden Layer ◽  
Privacy Budget

In this paper, we propose a novel Heterogeneous Gaussian Mechanism (HGM) to preserve differential privacy in deep neural networks, with provable robustness against adversarial examples. We first relax the constraint of the privacy budget in the traditional Gaussian Mechanism from (0, 1] to (0, infty), with a new bound of the noise scale to preserve differential privacy. The noise in our mechanism can be arbitrarily redistributed, offering a distinctive ability to address the trade-off between model utility and privacy loss. To derive provable robustness, our HGM is applied to inject Gaussian noise into the first hidden layer. Then, a tighter robustness bound is proposed. Theoretical analysis and thorough evaluations show that our mechanism notably improves the robustness of differentially private deep neural networks, compared with baseline approaches, under a variety of model attacks.

Sign in / Sign up

Export Citation Format

Share Document