scholarly journals Adversarially Robust Distillation

2020 ◽  
Vol 34 (04) ◽  
pp. 3996-4003
Author(s):  
Micah Goldblum ◽  
Liam Fowl ◽  
Soheil Feizi ◽  
Tom Goldstein
Keyword(s):  
Neural Networks ◽  
High Performance ◽  
State Of The Art ◽  
Test Accuracy ◽  
Training Methods ◽  
Knowledge Distillation ◽  
Adversarial Training ◽  
Student Models ◽  
Small Models ◽  
High Test

Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robustness may be inherited by the student even when distilled on only clean images. Second, we introduce Adversarially Robust Distillation (ARD) for distilling robustness onto student networks. In addition to producing small models with high test accuracy like conventional distillation, ARD also passes the superior robustness of large networks onto the student. In our experiments, we find that ARD student models decisively outperform adversarially trained networks of identical architecture in terms of robust accuracy, surpassing state-of-the-art methods on standard robustness benchmarks. Finally, we adapt recent fast adversarial training methods to ARD for accelerated robust distillation.

scholarly journals Communication Failure Resilient Distributed Neural Network for Edge Devices

Electronics ◽  
2021 ◽  
Vol 10 (14) ◽  
pp. 1614
Author(s):  
Jonghun Jeong ◽  
Jong Sung Park ◽  
Hoeseok Yang
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
High Performance ◽  
State Of The Art ◽  
Wearable Devices ◽  
Communication Failure ◽  
Canadian Institute ◽  
Multiple Devices ◽  
Knowledge Distillation ◽  
Partitioning Technique

Recently, the necessity to run high-performance neural networks (NN) is increasing even in resource-constrained embedded systems such as wearable devices. However, due to the high computational and memory requirements of the NN applications, it is typically infeasible to execute them on a single device. Instead, it has been proposed to run a single NN application cooperatively on top of multiple devices, a so-called distributed neural network. In the distributed neural network, workloads of a single big NN application are distributed over multiple tiny devices. While the computation overhead could effectively be alleviated by this approach, the existing distributed NN techniques, such as MoDNN, still suffer from large traffics between the devices and vulnerability to communication failures. In order to get rid of such big communication overheads, a knowledge distillation based distributed NN, called Network of Neural Networks (NoNN), was proposed, which partitions the filters in the final convolutional layer of the original NN into multiple independent subsets and derives smaller NNs out of each subset. However, NoNN also has limitations in that the partitioning result may be unbalanced and it considerably compromises the correlation between filters in the original NN, which may result in an unacceptable accuracy degradation in case of communication failure. In this paper, in order to overcome these issues, we propose to enhance the partitioning strategy of NoNN in two aspects. First, we enhance the redundancy of the filters that are used to derive multiple smaller NNs by means of averaging to increase the immunity of the distributed NN to communication failure. Second, we propose a novel partitioning technique, modified from Eigenvector-based partitioning, to preserve the correlation between filters as much as possible while keeping the consistent number of filters distributed to each device. Throughout extensive experiments with the CIFAR-100 (Canadian Institute For Advanced Research-100) dataset, it has been observed that the proposed approach maintains high inference accuracy (over 70%, 1.53× improvement over the state-of-the-art approach), on average, even when a half of eight devices in a distributed NN fail to deliver their partial inference results.

scholarly journals Robust Neural Networks are More Interpretable for Genomics

2019 ◽  
Author(s):  
Peter K. Koo ◽  
Sharon Qian ◽  
Gal Kaplun ◽  
Verena Volf ◽  
Dimitris Kalimeris
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Random Noise ◽  
Genomic Data ◽  
Training Methods ◽  
Generalization Performance ◽  
Regulatory Genomics ◽  
Adversarial Training

AbstractDeep neural networks (DNNs) have been applied to a variety of regulatory genomics tasks. For interpretability, attribution methods are employed to provide importance scores for each nucleotide in a given sequence. However, even with state-of-the-art DNNs, there is no guarantee that these methods can recover interpretable, biological representations. Here we perform systematic experiments on synthetic genomic data to raise awareness of this issue. We find that deeper networks have better generalization performance, but attribution methods recover less interpretable representations. Then, we show training methods promoting robustness – including regularization, injecting random noise into the data, and adversarial training – significantly improve interpretability of DNNs, especially for smaller datasets.

scholarly journals Robust CNN Compression Framework for Security-Sensitive Embedded Systems

2021 ◽  
Vol 11 (3) ◽  
pp. 1093
Author(s):  
Jeonghyun Lee ◽  
Sangkyun Lee
Keyword(s):  
Embedded Systems ◽  
Optimization Problem ◽  
State Of The Art ◽  
Classification Problems ◽  
Proximal Gradient Method ◽  
Knowledge Distillation ◽  
New Type ◽  
Adversarial Examples ◽  
Adversarial Training ◽  
Memory Efficient

Convolutional neural networks (CNNs) have achieved tremendous success in solving complex classification problems. Motivated by this success, there have been proposed various compression methods for downsizing the CNNs to deploy them on resource-constrained embedded systems. However, a new type of vulnerability of compressed CNNs known as the adversarial examples has been discovered recently, which is critical for security-sensitive systems because the adversarial examples can cause malfunction of CNNs and can be crafted easily in many cases. In this paper, we proposed a compression framework to produce compressed CNNs robust against such adversarial examples. To achieve the goal, our framework uses both pruning and knowledge distillation with adversarial training. We formulate our framework as an optimization problem and provide a solution algorithm based on the proximal gradient method, which is more memory-efficient than the popular ADMM-based compression approaches. In experiments, we show that our framework can improve the trade-off between adversarial robustness and compression rate compared to the existing state-of-the-art adversarial pruning approach.

scholarly journals Modified Deep Neural Networks for Dog Breeds Identification

2018 ◽  
Author(s):  
Aydin Ayanzadeh ◽  
Sahand Vahidnia
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
The State ◽  
Fine Tuning ◽  
Test Accuracy ◽  
Data Sets ◽  
Data Set

In this paper, we leverage state of the art models on Imagenet data-sets. We use the pre-trained model and learned weighs to extract the feature from the Dog breeds identification data-set. Afterwards, we applied fine-tuning and dataaugmentation to increase the performance of our test accuracy in classification of dog breeds datasets. The performance of the proposed approaches are compared with the state of the art models of Image-Net datasets such as ResNet-50, DenseNet-121, DenseNet-169 and GoogleNet. we achieved 89.66% , 85.37% 84.01% and 82.08% test accuracy respectively which shows thesuperior performance of proposed method to the previous works on Stanford dog breeds datasets.

scholarly journals Re-Identification in Urban Scenarios: A Review of Tools and Methods

2021 ◽  
Vol 11 (22) ◽  
pp. 10809
Author(s):  
Hugo S. Oliveira ◽  
José J. M. Machado ◽  
João Manuel R. S. Tavares
Keyword(s):  
Neural Networks ◽  
High Performance ◽  
Object Identification ◽  
Future Research ◽  
Great Effort ◽  
Data Limitations ◽  
Video Feed ◽  
Adversarial Training ◽  
Future Research Directions ◽  
Model Training

With the widespread use of surveillance image cameras and enhanced awareness of public security, objects, and persons Re-Identification (ReID), the task of recognizing objects in non-overlapping camera networks has attracted particular attention in computer vision and pattern recognition communities. Given an image or video of an object-of-interest (query), object identification aims to identify the object from images or video feed taken from different cameras. After many years of great effort, object ReID remains a notably challenging task. The main reason is that an object’s appearance may dramatically change across camera views due to significant variations in illumination, poses or viewpoints, or even cluttered backgrounds. With the advent of Deep Neural Networks (DNN), there have been many proposals for different network architectures achieving high-performance levels. With the aim of identifying the most promising methods for ReID for future robust implementations, a review study is presented, mainly focusing on the person and multi-object ReID and auxiliary methods for image enhancement. Such methods are crucial for robust object ReID, while highlighting limitations of the identified methods. This is a very active field, evidenced by the dates of the publications found. However, most works use data from very different datasets and genres, which presents an obstacle to wide generalized DNN model training and usage. Although the model’s performance has achieved satisfactory results on particular datasets, a particular trend was observed in the use of 3D Convolutional Neural Networks (CNN), attention mechanisms to capture object-relevant features, and generative adversarial training to overcome data limitations. However, there is still room for improvement, namely in using images from urban scenarios among anonymized images to comply with public privacy legislation. The main challenges that remain in the ReID field, and prospects for future research directions towards ReID in dense urban scenarios, are also discussed.

scholarly journals Effective training of convolutional neural networks for age estimation based on knowledge distillation

2021 ◽  
Author(s):  
Antonio Greco ◽  
Alessia Saggese ◽  
Mario Vento ◽  
Vincenzo Vigilante
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Age Estimation ◽  
Resource Constraints ◽  
Training Procedure ◽  
Face Images ◽  
Effective Training ◽  
Knowledge Distillation ◽  
Student Models ◽  
Teacher Model

AbstractAge estimation from face images can be profitably employed in several applications, ranging from digital signage to social robotics, from business intelligence to access control. Only in recent years, the advent of deep learning allowed for the design of extremely accurate methods based on convolutional neural networks (CNNs) that achieve a remarkable performance in various face analysis tasks. However, these networks are not always applicable in real scenarios, due to both time and resource constraints that the most accurate approaches often do not meet. Moreover, in case of age estimation, there is the lack of a large and reliably annotated dataset for training deep neural networks. Within this context, we propose in this paper an effective training procedure of CNNs for age estimation based on knowledge distillation, able to allow smaller and simpler “student” models to be trained to match the predictions of a larger “teacher” model. We experimentally show that such student models are able to almost reach the performance of the teacher, obtaining high accuracy over the LFW+, LAP 2016 and Adience datasets, but being up to 15 times faster. Furthermore, we evaluate the performance of the student models in the presence of image corruptions, and we demonstrate that some of them are even more resilient to these corruptions than the teacher model.

scholarly journals Detection of Melanoma Skin Cancer with Deep Neural Networks

2019 ◽  
Vol 4 (4) ◽  
Keyword(s):  
Neural Networks ◽  
Skin Cancer ◽  
High Performance ◽  
Deep Neural Networks ◽  
Histopathological Examination ◽  
Skin Lesions ◽  
Image Features ◽  
Test Accuracy ◽  
Data Sets ◽  
Deep Convolutional Neural Networks

Detection of skin cancer involves several steps of examinations first being visual diagnosis that is followed by dermoscopic analysis, a biopsy, and histopathological examination. The classification of skin lesions in the first step is critical and challenging as classes vary by minute appearance in skin lesions. Deep convolutional neural networks (CNNs) have great potential in multicategory image-based classification by considering coarse-to-fine image features. This study aims to demonstrate how to classify skin lesions, in particular, melanoma, using CNN trained on data sets with disease labels. We developed and trained our own CNN model using a subset of the images from International Skin Imaging Collaboration (ISIC) Dermoscopic Archive. To test the performance of the proposed model, we used a different subset of images from the same archive as the test set. Our model is trained to classify images into two categories: malignant melanoma and nevus and is shown to achieve excellent classification results with high test accuracy (91.16%) and high performance as measured by various metrics. Our study demonstrated the potential of using deep neural networks to assist early detection of melanoma and thereby improve the patient survival rate from this aggressive skin cancer.

scholarly journals High Performance Gesture Recognition via Effective and Efficient Temporal Modeling

2019 ◽  
Author(s):  
Yang Yi ◽  
Feng Ni ◽  
Yuexin Ma ◽  
Xinge Zhu ◽  
Yuankai Qi ◽  
...  
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Gesture Recognition ◽  
High Performance ◽  
Short Term Memory ◽  
State Of The Art ◽  
Computational Cost ◽  
Temporal Modeling ◽  
Spatiotemporal Features ◽  
Public Datasets

State-of-the-art hand gesture recognition methods have investigated the spatiotemporal features based on 3D convolutional neural networks (3DCNNs) or convolutional long short-term memory (ConvLSTM). However, they often suffer from the inefficiency due to the high computational complexity of their network structures. In this paper, we focus instead on the 1D convolutional neural networks and propose a simple and efficient architectural unit, Multi-Kernel Temporal Block (MKTB), that models the multi-scale temporal responses by explicitly applying different temporal kernels. Then, we present a Global Refinement Block (GRB), which is an attention module for shaping the global temporal features based on the cross-channel similarity. By incorporating the MKTB and GRB, our architecture can effectively explore the spatiotemporal features within tolerable computational cost. Extensive experiments conducted on public datasets demonstrate that our proposed model achieves the state-of-the-art with higher efficiency. Moreover, the proposed MKTB and GRB are plug-and-play modules and the experiments on other tasks, like video understanding and video-based person re-identification, also display their good performance in efficiency and capability of generalization.

scholarly journals Invariant Representations through Adversarial Forgetting

2020 ◽  
Vol 34 (04) ◽  
pp. 4272-4279
Author(s):  
Ayush Jaiswal ◽  
Daniel Moyer ◽  
Greg Ver Steeg ◽  
Wael AbdAlmageed ◽  
Premkumar Natarajan
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Empirical Results ◽  
Information Bottleneck ◽  
Novel Approach ◽  
Adversarial Training ◽  
Invariant Representations ◽  
Art Performance ◽  
Forgetting Mechanism

We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism. We show that the forgetting mechanism serves as an information-bottleneck, which is manipulated by the adversarial training to learn invariance to unwanted factors. Empirical results show that the proposed framework achieves state-of-the-art performance at learning invariance in both nuisance and bias settings on a diverse collection of datasets and tasks.

scholarly journals Neural Belief Reasoner

2020 ◽  
Author(s):  
Haifeng Qian
Keyword(s):  
Neural Networks ◽  
State Of The Art ◽  
Belief Function ◽  
Learning Task ◽  
Natural Images ◽  
Training Algorithms ◽  
Conflicting Information ◽  
Set Operations ◽  
Adversarial Training ◽  
And Training

This paper proposes a new generative model called neural belief reasoner (NBR). It differs from previous models in that it specifies a belief function rather than a probability distribution. Its implementation consists of neural networks, fuzzy-set operations and belief-function operations, and query-answering, sample-generation and training algorithms are presented. This paper studies NBR in two tasks. The first is a synthetic unsupervised-learning task, which demonstrates NBR's ability to perform multi-hop reasoning, reasoning with uncertainty and reasoning about conflicting information. The second is supervised learning: a robust MNIST classifier for 4 and 9, which is the most challenging pair of digits. This classifier needs no adversarial training, and it substantially exceeds the state of the art in adversarial robustness as measured by the L2 metric, while at the same time maintains 99.1% accuracy on natural images.

Sign in / Sign up

Export Citation Format

Share Document