scholarly journals Improved Knowledge Distillation via Teacher Assistant

2020 ◽  
Vol 34 (04) ◽  
pp. 5191-5198 ◽  
Author(s):  
Seyed Iman Mirzadeh ◽  
Mehrdad Farajtabar ◽  
Ang Li ◽  
Nir Levine ◽  
Akihiro Matsukawa ◽  
...  
Keyword(s):  
Neural Networks ◽  
Theoretical Analysis ◽  
Network Performance ◽  
Deep Neural Networks ◽  
Sensor Nodes ◽  
Popular Method ◽  
Knowledge Distillation ◽  
Teacher Assistant ◽  
Trained Network

Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too large to be deployed on edge devices like smartphones or embedded sensor nodes. There have been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teacher) pre-trained network is used to train a smaller (student) network. However, in this paper, we show that the student network performance degrades when the gap between student and teacher is large. Given a fixed student network, one cannot employ an arbitrarily large teacher, or in other words, a teacher can effectively transfer its knowledge to students up to a certain size, not smaller. To alleviate this shortcoming, we introduce multi-step knowledge distillation, which employs an intermediate-sized network (teacher assistant) to bridge the gap between the student and the teacher. Moreover, we study the effect of teacher assistant size and extend the framework to multi-step distillation. Theoretical analysis and extensive experiments on CIFAR-10,100 and ImageNet datasets and on CNN and ResNet architectures substantiate the effectiveness of our proposed approach.

scholarly journals A theoretical analysis of Deep Neural Networks for texture classification

2016 ◽  
Author(s):  
Saikat Basu ◽  
Manohar Karki ◽  
Supratik Mukhopadhyay ◽  
Sangram Ganguly ◽  
Ramakrishna Nemani ◽  
...  
Keyword(s):  
Neural Networks ◽  
Theoretical Analysis ◽  
Deep Neural Networks ◽  
Texture Classification

scholarly journals Generative Adversarial Positive-Unlabelled Learning

2018 ◽  
Author(s):  
Ming Hou ◽  
Brahim Chaib-draa ◽  
Chao Li ◽  
Qibin Zhao
Keyword(s):  
Neural Networks ◽  
Theoretical Analysis ◽  
Supervised Classification ◽  
Deep Neural Networks ◽  
Generative Adversarial Networks ◽  
Discriminative Learning ◽  
Decision Boundary ◽  
Negative Data ◽  
New Paradigm ◽  
Adversarial Networks

 In this work, we consider the task of classifying binary positive-unlabeled (PU) data. The existing discriminative learning based PU models attempt to seek an optimal reweighting strategy for U data, so that a decent decision boundary can be found. However, given limited P data, the conventional PU models tend to suffer from overfitting when adapted to very flexible deep neural networks. In contrast, we are the first to innovate a totally new paradigm to attack the binary PU task, from perspective of generative learning by leveraging the powerful generative adversarial networks (GAN). Our generative positive-unlabeled (GenPU) framework incorporates an array of discriminators and generators that are endowed with different roles in simultaneously producing positive and negative realistic samples. We provide theoretical analysis to justify that, at equilibrium, GenPU is capable of recovering both positive and negative data distributions. Moreover, we show GenPU is generalizable and closely related to the semi-supervised classification. Given rather limited P data, experiments on both synthetic and real-world dataset demonstrate the effectiveness of our proposed framework. With infinite realistic and diverse sample streams generated from GenPU, a very flexible classifier can then be trained using deep neural networks.

scholarly journals Heterogeneous Gaussian Mechanism: Preserving Differential Privacy in Deep Learning with Provable Robustness

2019 ◽  
Author(s):  
NhatHai Phan ◽  
Minh N. Vu ◽  
Yang Liu ◽  
Ruoming Jin ◽  
Dejing Dou ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Theoretical Analysis ◽  
Gaussian Noise ◽  
Deep Neural Networks ◽  
Differential Privacy ◽  
Trade Off ◽  
Adversarial Examples ◽  
Hidden Layer ◽  
Privacy Budget

In this paper, we propose a novel Heterogeneous Gaussian Mechanism (HGM) to preserve differential privacy in deep neural networks, with provable robustness against adversarial examples. We first relax the constraint of the privacy budget in the traditional Gaussian Mechanism from (0, 1] to (0, infty), with a new bound of the noise scale to preserve differential privacy. The noise in our mechanism can be arbitrarily redistributed, offering a distinctive ability to address the trade-off between model utility and privacy loss. To derive provable robustness, our HGM is applied to inject Gaussian noise into the first hidden layer. Then, a tighter robustness bound is proposed. Theoretical analysis and thorough evaluations show that our mechanism notably improves the robustness of differentially private deep neural networks, compared with baseline approaches, under a variety of model attacks.

scholarly journals Adam and the Ants: On the Influence of the Optimization Algorithm on the Detectability of DNN Watermarks

Entropy ◽  
2020 ◽  
Vol 22 (12) ◽  
pp. 1379
Author(s):  
Betty Cortiñas-Lorenzo ◽  
Fernando Pérez-González
Keyword(s):  
Neural Networks ◽  
Theoretical Analysis ◽  
Digital Watermarking ◽  
Optimization Algorithm ◽  
Gradient Descent ◽  
Deep Neural Networks ◽  
Stochastic Gradient Descent ◽  
Secret Message ◽  
Weight Distributions ◽  
A Minor

As training Deep Neural Networks (DNNs) becomes more expensive, the interest in protecting the ownership of the models with watermarking techniques increases. Uchida et al. proposed a digital watermarking algorithm that embeds the secret message into the model coefficients. However, despite its appeal, in this paper, we show that its efficacy can be compromised by the optimization algorithm being used. In particular, we found through a theoretical analysis that, as opposed to Stochastic Gradient Descent (SGD), the update direction given by Adam optimization strongly depends on the sign of a combination of columns of the projection matrix used for watermarking. Consequently, as observed in the empirical results, this makes the coefficients move in unison giving rise to heavily spiked weight distributions that can be easily detected by adversaries. As a way to solve this problem, we propose a new method called Block-Orthonormal Projections (BOP) that allows one to combine watermarking with Adam optimization with a minor impact on the detectability of the watermark and an increased robustness.

Sign in / Sign up

Export Citation Format

Share Document