Model Compression Hardens Deep Neural Networks: A New Perspective to Prevent Adversarial Attacks

2021 ◽  
pp. 1-12
Author(s):  
Qi Liu ◽  
Wujie Wen
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Model Compression ◽  
New Perspective

scholarly journals Deep neural networks for waves assisted by the Wiener–Hopf method

2020 ◽  
Vol 476 (2235) ◽  
pp. 20190846 ◽  
Author(s):  
Xun Huang
Keyword(s):  
Neural Networks ◽  
Wave Equations ◽  
Deep Neural Networks ◽  
Research Strategy ◽  
Propagation Model ◽  
Physical Constraints ◽  
Aerospace Applications ◽  
Feed Forward Network ◽  
New Perspective ◽  
Hopf Method

In this work, the classical Wiener–Hopf method is incorporated into the emerging deep neural networks for the study of certain wave problems. The essential idea is to use the first-principle-based analytical method to efficiently produce a large volume of datasets that would supervise the learning of data-hungry deep neural networks, and to further explain the working mechanisms on underneath. To demonstrate such a combinational research strategy, a deep feed-forward network is first used to approximate the forward propagation model of a duct acoustic problem, which can find important aerospace applications in aeroengine noise tests. Next, a convolutional type U-net is developed to learn spatial derivatives in wave equations, which could help to promote computational paradigm in mathematical physics and engineering applications. A couple of extensions of the U-net architecture are proposed to further impose possible physical constraints. Finally, after giving the implementation details, the performance of the neural networks are studied by comparing with analytical solutions from the Wiener–Hopf method. Overall, the Wiener–Hopf method is used here from a totally new perspective and such a combinational research strategy shall represent the key achievement of this work.

Evolutionary Multi-Objective Model Compression for Deep Neural Networks

2021 ◽  
Vol 16 (3) ◽  
pp. 10-21
Author(s):  
Zhehui Wang ◽  
Tao Luo ◽  
Miqing Li ◽  
Joey Tianyi Zhou ◽  
Rick Siow Mong Goh ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Multi Objective ◽  
Model Compression ◽  
Objective Model

scholarly journals Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks

2020 ◽  
Vol 34 (03) ◽  
pp. 2501-2508 ◽  
Author(s):  
Woo-Jeoung Nam ◽  
Shir Gur ◽  
Jaesik Choi ◽  
Lior Wolf ◽  
Seong-Whan Lee
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Relative Influence ◽  
Evaluation Metrics ◽  
Importance Score ◽  
Relative Priority ◽  
New Perspective ◽  
Conservation Rule

As Deep Neural Networks (DNNs) have demonstrated superhuman performance in a variety of fields, there is an increasing interest in understanding the complex internal mechanisms of DNNs. In this paper, we propose Relative Attributing Propagation (RAP), which decomposes the output predictions of DNNs with a new perspective of separating the relevant (positive) and irrelevant (negative) attributions according to the relative influence between the layers. The relevance of each neuron is identified with respect to its degree of contribution, separated into positive and negative, while preserving the conservation rule. Considering the relevance assigned to neurons in terms of relative priority, RAP allows each neuron to be assigned with a bi-polar importance score concerning the output: from highly relevant to highly irrelevant. Therefore, our method makes it possible to interpret DNNs with much clearer and attentive visualizations of the separated attributions than the conventional explaining methods. To verify that the attributions propagated by RAP correctly account for each meaning, we utilize the evaluation metrics: (i) Outside-inside relevance ratio, (ii) Segmentation mIOU and (iii) Region perturbation. In all experiments and metrics, we present a sizable gap in comparison to the existing literature.

scholarly journals Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems

2019 ◽  
Vol 9 (8) ◽  
pp. 1669
Author(s):  
Sangkyun Lee ◽  
Jeonghyun Lee
Keyword(s):  
Neural Networks ◽  
Embedded Systems ◽  
Sparse Coding ◽  
Deep Neural Networks ◽  
Sparse Matrices ◽  
Learning Problems ◽  
Proximal Point ◽  
Embedded Devices ◽  
Model Compression ◽  
Large Memory

Deep neural networks (DNNs) have been quite successful in solving many complex learning problems. However, DNNs tend to have a large number of learning parameters, leading to a large memory and computation requirement. In this paper, we propose a model compression framework for efficient training and inference of deep neural networks on embedded systems. Our framework provides data structures and kernels for OpenCL-based parallel forward and backward computation in a compressed form. In particular, our method learns sparse representations of parameters using ℓ 1 -based sparse coding while training, storing them in compressed sparse matrices. Unlike the previous works, our method does not require a pre-trained model as an input and therefore can be more versatile for different application environments. Even though the use of ℓ 1 -based sparse coding for model compression is not new, we show that it can be far more effective than previously reported when we use proximal point algorithms and the technique of debiasing. Our experiments show that our method can produce minimal learning models suitable for small embedded devices.

scholarly journals Improving Deep Neural Network Sparsity through Decorrelation Regularization

2018 ◽  
Author(s):  
Xiaotian Zhu ◽  
Wengang Zhou ◽  
Houqiang Li
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Neural Network ◽  
Deep Neural Networks ◽  
Group Lasso ◽  
High Complexity ◽  
Model Compression ◽  
Model Size ◽  
Sparse Networks ◽  
Convolution Filters

Modern deep learning models usually suffer high complexity in model size and computation when transplanted to resource constrained platforms. To this end, many works are dedicated to compressing deep neural networks. Adding group LASSO regularization is one of the most effective model compression methods since it generates structured sparse networks. We investigate the deep neural networks trained by group LASSO constraint and observe that even with strong sparsity regularization imposed, there still exists substantial filter correlation among the convolution filters, which is undesired for a compact neural network. We propose to suppress such correlation with a new kind of constraint called decorrelation regularization, which explicitly forces the network to learn a set of less correlated filters. The experiments on CIFAR10/100 and ILSVRC2012 datasets show that when combined our decorrelation regularization with group LASSO, the correlation between filters could be effectively weakened, which increases the sparsity of the resulting model and leads to better compressing performance.

Sign in / Sign up

Export Citation Format

Share Document