Weight Pruning Techniques towards Photonic Implementation of Nonlinear Impairment Compensation using Neural Networks

2021 ◽  
pp. 1-1
Author(s):  
Shinsuke Fujisawa ◽  
Fatih Yaman ◽  
Hussam G. Batshon ◽  
Masaaki Tanio ◽  
Naoto Ishii ◽  
...  
Keyword(s):  
Neural Networks ◽  
Weight Pruning ◽  
Pruning Techniques

scholarly journals Novel Pruning Techniques in Convolutional-Neural Networks

2020 ◽  
Vol 9 (4) ◽  
pp. 1541-1548
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Image Classification ◽  
Language Processing ◽  
Time Series Prediction ◽  
L1 Norm ◽  
Network Pruning ◽  
Weight Pruning ◽  
The Cost ◽  
Pruning Techniques

Deep Learning allows us to build powerful models to solve problems like image classification, time series prediction, natural language processing, etc. This is achieved at the cost of huge amounts of storage and processing requirements which are sometimes not possible in machines with limited resources. In this paper, we compare different methods which tackle this problem with network pruning. Selected few pruning methodologies from the deep learning literature were implemented to display their results. Modern neural architectures have a combination of different layers like convolutional layers, pooling layers, dense layers, etc. We compare pruning techniques for dense layers (such as unit/neuron pruning, and weight Pruning), and convolutional layers as well (using L1 norm, taylor expansion of loss to determine importance of convolutional filters, and Variable Importance in Projection using Partial Least Squares) for the image classification task. This study aims to ease the overhead in terms of optimization of the model for academic, as well as commercial, use of deep neural networks.

scholarly journals Neurons vs Weights Pruning in Artificial Neural Networks

2015 ◽  
Vol 3 ◽  
pp. 22 ◽  
Author(s):  
Andrey Bondarenko ◽  
Arkady Borisov ◽  
Ludmila Alekseeva
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Classification Accuracy ◽  
Classification Rate ◽  
Good Classification ◽  
Pruning Algorithms ◽  
Artificial Neural ◽  
Accuracy Rates ◽  
Pruning Techniques ◽  
Growing Neural Networks

<p class="R-AbstractKeywords">Artificial neural networks (ANN) are well known for their good classification abilities. Recent advances in deep learning imposed second ANN renaissance. But neural networks possesses some problems like choosing hyper parameters such as neuron layers count and sizes which can greatly influence classification rate. Thus pruning techniques were developed that can reduce network sizes, increase its generalization abilities and overcome overfitting. Pruning approaches, in contrast to growing neural networks approach, assume that sufficiently large ANN is already trained and can be simplified with acceptable classification accuracy loss.</p><p class="R-AbstractKeywords">Current paper compares nodes vs weights pruning algorithms and gives experimental results for pruned networks accuracy rates versus their non-pruned counterparts. We conclude that nodes pruning is more preferable solution, with some sidenotes.</p>

Optimizing and Learning Algorithm for Feed-forward Neural Networks

2001 ◽  
Vol 5 (1) ◽  
pp. 51-57
Author(s):  
Pilar Bachiller ◽  
◽  
Julia González
Keyword(s):  
Neural Networks ◽  
Learning Algorithm ◽  
Neural Model ◽  
Training Set ◽  
Feed Forward Neural Network ◽  
Feed Forward ◽  
Training Time ◽  
Network Pruning ◽  
Feed Forward Neural Networks ◽  
Pruning Techniques

Feed-forward neural networks have emerged as a good solution for many problems, such as classification, recognition and identification, and signal processing. However, the importance of selecting an adequate hidden structure for this neural model should not be underestimated. When the hidden structure of the network is too large and complex for the model being developed, the network may tend to memorize input and output sets rather than learning relationships between them. Such a network may train well but test poorly when inputs outside the training set are presented. In addition, training time will significantly increase when the network is unnecessarily large and complex. Most of the proposed solutions to this problem consist of training a larger than necessary network, pruning unnecessary links and nodes and retraining the reduced network. We propose a new method to optimize the size of a feed-forward neural network using orthogonal transformations. This approach prunes unnecessary nodes during the training process, avoiding the retraining phase of the reduced network, which is necessary in most pruning techniques.

Unbiased sensitivity analysis and pruning techniques in neural networks for surface ozone modelling

2005 ◽  
Vol 182 (2) ◽  
pp. 149-158 ◽  
Author(s):  
O. Pastor-Bárcenas ◽  
E. Soria-Olivas ◽  
J.D. Martín-Guerrero ◽  
G. Camps-Valls ◽  
J.L. Carrasco-Rodríguez ◽  
...  
Keyword(s):  
Neural Networks ◽  
Sensitivity Analysis ◽  
Surface Ozone ◽  
Ozone Modelling ◽  
Pruning Techniques

scholarly journals Fast Convolutional Neural Networks in Low Density FPGAs Using Zero-Skipping and Weight Pruning

Electronics ◽  
2019 ◽  
Vol 8 (11) ◽  
pp. 1321 ◽  
Author(s):  
Mário P. Véstias ◽  
Rui Policarpo Duarte ◽  
José T. de Sousa ◽  
Horácio C. Neto
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Low Density ◽  
Embedded Computing ◽  
System Architectures ◽  
Field Programmable ◽  
New Processing ◽  
On Chip ◽  
Weight Pruning

Edge devices are becoming smarter with the integration of machine learning methods, such as deep learning, and are therefore used in many application domains where decisions have to be made without human intervention. Deep learning and, in particular, convolutional neural networks (CNN) are more efficient than previous algorithms for several computer vision applications such as security and surveillance, where image and video analysis are required. This better efficiency comes with a cost of high computation and memory requirements. Hence, running CNNs in embedded computing devices is a challenge for both algorithm and hardware designers. New processing devices, dedicated system architectures and optimization of the networks have been researched to deal with these computation requirements. In this paper, we improve the inference execution times of CNNs in low density FPGAs (Field-Programmable Gate Arrays) using fixed-point arithmetic, zero-skipping and weight pruning. The developed architecture supports the execution of large CNNs in FPGA devices with reduced on-chip memory and computing resources. With the proposed architecture, it is possible to infer an image in AlexNet in 2.9 ms in a ZYNQ7020 and 1.0 ms in a ZYNQ7045 with less than 1% accuracy degradation. These results improve previous state-of-the-art architectures for CNN inference.

Sign in / Sign up

Export Citation Format

Share Document