scholarly journals NICE: Noise Injection and Clamping Estimation for Neural Network Quantization

Mathematics ◽  
2021 ◽  
Vol 9 (17) ◽  
pp. 2144
Author(s):  
Chaim Baskin ◽  
Evgenii Zheltonozhkii ◽  
Tal Rozen ◽  
Natan Liss ◽  
Yoav Chai ◽  
...  
Keyword(s):  
Neural Networks ◽  
Power Systems ◽  
Low Power ◽  
Real Time ◽  
Language Processing ◽  
Power Efficient ◽  
Noise Injection ◽  
Classification Tasks ◽  
Real Time Applications ◽  
The Cost

Convolutional Neural Networks (CNNs) are very popular in many fields including computer vision, speech recognition, natural language processing, etc. Though deep learning leads to groundbreaking performance in those domains, the networks used are very computationally demanding and are far from being able to perform in real-time applications even on a GPU, which is not power efficient and therefore does not suit low power systems such as mobile devices. To overcome this challenge, some solutions have been proposed for quantizing the weights and activations of these networks, which accelerate the runtime significantly. Yet, this acceleration comes at the cost of a larger error unless spatial adjustments are carried out. The method proposed in this work trains quantized neural networks by noise injection and a learned clamping, which improve accuracy. This leads to state-of-the-art results on various regression and classification tasks, e.g., ImageNet classification with architectures such as ResNet-18/34/50 with as low as 3 bit weights and activations. We implement the proposed solution on an FPGA to demonstrate its applicability for low-power real-time applications. The quantization code will become publicly available upon acceptance.

Meta-Heuristic Parameter Optimization for ANN and Real-Time Applications of ANN

2022 ◽  
pp. 166-201
Author(s):  
Asha Gowda Karegowda ◽  
Devika G.
Keyword(s):  
Neural Networks ◽  
Real Time ◽  
Parameter Optimization ◽  
Heuristic Algorithms ◽  
Search Space ◽  
Optimization Method ◽  
Classification Problems ◽  
Local Optima ◽  
Consistent Solution ◽  
Real Time Applications

Artificial neural networks (ANN) are often more suitable for classification problems. Even then, training of ANN is a surviving challenge task for large and high dimensional natured search space problems. These hitches are more for applications that involves process of fine tuning of ANN control parameters: weights and bias. There is no single search and optimization method that suits the weights and bias of ANN for all the problems. The traditional heuristic approach fails because of their poorer convergence speed and chances of ending up with local optima. In this connection, the meta-heuristic algorithms prove to provide consistent solution for optimizing ANN training parameters. This chapter will provide critics on both heuristics and meta-heuristic existing literature for training neural networks algorithms, applicability, and reliability on parameter optimization. In addition, the real-time applications of ANN will be presented. Finally, future directions to be explored in the field of ANN are presented which will of potential interest for upcoming researchers.

scholarly journals Real-time Neural Networks Implementation Proposal for Microcontrollers

Electronics ◽  
2020 ◽  
Vol 9 (10) ◽  
pp. 1597
Author(s):  
Caio José B. V. Guimarães ◽  
Marcelo A. C. Fernandes
Keyword(s):  
Neural Networks ◽  
Real Time ◽  
Intelligent Systems ◽  
Implementation Strategy ◽  
Processing Time ◽  
Mean Squared Error ◽  
Low Cost ◽  
Real Time Applications ◽  
Time Required ◽  
Process Classification

The adoption of intelligent systems with Artificial Neural Networks (ANNs) embedded in hardware for real-time applications currently faces a growing demand in fields such as the Internet of Things (IoT) and Machine to Machine (M2M). However, the application of ANNs in this type of system poses a significant challenge due to the high computational power required to process its basic operations. This paper aims to show an implementation strategy of a Multilayer Perceptron (MLP)-type neural network, in a microcontroller (a low-cost, low-power platform). A modular matrix-based MLP with the full classification process was implemented as was the backpropagation training in the microcontroller. The testing and validation were performed through Hardware-In-the-Loop (HIL) of the Mean Squared Error (MSE) of the training process, classification results, and the processing time of each implementation module. The results revealed a linear relationship between the values of the hyperparameters and the processing time required for classification, also the processing time concurs with the required time for many applications in the fields mentioned above. These findings show that this implementation strategy and this platform can be applied successfully in real-time applications that require the capabilities of ANNs.

scholarly journals A Homogeneous Algorithm for Motion Estimation and Compensation by Using Cellular Neural Networks

2010 ◽  
Vol 5 (5) ◽  
pp. 719 ◽  
Author(s):  
Cristian Grava ◽  
Alexandru Gacsádi ◽  
Ioan Buciu
Keyword(s):  
Neural Networks ◽  
Motion Estimation ◽  
Real Time ◽  
Cellular Neural Networks ◽  
Image Sequences ◽  
Artificial Vision ◽  
Real Time Image ◽  
Field Programmable ◽  
Real Time Applications ◽  
Time Image

In this paper we present an original implementation of a homogeneous algorithm for motion estimation and compensation in image sequences, by using Cellular Neural Networks (CNN). The CNN has been proven their efficiency in real-time image processing, because they can be implemented on a CNN chip or they can be emulated on Field Programmable Gate Array (FPGA). The motion information is obtained by using a CNN implementation of the well-known Horn & Schunck method. This information is further used in a CNN implementation of a motion-compensation method. Through our algorithm we obtain a homogeneous implementation for real-time applications in artificial vision or medical imaging. The algorithm is illustrated on some classical sequences and the results confirm the validity of our algorithm.

scholarly journals Wearable Fall Detector Using Recurrent Neural Networks

Sensors ◽  
2019 ◽  
Vol 19 (22) ◽  
pp. 4885 ◽  
Author(s):  
Francisco Luna-Perejón ◽  
Manuel Jesús Domínguez-Morales ◽  
Antón Civit-Balcells
Keyword(s):  
Neural Networks ◽  
Low Power ◽  
Real Time ◽  
Recurrent Neural Networks ◽  
Response Times ◽  
Fall Detection ◽  
High Energy ◽  
Great Accuracy ◽  
Public Health Issue ◽  
Fall Detector

Falls have become a relevant public health issue due to their high prevalence and negative effects in elderly people. Wearable fall detector devices allow the implementation of continuous and ubiquitous monitoring systems. The effectiveness for analyzing temporal signals with low energy consumption is one of the most relevant characteristics of these devices. Recurrent neural networks (RNNs) have demonstrated a great accuracy in some problems that require analyzing sequential inputs. However, getting appropriate response times in low power microcontrollers remains a difficult task due to their limited hardware resources. This work shows a feasibility study about using RNN-based deep learning models to detect both falls and falls’ risks in real time using accelerometer signals. The effectiveness of four different architectures was analyzed using the SisFall dataset at different frequencies. The resulting models were integrated into two different embedded systems to analyze the execution times and changes in the model effectiveness. Finally, a study of power consumption was carried out. A sensitivity of 88.2% and a specificity of 96.4% was obtained. The simplest models reached inference times lower than 34 ms, which implies the capability to detect fall events in real-time with high energy efficiency. This suggests that RNN models provide an effective method that can be implemented in low power microcontrollers for the creation of autonomous wearable fall detection systems in real-time.

scholarly journals Novel Pruning Techniques in Convolutional-Neural Networks

2020 ◽  
Vol 9 (4) ◽  
pp. 1541-1548
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Image Classification ◽  
Language Processing ◽  
Time Series Prediction ◽  
L1 Norm ◽  
Network Pruning ◽  
Weight Pruning ◽  
The Cost ◽  
Pruning Techniques

Deep Learning allows us to build powerful models to solve problems like image classification, time series prediction, natural language processing, etc. This is achieved at the cost of huge amounts of storage and processing requirements which are sometimes not possible in machines with limited resources. In this paper, we compare different methods which tackle this problem with network pruning. Selected few pruning methodologies from the deep learning literature were implemented to display their results. Modern neural architectures have a combination of different layers like convolutional layers, pooling layers, dense layers, etc. We compare pruning techniques for dense layers (such as unit/neuron pruning, and weight Pruning), and convolutional layers as well (using L1 norm, taylor expansion of loss to determine importance of convolutional filters, and Variable Importance in Projection using Partial Least Squares) for the image classification task. This study aims to ease the overhead in terms of optimization of the model for academic, as well as commercial, use of deep neural networks.

scholarly journals Noise and Performances Analysis of Commerical Aircrafts using Artificial Neural Networks

2015 ◽  
Vol 2 (1-2.) ◽  
Author(s):  
Şahin Yildirim
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Real Time ◽  
High Technology ◽  
Measured Parameter ◽  
The World ◽  
Real Time Applications ◽  
Artificial Neural ◽  
Parameter Values

Commerical aircrafts are very important part for airway travelling. In spite of high technology on aircrafts, there is still fatality accidents in the world. Because of this reason, it is very important criteria to analyse noises of main elements of the air-craft systems. In tis study, an aircraft’s main disturbances are analysed with proposed neural networks. Firstly, the noises of the jet, turbine and fan were measured from the aircraft. Secondly, the measured parameter values were predicted the proposed neural networks. The results of the proposed neuarl approaches were shown that this type of neural predictors will be employed to predict aircrafts unpredicted disturbances in real time applications.

Sign in / Sign up

Export Citation Format

Share Document