Sparse Deep Neural Network Optimization for Embedded Intelligence

2020 ◽  
Vol 29 (03n04) ◽  
pp. 2060002
Author(s):  
Jia Bi ◽  
Steve R. Gunn
Keyword(s):  
Neural Networks ◽  
Network Optimization ◽  
Deep Neural Networks ◽  
Training Process ◽  
Neural Network Optimization ◽  
Convex Problems ◽  
Fast Training ◽  
Regularization Techniques ◽  
Memory Resources ◽  
Stochastic Element

Deep neural networks become more popular as its ability to solve very complex pattern recognition problems. However, deep neural networks often need massive computational and memory resources, which is main reason resulting them to be difficult efficiently and entirely running on embedded platforms. This work addresses this problem by saving the computational and memory requirements of deep neural networks by proposing a variance reduced (VR)-based optimization with regularization techniques to compress the requirements of memory of models within fast training process. It is shown theoretically and experimentally that sparsity-inducing regularization can be effectively worked with the VR-based optimization whereby in the optimizer the behaviors of the stochastic element is controlled by a hyper-parameter to solve non-convex problems.

scholarly journals Diversity oriented Deep Reinforcement Learning for targeted molecule generation

2021 ◽  
Vol 13 (1) ◽  
Author(s):  
Tiago Pereira ◽  
Maryam Abbasi ◽  
Bernardete Ribeiro ◽  
Joel P. Arrais
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Reinforcement Learning ◽  
Deep Neural Networks ◽  
Chemical Space ◽  
Biological Properties ◽  
Training Process ◽  
Training Strategy ◽  
Inhibitory Power ◽  
Exploratory Strategy

AbstractIn this work, we explore the potential of deep learning to streamline the process of identifying new potential drugs through the computational generation of molecules with interesting biological properties. Two deep neural networks compose our targeted generation framework: the Generator, which is trained to learn the building rules of valid molecules employing SMILES strings notation, and the Predictor which evaluates the newly generated compounds by predicting their affinity for the desired target. Then, the Generator is optimized through Reinforcement Learning to produce molecules with bespoken properties. The innovation of this approach is the exploratory strategy applied during the reinforcement training process that seeks to add novelty to the generated compounds. This training strategy employs two Generators interchangeably to sample new SMILES: the initially trained model that will remain fixed and a copy of the previous one that will be updated during the training to uncover the most promising molecules. The evolution of the reward assigned by the Predictor determines how often each one is employed to select the next token of the molecule. This strategy establishes a compromise between the need to acquire more information about the chemical space and the need to sample new molecules, with the experience gained so far. To demonstrate the effectiveness of the method, the Generator is trained to design molecules with an optimized coefficient of partition and also high inhibitory power against the Adenosine $$A_{2A}$$ A 2 A and $$\kappa$$ κ opioid receptors. The results reveal that the model can effectively adjust the newly generated molecules towards the wanted direction. More importantly, it was possible to find promising sets of unique and diverse molecules, which was the main purpose of the newly implemented strategy.

scholarly journals Singular Learning of Deep Multilayer Perceptrons for EEG-Based Emotion Recognition

2021 ◽  
Vol 3 ◽  
Author(s):  
Weili Guo ◽  
Guangyu Li ◽  
Jianfeng Lu ◽  
Jian Yang
Keyword(s):  
Neural Networks ◽  
Emotion Recognition ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
High Reliability ◽  
Multilayer Perceptrons ◽  
Learning Technology ◽  
Human Computer Interactions ◽  
Training Process ◽  
Specific Influence

Human emotion recognition is an important issue in human–computer interactions, and electroencephalograph (EEG) has been widely applied to emotion recognition due to its high reliability. In recent years, methods based on deep learning technology have reached the state-of-the-art performance in EEG-based emotion recognition. However, there exist singularities in the parameter space of deep neural networks, which may dramatically slow down the training process. It is very worthy to investigate the specific influence of singularities when applying deep neural networks to EEG-based emotion recognition. In this paper, we mainly focus on this problem, and analyze the singular learning dynamics of deep multilayer perceptrons theoretically and numerically. The results can help us to design better algorithms to overcome the serious influence of singularities in deep neural networks for EEG-based emotion recognition.

Swarm-Based Nature-Inspired Metaheuristics for Neural Network Optimization

2018 ◽  
pp. 23-53
Author(s):  
Swathi Jamjala Narayanan ◽  
Boominathan Perumal ◽  
Jayant G. Rohra
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Optimization ◽  
Network Architectures ◽  
Neural Network Optimization ◽  
Local Optima ◽  
Gradient Based ◽  
Neural Network Architectures ◽  
Nature Inspired Algorithms ◽  
Nature Inspired Metaheuristics

Nature-inspired algorithms have been productively applied to train neural network architectures. There exist other mechanisms like gradient descent, second order methods, Levenberg-Marquardt methods etc. to optimize the parameters of neural networks. Compared to gradient-based methods, nature-inspired algorithms are found to be less sensitive towards the initial weights set and also it is less likely to become trapped in local optima. Despite these benefits, some nature-inspired algorithms also suffer from stagnation when applied to neural networks. The other challenge when applying nature inspired techniques for neural networks would be in handling large dimensional and correlated weight space. Hence, there arises a need for scalable nature inspired algorithms for high dimensional neural network optimization. In this chapter, the characteristics of nature inspired techniques towards optimizing neural network architectures along with its applicability, advantages and limitations/challenges are studied.

scholarly journals VECA: A Method for Detecting Overfitting in Neural Networks (Student Abstract)

2020 ◽  
Vol 34 (10) ◽  
pp. 13791-13792
Author(s):  
Liangzhu Ge ◽  
Yuexian Hou ◽  
Yaju Jiang ◽  
Shuai Yao ◽  
Chao Yang
Keyword(s):  
Neural Networks ◽  
Strong Correlation ◽  
Good Predictor ◽  
Deep Neural Networks ◽  
Training Data ◽  
Training Process ◽  
Generalization Performance ◽  
Validation Set ◽  
Fully Connected ◽  
Fully Connected Networks

Despite their widespread applications, deep neural networks often tend to overfit the training data. Here, we propose a measure called VECA (Variance of Eigenvalues of Covariance matrix of Activation matrix) and demonstrate that VECA is a good predictor of networks' generalization performance during the training process. Experiments performed on fully-connected networks and convolutional neural networks trained on benchmark image datasets show a strong correlation between test loss and VECA, which suggest that we can calculate the VECA to estimate generalization performance without sacrificing training data to be used as a validation set.

scholarly journals An Efficient Approach for Detecting Driver Drowsiness Based on Deep Learning

2021 ◽  
Vol 11 (18) ◽  
pp. 8441
Author(s):  
Anh-Cang Phan ◽  
Ngoc-Hoang-Quyen Nguyen  ◽  
Thanh-Ngoan Trieu ◽  
Thuong-Cang Phan
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Training Dataset ◽  
Economic Losses ◽  
Drowsy Driving ◽  
Training Time ◽  
Road Users ◽  
Fast Training ◽  
Learning Techniques

Drowsy driving is one of the common causes of road accidents resulting in injuries, even death, and significant economic losses to drivers, road users, families, and society. There have been many studies carried out in an attempt to detect drowsiness for alert systems. However, a majority of the studies focused on determining eyelid and mouth movements, which have revealed many limitations for drowsiness detection. Besides, physiological measures-based studies may not be feasible in practice because the measuring devices are often not available on vehicles and often uncomfortable for drivers. In this research, we therefore propose two efficient methods with three scenarios for doze alert systems. The former applies facial landmarks to detect blinks and yawns based on appropriate thresholds for each driver. The latter uses deep learning techniques with two adaptive deep neural networks based on MobileNet-V2 and ResNet-50V2. The second method analyzes the videos and detects driver’s activities in every frame to learn all features automatically. We leverage the advantage of the transfer learning technique to train the proposed networks on our training dataset. This solves the problem of limited training datasets, provides fast training time, and keeps the advantage of the deep neural networks. Experiments were conducted to test the effectiveness of our methods compared with other methods. Empirical results demonstrate that the proposed method using deep learning techniques can achieve a high accuracy of 97% . This study provides meaningful solutions in practice to prevent unfortunate automobile accidents caused by drowsiness.

scholarly journals Application of neural networks to the analysis of consumer opinions

2019 ◽  
Vol 13 ◽  
pp. 310-314
Author(s):  
Roman Mysan ◽  
Ivan Loichuk ◽  
Małgorzata Plechawska-Wójcik
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Optimization ◽  
Optimization Methods ◽  
User Feedback ◽  
Text Data ◽  
Neural Network Optimization ◽  
The Neural Network ◽  
Test Application

This paper presents an analysis of the possibilities of using neural networks to classify text data in the form of comments. Moreover, results of research of two neural network optimization methods: Adam and Gradient are presented. The aim of the work is to conduct research on the behavior of the neural network depending on the change of parameters and the amount of data used to teach the neural network. To achieve the goal, a test application was created. It uses a neural network to display the overall assessment of the accommodation facility based on the added user feedback.

scholarly journals Network as Regularization for Training Deep Neural Networks: Framework, Model and Performance

2020 ◽  
Vol 34 (04) ◽  
pp. 6013-6020
Author(s):  
Kai Tian ◽  
Yi Xu ◽  
Jihong Guan ◽  
Shuigeng Zhou
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Convex Combination ◽  
Network Training ◽  
Class Level ◽  
Framework Model ◽  
Target Network ◽  
Regularization Techniques ◽  
And Performance ◽  
Fitting Problem

Despite powerful representation ability, deep neural networks (DNNs) are prone to over-fitting, because of over-parametrization. Existing works have explored various regularization techniques to tackle the over-fitting problem. Some of them employed soft targets rather than one-hot labels to guide network training (e.g. label smoothing in classification tasks), which are called target-based regularization approaches in this paper. To alleviate the over-fitting problem, here we propose a new and general regularization framework that introduces an auxiliary network to dynamically incorporate guided semantic disturbance to the labels. We call it Network as Regularization (NaR in short). During training, the disturbance is constructed by a convex combination of the predictions of the target network and the auxiliary network. These two networks are initialized separately. And the auxiliary network is trained independently from the target network, while providing instance-level and class-level semantic information to the latter progressively. We conduct extensive experiments to validate the effectiveness of the proposed method. Experimental results show that NaR outperforms many state-of-the-art target-based regularization methods, and other regularization approaches (e.g. mixup) can also benefit from combining with NaR.

Sign in / Sign up

Export Citation Format

Share Document