scholarly journals On the Redundancy in the Rank of Neural Network Parameters and Its Controllability

2021 ◽  
Vol 11 (2) ◽  
pp. 725
Author(s):  
Chanhee Lee ◽  
Young-Bum Kim ◽  
Hyesung Ji ◽  
Yeonsoo Lee ◽  
Yuna Hur ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Objective Function ◽  
Regularization Method ◽  
Network Size ◽  
Experimental Results ◽  
Low Rank ◽  
Factorization Algorithm ◽  
Feature Correlation ◽  
Rank Factorization

In this paper, we show that parameters of a neural network can have redundancy in their ranks, both theoretically and empirically. When viewed as a function from one space to another, neural networks can exhibit feature correlation and slower training due to this redundancy. Motivated by this, we propose a novel regularization method to reduce the redundancy in the rank of parameters. It is a combination of an objective function that makes the parameter rank-deficient and a dynamic low-rank factorization algorithm that gradually reduces the size of this parameter by fusing linearly dependent vectors together. This regularization-by-pruning approach leads to a neural network with better training dynamics and fewer trainable parameters. We also present experimental results that verify our claims. When applied to a neural network trained to classify images, this method provides statistically significant improvement in accuracy and 7.1 times speedup in terms of number of steps required for training. Furthermore, this approach has the side benefit of reducing the network size, which led to a model with 30.65% fewer trainable parameters.

scholarly journals Convolutional Neural Networks for Leaf Image-Based Plant Disease Classification

2019 ◽  
Vol 8 (4) ◽  
pp. 328
Author(s):  
Sachin B. Jadhav
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Network ◽  
Convolutional Neural Networks ◽  
Plant Diseases ◽  
Experimental Results ◽  
Disease Classification ◽  
Soybean Leaves ◽  
Soybean Diseases ◽  
Validation Strategy

<span lang="EN-US">Plant pathologists desire soft computing technology for accurate and reliable diagnosis of plant diseases. In this study, we propose an efficient soybean disease identification method based on a transfer learning approach by using a pre-trained convolutional neural network (CNN’s) such as AlexNet, GoogleNet, VGG16, ResNet101, and DensNet201. The proposed convolutional neural networks were trained using 1200 plant village image dataset of diseased and healthy soybean leaves, to identify three soybean diseases out of healthy leaves. Pre-trained CNN used to enable a fast and easy system implementation in practice. We used the five-fold cross-validation strategy to analyze the performance of networks. In this study, we used a pre-trained convolutional neural network as feature extractors and classifiers. The experimental results based on the proposed approach using pre-trained AlexNet, GoogleNet, VGG16, ResNet101, and DensNet201 networks achieve an accuracy of 95%, 96.4 %, 96.4 %, 92.1%, 93.6% respectively. The experimental results for the identification of soybean diseases indicated that the proposed networks model achieves the highest accuracy</span>

Prediction of slip in cable-drum systems using structured neural networks

2013 ◽  
Vol 228 (3) ◽  
pp. 441-456 ◽  
Author(s):  
Ergin Kilic ◽  
Melik Dolen
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Size ◽  
Network Architectures ◽  
Network Development ◽  
Low Velocity ◽  
Error Band ◽  
Artificial Neural ◽  
Recurrent Type ◽  
Selection Of

This study focuses on the slip prediction in a cable-drum system using artificial neural networks for the prospect of developing linear motion sensing scheme for such mechanisms. Both feed-forward and recurrent-type artificial neural network architectures are considered to capture the slip dynamics of cable-drum mechanisms. In the article, the network development is presented in a progressive (step-by-step) fashion for the purpose of not only making the design process transparent to the readers but also highlighting the corresponding challenges associated with the design phase (i.e. selection of architecture, network size, training process parameters, etc.). Prediction performances of the devised networks are evaluated rigorously via an experimental study. Finally, a structured neural network, which embodies the network with the best prediction performance, is further developed to overcome the drift observed at low velocity. The study illustrates that the resulting structured neural network could predict the slip in the mechanism within an error band of 100 µm when an absolute reference is utilized.

scholarly journals PREDETERMINED MOVEMENT OF MOBILE ROBOT USING NEURAL NETWORKS

2014 ◽  
pp. 64-68
Author(s):  
Oleh Adamiv ◽  
Vasyl Koval ◽  
Iryna Turchenko
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Mobile Robot ◽  
Formation Process ◽  
Robot Control ◽  
Experimental Results ◽  
The Road ◽  
Mobile Robot Control ◽  
Different Parts ◽  
Training Sets

This paper describes the experimental results of neural networks application for mobile robot control on predetermined trajectory of the road. There is considered the formation process of training sets for neural network, their structure and simulating features. Researches have showed robust mobile robot movement on different parts of the road.

Genetic Algorithm (GA) and Internal Energy Transfer Calculations Using Neural Network (NN) Methods

2012 ◽  
Author(s):  
Lionel Raff ◽  
Ranga Komanduri ◽  
Martin Hagan ◽  
Satish Bukkapatnam
Keyword(s):  
Neural Network ◽  
Genetic Algorithm ◽  
Neural Networks ◽  
Objective Function ◽  
Fitness Function ◽  
Solution Space ◽  
Large Ensemble ◽  
Nonlinear Functional ◽  
Squared Error ◽  
Time Required

Genetic algorithms (GA), like NNs, can be used to fit highly nonlinear functional forms, such as empirical interatomic potentials from a large ensemble of data. Briefly, a genetic algorithm uses a stochastic global search method that mimics the process of natural biological evolution. GAs operate on a population of potential solutions applying the principle of survival of the fittest to generate progressively better approximations to a solution. A new set of approximations is generated in each iteration (also known as generation) of a GA through the process of selecting individuals from the solution space according to their fitness levels, and breeding them together using operators borrowed from natural genetics. This process leads to the evolution of populations of individuals that have a higher probability of being “fitter,” i.e., better approximations of the specified potential values, than the individuals they were created from, just as in natural adaptation. The most time-consuming part in implementing a GA is often the evaluation of the objective or the fitness function. The objective function O[P] is expressed as sum squared error computed over a given large ensemble of data. Consequently, the time required for evaluating the objective function becomes an important factor. Since a GA is well suited for implementing on parallel computers, the time required for evaluating the objective function can be reduced significantly by parallel processing. A better approach would be to map out the objective function using several possible solutions concurrently or beforehand to improve computational efficiency of the GA prior to its execution, and using this information to implement the GA. This will obviate the need for cumbersome direct evaluation of the objective function. Neural networks may be best suited to map the functional relationship between the objective function and the various parameters of the specific functional form. This study presents an approach that combines the universal function approximation capability of multilayer neural networks to accelerate a GA for fitting atomic system potentials. The approach involves evaluating the objective function, which for the present application is the mean squared error (MSE) between the computed and model-estimated potential, and training a multilayer neural network with decision variables as input and the objective function as output.

Contributions to Ranking an Ergonomic Workstation, Considering the Human Effort and the Microclimate Parameters, Using Neural Networks

2013 ◽  
Vol 371 ◽  
pp. 812-816 ◽  
Author(s):  
Daniel Constantin Anghel ◽  
Nadia Belu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Experimental Results ◽  
Work Place ◽  
Feed Forward Neural Network ◽  
Linear Relationships ◽  
The Neural Network ◽  
Manufacture Industry ◽  
Human Effort ◽  
Hidden Layer

The paper presents a method to use a feed forward neural network in order to rank a working place from the manufacture industry. Neural networks excel in gathering difficult non-linear relationships between the inputs and outputs of a system. The neural network is simulated with a simple simulator: SSNN. In this paper, we considered as relevant for a work place ranking, 6 input parameters: temperature, humidity, noise, luminosity, load and frequency. The neural network designed for the study presented in this paper has 6 input neurons, 13 neurons in the hidden layer and 1 neuron in the output layer. We present also some experimental results obtained through simulations.

scholarly journals Novel Convolutional Neural Network with Variational Information Bottleneck for P300 Detection

Entropy ◽  
2020 ◽  
Vol 23 (1) ◽  
pp. 39
Author(s):  
Hongpeng Liao ◽  
Jianwu Xu ◽  
Zhuliang Yu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Network ◽  
Character Recognition ◽  
Signal To Noise Ratio ◽  
Recognition Performance ◽  
Experimental Results ◽  
Redundant Information ◽  
Promising Tool ◽  
Information Bottleneck

In the area of brain-computer interfaces (BCI), the detection of P300 is a very important technique and has a lot of applications. Although this problem has been studied for decades, it is still a tough problem in electroencephalography (EEG) signal processing owing to its high dimension features and low signal-to-noise ratio (SNR). Recently, neural networks, like conventional neural networks (CNN), has shown excellent performance on many applications. However, standard convolutional neural networks suffer from performance degradation on dealing with noisy data or data with too many redundant information. In this paper, we proposed a novel convolutional neural network with variational information bottleneck for P300 detection. Wiht the CNN architecture and information bottleneck, the proposed network termed P300-VIB-Net could remove the redundant information in data effectively. The experimental results on BCI competition data sets show that P300-VIB-Net achieves cutting-edge character recognition performance. Furthermore, the proposed model is capable of restricting the flow of irrelevant information adaptively in the network from perspective of information theory. The experimental results show that P300-VIB-Net is a promising tool for P300 detection.

scholarly journals Research on Network Layer Recursive Reduction Model Compression for Image Recognition

2021 ◽  
Vol 2021 ◽  
pp. 1-11
Author(s):  
Hongfei Ling ◽  
Weiwei Zhang ◽  
Yingjie Tao ◽  
Mi Zhou
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Model ◽  
Computational Cost ◽  
Network Size ◽  
Compression Technique ◽  
Network Layer ◽  
Network Layers ◽  
Model Compression ◽  
The Neural Network

ResNet has been widely used in the field of machine learning since it was proposed. This network model is successful in extracting features from input data by superimposing multiple layers of neural networks and thus achieves high accuracy in many applications. However, the superposition of multilayer neural networks increases their computational cost. For this reason, we propose a network model compression technique that removes multiple neural network layers from ResNet without decreasing the accuracy rate. The key idea is to provide a priority term to identify the importance of each neural network layer, and then select the unimportant layers to be removed during the training process based on the priority of the neural network layers. In addition, this paper also retrains the network model to avoid the accuracy degradation caused by the deletion of network layers. Experiments demonstrate that the network size can be reduced by 24.00%–42.86% of the number of layers without reducing the classification accuracy when classification is performed on CIFAR-10/100 and ImageNet.

scholarly journals Personal Interest Attention Graph Neural Networks for Session-Based Recommendation

Entropy ◽  
2021 ◽  
Vol 23 (11) ◽  
pp. 1500
Author(s):  
Xiangde Zhang ◽  
Yuan Zhou ◽  
Jianping Wang ◽  
Xiaojun Lu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Objective Function ◽  
Cross Entropy ◽  
Personal Interest ◽  
Entropy Loss ◽  
Convolutional Networks ◽  
The Cross ◽  
Graph Neural Networks

Session-based recommendations aim to predict a user’s next click based on the user’s current and historical sessions, which can be applied to shopping websites and APPs. Existing session-based recommendation methods cannot accurately capture the complex transitions between items. In addition, some approaches compress sessions into a fixed representation vector without taking into account the user’s interest preferences at the current moment, thus limiting the accuracy of recommendations. Considering the diversity of items and users’ interests, a personalized interest attention graph neural network (PIA-GNN) is proposed for session-based recommendation. This approach utilizes personalized graph convolutional networks (PGNN) to capture complex transitions between items, invoking an interest-aware mechanism to activate users’ interest in different items adaptively. In addition, a self-attention layer is used to capture long-term dependencies between items when capturing users’ long-term preferences. In this paper, the cross-entropy loss is used as the objective function to train our model. We conduct rich experiments on two real datasets, and the results show that PIA-GNN outperforms existing personalized session-aware recommendation methods.

scholarly journals Using Attenuation Coefficient Generating Function in Parallel Execution of Neural Networks for Solving SAT

2005 ◽  
Vol 9 (2) ◽  
pp. 121-126 ◽  
Author(s):  
Kairong Zhang ◽  
◽  
Masahiro Nagamatu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Probability Density Function ◽  
Attenuation Coefficient ◽  
Parallel Execution ◽  
Experimental Results ◽  
Attenuation Coefficients ◽  
Execution Speed ◽  
Speedup Ratio ◽  
Analog Neural Network

The satisfiability problem (SAT) is one of the most basic and important problems in computer science. We have proposed a recurrent analog neural network called Lagrange Programming neural network with Polarized High-order connections (LPPH) for the SAT, together with a method of parallel execution of LPPH. Experimental results demonstrate a high speedup ratio. Furthermore this method is very easy to realize by hardware. LPPH dynamics has an important parameter, the attenuation coefficient, known to strongly affect LPPH execution speed, but determining a good value of attenuation coefficient is difficult. Experimental results show that the parallel execution reduces this difficulty. In this paper we propose a method to assign different values of attenuation coefficients to LPPHs used in the parallel execution. The values are generated uniformly randomly or randomly using a probability density function.

Sign in / Sign up

Export Citation Format

Share Document