scholarly journals Gradient-Sensitive Optimization for Convolutional Neural Networks

2021 ◽  
Vol 2021 ◽  
pp. 1-16
Author(s):  
Zhipeng Liu ◽  
Rui Feng ◽  
Xiuhan Li ◽  
Wei Wang ◽  
Xiaoling Wu
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Adaptive Learning ◽  
Gradient Descent ◽  
Recognition Rate ◽  
Learning Rate ◽  
Basic Algorithm ◽  
Descent Algorithms ◽  
Moment Estimation ◽  
Effective Models

Convolutional neural networks (CNNs) are effective models for image classification and recognition. Gradient descent optimization (GD) is the basic algorithm for CNN model optimization. Since GD appeared, a series of improved algorithms have been derived. Among these algorithms, adaptive moment estimation (Adam) has been widely recognized. However, local changes are ignored in Adam to some extent. In this paper, we introduce an adaptive learning rate factor based on current and recent gradients. According to this factor, we can dynamically adjust the learning rate of each independent parameter to adaptively adjust the global convergence process. We use the factor to adjust the learning rate for each parameter. The convergence of the proposed algorithm is proven by using the regret bound approach of the online learning framework. In the experimental section, comparisons are conducted between the proposed algorithm and other existing algorithms, such as AdaGrad, RMSprop, Adam, diffGrad, and AdaHMG, on test functions and the MNIST dataset. The results show that Adam and RMSprop combined with our algorithm can not only find the global minimum faster in the experiment using the test function but also have a better convergence curve and higher test set accuracy in experiments using datasets. Our algorithm is a supplement to the existing gradient descent algorithms, which can be combined with many other existing gradient descent algorithms to improve the efficiency of iteration, speed up the convergence of the cost function, and improve the final recognition rate.

scholarly journals Plant Diseases Identification through a Discount Momentum Optimizer in Deep Learning

2021 ◽  
Vol 11 (20) ◽  
pp. 9468
Author(s):  
Yunyun Sun ◽  
Yutong Liu ◽  
Haocheng Zhou ◽  
Huijuan Hu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Adaptive Learning ◽  
Learning Rate ◽  
Plant Diseases ◽  
Stochastic Gradient Descent ◽  
Automatic Identification ◽  
Deep Convolutional Neural Networks ◽  
Adaptive Learning Rate

Deep learning proves its promising results in various domains. The automatic identification of plant diseases with deep convolutional neural networks attracts a lot of attention at present. This article extends stochastic gradient descent momentum optimizer and presents a discount momentum (DM) deep learning optimizer for plant diseases identification. To examine the recognition and generalization capability of the DM optimizer, we discuss the hyper-parameter tuning and convolutional neural networks models across the plantvillage dataset. We further conduct comparison experiments on popular non-adaptive learning rate methods. The proposed approach achieves an average validation accuracy of no less than 97% for plant diseases prediction on several state-of-the-art deep learning models and holds a low sensitivity to hyper-parameter settings. Experimental results demonstrate that the DM method can bring a higher identification performance, while still maintaining a competitive performance over other non-adaptive learning rate methods in terms of both training speed and generalization.

scholarly journals RSPCN: Super-Resolution of Digital Elevation Model Based on Recursive Sub-Pixel Convolutional Neural Networks

2021 ◽  
Vol 10 (8) ◽  
pp. 501
Author(s):  
Ruichen Zhang ◽  
Shaofeng Bian ◽  
Houpu Li
Keyword(s):  
Neural Networks ◽  
Digital Elevation Model ◽  
Convolutional Neural Networks ◽  
Adaptive Learning ◽  
Nearest Neighbor ◽  
Super Resolution ◽  
Recursion Theory ◽  
Research Issues ◽  
Digital Elevation ◽  
Elevation Model

The digital elevation model (DEM) is known as one kind of the most significant fundamental geographical data models. The theory, method and application of DEM are hot research issues in geography, especially in geomorphology, hydrology, soil and other related fields. In this paper, we improve the efficient sub-pixel convolutional neural networks (ESPCN) and propose recursive sub-pixel convolutional neural networks (RSPCN) to generate higher-resolution DEMs (HRDEMs) from low-resolution DEMs (LRDEMs). Firstly, the structure of RSPCN is described in detail based on recursion theory. This paper explores the effects of different training datasets, with the self-adaptive learning rate Adam algorithm optimizing the model. Furthermore, the adding-“zero” boundary method is introduced into the RSPCN algorithm as a data preprocessing method, which improves the RSPCN method’s accuracy and convergence. Extensive experiments are conducted to train the method till optimality. Finally, comparisons are made with other traditional interpolation methods, such as bicubic, nearest-neighbor and bilinear methods. The results show that our method has obvious improvements in both accuracy and robustness and further illustrate the feasibility of deep learning methods in the DEM data processing area.

scholarly journals Hardware‐Friendly Stochastic and Adaptive Learning in Memristor Convolutional Neural Networks

2021 ◽  
pp. 2100041
Author(s):  
Wei Zhang ◽  
Lunshuai Pan ◽  
Xuelong Yan ◽  
Guangchao Zhao ◽  
Hong Chen ◽  
...  
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Adaptive Learning

scholarly journals A Hybrid GA-PSO Method for Evolving Architecture and Short Connections of Deep Convolutional Neural Networks

2020 ◽  
Author(s):  
B Wang ◽  
Y Sun ◽  
Bing Xue ◽  
Mengjie Zhang
Keyword(s):  
Neural Networks ◽  
Image Classification ◽  
Convolutional Neural Networks ◽  
Network Architecture ◽  
Learning Task ◽  
Fixed Number ◽  
Learning Rate ◽  
Current Layer ◽  
Training Process ◽  
Deep Convolutional Neural Networks

© 2019, Springer Nature Switzerland AG. Image classification is a difficult machine learning task, where Convolutional Neural Networks (CNNs) have been applied for over 20 years in order to solve the problem. In recent years, instead of the traditional way of only connecting the current layer with its next layer, shortcut connections have been proposed to connect the current layer with its forward layers apart from its next layer, which has been proved to be able to facilitate the training process of deep CNNs. However, there are various ways to build the shortcut connections, it is hard to manually design the best shortcut connections when solving a particular problem, especially given the design of the network architecture is already very challenging. In this paper, a hybrid evolutionary computation (EC) method is proposed to automatically evolve both the architecture of deep CNNs and the shortcut connections. Three major contributions of this work are: Firstly, a new encoding strategy is proposed to encode a CNN, where the architecture and the shortcut connections are encoded separately; Secondly, a hybrid two-level EC method, which combines particle swarm optimisation and genetic algorithms, is developed to search for the optimal CNNs; Lastly, an adjustable learning rate is introduced for the fitness evaluations, which provides a better learning rate for the training process given a fixed number of epochs. The proposed algorithm is evaluated on three widely used benchmark datasets of image classification and compared with 12 peer Non-EC based competitors and one EC based competitor. The experimental results demonstrate that the proposed method outperforms all of the peer competitors in terms of classification accuracy.

scholarly journals Layer-Wise Compressive Training for Convolutional Neural Networks

2018 ◽  
Vol 11 (1) ◽  
pp. 7 ◽  
Author(s):  
Matteo Grimaldi ◽  
Valerio Tenace ◽  
Andrea Calimera
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Gradient Descent ◽  
Computational Models ◽  
Stochastic Gradient Descent ◽  
Training Algorithm ◽  
Heuristic Rules ◽  
Human Capabilities ◽  
Model Size ◽  
Large Model

Convolutional Neural Networks (CNNs) are brain-inspired computational models designed to recognize patterns. Recent advances demonstrate that CNNs are able to achieve, and often exceed, human capabilities in many application domains. Made of several millions of parameters, even the simplest CNN shows large model size. This characteristic is a serious concern for the deployment on resource-constrained embedded-systems, where compression stages are needed to meet the stringent hardware constraints. In this paper, we introduce a novel accuracy-driven compressive training algorithm. It consists of a two-stage flow: first, layers are sorted by means of heuristic rules according to their significance; second, a modified stochastic gradient descent optimization is applied on less significant layers such that their representation is collapsed into a constrained subspace. Experimental results demonstrate that our approach achieves remarkable compression rates with low accuracy loss (<1%).

scholarly journals An Adaptive Learning Rate for RBFNN Using Time-Domain Feedback Analysis

2014 ◽  
Vol 2014 ◽  
pp. 1-9 ◽  
Author(s):  
Syed Saad Azhar Ali ◽  
Muhammad Moinuddin ◽  
Kamran Raza ◽  
Syed Hasan Adil
Keyword(s):  
Neural Networks ◽  
Radial Basis Function ◽  
Adaptive Learning ◽  
Basis Function ◽  
Learning Algorithm ◽  
Time Series Prediction ◽  
Learning Rate ◽  
Theoretical Development ◽  
Feedback Analysis ◽  
Radial Basis

Radial basis function neural networks are used in a variety of applications such as pattern recognition, nonlinear identification, control and time series prediction. In this paper, the learning algorithm of radial basis function neural networks is analyzed in a feedback structure. The robustness of the learning algorithm is discussed in the presence of uncertainties that might be due to noisy perturbations at the input or to modeling mismatch. An intelligent adaptation rule is developed for the learning rate of RBFNN which gives faster convergence via an estimate of error energy while giving guarantee to thel2stability governed by the upper bounding via small gain theorem. Simulation results are presented to support our theoretical development.

Sign in / Sign up

Export Citation Format

Share Document