An Adaptive Learning Rate Schedule for SIGNSGD Optimizer in Neural Networks

2021 ◽  
Author(s):  
Kang Wang ◽  
Tao Sun ◽  
Yong Dou
Keyword(s):  
Neural Networks ◽  
Adaptive Learning ◽  
Learning Rate ◽  
Adaptive Learning Rate

CONSTRUCTIVE APPROACHES FOR TRAINING OF WAVELET NEURAL NETWORKS USING ADAPTIVE LEARNING RATE

2013 ◽  
Vol 11 (03) ◽  
pp. 1350021
Author(s):  
MOHAMED ZINE EL ABIDINE SKHIRI ◽  
MOHAMED CHTOUROU
Keyword(s):  
Neural Networks ◽  
Adaptive Learning ◽  
Fixed Number ◽  
Stability Theorem ◽  
Learning Rate ◽  
Training Algorithms ◽  
Lyapunov Stability Theorem ◽  
Constructive Approach ◽  
Wavelet Neural Networks ◽  
Adaptive Learning Rate

This paper investigates the applicability of the constructive approach proposed in Ref. 1 to wavelet neural networks (WNN). In fact, two incremental training algorithms will be presented. The first one, known as one pattern at a time (OPAT) approach, is the WNN version of the method applied in Ref. 1. The second approach however proposes a modified version of Ref. 1, known as one epoch at a time (OEAT) approach. In the OPAT approach, the input patterns are trained incrementally one by one until all patterns are presented. If the algorithm gets stuck in a local minimum and could not escape after a fixed number of successive attempts, then a new wavelet called also wavelon, will be recruited. In the OEAT approach however, all the input patterns are presented one epoch at a time. During one epoch, each pattern is trained only once until all patterns are trained. If the resulting overall error is reduced, then all the patterns will be retrained for one more epoch. Otherwise, a new wavelon will be recruited. To guarantee the convergence of the trained networks, an adaptive learning rate has been introduced using the discrete Lyapunov stability theorem.

scholarly journals Plant Diseases Identification through a Discount Momentum Optimizer in Deep Learning

2021 ◽  
Vol 11 (20) ◽  
pp. 9468
Author(s):  
Yunyun Sun ◽  
Yutong Liu ◽  
Haocheng Zhou ◽  
Huijuan Hu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Adaptive Learning ◽  
Learning Rate ◽  
Plant Diseases ◽  
Stochastic Gradient Descent ◽  
Automatic Identification ◽  
Deep Convolutional Neural Networks ◽  
Adaptive Learning Rate

Deep learning proves its promising results in various domains. The automatic identification of plant diseases with deep convolutional neural networks attracts a lot of attention at present. This article extends stochastic gradient descent momentum optimizer and presents a discount momentum (DM) deep learning optimizer for plant diseases identification. To examine the recognition and generalization capability of the DM optimizer, we discuss the hyper-parameter tuning and convolutional neural networks models across the plantvillage dataset. We further conduct comparison experiments on popular non-adaptive learning rate methods. The proposed approach achieves an average validation accuracy of no less than 97% for plant diseases prediction on several state-of-the-art deep learning models and holds a low sensitivity to hyper-parameter settings. Experimental results demonstrate that the DM method can bring a higher identification performance, while still maintaining a competitive performance over other non-adaptive learning rate methods in terms of both training speed and generalization.

Active random noise control using adaptive learning rate neural networks with an immune feedback law

2005 ◽  
Author(s):  
Minoru Sasaki ◽  
Takumi Kuribayashi ◽  
Satoshi Ito
Keyword(s):  
Neural Networks ◽  
Adaptive Learning ◽  
Noise Control ◽  
Random Noise ◽  
Learning Rate ◽  
Adaptive Learning Rate

scholarly journals Adaptive Learning Rate and Momentum for Training Deep Neural Networks

2021 ◽  
pp. 381-396
Author(s):  
Zhiyong Hao ◽  
Yixuan Jiang ◽  
Huihua Yu ◽  
Hsiao-Dong Chiang
Keyword(s):  
Neural Networks ◽  
Adaptive Learning ◽  
Deep Neural Networks ◽  
Learning Rate ◽  
Adaptive Learning Rate

scholarly journals Training Deep Neural Networks Using Conjugate Gradient-like Methods

Electronics ◽  
2020 ◽  
Vol 9 (11) ◽  
pp. 1809
Author(s):  
Hideaki Iiduka ◽  
Yu Kobayashi
Keyword(s):  
Neural Networks ◽  
Nonconvex Optimization ◽  
Adaptive Learning ◽  
Stationary Point ◽  
Optimization Problem ◽  
Deep Neural Networks ◽  
Optimization Algorithms ◽  
Learning Rate ◽  
Adaptive Learning Rate ◽  
Rate Optimization

The goal of this article is to train deep neural networks that accelerate useful adaptive learning rate optimization algorithms such as AdaGrad, RMSProp, Adam, and AMSGrad. To reach this goal, we devise an iterative algorithm combining the existing adaptive learning rate optimization algorithms with conjugate gradient-like methods, which are useful for constrained optimization. Convergence analyses show that the proposed algorithm with a small constant learning rate approximates a stationary point of a nonconvex optimization problem in deep learning. Furthermore, it is shown that the proposed algorithm with diminishing learning rates converges to a stationary point of the nonconvex optimization problem. The convergence and performance of the algorithm are demonstrated through numerical comparisons with the existing adaptive learning rate optimization algorithms for image and text classification. The numerical results show that the proposed algorithm with a constant learning rate is superior for training neural networks.

Sign in / Sign up

Export Citation Format

Share Document