INITIAL IMPROVEMENT OF THE HYBRID ACCELERATED GRADIENT DESCENT PROCESS

2018 ◽  
Vol 98 (2) ◽  
pp. 331-338 ◽  
Author(s):  
STEFAN PANIĆ ◽  
MILENA J. PETROVIĆ ◽  
MIROSLAVA MIHAJLOV CAREVIĆ
Keyword(s):  
Gradient Descent ◽  
Line Search ◽  
Initial Step ◽  
Step Length ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Convergence Properties ◽  
Backtracking Line Search ◽  
Initial Improvement ◽  
Accelerated Gradient

We improve the convergence properties of the iterative scheme for solving unconstrained optimisation problems introduced in Petrovic et al. [‘Hybridization of accelerated gradient descent method’, Numer. Algorithms (2017), doi:10.1007/s11075-017-0460-4] by optimising the value of the initial step length parameter in the backtracking line search procedure. We prove the validity of the algorithm and illustrate its advantages by numerical experiments and comparisons.

scholarly journals Multiplicative parameters in gradient descent methods

Filomat ◽  
2009 ◽  
Vol 23 (3) ◽  
pp. 23-36 ◽  
Author(s):  
Predrag Stanimirovic ◽  
Marko Miladinovic ◽  
Snezana Djordjevic
Keyword(s):  
Gradient Descent ◽  
Line Search ◽  
Main Idea ◽  
Step Length ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Calculation Algorithm ◽  
Modified Newton Method ◽  
Backtracking Line Search ◽  
Algorithm Construction

We introduced an algorithm for unconstrained optimization based on the reduction of the modified Newton method with line search into a gradient descent method. Main idea used in the algorithm construction is approximation of Hessian by a diagonal matrix. The step length calculation algorithm is based on the Taylor's development in two successive iterative points and the backtracking line search procedure.

scholarly journals A Transformation of Accelerated Double Step Size Method for Unconstrained Optimization

2015 ◽  
Vol 2015 ◽  
pp. 1-8 ◽  
Author(s):  
Predrag S. Stanimirović ◽  
Gradimir V. Milovanović ◽  
Milena J. Petrović ◽  
Nataša Z. Kontrec
Keyword(s):  
Gradient Descent ◽  
Linear Convergence ◽  
Step Length ◽  
Descent Method ◽  
Single Step ◽  
Gradient Descent Method ◽  
Step Size ◽  
Double Step ◽  
Substantial Progress ◽  
Accelerated Gradient

A reduction of the originally double step size iteration into the single step length scheme is derived under the proposed condition that relates two step lengths in the accelerated double step size gradient descent scheme. The proposed transformation is numerically tested. Obtained results confirm the substantial progress in comparison with the single step size accelerated gradient descent method defined in a classical way regarding all analyzed characteristics: number of iterations, CPU time, and number of function evaluations. Linear convergence of derived method has been proved.

scholarly journals Adaptive Natural Gradient Method for Learning of Stochastic Neural Networks in Mini-Batch Mode

2019 ◽  
Vol 9 (21) ◽  
pp. 4568
Author(s):  
Hyeyoung Park ◽  
Kwanyong Lee
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Learning Algorithm ◽  
Descent Method ◽  
Benchmark Problems ◽  
Stochastic Neural Networks ◽  
Gradient Descent Method ◽  
Natural Gradient ◽  
Convergence Properties ◽  
Data Set

Gradient descent method is an essential algorithm for learning of neural networks. Among diverse variations of gradient descent method that have been developed for accelerating learning speed, the natural gradient learning is based on the theory of information geometry on stochastic neuromanifold, and is known to have ideal convergence properties. Despite its theoretical advantages, the pure natural gradient has some limitations that prevent its practical usage. In order to get the explicit value of the natural gradient, it is required to know true probability distribution of input variables, and to calculate inverse of a matrix with the square size of the number of parameters. Though an adaptive estimation of the natural gradient has been proposed as a solution, it was originally developed for online learning mode, which is computationally inefficient for the learning of large data set. In this paper, we propose a novel adaptive natural gradient estimation for mini-batch learning mode, which is commonly adopted for big data analysis. For two representative stochastic neural network models, we present explicit rules of parameter updates and learning algorithm. Through experiments on three benchmark problems, we confirm that the proposed method has superior convergence properties to the conventional methods.

scholarly journals A Three-Term Gradient Descent Method with Subspace Techniques

2021 ◽  
Vol 2021 ◽  
pp. 1-7
Author(s):  
Shengwei Yao ◽  
Yuping Wu ◽  
Jielan Yang ◽  
Jieqiong Xu
Keyword(s):  
Numerical Experiments ◽  
Gradient Descent ◽  
Line Search ◽  
Optimization Problems ◽  
Convergence Result ◽  
Descent Method ◽  
Search Direction ◽  
Gradient Descent Method ◽  
Approximation Model ◽  
Wolfe Line Search

We proposed a three-term gradient descent method that can be well applied to address the optimization problems in this article. The search direction of the obtained method is generated in a specific subspace. Specifically, a quadratic approximation model is applied in the process of generating the search direction. In order to reduce the amount of calculation and make the best use of the existing information, the subspace was made up of the gradient of the current and prior iteration point and the previous search direction. By using the subspace-based optimization technology, the global convergence result is established under Wolfe line search. The results of numerical experiments show that the new method is effective and robust.

Hybridization of accelerated gradient descent method

2017 ◽  
Vol 79 (3) ◽  
pp. 769-786 ◽  
Author(s):  
Milena Petrović ◽  
Vladimir Rakočević ◽  
Nataša Kontrec ◽  
Stefan Panić ◽  
Dejan Ilić
Keyword(s):  
Gradient Descent ◽  
Descent Method ◽  
Gradient Descent Method ◽  
Accelerated Gradient

scholarly journals A Distributed Conjugate Gradient Online Learning Method over Networks

Complexity ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-13
Author(s):  
Cuixia Xu ◽  
Junlong Zhu ◽  
Youlin Shang ◽  
Qingtao Wu
Keyword(s):  
Conjugate Gradient ◽  
Gradient Descent ◽  
Line Search ◽  
Descent Method ◽  
Convergence Speed ◽  
Gradient Algorithm ◽  
Gradient Descent Method ◽  
Objective Functions ◽  
Regret Bound ◽  
Number Of Iterations

In a distributed online optimization problem with a convex constrained set over an undirected multiagent network, the local objective functions are convex and vary over time. Most of the existing methods used to solve this problem are based on the fastest gradient descent method. However, the convergence speed of these methods is decreased with an increase in the number of iterations. To accelerate the convergence speed of the algorithm, we present a distributed online conjugate gradient algorithm, different from a gradient method, in which the search directions are a set of vectors that are conjugated to each other and the step sizes are obtained through an accurate line search. We analyzed the convergence of the algorithm theoretically and obtained a regret bound of OT, where T is the number of iterations. Finally, numerical experiments conducted on a sensor network demonstrate the performance of the proposed algorithm.

Sign in / Sign up

Export Citation Format

Share Document