scholarly journals New computational guarantees for solving convex optimization problems with first order methods, via a function growth condition measure

2017 ◽  
Vol 170 (2) ◽  
pp. 445-477 ◽  
Author(s):  
Robert M. Freund ◽  
Haihao Lu
Keyword(s):  
Convex Optimization ◽  
Growth Condition ◽  
Optimization Problems ◽  
First Order ◽  
Convex Optimization Problems ◽  
First Order Methods

scholarly journals On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Minimization

2019 ◽  
Author(s):  
Yi Xu ◽  
Zhuoning Yuan ◽  
Sen Yang ◽  
Rong Jin ◽  
Tianbao Yang
Keyword(s):  
Convex Optimization ◽  
Gradient Descent ◽  
Optimization Problems ◽  
Upper Bounds ◽  
Convex Minimization ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
First Order ◽  
Convex Optimization Problems ◽  
Learning Tasks

Extrapolation is a well-known technique for solving convex optimization and variational inequalities and recently attracts some attention for non-convex optimization. Several recent works have empirically shown its success in some machine learning tasks. However, it has not been analyzed for non-convex minimization and there still remains a gap between the theory and the practice. In this paper, we analyze gradient descent  and stochastic gradient descent with extrapolation for finding an approximate first-order stationary point in smooth non-convex optimization problems. Our convergence upper bounds show that the algorithms with extrapolation can be accelerated than without extrapolation.

scholarly journals From differential equation solvers to accelerated first-order methods for convex optimization

2021 ◽  
Author(s):  
Hao Luo ◽  
Long Chen
Keyword(s):  
Differential Equation ◽  
Lyapunov Function ◽  
Convex Optimization ◽  
Convergence Rates ◽  
Optimization Problems ◽  
Point Of View ◽  
First Order ◽  
Strongly Convex ◽  
Accelerated Gradient ◽  
First Order Methods

AbstractConvergence analysis of accelerated first-order methods for convex optimization problems are developed from the point of view of ordinary differential equation solvers. A new dynamical system, called Nesterov accelerated gradient (NAG) flow, is derived from the connection between acceleration mechanism and A-stability of ODE solvers, and the exponential decay of a tailored Lyapunov function along with the solution trajectory is proved. Numerical discretizations of NAG flow are then considered and convergence rates are established via a discrete Lyapunov function. The proposed differential equation solver approach can not only cover existing accelerated methods, such as FISTA, Güler’s proximal algorithm and Nesterov’s accelerated gradient method, but also produce new algorithms for composite convex optimization that possess accelerated convergence rates. Both the convex and the strongly convex cases are handled in a unified way in our approach.

Sign in / Sign up

Export Citation Format

Share Document