Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

2021 ◽  
Author(s):  
Jane J. Ye ◽  
Xiaoming Yuan ◽  
Shangzhi Zeng ◽  
Jin Zhang
Keyword(s):  
Convex Optimization ◽  
Variational Analysis ◽  
Optimization Problems ◽  
Linear Convergence ◽  
Nonsmooth Convex Optimization ◽  
First Order ◽  
Convex Optimization Problems ◽  
First Order Methods

scholarly journals On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Minimization

2019 ◽  
Author(s):  
Yi Xu ◽  
Zhuoning Yuan ◽  
Sen Yang ◽  
Rong Jin ◽  
Tianbao Yang
Keyword(s):  
Convex Optimization ◽  
Gradient Descent ◽  
Optimization Problems ◽  
Upper Bounds ◽  
Convex Minimization ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
First Order ◽  
Convex Optimization Problems ◽  
Learning Tasks

Extrapolation is a well-known technique for solving convex optimization and variational inequalities and recently attracts some attention for non-convex optimization. Several recent works have empirically shown its success in some machine learning tasks. However, it has not been analyzed for non-convex minimization and there still remains a gap between the theory and the practice. In this paper, we analyze gradient descent  and stochastic gradient descent with extrapolation for finding an approximate first-order stationary point in smooth non-convex optimization problems. Our convergence upper bounds show that the algorithms with extrapolation can be accelerated than without extrapolation.

scholarly journals Multivariate Spectral Gradient Algorithm for Nonsmooth Convex Optimization Problems

2015 ◽  
Vol 2015 ◽  
pp. 1-7
Author(s):  
Yaping Hu
Keyword(s):  
Convex Optimization ◽  
Optimization Problem ◽  
Optimization Problems ◽  
Gradient Algorithm ◽  
Nonsmooth Convex Optimization ◽  
Convex Optimization Problem ◽  
Convex Optimization Problems ◽  
Yosida Regularization ◽  
Approximate Function ◽  
Original Objective

We propose an extended multivariate spectral gradient algorithm to solve the nonsmooth convex optimization problem. First, by using Moreau-Yosida regularization, we convert the original objective function to a continuously differentiable function; then we use approximate function and gradient values of the Moreau-Yosida regularization to substitute the corresponding exact values in the algorithm. The global convergence is proved under suitable assumptions. Numerical experiments are presented to show the effectiveness of this algorithm.

Sign in / Sign up

Export Citation Format

Share Document