Global convergence rates for descent algorithms: Improved results for strictly convex problems

1978 ◽  
Vol 26 (3) ◽  
pp. 367-372 ◽  
Author(s):  
J. C. Allwright
Keyword(s):  
Global Convergence ◽  
Convergence Rates ◽  
Descent Algorithms ◽  
Strictly Convex ◽  
Convex Problems

scholarly journals The Strength of Nesterov's Extrapolation2019

2020 ◽  
Author(s):  
Qing Tao
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
Convergence Rates ◽  
Optimization Problems ◽  
Gradient Methods ◽  
Learning Problems ◽  
Smooth Convex ◽  
Simple Modification ◽  
Convex Problems ◽  
Hinge Loss

The extrapolation strategy raised by Nesterov, which can accelerate the convergence rate of gradient descent methods by orders of magnitude when dealing with smooth convex objective, has led to tremendous success in training machine learning tasks. In this paper, we theoretically study its strength in the convergence of individual iterates of general non-smooth convex optimization problems, which we name \textit{individual convergence}. We prove that Nesterov's extrapolation is capable of making the individual convergence of projected gradient methods optimal for general convex problems, which is now a challenging problem in the machine learning community. In light of this consideration, a simple modification of the gradient operation suffices to achieve optimal individual convergence for strongly convex problems, which can be regarded as making an interesting step towards the open question about SGD posed by Shamir \cite{shamir2012open}. Furthermore, the derived algorithms are extended to solve regularized non-smooth learning problems in stochastic settings. {\color{blue}They can serve as an alternative to the most basic SGD especially in coping with machine learning problems, where an individual output is needed to guarantee the regularization structure while keeping an optimal rate of convergence.} Typically, our method is applicable as an efficient tool for solving large-scale $l_1$-regularized hinge-loss learning problems. Several real experiments demonstrate that the derived algorithms not only achieve optimal individual convergence rates but also guarantee better sparsity than the averaged solution.

scholarly journals A Truncated Descent HS Conjugate Gradient Method and Its Global Convergence

2009 ◽  
Vol 2009 ◽  
pp. 1-13
Author(s):  
Wanyou Cheng ◽  
Zongguo Zhang
Keyword(s):  
Global Convergence ◽  
Line Search ◽  
Optimization Problems ◽  
Test Problems ◽  
Uniformly Convex ◽  
Convex Problems ◽  
Unconstrained Optimization Problems ◽  
Globally Convergent ◽  
Armijo Line Search ◽  
Numerical Performance

Recently, Zhang (2006) proposed a three-term modified HS (TTHS) method for unconstrained optimization problems. An attractive property of the TTHS method is that the direction generated by the method is always descent. This property is independent of the line search used. In order to obtain the global convergence of the TTHS method, Zhang proposed a truncated TTHS method. A drawback is that the numerical performance of the truncated TTHS method is not ideal. In this paper, we prove that the TTHS method with standard Armijo line search is globally convergent for uniformly convex problems. Moreover, we propose a new truncated TTHS method. Under suitable conditions, global convergence is obtained for the proposed method. Extensive numerical experiment show that the proposed method is very efficient for the test problems from the CUTE Library.

Global Convergence of a Cass of Quasi-Newton Methods on Convex Problems

1987 ◽  
Vol 24 (5) ◽  
pp. 1171-1190 ◽  
Author(s):  
Richard H. Byrd ◽  
Jorge Nocedal ◽  
Ya-Xiang Yuan
Keyword(s):  
Global Convergence ◽  
Newton Methods ◽  
Convex Problems ◽  
Quasi Newton

scholarly journals A New Homotopy Proximal Variable-Metric Framework for Composite Convex Minimization

2021 ◽  
Author(s):  
Quoc Tran-Dinh ◽  
Ling Liang ◽  
Kim-Chuan Toh
Keyword(s):  
Convergence Rates ◽  
Optimization Problems ◽  
Linear Convergence ◽  
Convex Minimization ◽  
Variable Metric ◽  
Complexity Bounds ◽  
Convex Problems ◽  
Convex Minimization Problems ◽  
Global Iteration ◽  
Primal Dual

This paper suggests two novel ideas to develop new proximal variable-metric methods for solving a class of composite convex optimization problems. The first idea is to utilize a new parameterization strategy of the optimality condition to design a class of homotopy proximal variable-metric algorithms that can achieve linear convergence and finite global iteration-complexity bounds. We identify at least three subclasses of convex problems in which our approach can apply to achieve linear convergence rates. The second idea is a new primal-dual-primal framework for implementing proximal Newton methods that has attractive computational features for a subclass of nonsmooth composite convex minimization problems. We specialize the proposed algorithm to solve a covariance estimation problem in order to demonstrate its computational advantages. Numerical experiments on the four concrete applications are given to illustrate the theoretical and computational advances of the new methods compared with other state-of-the-art algorithms.

scholarly journals Solving a 1-D inverse medium scattering problem using a new multi-frequency globally strictly convex objective functional

2020 ◽  
Vol 28 (5) ◽  
pp. 693-711
Author(s):  
Nguyen T. Thành ◽  
Michael V. Klibanov
Keyword(s):  
Global Convergence ◽  
Error Estimate ◽  
Scattering Problem ◽  
Projection Algorithm ◽  
New Approach ◽  
Numerical Examples ◽  
Strictly Convex ◽  
Inverse Medium ◽  
Inverse Medium Scattering ◽  
Objective Functional

AbstractWe propose a new approach to constructing globally strictly convex objective functional in a 1-D inverse medium scattering problem using multi-frequency backscattering data. The global convexity of the proposed objective functional is proved. We also prove the global convergence of the gradient projection algorithm and derive an error estimate. Numerical examples are presented to illustrate the performance of the proposed algorithm.

Sign in / Sign up

Export Citation Format

Share Document