scholarly journals A New Homotopy Proximal Variable-Metric Framework for Composite Convex Minimization

2021 ◽  
Author(s):  
Quoc Tran-Dinh ◽  
Ling Liang ◽  
Kim-Chuan Toh
Keyword(s):  
Convergence Rates ◽  
Optimization Problems ◽  
Linear Convergence ◽  
Convex Minimization ◽  
Variable Metric ◽  
Complexity Bounds ◽  
Convex Problems ◽  
Convex Minimization Problems ◽  
Global Iteration ◽  
Primal Dual

This paper suggests two novel ideas to develop new proximal variable-metric methods for solving a class of composite convex optimization problems. The first idea is to utilize a new parameterization strategy of the optimality condition to design a class of homotopy proximal variable-metric algorithms that can achieve linear convergence and finite global iteration-complexity bounds. We identify at least three subclasses of convex problems in which our approach can apply to achieve linear convergence rates. The second idea is a new primal-dual-primal framework for implementing proximal Newton methods that has attractive computational features for a subclass of nonsmooth composite convex minimization problems. We specialize the proposed algorithm to solve a covariance estimation problem in order to demonstrate its computational advantages. Numerical experiments on the four concrete applications are given to illustrate the theoretical and computational advances of the new methods compared with other state-of-the-art algorithms.

scholarly journals The Strength of Nesterov's Extrapolation2019

2020 ◽  
Author(s):  
Qing Tao
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
Convergence Rates ◽  
Optimization Problems ◽  
Gradient Methods ◽  
Learning Problems ◽  
Smooth Convex ◽  
Simple Modification ◽  
Convex Problems ◽  
Hinge Loss

The extrapolation strategy raised by Nesterov, which can accelerate the convergence rate of gradient descent methods by orders of magnitude when dealing with smooth convex objective, has led to tremendous success in training machine learning tasks. In this paper, we theoretically study its strength in the convergence of individual iterates of general non-smooth convex optimization problems, which we name \textit{individual convergence}. We prove that Nesterov's extrapolation is capable of making the individual convergence of projected gradient methods optimal for general convex problems, which is now a challenging problem in the machine learning community. In light of this consideration, a simple modification of the gradient operation suffices to achieve optimal individual convergence for strongly convex problems, which can be regarded as making an interesting step towards the open question about SGD posed by Shamir \cite{shamir2012open}. Furthermore, the derived algorithms are extended to solve regularized non-smooth learning problems in stochastic settings. {\color{blue}They can serve as an alternative to the most basic SGD especially in coping with machine learning problems, where an individual output is needed to guarantee the regularization structure while keeping an optimal rate of convergence.} Typically, our method is applicable as an efficient tool for solving large-scale $l_1$-regularized hinge-loss learning problems. Several real experiments demonstrate that the derived algorithms not only achieve optimal individual convergence rates but also guarantee better sparsity than the averaged solution.

scholarly journals A primal-dual dynamical approach to structured convex minimization problems

2020 ◽  
Vol 269 (12) ◽  
pp. 10717-10757 ◽  
Author(s):  
Radu Ioan Boţ ◽  
Ernö Robert Csetnek ◽  
Szilárd Csaba László
Keyword(s):  
Convex Minimization ◽  
Minimization Problems ◽  
Dynamical Approach ◽  
Convex Minimization Problems ◽  
Primal Dual

scholarly journals An interior point algorithm for solving linear optimization problems using a new trigonometric kernel function

Filomat ◽  
2020 ◽  
Vol 34 (5) ◽  
pp. 1471-1486
Author(s):  
S. Fathi-Hafshejani ◽  
Reza Peyghami
Keyword(s):  
Kernel Function ◽  
Interior Point ◽  
Optimization Problems ◽  
Linear Optimization ◽  
Interior Point Algorithm ◽  
Worst Case ◽  
Complexity Bounds ◽  
Linear Optimization Problems ◽  
Primal Dual ◽  
The Given

In this paper, a primal-dual interior point algorithm for solving linear optimization problems based on a new kernel function with a trigonometric barrier term which is not only used for determining the search directions but also for measuring the distance between the given iterate and the ?-center for the algorithm is proposed. Using some simple analysis tools and prove that our algorithm based on the new proposed trigonometric kernel function meets O (?n log n log n/?) and O (?n log n/?) as the worst case complexity bounds for large and small-update methods. Finally, some numerical results of performing our algorithm are presented.

A randomized incremental primal-dual method for decentralized consensus optimization

2019 ◽  
pp. 1-25
Author(s):  
Chenxi Chen ◽  
Yunmei Chen ◽  
Xiaojing Ye
Keyword(s):  
Convergence Rates ◽  
Optimization Problems ◽  
Dual Method ◽  
Selection Strategy ◽  
Consensus Algorithms ◽  
Uniform Sampling ◽  
Primal Dual ◽  
Multi Agent ◽  
Dual Variables ◽  
Consensus Optimization

We consider a class of convex decentralized consensus optimization problems over connected multi-agent networks. Each agent in the network holds its local objective function privately, and can only communicate with its directly connected agents during the computation to find the minimizer of the sum of all objective functions. We propose a randomized incremental primal-dual method to solve this problem, where the dual variable over the network in each iteration is only updated at a randomly selected node, whereas the dual variables elsewhere remain the same as in the previous iteration. Thus, the communication only occurs in the neighborhood of the selected node in each iteration and hence can greatly reduce the chance of communication delay and failure in the standard fully synchronized consensus algorithms. We provide comprehensive convergence analysis including convergence rates of the primal residual and consensus error of the proposed algorithm, and conduct numerical experiments to show its performance using both uniform sampling and important sampling as node selection strategy.

scholarly journals On the linear convergence of the general first order primal-dual algorithm

2021 ◽  
Vol 0 (0) ◽  
pp. 0
Author(s):  
Kai Wang ◽  
Deren Han
Keyword(s):  
Linear Convergence ◽  
Analytical Framework ◽  
Dual Algorithm ◽  
Linear Rate ◽  
First Order ◽  
Convex Problems ◽  
Primal Dual ◽  
The One ◽  
Weaker Conditions ◽  
Special Case

<p style='text-indent:20px;'>In this paper, we consider the general first order primal-dual algorithm, which covers several recent popular algorithms such as the one proposed in [Chambolle, A. and Pock T., A first-order primal-dual algorithm for convex problems with applications to imaging, J. Math. Imaging Vis., 40 (2011) 120-145] as a special case. Under suitable conditions, we prove its global convergence and analyze its linear rate of convergence. As compared to the results in the literature, we derive the corresponding results for the general case and under weaker conditions. Furthermore, the global linear rate of the linearized primal-dual algorithm is established in the same analytical framework.</p>

scholarly journals The Strength of Nesterov's Extrapolation2019

2020 ◽  
Author(s):  
Qing Tao
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
Convergence Rates ◽  
Optimization Problems ◽  
Gradient Methods ◽  
Learning Problems ◽  
Smooth Convex ◽  
Simple Modification ◽  
Convex Problems ◽  
Hinge Loss

The extrapolation strategy raised by Nesterov, which can accelerate the convergence rate of gradient descent methods by orders of magnitude when dealing with smooth convex objective, has led to tremendous success in training machine learning tasks. In this paper, we theoretically study its strength in the convergence of individual iterates of general non-smooth convex optimization problems, which we name \textit{individual convergence}. We prove that Nesterov's extrapolation is capable of making the individual convergence of projected gradient methods optimal for general convex problems, which is now a challenging problem in the machine learning community. In light of this consideration, a simple modification of the gradient operation suffices to achieve optimal individual convergence for strongly convex problems, which can be regarded as making an interesting step towards the open question about SGD posed by Shamir \cite{shamir2012open}. Furthermore, the derived algorithms are extended to solve regularized non-smooth learning problems in stochastic settings. {\color{blue}They can serve as an alternative to the most basic SGD especially in coping with machine learning problems, where an individual output is needed to guarantee the regularization structure while keeping an optimal rate of convergence.} Typically, our method is applicable as an efficient tool for solving large-scale $l_1$-regularized hinge-loss learning problems. Several real experiments demonstrate that the derived algorithms not only achieve optimal individual convergence rates but also guarantee better sparsity than the averaged solution.

Sign in / Sign up

Export Citation Format

Share Document