scholarly journals Augmented Lagrangian–Based First-Order Methods for Convex-Constrained Programs with Weakly Convex Objective

2021 ◽  
Author(s):  
Zichong Li ◽  
Yangyang Xu
Keyword(s):  
Large Scale ◽  
Augmented Lagrangian ◽  
Proximal Point ◽  
First Order ◽  
Kkt Point ◽  
Large Scale Problems ◽  
Complexity Result ◽  
Linearly Constrained ◽  
First Order Methods ◽  
Better Than

First-order methods (FOMs) have been widely used for solving large-scale problems. A majority of existing works focus on problems without constraint or with simple constraints. Several recent works have studied FOMs for problems with complicated functional constraints. In this paper, we design a novel augmented Lagrangian (AL)–based FOM for solving problems with nonconvex objective and convex constraint functions. The new method follows the framework of the proximal point (PP) method. On approximately solving PP subproblems, it mixes the usage of the inexact AL method (iALM) and the quadratic penalty method, whereas the latter is always fed with estimated multipliers by the iALM. The proposed method achieves the best-known complexity result to produce a near Karush–Kuhn–Tucker (KKT) point. Theoretically, the hybrid method has a lower iteration-complexity requirement than its counterpart that only uses iALM to solve PP subproblems; numerically, it can perform significantly better than a pure-penalty-based method. Numerical experiments are conducted on nonconvex linearly constrained quadratic programs. The numerical results demonstrate the efficiency of the proposed methods over existing ones.

scholarly journals First-Order Methods for Constrained Convex Programming Based on Linearized Augmented Lagrangian Function

2021 ◽  
Vol 3 (1) ◽  
pp. 89-117
Author(s):  
Yangyang Xu
Keyword(s):  
Large Scale ◽  
Augmented Lagrangian ◽  
Augmented Lagrangian Method ◽  
Linear Convergence ◽  
Classification Problem ◽  
Lagrangian Function ◽  
Augmented Lagrangian Function ◽  
Convex Programs ◽  
First Order ◽  
First Order Methods

First-order methods (FOMs) have been popularly used for solving large-scale problems. However, many existing works only consider unconstrained problems or those with simple constraint. In this paper, we develop two FOMs for constrained convex programs, where the constraint set is represented by affine equations and smooth nonlinear inequalities. Both methods are based on the classical augmented Lagrangian function. They update the multipliers in the same way as the augmented Lagrangian method (ALM) but use different primal updates. The first method, at each iteration, performs a single proximal gradient step to the primal variable, and the second method is a block update version of the first one. For the first method, we establish its global iterate convergence and global sublinear and local linear convergence, and for the second method, we show a global sublinear convergence result in expectation. Numerical experiments are carried out on the basis pursuit denoising, convex quadratically constrained quadratic programs, and the Neyman-Pearson classification problem to show the empirical performance of the proposed methods. Their numerical behaviors closely match the established theoretical results.

In Support of Second Order Optimization

1991 ◽  
Author(s):  
Qiushi Cao ◽  
Prakash Krishnaswami
Keyword(s):  
Second Order ◽  
Optimization Techniques ◽  
Future Research ◽  
Order Method ◽  
First Order ◽  
Applied Optimization ◽  
Minimum Sensitivity ◽  
First Order Methods ◽  
Better Than ◽  
Second Order Methods

Abstract The vast majority of applied optimization falls into the category of first order optimization. This paper attempts to make the case for increased use of second order optimization techniques. Some of the most serious criticisms against second order methods are discussed and are shown to have lost some of their validity in recent years. In addition, some positive advantages of second order methods are also presented. These advantages include computational efficiency, compatibility with new advances in hardware and spill-over benefits in areas such as minimum sensitivity design. A simple second order constrained optimization algorithm is developed and several examples are solved using this method. A comparison is made with first order methods in terms of the number of function evaluations. The results show that the second order method performs much better than the first order methods in this regard. The paper also suggests some directions for future research in second order optimization.

The SEISCOPE optimization toolbox: A large-scale nonlinear optimization library based on reverse communication

Geophysics ◽  
2016 ◽  
Vol 81 (2) ◽  
pp. F1-F15 ◽  
Author(s):  
Ludovic Métivier ◽  
Romain Brossier
Keyword(s):  
Nonlinear Optimization ◽  
Large Scale ◽  
Convergence Rates ◽  
Optimization Problems ◽  
Waveform Inversion ◽  
Second Order ◽  
First Order ◽  
Truncated Newton ◽  
First Order Methods ◽  
Second Order Methods

The SEISCOPE optimization toolbox is a set of FORTRAN 90 routines, which implement first-order methods (steepest-descent and nonlinear conjugate gradient) and second-order methods ([Formula: see text]-BFGS and truncated Newton), for the solution of large-scale nonlinear optimization problems. An efficient line-search strategy ensures the robustness of these implementations. The routines are proposed as black boxes easy to interface with any computational code, where such large-scale minimization problems have to be solved. Traveltime tomography, least-squares migration, or full-waveform inversion are examples of such problems in the context of geophysics. Integrating the toolbox for solving this class of problems presents two advantages. First, it helps to separate the routines depending on the physics of the problem from the ones related to the minimization itself, thanks to the reverse communication protocol. This enhances flexibility in code development and maintenance. Second, it allows us to switch easily between different optimization algorithms. In particular, it reduces the complexity related to the implementation of second-order methods. Because the latter benefit from faster convergence rates compared to first-order methods, significant improvements in terms of computational efforts can be expected.

scholarly journals Heavy-ball Algorithms Always Escape Saddle Points

2019 ◽  
Author(s):  
Tao Sun ◽  
Dongsheng Li ◽  
Zhe Quan ◽  
Hao Jiang ◽  
Shengguo Li ◽  
...  
Keyword(s):  
Saddle Point ◽  
Gradient Descent ◽  
Proximal Point Algorithm ◽  
Saddle Points ◽  
Proof Technique ◽  
Proximal Point ◽  
First Order ◽  
Random Initialization ◽  
New Space ◽  
First Order Methods

Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoid saddle points? The answer is yes! Direct using the existing proof technique for the heavy-ball algorithms is hard due to that each iteration of the heavy-ball algorithm consists of current and last points. It is impossible to formulate the algorithms as iteration like xk+1= g(xk) under some mapping g. To this end, we design a new mapping on a new space. With some transfers, the heavy-ball algorithm can be interpreted as iterations after this mapping. Theoretically, we prove that heavy-ball gradient descent enjoys larger stepsize than the gradient descent to escape saddle points to escape the saddle point. And the heavy-ball proximal point algorithm is also considered; we also proved that the algorithm can always escape the saddle point.

Sign in / Sign up

Export Citation Format

Share Document