Accelerated Primal-Dual Gradient Descent with Linesearch for Convex, Nonconvex, and Nonsmooth Optimization Problems

2019 ◽  
Vol 99 (2) ◽  
pp. 125-128 ◽  
Author(s):  
S. V. Guminov ◽  
Yu. E. Nesterov ◽  
P. E. Dvurechensky ◽  
A. V. Gasnikov
Keyword(s):  
Nonsmooth Optimization ◽  
Gradient Descent ◽  
Optimization Problems ◽  
Primal Dual ◽  
Nonconvex And Nonsmooth Optimization ◽  
Dual Gradient

scholarly journals Primal-dual accelerated gradient descent with line search for convex and nonconvex optimization problems

2019 ◽  
Vol 485 (1) ◽  
pp. 15-18
Author(s):  
S. V. Guminov ◽  
Yu. E. Nesterov ◽  
P. E. Dvurechensky ◽  
A. V. Gasnikov
Keyword(s):  
Lower Bounds ◽  
Gradient Descent ◽  
Line Search ◽  
Optimization Problems ◽  
Objective Functions ◽  
Nonconvex Optimization Problems ◽  
Exact Line Search ◽  
New Variant ◽  
Primal Dual ◽  
Accelerated Gradient

In this paper a new variant of accelerated gradient descent is proposed. The proposed method does not require any information about the objective function, uses exact line search for the practical accelerations of convergence, converges according to the well-known lower bounds for both convex and non-convex objective functions and possesses primal-dual properties. We also provide a universal version of said method, which converges according to the known lower bounds for both smooth and non-smooth problems.

A Smooth Double Proximal Primal-Dual Algorithm for a Class of Distributed Nonsmooth Optimization Problems

2020 ◽  
Vol 65 (4) ◽  
pp. 1800-1806 ◽  
Author(s):  
Yue Wei ◽  
Hao Fang ◽  
Xianlin Zeng ◽  
Jie Chen ◽  
Panos Pardalos
Keyword(s):  
Nonsmooth Optimization ◽  
Optimization Problems ◽  
Dual Algorithm ◽  
Primal Dual

scholarly journals Canonical Duality for Box Constrained Nonconvex and Nonsmooth Optimization Problems

2015 ◽  
Vol 2015 ◽  
pp. 1-9
Author(s):  
Jing Liu ◽  
Huicheng Liu
Keyword(s):  
Nonsmooth Optimization ◽  
Optimization Problems ◽  
Transformation Method ◽  
Constrained Optimization Problems ◽  
Canonical Duality Theory ◽  
Canonical Duality ◽  
Local Extrema ◽  
Triality Theory ◽  
Nonconvex And Nonsmooth Optimization ◽  
Global And Local

This paper presents an application of the canonical duality theory for box constrained nonconvex and nonsmooth optimization problems. By use of the canonical dual transformation method, which is developed recently, these very difficult constrained optimization problems inRncan be converted into the canonical dual problems, which can be solved by deterministic methods. The global and local extrema can be identified by the triality theory. Some examples are listed to illustrate the applications of the theory presented in the paper.

scholarly journals The Space Decomposition Theory for a Class of Semi-Infinite Maximum Eigenvalue Optimizations

2014 ◽  
Vol 2014 ◽  
pp. 1-12 ◽  
Author(s):  
Ming Huang ◽  
Li-Ping Pang ◽  
Xi-Jun Liang ◽  
Zun-Quan Xia
Keyword(s):  
Optimization Problems ◽  
Optimization Technique ◽  
Second Order ◽  
Performance Criteria ◽  
Dual Function ◽  
Space Decomposition ◽  
Decision Variables ◽  
Primal Dual ◽  
Dual Gradient ◽  
Derivatives Of

We study optimization problems involving eigenvalues of symmetric matrices. We present a nonsmooth optimization technique for a class of nonsmooth functions which are semi-infinite maxima of eigenvalue functions. Our strategy uses generalized gradients and𝒰𝒱space decomposition techniques suited for the norm and other nonsmooth performance criteria. For the class of max-functions, which possesses the so-called primal-dual gradient structure, we compute smooth trajectories along which certain second-order expansions can be obtained. We also give the first- and second-order derivatives of primal-dual function in the space of decision variablesRmunder some assumptions.

scholarly journals A Novel Approach for Solving Nonsmooth Optimization Problems with Application to Nonsmooth Equations

2013 ◽  
Vol 2013 ◽  
pp. 1-10
Author(s):  
Hamid Reza Erfanian ◽  
M. H. Noori Skandari ◽  
A. V. Kamyad
Keyword(s):  
Nonsmooth Optimization ◽  
Optimization Problems ◽  
Piecewise Linear ◽  
Linear Optimization ◽  
Optimal Solution ◽  
Piecewise Linear Function ◽  
Nonsmooth Equations ◽  
Smooth Functions ◽  
Generalized Derivative ◽  
Novel Approach

We present a new approach for solving nonsmooth optimization problems and a system of nonsmooth equations which is based on generalized derivative. For this purpose, we introduce the first order of generalized Taylor expansion of nonsmooth functions and replace it with smooth functions. In other words, nonsmooth function is approximated by a piecewise linear function based on generalized derivative. In the next step, we solve smooth linear optimization problem whose optimal solution is an approximate solution of main problem. Then, we apply the results for solving system of nonsmooth equations. Finally, for efficiency of our approach some numerical examples have been presented.

scholarly journals Why Does Large Batch Training Result in Poor Generalization? A Comprehensive Explanation and a Better Strategy from the Viewpoint of Stochastic Optimization

2018 ◽  
Vol 30 (7) ◽  
pp. 2005-2023 ◽  
Author(s):  
Tomoumi Takase ◽  
Satoshi Oyama ◽  
Masahito Kurihara
Keyword(s):  
Gradient Descent ◽  
Optimization Problems ◽  
Descent Method ◽  
Batch Size ◽  
Gradient Descent Method ◽  
Neural Network Training ◽  
Nonconvex Optimization Problems ◽  
Large Batch ◽  
Network Training ◽  
Comprehensive Framework

We present a comprehensive framework of search methods, such as simulated annealing and batch training, for solving nonconvex optimization problems. These methods search a wider range by gradually decreasing the randomness added to the standard gradient descent method. The formulation that we define on the basis of this framework can be directly applied to neural network training. This produces an effective approach that gradually increases batch size during training. We also explain why large batch training degrades generalization performance, which previous studies have not clarified.

Sign in / Sign up

Export Citation Format

Share Document