scholarly journals Adaptive cubic regularization methods with dynamic inexact Hessian information and applications to finite-sum minimization

2020 ◽  
Author(s):  
Stefania Bellavia ◽  
Gianmarco Gurioli ◽  
Benedetta Morini
Keyword(s):  
Theoretical Analysis ◽  
Numerical Experiments ◽  
Large Scale ◽  
Optimization Problems ◽  
Regularization Methods ◽  
Worst Case ◽  
Nonconvex Optimization Problems ◽  
New Variant ◽  
Cubic Regularization ◽  
Finite Sum

Abstract We consider the adaptive regularization with cubics approach for solving nonconvex optimization problems and propose a new variant based on inexact Hessian information chosen dynamically. The theoretical analysis of the proposed procedure is given. The key property of ARC framework, constituted by optimal worst-case function/derivative evaluation bounds for first- and second-order critical point, is guaranteed. Application to large-scale finite-sum minimization based on subsampled Hessian is discussed and analyzed in both a deterministic and probabilistic manner, and equipped with numerical experiments on synthetic and real datasets.

scholarly journals Investigation of Improved Cooperative Coevolution for Large-Scale Global Optimization Problems

Algorithms ◽  
2021 ◽  
Vol 14 (5) ◽  
pp. 146
Author(s):  
Aleksei Vakhnin ◽  
Evgenii Sopov
Keyword(s):  
Global Optimization ◽  
Evolutionary Algorithms ◽  
Numerical Experiments ◽  
Large Scale ◽  
Optimization Problems ◽  
State Of The Art ◽  
Fixed Number ◽  
High Dimensional ◽  
Cooperative Coevolution ◽  
Large Scale Problems

Modern real-valued optimization problems are complex and high-dimensional, and they are known as “large-scale global optimization (LSGO)” problems. Classic evolutionary algorithms (EAs) perform poorly on this class of problems because of the curse of dimensionality. Cooperative Coevolution (CC) is a high-performed framework for performing the decomposition of large-scale problems into smaller and easier subproblems by grouping objective variables. The efficiency of CC strongly depends on the size of groups and the grouping approach. In this study, an improved CC (iCC) approach for solving LSGO problems has been proposed and investigated. iCC changes the number of variables in subcomponents dynamically during the optimization process. The SHADE algorithm is used as a subcomponent optimizer. We have investigated the performance of iCC-SHADE and CC-SHADE on fifteen problems from the LSGO CEC’13 benchmark set provided by the IEEE Congress of Evolutionary Computation. The results of numerical experiments have shown that iCC-SHADE outperforms, on average, CC-SHADE with a fixed number of subcomponents. Also, we have compared iCC-SHADE with some state-of-the-art LSGO metaheuristics. The experimental results have shown that the proposed algorithm is competitive with other efficient metaheuristics.

scholarly journals A Simple Sufficient Descent Method for Unconstrained Optimization

2010 ◽  
Vol 2010 ◽  
pp. 1-9 ◽  
Author(s):  
Ming-Liang Zhang ◽  
Yun-Hai Xiao ◽  
Dangzhen Zhou
Keyword(s):  
Unconstrained Optimization ◽  
Numerical Experiments ◽  
Large Scale ◽  
Optimization Problems ◽  
Unconstrained Minimization ◽  
Descent Method ◽  
Minimization Problems ◽  
Unconstrained Optimization Problems ◽  
Sufficient Descent ◽  
Large Scale Unconstrained Optimization

We develop a sufficient descent method for solving large-scale unconstrained optimization problems. At each iteration, the search direction is a linear combination of the gradient at the current and the previous steps. An attractive property of this method is that the generated directions are always descent. Under some appropriate conditions, we show that the proposed method converges globally. Numerical experiments on some unconstrained minimization problems from CUTEr library are reported, which illustrate that the proposed method is promising.

Research on MPI-Based Parallel Max-Min Ant System

2012 ◽  
Vol 198-199 ◽  
pp. 1321-1326 ◽  
Author(s):  
Yu Liu ◽  
Guo Dong Wu
Keyword(s):  
Combinatorial Optimization ◽  
Numerical Experiments ◽  
Large Scale ◽  
Optimization Problems ◽  
Computation Time ◽  
Traveling Salesman ◽  
Ant System ◽  
Combinatorial Optimization Problems ◽  
The Traveling Salesman Problem

When solving large scale combinatorial optimization problems, Max-Min Ant System requires long computation time. MPI-based Parallel Max-Min Ant System described in this paper can ensure the quality of the solution, as well as reduce the computation time. Numerical experiments on the multi-node cluster system show that when solving the traveling salesman problem, MPI-based Parallel Max-Min Ant System can get better computational efficiency.

scholarly journals Primal-dual accelerated gradient descent with line search for convex and nonconvex optimization problems

2019 ◽  
Vol 485 (1) ◽  
pp. 15-18
Author(s):  
S. V. Guminov ◽  
Yu. E. Nesterov ◽  
P. E. Dvurechensky ◽  
A. V. Gasnikov
Keyword(s):  
Lower Bounds ◽  
Gradient Descent ◽  
Line Search ◽  
Optimization Problems ◽  
Objective Functions ◽  
Nonconvex Optimization Problems ◽  
Exact Line Search ◽  
New Variant ◽  
Primal Dual ◽  
Accelerated Gradient

In this paper a new variant of accelerated gradient descent is proposed. The proposed method does not require any information about the objective function, uses exact line search for the practical accelerations of convergence, converges according to the well-known lower bounds for both convex and non-convex objective functions and possesses primal-dual properties. We also provide a universal version of said method, which converges according to the known lower bounds for both smooth and non-smooth problems.

scholarly journals An inexact regularized Newton framework with a worst-case iteration complexity of $ {\mathscr O}(\varepsilon^{-3/2}) $ for nonconvex optimization

2018 ◽  
Vol 39 (3) ◽  
pp. 1296-1327 ◽  
Author(s):  
Frank E Curtis ◽  
Daniel P Robinson ◽  
Mohammadreza Samadi
Keyword(s):  
Nonconvex Optimization ◽  
Optimization Problems ◽  
Trust Region ◽  
Newton Methods ◽  
Positive Real ◽  
Worst Case ◽  
Nonconvex Optimization Problems ◽  
Trust Region Algorithm ◽  
Wide Range ◽  
Regularized Newton Methods

Abstract An algorithm for solving smooth nonconvex optimization problems is proposed that, in the worst-case, takes ${\mathscr O}(\varepsilon ^{-3/2})$ iterations to drive the norm of the gradient of the objective function below a prescribed positive real number $\varepsilon $ and can take ${\mathscr O}(\varepsilon ^{-3})$ iterations to drive the leftmost eigenvalue of the Hessian of the objective above $-\varepsilon $. The proposed algorithm is a general framework that covers a wide range of techniques including quadratically and cubically regularized Newton methods, such as the Adaptive Regularization using Cubics (arc) method and the recently proposed Trust-Region Algorithm with Contractions and Expansions (trace). The generality of our method is achieved through the introduction of generic conditions that each trial step is required to satisfy, which in particular allows for inexact regularized Newton steps to be used. These conditions center around a new subproblem that can be approximately solved to obtain trial steps that satisfy the conditions. A new instance of the framework, distinct from arc and trace, is described that may be viewed as a hybrid between quadratically and cubically regularized Newton methods. Numerical results demonstrate that our hybrid algorithm outperforms a cubically regularized Newton method.

scholarly journals Inexact Nonconvex Newton-Type Methods

2021 ◽  
pp. ijoo.2019.0043
Author(s):  
Zhewei Yao ◽  
Peng Xu ◽  
Fred Roosta ◽  
Michael W. Mahoney
Keyword(s):  
Machine Learning ◽  
Research Program ◽  
Large Scale ◽  
Trust Region ◽  
Optimization Methods ◽  
Worst Case ◽  
The Third ◽  
Cubic Regularization ◽  
Machine Learning Applications ◽  
Fourth Author

The paper aims to extend the theory and application of nonconvex Newton-type methods, namely trust region and cubic regularization, to the settings in which, in addition to the solution of subproblems, the gradient and the Hessian of the objective function are approximated. Using certain conditions on such approximations, the paper establishes optimal worst-case iteration complexities as the exact counterparts. This paper is part of a broader research program on designing, analyzing, and implementing efficient second-order optimization methods for large-scale machine learning applications. The authors were based at UC Berkeley when the idea of the project was conceived. The first two authors were PhD students, the third author was a postdoc, all supervised by the fourth author.

scholarly journals The application of new conjugate gradient methods in estimating data

2018 ◽  
Vol 7 (2.14) ◽  
pp. 25 ◽  
Author(s):  
Syazni Shoid ◽  
Norrlaili Shapiee ◽  
Norhaslinda Zull ◽  
Nur Hamizah Abdul Ghani ◽  
Nur Syarafina Mohamed ◽  
...  
Keyword(s):  
Conjugate Gradient ◽  
Numerical Experiments ◽  
Large Scale ◽  
Line Search ◽  
Optimization Problems ◽  
Real Life ◽  
Gradient Methods ◽  
Conjugate Gradient Methods ◽  
Large Scale Optimization ◽  
Scale Optimization

Many researchers are intended to improve the conjugate gradient (CG) methods as well as their applications in real life. Besides, CG become more interesting and useful in many disciplines and has important role for solving large-scale optimization problems. In this paper, three types of new CG coefficients are presented with application in estimating data. Numerical experiments show that the proposed methods have succeeded in solving problems under strong Wolfe Powell line search conditions. 

Sign in / Sign up

Export Citation Format

Share Document