scholarly journals Proximal-Like Incremental Aggregated Gradient Method with Linear Convergence Under Bregman Distance Growth Conditions

2019 ◽  
Author(s):  
Hui Zhang ◽  
Yu-Hong Dai ◽  
Lei Guo ◽  
Wei Peng
Keyword(s):  
Growth Condition ◽  
Gradient Method ◽  
Legendre Function ◽  
Lower Semicontinuous ◽  
Gradient Algorithm ◽  
Bregman Distance ◽  
Proximal Gradient Method ◽  
Step Size ◽  
Convergence Results ◽  
Special Cases

We introduce a unified algorithmic framework, called the proximal-like incremental aggregated gradient (PLIAG) method, for minimizing the sum of a convex function that consists of additive relatively smooth convex components and a proper lower semicontinuous convex regularization function over an abstract feasible set whose geometry can be captured by using the domain of a Legendre function. The PLIAG method includes many existing algorithms in the literature as special cases, such as the proximal gradient method, the Bregman proximal gradient method (also called the NoLips algorithm), the incremental aggregated gradient method, the incremental aggregated proximal method, and the proximal incremental aggregated gradient method. It also includes some novel interesting iteration schemes. First, we show that the PLIAG method is globally sublinearly convergent without requiring a growth condition, which extends the sublinear convergence result for the proximal gradient algorithm to incremental aggregated-type first-order methods. Then, by embedding a so-called Bregman distance growth condition into a descent-type lemma to construct a special Lyapunov function, we show that the PLIAG method is globally linearly convergent in terms of both function values and Bregman distances to the optimal solution set, provided that the step size is not greater than some positive constant. The convergence results derived in this paper are all established beyond the standard assumptions in the literature (i.e., without requiring the strong convexity and the Lipschitz gradient continuity of the smooth part of the objective). When specialized to many existing algorithms, our results recover or supplement their convergence results under strictly weaker conditions.

scholarly journals PROXIMAL ALGORITHMS FOR BI-LEVEL CONVEX OPTIMIZATION PROBLEMS

2021 ◽  
pp. 145-150
Author(s):  
A. V. Luita ◽  
S. O. Zhilina ◽  
V. V. Semenov
Keyword(s):  
Weak Convergence ◽  
Convex Function ◽  
Gradient Method ◽  
Optimization Problems ◽  
Rates Of Convergence ◽  
Convex Minimization ◽  
Gradient Algorithm ◽  
Penalty Function Method ◽  
Proximal Gradient Method ◽  
Proximal Algorithms

In this paper, problems of bi-level convex minimization in a Hilbert space are considered. The bi-level convex minimization problem is to minimize the first convex function on the set of minima of the second convex function. This setting has many applications, but the implicit constraints generated by the internal problem make it difficult to obtain optimality conditions and construct algorithms. Multilevel optimization problems are formulated in a similar way, the source of which is the operation research problems (optimization according to sequentially specified criteria or lexicographic optimization). Attention is focused on problem solving using two proximal methods. The main theoretical results are theorems on the convergence of methods in various situations. The first of the methods is obtained by combining the penalty function method and the proximal method. Strong convergence is proved in the case of strong convexity of the function of the exterior problem. In the general case, only weak convergence has been proved. The second, the so-called proximal-gradient method, is a combination of one of the variants of the fast proximal-gradient algorithm with the method of penalty functions. The rates of convergence of the proximal-gradient method and its weak convergence are proved.

scholarly journals A dual Bregman proximal gradient method for relatively-strongly convex optimization

2021 ◽  
Vol 0 (0) ◽  
pp. 0
Author(s):  
Jin-Zan Liu ◽  
Xin-Wei Liu
Keyword(s):  
Convex Optimization ◽  
Convergence Rate ◽  
Convex Function ◽  
Minimization Problem ◽  
Gradient Method ◽  
Bregman Distance ◽  
Proximal Gradient Method ◽  
Strongly Convex ◽  
Proper Convex ◽  
Strongly Convex Optimization

<p style='text-indent:20px;'>We consider a convex composite minimization problem, whose objective is the sum of a relatively-strongly convex function and a closed proper convex function. A dual Bregman proximal gradient method is proposed for solving this problem and is shown that the convergence rate of the primal sequence is <inline-formula><tex-math id="M1">\begin{document}$ O(\frac{1}{k}) $\end{document}</tex-math></inline-formula>. Moreover, based on the acceleration scheme, we prove that the convergence rate of the primal sequence is <inline-formula><tex-math id="M2">\begin{document}$ O(\frac{1}{k^{\gamma}}) $\end{document}</tex-math></inline-formula>, where <inline-formula><tex-math id="M3">\begin{document}$ \gamma\in[1,2] $\end{document}</tex-math></inline-formula> is determined by the triangle scaling property of the Bregman distance.</p>

Distributed proximal-gradient method for convex optimization with inequality constraints

2015 ◽  
Vol 56 ◽  
pp. 160 ◽  
Author(s):  
Jueyou Li ◽  
Changzhi Wu ◽  
Zhiyou Wu ◽  
Qiang Long ◽  
Xiangyu Wang
Keyword(s):  
Convex Optimization ◽  
Gradient Method ◽  
Inequality Constraints ◽  
Proximal Gradient Method

scholarly journals Learned Collaborative Stereo Refinement

2021 ◽  
Author(s):  
Patrick Knöbelreiter ◽  
Thomas Pock
Keyword(s):  
Structural Properties ◽  
Gradient Method ◽  
Color Image ◽  
Image Space ◽  
Activation Functions ◽  
Disparity Map ◽  
Proximal Gradient Method ◽  
Variational Structure ◽  
Disparity Maps ◽  
The Individual

AbstractIn this work, we propose a learning-based method to denoise and refine disparity maps. The proposed variational network arises naturally from unrolling the iterates of a proximal gradient method applied to a variational energy defined in a joint disparity, color, and confidence image space. Our method allows to learn a robust collaborative regularizer leveraging the joint statistics of the color image, the confidence map and the disparity map. Due to the variational structure of our method, the individual steps can be easily visualized, thus enabling interpretability of the method. We can therefore provide interesting insights into how our method refines and denoises disparity maps. To this end, we can visualize and interpret the learned filters and activation functions and prove the increased reliability of the predicted pixel-wise confidence maps. Furthermore, the optimization based structure of our refinement module allows us to compute eigen disparity maps, which reveal structural properties of our refinement module. The efficiency of our method is demonstrated on the publicly available stereo benchmarks Middlebury 2014 and Kitti 2015.

scholarly journals A multi-step Lagrangian scheme for spatially inhomogeneous evolutionary games

2021 ◽  
Author(s):  
Stefano Almi ◽  
Marco Morandotti ◽  
Francesco Solombrino
Keyword(s):  
Mean Field ◽  
Evolutionary Games ◽  
Type Systems ◽  
Convergence Result ◽  
Performance Criterion ◽  
Mean Field Limit ◽  
Spatially Inhomogeneous ◽  
Convergence Results ◽  
Special Cases ◽  
Lagrangian Scheme

AbstractA multi-step Lagrangian scheme at discrete times is proposed for the approximation of a nonlinear continuity equation arising as a mean-field limit of spatially inhomogeneous evolutionary games, describing the evolution of a system of spatially distributed agents with strategies, or labels, whose payoff depends also on the current position of the agents. The scheme is Lagrangian, as it traces the evolution of position and labels along characteristics, and is a multi-step scheme, as it develops on the following two stages: First, the distribution of strategies or labels is updated according to a best performance criterion, and then, this is used by the agents to evolve their position. A general convergence result is provided in the space of probability measures. In the special cases of replicator-type systems and reversible Markov chains, variants of the scheme, where the explicit step in the evolution of the labels is replaced by an implicit one, are also considered and convergence results are provided.

A new adaptive variable step size natural gradient BSS algorithm

2021 ◽  
pp. 1-12
Author(s):  
Junqing Ji ◽  
Xiaojia Kong ◽  
Yajing Zhang ◽  
Tongle Xu ◽  
Jing Zhang
Keyword(s):  
Signal To Noise Ratio ◽  
Wavelet Coefficient ◽  
Gradient Algorithm ◽  
Variable Step Size ◽  
Natural Gradient ◽  
Threshold Method ◽  
Step Size ◽  
Morphological Component Analysis ◽  
Variable Step ◽  
Separation Degree

The traditional blind source separation (BSS) algorithm is mainly used to deal with signal separation under the noiseless model, but it does not apply to data with the low signal to noise ratio (SNR). To solve the problem, an adaptive variable step size natural gradient BSS algorithm based on an improved wavelet threshold is proposed in this paper. Firstly, an improved wavelet threshold method is used to reduce the noise of the signal. Secondly, the wavelet coefficient layer with obvious periodicity is denoised using a morphological component analysis (MCA) algorithm, and the processed wavelet coefficients are recombined to obtain the ideal model. Thirdly, the recombined signal is pre-whitened, and a new separation matrix update formula of natural gradient algorithm is constructed by defining a new separation degree estimation function. Finally, the adaptive variable step size natural gradient blind source algorithm is used to separate the noise reduction signal. The results show that the algorithm can not only adaptively adjust the step size according to different signals, but also improve the convergence speed, stability and separation accuracy.

scholarly journals A Modified Three-Term Type CD Conjugate Gradient Algorithm for Unconstrained Optimization Problems

2020 ◽  
Vol 2020 ◽  
pp. 1-14
Author(s):  
Zhan Wang ◽  
Pengyuan Li ◽  
Xiangrong Li ◽  
Hongtruong Pham
Keyword(s):  
Conjugate Gradient Method ◽  
Conjugate Gradient ◽  
Gradient Method ◽  
Optimization Problems ◽  
Gradient Methods ◽  
Trust Region ◽  
Conjugate Gradient Algorithm ◽  
Gradient Algorithm ◽  
General Function ◽  
Term Type

Conjugate gradient methods are well-known methods which are widely applied in many practical fields. CD conjugate gradient method is one of the classical types. In this paper, a modified three-term type CD conjugate gradient algorithm is proposed. Some good features are presented as follows: (i) A modified three-term type CD conjugate gradient formula is presented. (ii) The given algorithm possesses sufficient descent property and trust region property. (iii) The algorithm has global convergence with the modified weak Wolfe–Powell (MWWP) line search technique and projection technique for general function. The new algorithm has made great progress in numerical experiments. It shows that the modified three-term type CD conjugate gradient method is more competitive than the classical CD conjugate gradient method.

Sign in / Sign up

Export Citation Format

Share Document