scholarly journals Learning Deep Models: Critical Points and Local Openness

2021 ◽  
Author(s):  
Maher Nouiehed ◽  
Meisam Razaviyayn
Keyword(s):  
Optimization Problems ◽  
Matrix Multiplication ◽  
Sufficient Conditions ◽  
Full Rank ◽  
Complete Characterization ◽  
Data Matrix ◽  
Local Optimum ◽  
Local Optima ◽  
Training Models ◽  
Matrix Formula

With the increasing popularity of nonconvex deep models, developing a unifying theory for studying the optimization problems that arise from training these models becomes very significant. Toward this end, we present in this paper a unifying landscape analysis framework that can be used when the training objective function is the composite of simple functions. Using the local openness property of the underlying training models, we provide simple sufficient conditions under which any local optimum of the resulting optimization problem is globally optimal. We first completely characterize the local openness of the symmetric and nonsymmetric matrix multiplication mapping. Then we use our characterization to (1) provide a simple proof for the classical result of Burer-Monteiro and extend it to noncontinuous loss functions; (2) show that every local optimum of two-layer linear networks is globally optimal. Unlike many existing results in the literature, our result requires no assumption on the target data matrix [Formula: see text], and input data matrix [Formula: see text]; (3) develop a complete characterization of the local/global optima equivalence of multilayer linear neural networks (we provide various counterexamples to show the necessity of each of our assumptions); and (4) show global/local optima equivalence of overparameterized nonlinear deep models having a certain pyramidal structure. In contrast to existing works, our result requires no assumption on the differentiability of the activation functions and can go beyond “full-rank” cases.

An Improved Differential Evolution Algorithm for Numerical Optimization Problems

2013 ◽  
Vol 415 ◽  
pp. 349-352
Author(s):  
Hong Wei Zhao ◽  
Hong Gang Xia
Keyword(s):  
Differential Evolution ◽  
Numerical Optimization ◽  
Optimization Problems ◽  
Differential Evolution Algorithm ◽  
Population Based ◽  
Local Optimum ◽  
Local Optima ◽  
De Algorithm ◽  
Evolution Algorithm ◽  
Improved Differential Evolution Algorithm

Differential evolution (DE) is a population-based stochastic function minimizer (or maximizer), whose simple yet powerful and straightforward features make it very attractive for numerical optimization. However, DE is easy to trapped into local optima. In this paper, an improved differential evolution algorithm (IDE) proposed to speed the convergence rate of DE and enhance the global search of DE. The IDE employed a new mutation operation and modified crossover operation. The former can rapidly enhance the convergence of the MDE, and the latter can prevent the MDE from being trapped into the local optimum effectively. Besides, we dynamic adjust the scaling factor (F) and the crossover rate (CR), which is aimed at further improving algorithm performance. Based on several benchmark experiment simulations, the IDE has demonstrated stronger convergence and stability than original differential (DE) algorithm and other algorithms (PSO and JADE) that reported in recent literature.

scholarly journals Spacecraft Multiple-Impulse Trajectory Optimization Using Differential Evolution Algorithm with Combined Mutation Strategies and Boundary-Handling Schemes

2015 ◽  
Vol 2015 ◽  
pp. 1-13 ◽  
Author(s):  
Yuehe Zhu ◽  
Hua Wang ◽  
Jin Zhang
Keyword(s):  
Differential Evolution ◽  
Trajectory Optimization ◽  
Optimization Problems ◽  
Differential Evolution Algorithm ◽  
Optimal Solution ◽  
Global Optimum ◽  
Local Optimum ◽  
Local Optima ◽  
Trial Vector ◽  
Mutation Strategies

Since most spacecraft multiple-impulse trajectory optimization problems are complex multimodal problems with boundary constraint, finding the global optimal solution based on the traditional differential evolution (DE) algorithms becomes so difficult due to the deception of many local optima and the probable existence of a bias towards suboptimal solution. In order to overcome this issue and enhance the global searching ability, an improved DE algorithm with combined mutation strategies and boundary-handling schemes is proposed. In the first stage, multiple mutation strategies are utilized, and each strategy creates a mutant vector. In the second stage, multiple boundary-handling schemes are used to simultaneously address the same infeasible trial vector. Two typical spacecraft multiple-impulse trajectory optimization problems are studied and optimized using the proposed DE method. The experimental results demonstrate that the proposed DE method efficiently overcomes the problem created by the convergence to a local optimum and obtains the global optimum with a higher reliability and convergence rate compared with some other popular evolutionary methods.

scholarly journals An Improved Moth-Flame Optimization Algorithm with Adaptation Mechanism to Solve Numerical and Mechanical Engineering Problems

Entropy ◽  
2021 ◽  
Vol 23 (12) ◽  
pp. 1637
Author(s):  
Mohammad H. Nadimi-Shahraki ◽  
Ali Fatahi ◽  
Hoda Zamani ◽  
Seyedali Mirjalili ◽  
Laith Abualigah
Keyword(s):  
Mechanical Engineering ◽  
Optimization Problems ◽  
Population Diversity ◽  
Local Optimum ◽  
Friedman Test ◽  
Exploration And Exploitation ◽  
Adaptation Mechanism ◽  
Local Optima ◽  
Engineering Problems ◽  
Moth Flame Optimization Algorithm

Moth-flame optimization (MFO) algorithm inspired by the transverse orientation of moths toward the light source is an effective approach to solve global optimization problems. However, the MFO algorithm suffers from issues such as premature convergence, low population diversity, local optima entrapment, and imbalance between exploration and exploitation. In this study, therefore, an improved moth-flame optimization (I-MFO) algorithm is proposed to cope with canonical MFO’s issues by locating trapped moths in local optimum via defining memory for each moth. The trapped moths tend to escape from the local optima by taking advantage of the adapted wandering around search (AWAS) strategy. The efficiency of the proposed I-MFO is evaluated by CEC 2018 benchmark functions and compared against other well-known metaheuristic algorithms. Moreover, the obtained results are statistically analyzed by the Friedman test on 30, 50, and 100 dimensions. Finally, the ability of the I-MFO algorithm to find the best optimal solutions for mechanical engineering problems is evaluated with three problems from the latest test-suite CEC 2020. The experimental and statistical results demonstrate that the proposed I-MFO is significantly superior to the contender algorithms and it successfully upgrades the shortcomings of the canonical MFO.

A fuzzy imperialistic competitive algorithm for optimizing convex functions

2014 ◽  
Vol 7 (2) ◽  
pp. 192-208 ◽  
Author(s):  
Mahsan Esmaeilzadeh Tarei ◽  
Bijan Abdollahi ◽  
Mohammad Nakhaei
Keyword(s):  
Fuzzy Logic ◽  
Convex Functions ◽  
Optimization Problems ◽  
Imperialist Competitive Algorithm ◽  
Optimization Techniques ◽  
Local Optimum ◽  
Optimization Strategy ◽  
Local Optima ◽  
Content Type ◽  
Competitive Algorithm

Purpose – The purpose of this paper is to describe imperialist competitive algorithm (ICA), a novel socio-politically inspired optimization strategy for proposing a fuzzy variant of this algorithm. ICA is a meta-heuristic algorithm for dealing with different optimization tasks. The basis of the algorithm is inspired by imperialistic competition. It attempts to present the social policy of imperialisms (referred to empires) to control more countries (referred to colonies) and use their sources. If one empire loses its power, among the others making a competition to take possession of it. Design/methodology/approach – In fuzzy imperialist competitive algorithm (FICA), the colonies have a degree of belonging to their imperialists and the top imperialist, as in fuzzy logic, rather than belonging completely to just one empire therefore the colonies move toward the superior empire and their relevant empires. Simultaneously for balancing the exploration and exploitation abilities of the ICA. The algorithms are used for optimization have shortcoming to deal with accuracy rate and local optimum trap and they need complex tuning procedures. FICA is proposed a way for optimizing convex function with high accuracy and avoiding to trap in local optima rather than using original ICA algorithm by implementing fuzzy logic on it. Findings – Therefore several solution procedures, including ICA, FICA, genetic algorithm, particle swarm optimization, tabu search and simulated annealing optimization algorithm are considered. Finally numerical experiments are carried out to evaluate the effectiveness of models as well as solution procedures. Test results present the suitability of the proposed fuzzy ICA for convex functions with little fluctuations. Originality/value – The proposed evolutionary algorithm, FICA, can be used in diverse areas of optimization problems where convex functions properties are appeared including, industrial planning, resource allocation, scheduling, decision making, pattern recognition and machine learning (optimization techniques; fuzzy logic; convex functions).

scholarly journals A Modified Slime Mould Algorithm for Global Optimization

2021 ◽  
Vol 2021 ◽  
pp. 1-22
Author(s):  
An-Di Tang ◽  
Shang-Qin Tang ◽  
Tong Han ◽  
Huan Zhou ◽  
Lei Xie
Keyword(s):  
Learning Strategy ◽  
Optimization Problems ◽  
Control Strategies ◽  
Population Based ◽  
Population Diversity ◽  
Local Optimum ◽  
Test Functions ◽  
Local Optima ◽  
Slime Mould ◽  
Exploitation And Exploration

Slime mould algorithm (SMA) is a population-based metaheuristic algorithm inspired by the phenomenon of slime mould oscillation. The SMA is competitive compared to other algorithms but still suffers from the disadvantages of unbalanced exploitation and exploration and is easy to fall into local optima. To address these shortcomings, an improved variant of SMA named MSMA is proposed in this paper. Firstly, a chaotic opposition-based learning strategy is used to enhance population diversity. Secondly, two adaptive parameter control strategies are proposed to balance exploitation and exploration. Finally, a spiral search strategy is used to help SMA get rid of local optimum. The superiority of MSMA is verified in 13 multidimensional test functions and 10 fixed-dimensional test functions. In addition, two engineering optimization problems are used to verify the potential of MSMA to solve real-world optimization problems. The simulation results show that the proposed MSMA outperforms other comparative algorithms in terms of convergence accuracy, convergence speed, and stability.

Optimum Design Using Search Agents

2001 ◽  
Author(s):  
Tomoyuki Miyashita ◽  
Hiroshi Yamakawa
Keyword(s):  
Information Exchange ◽  
Design Space ◽  
Nearest Neighbor ◽  
Optimization Problems ◽  
Optimization Methods ◽  
Stochastic Search ◽  
Local Optimum ◽  
Objective Functions ◽  
Local Optima ◽  
Optimization Methodology

Abstract Many optimization methods and practical softwares have been developing for many years and most of them are very effective, especially to solve practical problems. But, non-linearity of objective functions and constraint functions, which have frequently seen in practical problems, has caused a difficulty in optimization. This difficulty mainly lies in the existence of several local optimum solutions. In this study, we have proposed a new global optimization methodology that provides an information exchange mechanism in the nearest neighbor method. We have developed a simple software system, which treated each design point in optimization as an agent. Many agents can search the optima simultaneously exchanging the their information. We have defined two roles of the agents. Local search agents have roles on searching local optima by such an existing method as the steepest decent method and so on. Stochastic search agents investigate the design space by making use of the information from other agents. Through simple and several structural optimization problems, we have confirmed the advantages of the method.

Topology and Sizing Optimization of Micromixers Using Graph-Theoretical Representation and Genetic Algorithm

2017 ◽  
Author(s):  
Mitsuo Yoshimura ◽  
Koji Shimoyama ◽  
Takashi Misaka ◽  
Shigeru Obayashi
Keyword(s):  
Genetic Algorithm ◽  
Topology Optimization ◽  
Optimization Problems ◽  
Computational Cost ◽  
Kriging Model ◽  
Local Optimum ◽  
Sizing Optimization ◽  
Local Optima ◽  
Novel Approach ◽  
Design Variables

This paper proposes a novel approach for fluid topology optimization using genetic algorithm. In this study, the enhancement of mixing in the passive micromixers is considered. The efficient mixing is achieved by the grooves attached on the bottom of the microchannel and the optimal configuration of grooves is investigated. The grooves are represented based on the graph theory. The micromixers are analyzed by a CFD solver and the exploration by genetic algorithm is assisted by the Kriging model in order to reduce the computational cost. Three cases with different constraint and treatment for design variables are considered. In each case, GA found several local optima since the objective function is a multi-modal function and each local optimum revealed the specific characteristic for efficient mixing in micromixers. Moreover, we discuss the validity of the constraint for optimization problems. The results show a novel insight for design of micromixer and fluid topology optimization using genetic algorithm.

scholarly journals Solving knapsack problems using a binary gaining sharing knowledge-based optimization algorithm

2021 ◽  
Author(s):  
Prachi Agrawal ◽  
Talari Ganesh ◽  
Ali Wagdy Mohamed
Keyword(s):  
Life Span ◽  
Population Size ◽  
Linear Function ◽  
Optimization Algorithm ◽  
Optimization Problems ◽  
Search Space ◽  
Knapsack Problems ◽  
Local Optima ◽  
Knowledge Based ◽  
Two Stages

AbstractThis article proposes a novel binary version of recently developed Gaining Sharing knowledge-based optimization algorithm (GSK) to solve binary optimization problems. GSK algorithm is based on the concept of how humans acquire and share knowledge during their life span. A binary version of GSK named novel binary Gaining Sharing knowledge-based optimization algorithm (NBGSK) depends on mainly two binary stages: binary junior gaining sharing stage and binary senior gaining sharing stage with knowledge factor 1. These two stages enable NBGSK for exploring and exploitation of the search space efficiently and effectively to solve problems in binary space. Moreover, to enhance the performance of NBGSK and prevent the solutions from trapping into local optima, NBGSK with population size reduction (PR-NBGSK) is introduced. It decreases the population size gradually with a linear function. The proposed NBGSK and PR-NBGSK applied to set of knapsack instances with small and large dimensions, which shows that NBGSK and PR-NBGSK are more efficient and effective in terms of convergence, robustness, and accuracy.

scholarly journals R-Regularity of Set-Valued Mappings Under the Relaxed Constant Positive Linear Dependence Constraint Qualification with Applications to Parametric and Bilevel Optimization

2021 ◽  
Author(s):  
Patrick Mehlitz ◽  
Leonid I. Minchenko
Keyword(s):  
Linear Dependence ◽  
Constraint Qualification ◽  
Parametric Optimization ◽  
Optimization Problems ◽  
Sufficient Conditions ◽  
Bilevel Optimization ◽  
Regularity Property ◽  
Constant Rank Constraint Qualification ◽  
Existence Criterion ◽  
Set Valued Mappings

AbstractThe presence of Lipschitzian properties for solution mappings associated with nonlinear parametric optimization problems is desirable in the context of, e.g., stability analysis or bilevel optimization. An example of such a Lipschitzian property for set-valued mappings, whose graph is the solution set of a system of nonlinear inequalities and equations, is R-regularity. Based on the so-called relaxed constant positive linear dependence constraint qualification, we provide a criterion ensuring the presence of the R-regularity property. In this regard, our analysis generalizes earlier results of that type which exploited the stronger Mangasarian–Fromovitz or constant rank constraint qualification. Afterwards, we apply our findings in order to derive new sufficient conditions which guarantee the presence of R-regularity for solution mappings in parametric optimization. Finally, our results are used to derive an existence criterion for solutions in pessimistic bilevel optimization and a sufficient condition for the presence of the so-called partial calmness property in optimistic bilevel optimization.

scholarly journals Exact Markov chain-based runtime analysis of a discrete particle swarm optimization algorithm on sorting and OneMax

2021 ◽  
Author(s):  
Moritz Mühlenthaler ◽  
Alexander Raß ◽  
Manuel Schmitt ◽  
Rolf Wanka
Keyword(s):  
Particle Swarm Optimization ◽  
Markov Chain ◽  
Markov Model ◽  
Optimization Problems ◽  
Formal Analysis ◽  
Particle Swarm ◽  
Upper And Lower Bounds ◽  
Swarm Optimization ◽  
Local Optima ◽  
Sorting Problem

AbstractMeta-heuristics are powerful tools for solving optimization problems whose structural properties are unknown or cannot be exploited algorithmically. We propose such a meta-heuristic for a large class of optimization problems over discrete domains based on the particle swarm optimization (PSO) paradigm. We provide a comprehensive formal analysis of the performance of this algorithm on certain “easy” reference problems in a black-box setting, namely the sorting problem and the problem OneMax. In our analysis we use a Markov model of the proposed algorithm to obtain upper and lower bounds on its expected optimization time. Our bounds are essentially tight with respect to the Markov model. We show that for a suitable choice of algorithm parameters the expected optimization time is comparable to that of known algorithms and, furthermore, for other parameter regimes, the algorithm behaves less greedy and more explorative, which can be desirable in practice in order to escape local optima. Our analysis provides a precise insight on the tradeoff between optimization time and exploration. To obtain our results we introduce the notion of indistinguishability of states of a Markov chain and provide bounds on the solution of a recurrence equation with non-constant coefficients by integration.

Sign in / Sign up

Export Citation Format

Share Document