scholarly journals An Accelerated Proximal Gradient Algorithm for Singly Linearly Constrained Quadratic Programs with Box Constraints

2013 ◽  
Vol 2013 ◽  
pp. 1-6
Author(s):  
Congying Han ◽  
Mingqiang Li ◽  
Tong Zhao ◽  
Tiande Guo
Keyword(s):  
Optimization Problems ◽  
Piecewise Linear ◽  
Optimal Solution ◽  
Gradient Algorithm ◽  
Support Vector ◽  
Box Constraints ◽  
Quadratic Programs ◽  
Wide Range ◽  
Proximal Gradient Algorithm ◽  
Linearly Constrained

Recently, the existed proximal gradient algorithms had been used to solve non-smooth convex optimization problems. As a special nonsmooth convex problem, the singly linearly constrained quadratic programs with box constraints appear in a wide range of applications. Hence, we propose an accelerated proximal gradient algorithm for singly linearly constrained quadratic programs with box constraints. At each iteration, the subproblem whose Hessian matrix is diagonal and positive definite is an easy model which can be solved efficiently via searching a root of a piecewise linear function. It is proved that the new algorithm can terminate at anε-optimal solution withinO(1/ε)iterations. Moreover, no line search is needed in this algorithm, and the global convergence can be proved under mild conditions. Numerical results are reported for solving quadratic programs arising from the training of support vector machines, which show that the new algorithm is efficient.

scholarly journals A Novel Approach for Solving Nonsmooth Optimization Problems with Application to Nonsmooth Equations

2013 ◽  
Vol 2013 ◽  
pp. 1-10
Author(s):  
Hamid Reza Erfanian ◽  
M. H. Noori Skandari ◽  
A. V. Kamyad
Keyword(s):  
Nonsmooth Optimization ◽  
Optimization Problems ◽  
Piecewise Linear ◽  
Linear Optimization ◽  
Optimal Solution ◽  
Piecewise Linear Function ◽  
Nonsmooth Equations ◽  
Smooth Functions ◽  
Generalized Derivative ◽  
Novel Approach

We present a new approach for solving nonsmooth optimization problems and a system of nonsmooth equations which is based on generalized derivative. For this purpose, we introduce the first order of generalized Taylor expansion of nonsmooth functions and replace it with smooth functions. In other words, nonsmooth function is approximated by a piecewise linear function based on generalized derivative. In the next step, we solve smooth linear optimization problem whose optimal solution is an approximate solution of main problem. Then, we apply the results for solving system of nonsmooth equations. Finally, for efficiency of our approach some numerical examples have been presented.

Nondegenerate Piecewise Linear Systems: A Finite Newton Algorithm and Applications in Machine Learning

2012 ◽  
Vol 24 (4) ◽  
pp. 1047-1084 ◽  
Author(s):  
Xiao-Tong Yuan ◽  
Shuicheng Yan
Keyword(s):  
Linear Systems ◽  
Optimization Problems ◽  
Piecewise Linear ◽  
Optimization Methods ◽  
Coefficient Matrix ◽  
Learning Problems ◽  
Support Vector ◽  
Data Sets ◽  
Piecewise Linear Systems ◽  
Vector Machines

We investigate Newton-type optimization methods for solving piecewise linear systems (PLSs) with nondegenerate coefficient matrix. Such systems arise, for example, from the numerical solution of linear complementarity problem, which is useful to model several learning and optimization problems. In this letter, we propose an effective damped Newton method, PLS-DN, to find the exact (up to machine precision) solution of nondegenerate PLSs. PLS-DN exhibits provable semiiterative property, that is, the algorithm converges globally to the exact solution in a finite number of iterations. The rate of convergence is shown to be at least linear before termination. We emphasize the applications of our method in modeling, from a novel perspective of PLSs, some statistical learning problems such as box-constrained least squares, elitist Lasso (Kowalski & Torreesani, 2008 ), and support vector machines (Cortes & Vapnik, 1995 ). Numerical results on synthetic and benchmark data sets are presented to demonstrate the effectiveness and efficiency of PLS-DN on these problems.

scholarly journals HyP-ABC: A Novel Automated Hyper-Parameter Tuning Algorithm Using Evolutionary Optimization

2021 ◽  
Author(s):  
Leila Zahedi ◽  
Farid Ghareh Mohammadi ◽  
M. Hadi Amini
Keyword(s):  
Parameter Optimization ◽  
Real World ◽  
Optimization Problems ◽  
State Of The Art ◽  
Parameter Tuning ◽  
Gradient Boosting ◽  
Support Vector ◽  
Wide Range ◽  
Extreme Gradient Boosting ◽  
Art Techniques

Machine learning techniques lend themselves as promising decision-making and analytic tools in a wide range of applications. Different ML algorithms have various hyper-parameters. In order to tailor an ML model towards a specific application, a large number of hyper-parameters should be tuned. Tuning the hyper-parameters directly affects the performance (accuracy and run-time). However, for large-scale search spaces, efficiently exploring the ample number of combinations of hyper-parameters is computationally challenging. Existing automated hyper-parameter tuning techniques suffer from high time complexity. In this paper, we propose HyP-ABC, an automatic innovative hybrid hyper-parameter optimization algorithm using the modified artificial bee colony approach, to measure the classification accuracy of three ML algorithms, namely random forest, extreme gradient boosting, and support vector machine. Compared to the state-of-the-art techniques, HyP-ABC is more efficient and has a limited number of parameters to be tuned, making it worthwhile for real-world hyper-parameter optimization problems. We further compare our proposed HyP-ABC algorithm with state-of-the-art techniques. In order to ensure the robustness of the proposed method, the algorithm takes a wide range of feasible hyper-parameter values, and is tested using a real-world educational dataset.

Numerical experiments on stochastic block proximal-gradient type method for convex constrained optimization involving coordinatewise separable problems

2019 ◽  
Vol 35 (3) ◽  
pp. 371-378
Author(s):  
PORNTIP PROMSINCHAI ◽  
NARIN PETROT ◽  
◽  
◽  
Keyword(s):  
Constrained Optimization ◽  
Optimization Problems ◽  
Gradient Algorithm ◽  
Objective Functions ◽  
Constrained Optimization Problems ◽  
Type Method ◽  
Coordinate Descent Algorithm ◽  
Proximal Gradient Algorithm ◽  
Separable Problems ◽  
Gradient Type

In this paper, we consider convex constrained optimization problems with composite objective functions over the set of a minimizer of another function. The main aim is to test numerically a new algorithm, namely a stochastic block coordinate proximal-gradient algorithm with penalization, by comparing both the number of iterations and CPU times between this introduced algorithm and the other well-known types of block coordinate descent algorithm for finding solutions of the randomly generated optimization problems with regularization term.

scholarly journals Numerical Performance of Different Formulations for Alternating Current Optimal Power Flow

2021 ◽  
Author(s):  
Sayed Abdullah Sadat ◽  
Kibaek Kim
Keyword(s):  
Nonlinear Optimization ◽  
Power Flow ◽  
Optimal Power Flow ◽  
Large Scale ◽  
Optimization Problems ◽  
Alternating Current ◽  
Box Constraints ◽  
Optimal Power ◽  
Wide Range ◽  
The Impact

<div>Alternating current optimal power flow (ACOPF) problems are nonconvex and nonlinear optimization problems. Utilities and independent service operators (ISO) require ACOPF to be solved in almost real time. Interior point methods (IPMs) are one of the powerful methods for solving large-scale nonlinear optimization problems and are a suitable approach for solving ACOPF with large-scale real-world transmission networks. Moreover, the choice of the formulation is as important as choosing the algorithm for solving an ACOPF problem. In this paper, different ACOPF formulations with various linear solvers and the impact of employing box constraints are evaluated for computational viability and best performance when using IPMs. Different optimization structures are used in these formulations to model the ACOPF problem representing a range of sparsity. The numerical experiments suggest that the least sparse ACOPF formulations with polar voltages yield the best computational results. Additionally, nodal injected models and current-based branch flow models are improved by enforcing box constraints. A wide range of test cases, ranging from 500-bus systems to 9591-bus systems, are used to verify the test results.</div>

scholarly journals Support Vector Ordinal Regression

2007 ◽  
Vol 19 (3) ◽  
pp. 792-815 ◽  
Author(s):  
Wei Chu ◽  
S. Sathiya Keerthi
Keyword(s):  
Optimization Problems ◽  
Optimal Solution ◽  
Quadratic Function ◽  
Ordinal Regression ◽  
Support Vector ◽  
Data Sets ◽  
Sequential Minimal Optimization ◽  
Real World Data ◽  
Training Samples ◽  
Multiple Thresholds

In this letter, we propose two new support vector approaches for ordinal regression, which optimize multiple thresholds to define parallel discriminant hyperplanes for the ordinal scales. Both approaches guarantee that the thresholds are properly ordered at the optimal solution. The size of these optimization problems is linear in the number of training samples. The sequential minimal optimization algorithm is adapted for the resulting optimization problems; it is extremely easy to implement and scales efficiently as a quadratic function of the number of examples. The results of numerical experiments on some benchmark and real-world data sets, including applications of ordinal regression to information retrieval, verify the usefulness of these approaches.

scholarly journals Multi-Objective Optimization Through Pareto Minimal Correction Subsets

2018 ◽  
Author(s):  
Miguel Terra-Neves ◽  
Inês Lynce ◽  
Vasco Manquinho
Keyword(s):  
Constrained Optimization ◽  
Optimization Problems ◽  
State Of The Art ◽  
Optimal Solution ◽  
Approximate Solutions ◽  
Pareto Optimal ◽  
Constrained Optimization Problems ◽  
Multi Objective ◽  
Wide Range ◽  
Constraint Set

A Minimal Correction Subset (MCS) of an unsatisfiable constraint set is a minimal subset of constraints that, if removed, makes the constraint set satisfiable. MCSs enjoy a wide range of applications, such as finding approximate solutions to constrained optimization problems. However, existing work on applying MCS enumeration to optimization problems focuses on the single-objective case. In this work, Pareto Minimal Correction Subsets (Pareto-MCSs) are proposed for approximating the Pareto-optimal solution set of multi-objective constrained optimization problems. We formalize and prove an equivalence relationship between Pareto-optimal solutions and Pareto-MCSs. Moreover, Pareto-MCSs and MCSs can be connected in such a way that existing state-of-the-art MCS enumeration algorithms can be used to enumerate Pareto-MCSs. Finally, experimental results on the multi-objective virtual machine consolidation problem show that the Pareto-MCS approach is competitive with state-of-the-art algorithms.

EA Multi-Model Selection for SVM

2011 ◽  
pp. 520-525
Author(s):  
Gilles Lebrun ◽  
Olivier Lezoray ◽  
Christopher Charrier ◽  
Hubert Cardot
Keyword(s):  
Model Selection ◽  
Efficient Solution ◽  
Optimization Problems ◽  
Fitness Function ◽  
Optimal Solution ◽  
Efficient Solutions ◽  
Support Vector ◽  
Natural Processes ◽  
Open Question ◽  
Global Selection

Evolutionary algorithms (EA) (Rechenberg, 1965) belong to a family of stochastic search algorithms inspired by natural evolution. In the last years, EA were used successfully to produce efficient solutions for a great number of hard optimization problems (Beasley, 1997). These algorithms operate on a population of potential solutions and apply a survival principle according to a fitness measure associated to each solution to produce better approximations of the optimal solution. At each iteration, a new set of solutions is created by selecting individuals according to their level of fitness and by applying to them several operators. These operators model natural processes, such as selection, recombination, mutation, migration, locality and neighborhood. Although the basic idea of EA is straightforward, solutions coding, size of population, fitness function and operators must be defined in compliance with the kind of problem to optimize. Multi-class problems with binary SVM (Support Vector Machine) classifiers are commonly treated as a decomposition in several binary sub-problems. An open question is how to properly choose all models for these sub-problems in order to have the lowest error rate for a specific SVM multi-class scheme. In this paper, we propose a new approach to optimize the generalization capacity of such SVM multi-class schemes. This approach consists in a global selection of models for sub-problems altogether and is denoted as multi-model selection. A multi-model selection can outperform the classical individual model selection used until now in the literature, but this type of selection defines a hard optimisation problem, because it corresponds to a search a efficient solution into a huge space. Therefore, we propose an adapted EA to achieve that multi-model selection by defining specific fitness function and recombination operator.

scholarly journals Hybrid Metaheuristics to the Automatic Selection of Features and Members of Classifier Ensembles

Information ◽  
2018 ◽  
Vol 9 (11) ◽  
pp. 268 ◽  
Author(s):  
Antonino Feitosa Neto ◽  
Anne Canuto ◽  
João Xavier-Junior
Keyword(s):  
Optimization Problems ◽  
Optimal Solution ◽  
Hybrid Technique ◽  
Classifier Ensembles ◽  
Hybrid Metaheuristics ◽  
Hybrid Techniques ◽  
Wide Range ◽  
Ensemble Systems ◽  
Two Hybrid ◽  
Selection Of

Metaheuristic algorithms have been applied to a wide range of global optimization problems. Basically, these techniques can be applied to problems in which a good solution must be found, providing imperfect or incomplete knowledge about the optimal solution. However, the concept of combining metaheuristics in an efficient way has emerged recently, in a field called hybridization of metaheuristics or, simply, hybrid metaheuristics. As a result of this, hybrid metaheuristics can be successfully applied in different optimization problems. In this paper, two hybrid metaheuristics, MAMH (Multiagent Metaheuristic Hybridization) and MAGMA (Multiagent Metaheuristic Architecture), are adapted to be applied in the automatic design of ensemble systems, in both mono- and multi-objective versions. To validate the feasibility of these hybrid techniques, we conducted an empirical investigation, performing a comparative analysis between them and traditional metaheuristics as well as existing existing ensemble generation methods. Our findings demonstrate a competitive performance of both techniques, in which a hybrid technique provided the lowest error rate for most of the analyzed objective functions.

Sign in / Sign up

Export Citation Format

Share Document