Nondegenerate Piecewise Linear Systems: A Finite Newton Algorithm and Applications in Machine Learning

2012 ◽  
Vol 24 (4) ◽  
pp. 1047-1084 ◽  
Author(s):  
Xiao-Tong Yuan ◽  
Shuicheng Yan
Keyword(s):  
Linear Systems ◽  
Optimization Problems ◽  
Piecewise Linear ◽  
Optimization Methods ◽  
Coefficient Matrix ◽  
Learning Problems ◽  
Support Vector ◽  
Data Sets ◽  
Piecewise Linear Systems ◽  
Vector Machines

We investigate Newton-type optimization methods for solving piecewise linear systems (PLSs) with nondegenerate coefficient matrix. Such systems arise, for example, from the numerical solution of linear complementarity problem, which is useful to model several learning and optimization problems. In this letter, we propose an effective damped Newton method, PLS-DN, to find the exact (up to machine precision) solution of nondegenerate PLSs. PLS-DN exhibits provable semiiterative property, that is, the algorithm converges globally to the exact solution in a finite number of iterations. The rate of convergence is shown to be at least linear before termination. We emphasize the applications of our method in modeling, from a novel perspective of PLSs, some statistical learning problems such as box-constrained least squares, elitist Lasso (Kowalski & Torreesani, 2008 ), and support vector machines (Cortes & Vapnik, 1995 ). Numerical results on synthetic and benchmark data sets are presented to demonstrate the effectiveness and efficiency of PLS-DN on these problems.

scholarly journals RECENT ADVANCES ON SUPPORT VECTOR MACHINES RESEARCH

2012 ◽  
Vol 18 (1) ◽  
pp. 5-33 ◽  
Author(s):  
Yingjie Tian ◽  
Yong Shi ◽  
Xiaohui Liu
Keyword(s):  
Machine Learning ◽  
Support Vector Machines ◽  
Optimization Problems ◽  
Optimization Methods ◽  
Bankruptcy Prediction ◽  
Optimization Models ◽  
Support Vector ◽  
Problem Solution ◽  
Vector Machines ◽  
Credit Risk Analysis

Support vector machines (SVMs), with their roots in Statistical Learning Theory (SLT) and optimization methods, have become powerful tools for problem solution in machine learning. SVMs reduce most machine learning problems to optimization problems and optimization lies at the heart of SVMs. Lots of SVM algorithms involve solving not only convex problems, such as linear programming, quadratic programming, second order cone programming, semi-definite programming, but also non-convex and more general optimization problems, such as integer programming, semi-infinite programming, bi-level programming and so on. The purpose of this paper is to understand SVM from the optimization point of view, review several representative optimization models in SVMs, their applications in economics, in order to promote the research interests in both optimization-based SVMs theory and economics applications. This paper starts with summarizing and explaining the nature of SVMs. It then proceeds to discuss optimization models for SVM following three major themes. First, least squares SVM, twin SVM, AUC Maximizing SVM, and fuzzy SVM are discussed for standard problems. Second, support vector ordinal machine, semisupervised SVM, Universum SVM, robust SVM, knowledge based SVM and multi-instance SVM are then presented for nonstandard problems. Third, we explore other important issues such as lp-norm SVM for feature selection, LOOSVM based on minimizing LOO error bound, probabilistic outputs for SVM, and rule extraction from SVM. At last, several applications of SVMs to financial forecasting, bankruptcy prediction, credit risk analysis are introduced.

Arbitrary Norm Support Vector Machines

2009 ◽  
Vol 21 (2) ◽  
pp. 560-582 ◽  
Author(s):  
Kaizhu Huang ◽  
Danian Zheng ◽  
Irwin King ◽  
Michael R. Lyu
Keyword(s):  
Support Vector Machines ◽  
Bayesian Learning ◽  
Optimization Problems ◽  
Relevance Vector Machine ◽  
Significant Feature ◽  
Support Vector ◽  
Data Sets ◽  
Vector Machines ◽  
Series Of Experiments ◽  
Arbitrary Norm

Support vector machines (SVM) are state-of-the-art classifiers. Typically L2-norm or L1-norm is adopted as a regularization term in SVMs, while other norm-based SVMs, for example, the L0-norm SVM or even the L∞-norm SVM, are rarely seen in the literature. The major reason is that L0-norm describes a discontinuous and nonconvex term, leading to a combinatorially NP-hard optimization problem. In this letter, motivated by Bayesian learning, we propose a novel framework that can implement arbitrary norm-based SVMs in polynomial time. One significant feature of this framework is that only a sequence of sequential minimal optimization problems needs to be solved, thus making it practical in many real applications. The proposed framework is important in the sense that Bayesian priors can be efficiently plugged into most learning methods without knowing the explicit form. Hence, this builds a connection between Bayesian learning and the kernel machines. We derive the theoretical framework, demonstrate how our approach works on the L0-norm SVM as a typical example, and perform a series of experiments to validate its advantages. Experimental results on nine benchmark data sets are very encouraging. The implemented L0-norm is competitive with or even better than the standard L2-norm SVM in terms of accuracy but with a reduced number of support vectors, − 9.46% of the number on average. When compared with another sparse model, the relevance vector machine, our proposed algorithm also demonstrates better sparse properties with a training speed over seven times faster.

scholarly journals SCALING LARGE LEARNING PROBLEMS WITH HARD PARALLEL MIXTURES

2003 ◽  
Vol 17 (03) ◽  
pp. 349-365 ◽  
Author(s):  
RONAN COLLOBERT ◽  
YOSHUA BENGIO ◽  
SAMY BENGIO
Keyword(s):  
Large Data ◽  
Generative Models ◽  
Training Data ◽  
Learning Problems ◽  
Support Vector ◽  
Data Sets ◽  
Training Time ◽  
Vector Machines ◽  
Research Challenge ◽  
Probabilistic Version

A challenge for statistical learning is to deal with large data sets, e.g. in data mining. The training time of ordinary Support Vector Machines is at least quadratic, which raises a serious research challenge if we want to deal with data sets of millions of examples. We propose a "hard parallelizable mixture" methodology which yields significantly reduced training time through modularization and parallelization: the training data is iteratively partitioned by a "gater" model in such a way that it becomes easy to learn an "expert" model separately in each region of the partition. A probabilistic extension and the use of a set of generative models allows representing the gater so that all pieces of the model are locally trained. For SVMs, time complexity appears empirically to local growth linearly with the number of examples, while generalization performance can be enhanced. For the probabilistic version of the algorithm, the iterative algorithm probably goes down in a cost function that is an upper bound on the negative log-likelihood.

scholarly journals Multistability in Piecewise Linear Systems versus Eigenspectra Variation and Round Function

2017 ◽  
Vol 27 (09) ◽  
pp. 1730031 ◽  
Author(s):  
H. E. Gilardi-Velázquez ◽  
L. J. Ontañón-García ◽  
D. G. Hurtado-Rodriguez ◽  
E. Campos-Cantón
Keyword(s):  
Numerical Simulations ◽  
Linear Systems ◽  
Bifurcation Analysis ◽  
Piecewise Linear ◽  
Stable State ◽  
Coefficient Matrix ◽  
Switching Function ◽  
Bifurcation Parameter ◽  
Piecewise Linear Systems ◽  
Round Function

A multistable system generated by Piecewise Linear (PWL) subsystems based on the jerk equation is presented. The system’s behavior is characterized by means of the Nearest Integer or the [Formula: see text] function to control the switching events and to locate the corresponding equilibria on each of the commutation surfaces. These surfaces are generated through the switching function dividing the space into regions equally distributed along one axis. The trajectory of the system is governed by the eigenspectrum of the coefficient matrix, which can be adjusted by a bifurcation parameter. The behavior of the system can change from multiscroll oscillations in a mono-stable state into the coexistence of several single-scroll attractors in multistable states. The dynamics and bifurcation analysis are illustrated by numerical simulations to depict the multistable states.

scholarly journals TRAINING SUPPORT VECTOR MACHINES USING FRANK–WOLFE OPTIMIZATION METHODS

2013 ◽  
Vol 27 (03) ◽  
pp. 1360003 ◽  
Author(s):  
EMANUELE FRANDI ◽  
RICARDO ÑANCULEF ◽  
MARIA GRAZIA GASPARO ◽  
STEFANO LODI ◽  
CLAUDIO SARTORI
Keyword(s):  
Large Scale ◽  
Binary Classification ◽  
Feature Space ◽  
Optimization Methods ◽  
Learning Problems ◽  
Support Vector ◽  
Fast Method ◽  
Training Support ◽  
Lower Accuracy ◽  
Vector Machines

Training a support vector machine (SVM) requires the solution of a quadratic programming problem (QP) whose computational complexity becomes prohibitively expensive for large scale datasets. Traditional optimization methods cannot be directly applied in these cases, mainly due to memory restrictions. By adopting a slightly different objective function and under mild conditions on the kernel used within the model, efficient algorithms to train SVMs have been devised under the name of core vector machines (CVMs). This framework exploits the equivalence of the resulting learning problem with the task of building a minimal enclosing ball (MEB) problem in a feature space, where data is implicitly embedded by a kernel function. In this paper, we improve on the CVM approach by proposing two novel methods to build SVMs based on the Frank–Wolfe algorithm, recently revisited as a fast method to approximate the solution of a MEB problem. In contrast to CVMs, our algorithms do not require to compute the solutions of a sequence of increasingly complex QPs and are defined by using only analytic optimization steps. Experiments on a large collection of datasets show that our methods scale better than CVMs in most cases, sometimes at the price of a slightly lower accuracy. As CVMs, the proposed methods can be easily extended to machine learning problems other than binary classification. However, effective classifiers are also obtained using kernels which do not satisfy the condition required by CVMs, and thus our methods can be used for a wider set of problems.

scholarly journals Limit cycles from a monodromic infinity in planar piecewise linear systems

2021 ◽  
Vol 496 (2) ◽  
pp. 124818
Author(s):  
Emilio Freire ◽  
Enrique Ponce ◽  
Joan Torregrosa ◽  
Francisco Torres
Keyword(s):  
Linear Systems ◽  
Limit Cycles ◽  
Piecewise Linear ◽  
Piecewise Linear Systems

scholarly journals Canards in piecewise-linear systems: explosions and super-explosions

2013 ◽  
Vol 469 (2154) ◽  
pp. 20120603 ◽  
Author(s):  
Mathieu Desroches ◽  
Emilio Freire ◽  
S. John Hogan ◽  
Enrique Ponce ◽  
Phanikrishna Thota
Keyword(s):  
Phase Space ◽  
Linear Systems ◽  
Limit Cycles ◽  
Piecewise Linear ◽  
Single Parameter ◽  
Van Der Pol ◽  
Piecewise Linear Systems ◽  
Homoclinic Connections ◽  
Limiting Case

We show that a planar slow–fast piecewise-linear (PWL) system with three zones admits limit cycles that share a lot of similarity with van der Pol canards, in particular an explosive growth. Using phase-space compactification, we show that these quasi-canard cycles are strongly related to a bifurcation at infinity. Furthermore, we investigate a limiting case in which we show the existence of a continuum of canard homoclinic connections that coexist for a single-parameter value and with amplitude ranging from an order of ε to an order of 1, a phenomenon truly associated with the non-smooth character of this system and which we call super-explosion .

Sign in / Sign up

Export Citation Format

Share Document