A comparative study of multi-class support vector machines in the unifying framework of large margin classifiers

This paper has two main focal points. We first consider an important class of machine learning algorithms: large margin classifiers, such as Support Vector Machines. The notion of margin complexity quantifies the extent to which a given class of functions can be learned by large margin classifiers. We prove that up to a small multiplicative constant, margin complexity is equal to the inverse of discrepancy. This establishes a strong tie between seemingly very different notions from two distinct areas.In the same way that matrix rigidity is related to rank, we introduce the notion of rigidity of margin complexity. We prove that sign matrices with small margin complexity rigidity are very rare. This leads to the question of proving lower bounds on the rigidity of margin complexity. Quite surprisingly, this question turns out to be closely related to basic open problems in communication complexity, e.g., whether PSPACE can be separated from the polynomial hierarchy in communication complexity.Communication is a key ingredient in many types of learning. This explains the relations between the field of learning theory and that of communication complexity [6, l0, 16, 26]. The results of this paper constitute another link in this rich web of relations. These new results have already been applied toward the solution of several open problems in communication complexity [18, 20, 29].

Download Full-text

Revisiting probabilistic neural networks: a comparative study with support vector machines and the microhabitat suitability for the Eastern Iberian chub (Squalius valentinus)

Ecological Informatics ◽

10.1016/j.ecoinf.2017.10.008 ◽

2018 ◽

Vol 43 ◽

pp. 24-37 ◽

Cited By ~ 9

Author(s):

Rafael Muñoz-Mas ◽

Shinji Fukuda ◽

Javier Pórtoles ◽

Francisco Martínez-Capel

Keyword(s):

Neural Networks ◽

Support Vector Machines ◽

Comparative Study ◽

Support Vector ◽

Probabilistic Neural Networks ◽

Vector Machines ◽

Iberian Chub

Download Full-text

Comparative study of two uncertain support vector machines

2012 IEEE Fifth International Conference on Advanced Computational Intelligence (ICACI) ◽

10.1109/icaci.2012.6463192 ◽

2012 ◽

Cited By ~ 4

Author(s):

Hu Han ◽

Jianwu Dang ◽

Enen Ren

Keyword(s):

Support Vector Machines ◽

Comparative Study ◽

Support Vector ◽

Vector Machines

Download Full-text

Margin Error Bounds for Support Vector Machines on Reproducing Kernel Banach Spaces

Neural Computation ◽

10.1162/neco_a_01013 ◽

2017 ◽

Vol 29 (11) ◽

pp. 3078-3093 ◽

Cited By ~ 1

Author(s):

Liangzhi Chen ◽

Haizhang Zhang

Keyword(s):

Machine Learning ◽

Support Vector Machines ◽

Banach Spaces ◽

Reproducing Kernel ◽

Support Vector ◽

Large Margin ◽

Large Margin Classification ◽

Vector Machines ◽

Reproducing Kernel Banach Spaces ◽

Margin Classification

Support vector machines, which maximize the margin from patterns to the separation hyperplane subject to correct classification, have received remarkable success in machine learning. Margin error bounds based on Hilbert spaces have been introduced in the literature to justify the strategy of maximizing the margin in SVM. Recently, there has been much interest in developing Banach space methods for machine learning. Large margin classification in Banach spaces is a focus of such attempts. In this letter we establish a margin error bound for the SVM on reproducing kernel Banach spaces, thus supplying statistical justification for large-margin classification in Banach spaces.

Download Full-text

Robust Support Vector Machines for Classification with Nonconvex and Smooth Losses

Neural Computation ◽

10.1162/neco_a_00837 ◽

2016 ◽

Vol 28 (6) ◽

pp. 1217-1247 ◽

Cited By ~ 9

Author(s):

Yunlong Feng ◽

Yuning Yang ◽

Xiaolin Huang ◽

Siamak Mehrkanoon ◽

Johan A. K. Suykens

Keyword(s):

Support Vector Machines ◽

Robust Statistics ◽

Real Data ◽

Constructive Proof ◽

Support Vector ◽

Data Sets ◽

Large Margin ◽

Hinge Loss ◽

Vector Machines ◽

Smooth Classification

This letter addresses the robustness problem when learning a large margin classifier in the presence of label noise. In our study, we achieve this purpose by proposing robustified large margin support vector machines. The robustness of the proposed robust support vector classifiers (RSVC), which is interpreted from a weighted viewpoint in this work, is due to the use of nonconvex classification losses. Besides the robustness, we also show that the proposed RSCV is simultaneously smooth, which again benefits from using smooth classification losses. The idea of proposing RSVC comes from M-estimation in statistics since the proposed robust and smooth classification losses can be taken as one-sided cost functions in robust statistics. Its Fisher consistency property and generalization ability are also investigated. Besides the robustness and smoothness, another nice property of RSVC lies in the fact that its solution can be obtained by solving weighted squared hinge loss–based support vector machine problems iteratively. We further show that in each iteration, it is a quadratic programming problem in its dual space and can be solved by using state-of-the-art methods. We thus propose an iteratively reweighted type algorithm and provide a constructive proof of its convergence to a stationary point. Effectiveness of the proposed classifiers is verified on both artificial and real data sets.

Download Full-text