Generalization error of random feature and kernel methods: hypercontractivity and kernel matrix concentration

2021 ◽  
Author(s):  
Song Mei ◽  
Theodor Misiakiewicz ◽  
Andrea Montanari
Keyword(s):  
Kernel Methods ◽  
Kernel Matrix ◽  
Generalization Error ◽  
Matrix Concentration

An Asymptotic Statistical Theory of Polynomial Kernel Methods

2004 ◽  
Vol 16 (8) ◽  
pp. 1705-1719 ◽  
Author(s):  
Kazushi Ikeda
Keyword(s):  
Kernel Methods ◽  
Feature Space ◽  
Upper Bounds ◽  
High Dimensional ◽  
Polynomial Kernel ◽  
Generalization Error ◽  
Essential Dimension ◽  
Input Surface ◽  
Separating Hyperplane ◽  
The Relationship

The generalization properties of learning classifiers with a polynomial kernel function are examined. In kernel methods, input vectors are mapped into a high-dimensional feature space where the mapped vectors are linearly separated. It is well-known that a linear dichotomy has an average generalization error or a learning curve proportional to the dimension of the input space and inversely proportional to the number of given examples in the asymptotic limit. However, it does not hold in the case of kernel methods since the feature vectors lie on a submanifold in the feature space, called the input surface. In this letter, we discuss how the asymptotic average generalization error depends on the relationship between the input surface and the true separating hyperplane in the feature space where the essential dimension of the true separating polynomial, named the class, is important. We show its upper bounds in several cases and confirm these using computer simulations.

scholarly journals Generalization Error Bounds for Kernel Matrix Completion and Extrapolation

2020 ◽  
Vol 27 ◽  
pp. 326-330 ◽  
Author(s):  
Pere Gimenez-Febrer ◽  
Alba Pages-Zamora ◽  
Georgios B. Giannakis
Keyword(s):  
Error Bounds ◽  
Matrix Completion ◽  
Kernel Matrix ◽  
Generalization Error

scholarly journals Automated Spectral Kernel Learning

2020 ◽  
Vol 34 (04) ◽  
pp. 4618-4625
Author(s):  
Jian Li ◽  
Yong Liu ◽  
Weiping Wang
Keyword(s):  
Error Bounds ◽  
Kernel Methods ◽  
Experimental Results ◽  
Kernel Learning ◽  
Generalization Error ◽  
Learning Framework ◽  
Model Training ◽  
Efficient Learning ◽  
Spectral Representations ◽  
Spectral Kernel

The generalization performance of kernel methods is largely determined by the kernel, but spectral representations of stationary kernels are both input-independent and output-independent, which limits their applications on complicated tasks. In this paper, we propose an efficient learning framework that incorporates the process of finding suitable kernels and model training. Using non-stationary spectral kernels and backpropagation w.r.t. the objective, we obtain favorable spectral representations that depends on both inputs and outputs. Further, based on Rademacher complexity, we derive data-dependent generalization error bounds, where we investigate the effect of those factors and introduce regularization terms to improve the performance. Extensive experimental results validate the effectiveness of the proposed algorithm and coincide with our theoretical findings.

Imbalanced Data Detection Kernel Method in Closed Systems

2013 ◽  
Vol 756-759 ◽  
pp. 3652-3658
Author(s):  
You Li Lu ◽  
Jun Luo
Keyword(s):  
Kernel Methods ◽  
Kernel Method ◽  
Imbalanced Data ◽  
Data Detection ◽  
Data Sets ◽  
System Call ◽  
Data Set ◽  
Imbalanced Data Sets ◽  
Lower Complexity ◽  
Closed Systems

Under the study of Kernel Methods, this paper put forward two improved algorithm which called R-SVM & I-SVDD in order to cope with the imbalanced data sets in closed systems. R-SVM used K-means algorithm clustering space samples while I-SVDD improved the performance of original SVDD by imbalanced sample training. Experiment of two sets of system call data set shows that these two algorithms are more effectively and R-SVM has a lower complexity.

scholarly journals Kernel methods in system identification, machine learning and function estimation: A survey

Automatica ◽  
2014 ◽  
Vol 50 (3) ◽  
pp. 657-682 ◽  
Author(s):  
Gianluigi Pillonetto ◽  
Francesco Dinuzzo ◽  
Tianshi Chen ◽  
Giuseppe De Nicolao ◽  
Lennart Ljung
Keyword(s):  
Machine Learning ◽  
System Identification ◽  
Kernel Methods ◽  
Function Estimation ◽  
And Function

scholarly journals Competitive Caching with Machine Learned Advice

2021 ◽  
Vol 68 (4) ◽  
pp. 1-25
Author(s):  
Thodoris Lykouris ◽  
Sergei Vassilvitskii
Keyword(s):  
Online Algorithms ◽  
Empirical Evaluation ◽  
Optimal Solution ◽  
Poor Performance ◽  
Machine Learning Algorithms ◽  
Average Error ◽  
Generalization Error ◽  
Worst Case ◽  
Future Events ◽  
Real World Datasets

Traditional online algorithms encapsulate decision making under uncertainty, and give ways to hedge against all possible future events, while guaranteeing a nearly optimal solution, as compared to an offline optimum. On the other hand, machine learning algorithms are in the business of extrapolating patterns found in the data to predict the future, and usually come with strong guarantees on the expected generalization error. In this work, we develop a framework for augmenting online algorithms with a machine learned predictor to achieve competitive ratios that provably improve upon unconditional worst-case lower bounds when the predictor has low error. Our approach treats the predictor as a complete black box and is not dependent on its inner workings or the exact distribution of its errors. We apply this framework to the traditional caching problem—creating an eviction strategy for a cache of size k . We demonstrate that naively following the oracle’s recommendations may lead to very poor performance, even when the average error is quite low. Instead, we show how to modify the Marker algorithm to take into account the predictions and prove that this combined approach achieves a competitive ratio that both (i) decreases as the predictor’s error decreases and (ii) is always capped by O (log k ), which can be achieved without any assistance from the predictor. We complement our results with an empirical evaluation of our algorithm on real-world datasets and show that it performs well empirically even when using simple off-the-shelf predictions.

scholarly journals Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Abdulkadir Canatar ◽  
Blake Bordelon ◽  
Cengiz Pehlevan
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Kernel Regression ◽  
Learning Task ◽  
Learning Curves ◽  
Generalization Error ◽  
Theoretical Understanding ◽  
Classical Statistics ◽  
Deep Networks ◽  
Model Alignment

AbstractA theoretical understanding of generalization remains an open problem for many machine learning models, including deep networks where overparameterization leads to better performance, contradicting the conventional wisdom from classical statistics. Here, we investigate generalization error for kernel regression, which, besides being a popular machine learning method, also describes certain infinitely overparameterized neural networks. We use techniques from statistical mechanics to derive an analytical expression for generalization error applicable to any kernel and data distribution. We present applications of our theory to real and synthetic datasets, and for many kernels including those that arise from training deep networks in the infinite-width limit. We elucidate an inductive bias of kernel regression to explain data with simple functions, characterize whether a kernel is compatible with a learning task, and show that more data may impair generalization when noisy or not expressible by the kernel, leading to non-monotonic learning curves with possibly many peaks.

Sign in / Sign up

Export Citation Format

Share Document