Generalization Error of Automatic Relevance Determination

2007 ◽  
pp. 1-10 ◽  
Author(s):  
Shinichi Nakajima ◽  
Sumio Watanabe
Keyword(s):  
Generalization Error ◽  
Automatic Relevance Determination

scholarly journals Competitive Caching with Machine Learned Advice

2021 ◽  
Vol 68 (4) ◽  
pp. 1-25
Author(s):  
Thodoris Lykouris ◽  
Sergei Vassilvitskii
Keyword(s):  
Online Algorithms ◽  
Empirical Evaluation ◽  
Optimal Solution ◽  
Poor Performance ◽  
Machine Learning Algorithms ◽  
Average Error ◽  
Generalization Error ◽  
Worst Case ◽  
Future Events ◽  
Real World Datasets

Traditional online algorithms encapsulate decision making under uncertainty, and give ways to hedge against all possible future events, while guaranteeing a nearly optimal solution, as compared to an offline optimum. On the other hand, machine learning algorithms are in the business of extrapolating patterns found in the data to predict the future, and usually come with strong guarantees on the expected generalization error. In this work, we develop a framework for augmenting online algorithms with a machine learned predictor to achieve competitive ratios that provably improve upon unconditional worst-case lower bounds when the predictor has low error. Our approach treats the predictor as a complete black box and is not dependent on its inner workings or the exact distribution of its errors. We apply this framework to the traditional caching problem—creating an eviction strategy for a cache of size k . We demonstrate that naively following the oracle’s recommendations may lead to very poor performance, even when the average error is quite low. Instead, we show how to modify the Marker algorithm to take into account the predictions and prove that this combined approach achieves a competitive ratio that both (i) decreases as the predictor’s error decreases and (ii) is always capped by O (log k ), which can be achieved without any assistance from the predictor. We complement our results with an empirical evaluation of our algorithm on real-world datasets and show that it performs well empirically even when using simple off-the-shelf predictions.

scholarly journals Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Abdulkadir Canatar ◽  
Blake Bordelon ◽  
Cengiz Pehlevan
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Kernel Regression ◽  
Learning Task ◽  
Learning Curves ◽  
Generalization Error ◽  
Theoretical Understanding ◽  
Classical Statistics ◽  
Deep Networks ◽  
Model Alignment

AbstractA theoretical understanding of generalization remains an open problem for many machine learning models, including deep networks where overparameterization leads to better performance, contradicting the conventional wisdom from classical statistics. Here, we investigate generalization error for kernel regression, which, besides being a popular machine learning method, also describes certain infinitely overparameterized neural networks. We use techniques from statistical mechanics to derive an analytical expression for generalization error applicable to any kernel and data distribution. We present applications of our theory to real and synthetic datasets, and for many kernels including those that arise from training deep networks in the infinite-width limit. We elucidate an inductive bias of kernel regression to explain data with simple functions, characterize whether a kernel is compatible with a learning task, and show that more data may impair generalization when noisy or not expressible by the kernel, leading to non-monotonic learning curves with possibly many peaks.

scholarly journals FEATURE SELECTION AND THE CHESSBOARD PROBLEM

2015 ◽  
Vol 1 (311) ◽  
Author(s):  
Mariusz Kubus
Keyword(s):  
Feature Selection ◽  
Data Structure ◽  
Important Criterion ◽  
Multivariate Approach ◽  
Generalization Error ◽  
Selection Methods ◽  
Embedded Methods ◽  
Feature Relevance

Feature selection methods are usually classified into three groups: filters, wrappers and embedded methods. The second important criterion of their classification is an individual or multivariate approach to evaluation of the feature relevance. The chessboard problem is an illustrative example, where two variables which have no individual influence on the dependent variable can be essential to separate the classes. The classifiers which deal well with such data structure are sensitive to irrelevant variables. The generalization error increases with the number of noisy variables. We discuss the feature selection methods in the context of chessboard-like structure in the data with numerous irrelevant variables.

Sign in / Sign up

Export Citation Format

Share Document