Distributed learning with partial coefficients regularization

2018 ◽  
Vol 16 (04) ◽  
pp. 1850025 ◽  
Author(s):  
Mengjuan Pang ◽  
Hongwei Sun
Keyword(s):  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Distributed Learning ◽  
Regression Function ◽  
Operator Technique ◽  
Interesting Phenomenon ◽  
Regularization Scheme ◽  
Complexity Of Algorithms ◽  
Learning Rates ◽  
The Individual

We study distributed learning with partial coefficients regularization scheme in a reproducing kernel Hilbert space (RKHS). The algorithm randomly partitions the sample set [Formula: see text] into [Formula: see text] disjoint sample subsets of equal size. In order to reduce the complexity of algorithms, we apply a partial coefficients regularization scheme to each sample subset to produce an output function, and average the individual output functions to get the final global estimator. The error bound in the [Formula: see text]-metric is deduced and the asymptotic convergence for this distributed learning with partial coefficients regularization is proved by the integral operator technique. Satisfactory learning rates are then derived under a standard regularity condition on the regression function, which reveals an interesting phenomenon that when [Formula: see text] and [Formula: see text] is small enough, this distributed learning has the same convergence rate with the algorithm processing the whole data in one single machine.

scholarly journals KTBoost: Combined Kernel and Tree Boosting

2021 ◽  
Author(s):  
Fabio Sigrist
Keyword(s):  
Hilbert Space ◽  
Reproducing Kernel ◽  
Predictive Accuracy ◽  
Reproducing Kernel Hilbert Space ◽  
Regression Tree ◽  
Regression Function ◽  
Data Sets ◽  
Regression Functions ◽  
Boosting Algorithm

AbstractWe introduce a novel boosting algorithm called ‘KTBoost’ which combines kernel boosting and tree boosting. In each boosting iteration, the algorithm adds either a regression tree or reproducing kernel Hilbert space (RKHS) regression function to the ensemble of base learners. Intuitively, the idea is that discontinuous trees and continuous RKHS regression functions complement each other, and that this combination allows for better learning of functions that have parts with varying degrees of regularity such as discontinuities and smooth parts. We empirically show that KTBoost significantly outperforms both tree and kernel boosting in terms of predictive accuracy in a comparison on a wide array of data sets.

Learning rates for regularized least squares ranking algorithm

2017 ◽  
Vol 15 (06) ◽  
pp. 815-836 ◽  
Author(s):  
Yulong Zhao ◽  
Jun Fan ◽  
Lei Shi
Keyword(s):  
Least Squares ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Ranking Algorithm ◽  
Generalization Error ◽  
Ranking Problem ◽  
Regularized Least Squares ◽  
U Statistics ◽  
Learning Rates ◽  
Error Decomposition

The ranking problem aims at learning real-valued functions to order instances, which has attracted great interest in statistical learning theory. In this paper, we consider the regularized least squares ranking algorithm within the framework of reproducing kernel Hilbert space. In particular, we focus on analysis of the generalization error for this ranking algorithm, and improve the existing learning rates by virtue of an error decomposition technique from regression and Hoeffding’s decomposition for U-statistics.

scholarly journals Learning rates for the risk of kernel-based quantile regression estimators in additive models

2016 ◽  
Vol 14 (03) ◽  
pp. 449-477 ◽  
Author(s):  
Andreas Christmann ◽  
Ding-Xuan Zhou
Keyword(s):  
Hilbert Space ◽  
Quantile Regression ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Gaussian Function ◽  
Additive Model ◽  
Additive Models ◽  
Gaussian Kernel ◽  
Optimal Learning ◽  
Learning Rates

Additive models play an important role in semiparametric statistics. This paper gives learning rates for regularized kernel-based methods for additive models. These learning rates compare favorably in particular in high dimensions to recent results on optimal learning rates for purely nonparametric regularized kernel-based quantile regression using the Gaussian radial basis function kernel, provided the assumption of an additive model is valid. Additionally, a concrete example is presented to show that a Gaussian function depending only on one variable lies in a reproducing kernel Hilbert space generated by an additive Gaussian kernel, but does not belong to the reproducing kernel Hilbert space generated by the multivariate Gaussian kernel of the same variance.

LEARNING RATES OF REGULARIZED REGRESSION FOR FUNCTIONAL DATA

2009 ◽  
Vol 07 (06) ◽  
pp. 839-850 ◽  
Author(s):  
YONG-LI XU ◽  
DI-RONG CHEN
Keyword(s):  
Functional Data Analysis ◽  
Functional Data ◽  
Input Data ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Natural Extension ◽  
Least Square ◽  
Regularized Regression ◽  
Fast Learning ◽  
Learning Rates

The study of regularized learning algorithms is a very important issue and functional data analysis extends classical methods. We establish the learning rates of the least square regularized regression algorithm in reproducing kernel Hilbert space for functional data. With the iteration method, we obtain fast learning rate for functional data. Our result is a natural extension for least square regularized regression algorithm when the dimension of input data is finite.

scholarly journals Convergence rates of Kernel Conjugate Gradient for random design regression

2016 ◽  
Vol 14 (06) ◽  
pp. 763-794 ◽  
Author(s):  
Gilles Blanchard ◽  
Nicole Krämer
Keyword(s):  
Least Squares ◽  
Conjugate Gradient ◽  
Reproducing Kernel ◽  
Convergence Rates ◽  
Reproducing Kernel Hilbert Space ◽  
Regression Function ◽  
Rates Of Convergence ◽  
Least Squares Regression ◽  
Hilbert Norm ◽  
Effective Dimensionality

We prove statistical rates of convergence for kernel-based least squares regression from i.i.d. data using a conjugate gradient (CG) algorithm, where regularization against overfitting is obtained by early stopping. This method is related to Kernel Partial Least Squares, a regression method that combines supervised dimensionality reduction with least squares projection. Following the setting introduced in earlier related literature, we study so-called “fast convergence rates” depending on the regularity of the target regression function (measured by a source condition in terms of the kernel integral operator) and on the effective dimensionality of the data mapped into the kernel space. We obtain upper bounds, essentially matching known minimax lower bounds, for the ℒ2 (prediction) norm as well as for the stronger Hilbert norm, if the true regression function belongs to the reproducing kernel Hilbert space. If the latter assumption is not fulfilled, we obtain similar convergence rates for appropriate norms, provided additional unlabeled data are available.

A NOTE ON STABILITY OF ERROR BOUNDS IN STATISTICAL LEARNING THEORY

2011 ◽  
Vol 09 (04) ◽  
pp. 369-382
Author(s):  
MING LI ◽  
ANDREA CAPONNETTO
Keyword(s):  
Hilbert Space ◽  
Statistical Learning ◽  
Learning Theory ◽  
Error Bounds ◽  
Wide Class ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Statistical Learning Theory ◽  
Regression Function ◽  
Functional Space

We consider a wide class of error bounds developed in the context of statistical learning theory which are expressed in terms of functionals of the regression function, for instance, its norm in a reproducing kernel Hilbert space or other functional space. These bounds are unstable in the sense that a small perturbation of the regression function can induce an arbitrary large increase of the relevant functional and make the error bound useless. Using a known result involving Fano inequality, we show how stability can be recovered.

scholarly journals Analysis of regularized Nyström subsampling for regression functions of low smoothness

2019 ◽  
Vol 17 (06) ◽  
pp. 931-946 ◽  
Author(s):  
Shuai Lu ◽  
Peter Mathé ◽  
Sergiy Pereverzyev
Keyword(s):  
Reproducing Kernel ◽  
Regularization Parameter ◽  
Reproducing Kernel Hilbert Space ◽  
Practical Importance ◽  
Target Function ◽  
Mild Conditions ◽  
Learning Rates ◽  
Regression Functions ◽  
Large Kernel ◽  
General Source

This paper studies a Nyström-type subsampling approach to large kernel learning methods in the misspecified case, where the target function is not assumed to belong to the reproducing kernel Hilbert space generated by the underlying kernel. This case is less understood in spite of its practical importance. To model such a case, the smoothness of target functions is described in terms of general source conditions. It is surprising that almost for the whole range of the source conditions, describing the misspecified case, the corresponding learning rate bounds can be achieved with just one value of the regularization parameter. This observation allows a formulation of mild conditions under which the plain Nyström subsampling can be realized with subquadratic cost maintaining the guaranteed learning rates.

Regularization schemes for minimum error entropy principle

2015 ◽  
Vol 13 (04) ◽  
pp. 437-455 ◽  
Author(s):  
Ting Hu ◽  
Jun Fan ◽  
Qiang Wu ◽  
Ding-Xuan Zhou
Keyword(s):  
Hilbert Spaces ◽  
Reproducing Kernel ◽  
Learning Algorithm ◽  
Regression Function ◽  
Explicit Learning ◽  
Minimum Error ◽  
Scaling Parameter ◽  
Learning Rates ◽  
Entropy Principle ◽  
Kernel Hilbert Spaces

We introduce a learning algorithm for regression generated by a minimum error entropy (MEE) principle and regularization schemes in reproducing kernel Hilbert spaces. This empirical MEE algorithm is highly related to a scaling parameter arising from Parzen windowing. The purpose of this paper is to carry out consistency analysis when the scaling parameter is large. Explicit learning rates are provided. Novel approaches are proposed to overcome the difficulties in bounding the output function uniformly and in the special MEE feature that the regression function may not be a minimizer of the error entropy.

scholarly journals Learning rates of regression with q-norm loss and threshold

2016 ◽  
Vol 14 (06) ◽  
pp. 809-827
Author(s):  
Ting Hu ◽  
Yuan Yao
Keyword(s):  
Hilbert Space ◽  
Reproducing Kernel ◽  
Robust Regression ◽  
Reproducing Kernel Hilbert Space ◽  
A Priori ◽  
Explicit Learning ◽  
Noise Condition ◽  
Learning Rates ◽  
Regression Problems ◽  
Loss Formula

This paper studies some robust regression problems associated with the [Formula: see text]-norm loss ([Formula: see text]) and the [Formula: see text]-insensitive [Formula: see text]-norm loss in the reproducing kernel Hilbert space. We establish a variance-expectation bound under a priori noise condition on the conditional distribution, which is the key technique to measure the error bound. Explicit learning rates will be given under the approximation ability assumptions on the reproducing kernel Hilbert space.

scholarly journals Lebesgue Decomposition of Non-Commutative Measures

2020 ◽  
Author(s):  
Michael T Jury ◽  
Robert T W Martin
Keyword(s):  
Hilbert Space ◽  
Reproducing Kernel ◽  
Fock Space ◽  
Reproducing Kernel Hilbert Space ◽  
Semigroup Algebra ◽  
Space Theory ◽  
Lebesgue Decomposition ◽  
Sesquilinear Forms ◽  
Positive Linear Functionals ◽  
Algebra Theory

Abstract We extend the Lebesgue decomposition of positive measures with respect to Lebesgue measure on the complex unit circle to the non-commutative (NC) multi-variable setting of (positive) NC measures. These are positive linear functionals on a certain self-adjoint subspace of the Cuntz–Toeplitz $C^{\ast }-$algebra, the $C^{\ast }-$algebra of the left creation operators on the full Fock space. This theory is fundamentally connected to the representation theory of the Cuntz and Cuntz–Toeplitz $C^{\ast }-$algebras; any *−representation of the Cuntz–Toeplitz $C^{\ast }-$algebra is obtained (up to unitary equivalence), by applying a Gelfand–Naimark–Segal construction to a positive NC measure. Our approach combines the theory of Lebesgue decomposition of sesquilinear forms in Hilbert space, Lebesgue decomposition of row isometries, free semigroup algebra theory, NC reproducing kernel Hilbert space theory, and NC Hardy space theory.

Sign in / Sign up

Export Citation Format

Share Document