Learning rates for regularized least squares ranking algorithm

2017 ◽  
Vol 15 (06) ◽  
pp. 815-836 ◽  
Author(s):  
Yulong Zhao ◽  
Jun Fan ◽  
Lei Shi
Keyword(s):  
Least Squares ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Ranking Algorithm ◽  
Generalization Error ◽  
Ranking Problem ◽  
Regularized Least Squares ◽  
U Statistics ◽  
Learning Rates ◽  
Error Decomposition

The ranking problem aims at learning real-valued functions to order instances, which has attracted great interest in statistical learning theory. In this paper, we consider the regularized least squares ranking algorithm within the framework of reproducing kernel Hilbert space. In particular, we focus on analysis of the generalization error for this ranking algorithm, and improve the existing learning rates by virtue of an error decomposition technique from regression and Hoeffding’s decomposition for U-statistics.

scholarly journals Coefficient-Based Regression with Non-Identical Unbounded Sampling

2013 ◽  
Vol 2013 ◽  
pp. 1-8 ◽  
Author(s):  
Jia Cai
Keyword(s):  
Hilbert Space ◽  
Integral Operator ◽  
Error Analysis ◽  
Least Squares ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Least Squares Regression ◽  
Regression Problem ◽  
Novel Technique ◽  
Sample Error

We investigate a coefficient-based least squares regression problem with indefinite kernels from non-identical unbounded sampling processes. Here non-identical unbounded sampling means the samples are drawn independently but not identically from unbounded sampling processes. The kernel is not necessarily symmetric or positive semi-definite. This leads to additional difficulty in the error analysis. By introducing a suitable reproducing kernel Hilbert space (RKHS) and a suitable intermediate integral operator, elaborate analysis is presented by means of a novel technique for the sample error. This leads to satisfactory results.

Online regularized pairwise learning with least squares loss

2019 ◽  
Vol 18 (01) ◽  
pp. 49-78 ◽  
Author(s):  
Cheng Wang ◽  
Ting Hu
Keyword(s):  
Hilbert Space ◽  
Least Squares ◽  
Online Algorithm ◽  
Reproducing Kernel ◽  
Sharp Estimate ◽  
Reproducing Kernel Hilbert Space ◽  
Learning Sequence ◽  
Pairwise Learning ◽  
Regularization Parameters

In this paper, we study online algorithm for pairwise problems generated from the Tikhonov regularization scheme associated with the least squares loss function and a reproducing kernel Hilbert space (RKHS). This work establishes the convergence for the last iterate of the online pairwise algorithm with the polynomially decaying step sizes and varying regularization parameters. We show that the obtained error rate in [Formula: see text]-norm can be nearly optimal in the minimax sense under some mild conditions. Our analysis is achieved by a sharp estimate for the norms of the learning sequence and the characterization of RKHS using its associated integral operators and probability inequalities for random variables with values in a Hilbert space.

scholarly journals Learning rates for the risk of kernel-based quantile regression estimators in additive models

2016 ◽  
Vol 14 (03) ◽  
pp. 449-477 ◽  
Author(s):  
Andreas Christmann ◽  
Ding-Xuan Zhou
Keyword(s):  
Hilbert Space ◽  
Quantile Regression ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Gaussian Function ◽  
Additive Model ◽  
Additive Models ◽  
Gaussian Kernel ◽  
Optimal Learning ◽  
Learning Rates

Additive models play an important role in semiparametric statistics. This paper gives learning rates for regularized kernel-based methods for additive models. These learning rates compare favorably in particular in high dimensions to recent results on optimal learning rates for purely nonparametric regularized kernel-based quantile regression using the Gaussian radial basis function kernel, provided the assumption of an additive model is valid. Additionally, a concrete example is presented to show that a Gaussian function depending only on one variable lies in a reproducing kernel Hilbert space generated by an additive Gaussian kernel, but does not belong to the reproducing kernel Hilbert space generated by the multivariate Gaussian kernel of the same variance.

Distributed learning with partial coefficients regularization

2018 ◽  
Vol 16 (04) ◽  
pp. 1850025 ◽  
Author(s):  
Mengjuan Pang ◽  
Hongwei Sun
Keyword(s):  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Distributed Learning ◽  
Regression Function ◽  
Operator Technique ◽  
Interesting Phenomenon ◽  
Regularization Scheme ◽  
Complexity Of Algorithms ◽  
Learning Rates ◽  
The Individual

We study distributed learning with partial coefficients regularization scheme in a reproducing kernel Hilbert space (RKHS). The algorithm randomly partitions the sample set [Formula: see text] into [Formula: see text] disjoint sample subsets of equal size. In order to reduce the complexity of algorithms, we apply a partial coefficients regularization scheme to each sample subset to produce an output function, and average the individual output functions to get the final global estimator. The error bound in the [Formula: see text]-metric is deduced and the asymptotic convergence for this distributed learning with partial coefficients regularization is proved by the integral operator technique. Satisfactory learning rates are then derived under a standard regularity condition on the regression function, which reveals an interesting phenomenon that when [Formula: see text] and [Formula: see text] is small enough, this distributed learning has the same convergence rate with the algorithm processing the whole data in one single machine.

LEARNING RATES OF REGULARIZED REGRESSION FOR FUNCTIONAL DATA

2009 ◽  
Vol 07 (06) ◽  
pp. 839-850 ◽  
Author(s):  
YONG-LI XU ◽  
DI-RONG CHEN
Keyword(s):  
Functional Data Analysis ◽  
Functional Data ◽  
Input Data ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Natural Extension ◽  
Least Square ◽  
Regularized Regression ◽  
Fast Learning ◽  
Learning Rates

The study of regularized learning algorithms is a very important issue and functional data analysis extends classical methods. We establish the learning rates of the least square regularized regression algorithm in reproducing kernel Hilbert space for functional data. With the iteration method, we obtain fast learning rate for functional data. Our result is a natural extension for least square regularized regression algorithm when the dimension of input data is finite.

scholarly journals Convergence rates of Kernel Conjugate Gradient for random design regression

2016 ◽  
Vol 14 (06) ◽  
pp. 763-794 ◽  
Author(s):  
Gilles Blanchard ◽  
Nicole Krämer
Keyword(s):  
Least Squares ◽  
Conjugate Gradient ◽  
Reproducing Kernel ◽  
Convergence Rates ◽  
Reproducing Kernel Hilbert Space ◽  
Regression Function ◽  
Rates Of Convergence ◽  
Least Squares Regression ◽  
Hilbert Norm ◽  
Effective Dimensionality

We prove statistical rates of convergence for kernel-based least squares regression from i.i.d. data using a conjugate gradient (CG) algorithm, where regularization against overfitting is obtained by early stopping. This method is related to Kernel Partial Least Squares, a regression method that combines supervised dimensionality reduction with least squares projection. Following the setting introduced in earlier related literature, we study so-called “fast convergence rates” depending on the regularity of the target regression function (measured by a source condition in terms of the kernel integral operator) and on the effective dimensionality of the data mapped into the kernel space. We obtain upper bounds, essentially matching known minimax lower bounds, for the ℒ2 (prediction) norm as well as for the stronger Hilbert norm, if the true regression function belongs to the reproducing kernel Hilbert space. If the latter assumption is not fulfilled, we obtain similar convergence rates for appropriate norms, provided additional unlabeled data are available.

Distributed least squares prediction for functional linear regression

2021 ◽  
Author(s):  
Hongzhi Tong
Keyword(s):  
Least Squares ◽  
Reproducing Kernel ◽  
Learning Strategy ◽  
Reproducing Kernel Hilbert Space ◽  
Target Function ◽  
Regularity Conditions ◽  
Least Squares Regression ◽  
Functional Linear Model ◽  
Data Set ◽  
Functional Linear Regression

Abstract To cope with the challenges of memory bottleneck and algorithmic scalability when massive data sets are involved, we propose a distributed least squares procedure in the framework of functional linear model and reproducing kernel Hilbert space. This approach divides the big data set into multiple subsets, applies regularized least squares regression on each of them, and then averages the individual outputs as a final prediction. We establish the non-asymptotic prediction error bounds for the proposed learning strategy under some regularity conditions. When the target function only has weak regularity, we also introduce some unlabelled data to construct a semi-supervised approach to enlarge the number of the partitioned subsets. Results in present paper provide a theoretical guarantee that the distributed algorithm can achieve the optimal rate of convergence while allowing the whole data set to be partitioned into a large number of subsets for parallel processing.

scholarly journals Analysis of regularized Nyström subsampling for regression functions of low smoothness

2019 ◽  
Vol 17 (06) ◽  
pp. 931-946 ◽  
Author(s):  
Shuai Lu ◽  
Peter Mathé ◽  
Sergiy Pereverzyev
Keyword(s):  
Reproducing Kernel ◽  
Regularization Parameter ◽  
Reproducing Kernel Hilbert Space ◽  
Practical Importance ◽  
Target Function ◽  
Mild Conditions ◽  
Learning Rates ◽  
Regression Functions ◽  
Large Kernel ◽  
General Source

This paper studies a Nyström-type subsampling approach to large kernel learning methods in the misspecified case, where the target function is not assumed to belong to the reproducing kernel Hilbert space generated by the underlying kernel. This case is less understood in spite of its practical importance. To model such a case, the smoothness of target functions is described in terms of general source conditions. It is surprising that almost for the whole range of the source conditions, describing the misspecified case, the corresponding learning rate bounds can be achieved with just one value of the regularization parameter. This observation allows a formulation of mild conditions under which the plain Nyström subsampling can be realized with subquadratic cost maintaining the guaranteed learning rates.

Sign in / Sign up

Export Citation Format

Share Document