scholarly journals Convergence rates of Kernel Conjugate Gradient for random design regression

2016 ◽  
Vol 14 (06) ◽  
pp. 763-794 ◽  
Author(s):  
Gilles Blanchard ◽  
Nicole Krämer
Keyword(s):  
Least Squares ◽  
Conjugate Gradient ◽  
Reproducing Kernel ◽  
Convergence Rates ◽  
Reproducing Kernel Hilbert Space ◽  
Regression Function ◽  
Rates Of Convergence ◽  
Least Squares Regression ◽  
Hilbert Norm ◽  
Effective Dimensionality

We prove statistical rates of convergence for kernel-based least squares regression from i.i.d. data using a conjugate gradient (CG) algorithm, where regularization against overfitting is obtained by early stopping. This method is related to Kernel Partial Least Squares, a regression method that combines supervised dimensionality reduction with least squares projection. Following the setting introduced in earlier related literature, we study so-called “fast convergence rates” depending on the regularity of the target regression function (measured by a source condition in terms of the kernel integral operator) and on the effective dimensionality of the data mapped into the kernel space. We obtain upper bounds, essentially matching known minimax lower bounds, for the ℒ2 (prediction) norm as well as for the stronger Hilbert norm, if the true regression function belongs to the reproducing kernel Hilbert space. If the latter assumption is not fulfilled, we obtain similar convergence rates for appropriate norms, provided additional unlabeled data are available.

scholarly journals Coefficient-Based Regression with Non-Identical Unbounded Sampling

2013 ◽  
Vol 2013 ◽  
pp. 1-8 ◽  
Author(s):  
Jia Cai
Keyword(s):  
Hilbert Space ◽  
Integral Operator ◽  
Error Analysis ◽  
Least Squares ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Least Squares Regression ◽  
Regression Problem ◽  
Novel Technique ◽  
Sample Error

We investigate a coefficient-based least squares regression problem with indefinite kernels from non-identical unbounded sampling processes. Here non-identical unbounded sampling means the samples are drawn independently but not identically from unbounded sampling processes. The kernel is not necessarily symmetric or positive semi-definite. This leads to additional difficulty in the error analysis. By introducing a suitable reproducing kernel Hilbert space (RKHS) and a suitable intermediate integral operator, elaborate analysis is presented by means of a novel technique for the sample error. This leads to satisfactory results.

Distributed least squares prediction for functional linear regression

2021 ◽  
Author(s):  
Hongzhi Tong
Keyword(s):  
Least Squares ◽  
Reproducing Kernel ◽  
Learning Strategy ◽  
Reproducing Kernel Hilbert Space ◽  
Target Function ◽  
Regularity Conditions ◽  
Least Squares Regression ◽  
Functional Linear Model ◽  
Data Set ◽  
Functional Linear Regression

Abstract To cope with the challenges of memory bottleneck and algorithmic scalability when massive data sets are involved, we propose a distributed least squares procedure in the framework of functional linear model and reproducing kernel Hilbert space. This approach divides the big data set into multiple subsets, applies regularized least squares regression on each of them, and then averages the individual outputs as a final prediction. We establish the non-asymptotic prediction error bounds for the proposed learning strategy under some regularity conditions. When the target function only has weak regularity, we also introduce some unlabelled data to construct a semi-supervised approach to enlarge the number of the partitioned subsets. Results in present paper provide a theoretical guarantee that the distributed algorithm can achieve the optimal rate of convergence while allowing the whole data set to be partitioned into a large number of subsets for parallel processing.

scholarly journals KTBoost: Combined Kernel and Tree Boosting

2021 ◽  
Author(s):  
Fabio Sigrist
Keyword(s):  
Hilbert Space ◽  
Reproducing Kernel ◽  
Predictive Accuracy ◽  
Reproducing Kernel Hilbert Space ◽  
Regression Tree ◽  
Regression Function ◽  
Data Sets ◽  
Regression Functions ◽  
Boosting Algorithm

AbstractWe introduce a novel boosting algorithm called ‘KTBoost’ which combines kernel boosting and tree boosting. In each boosting iteration, the algorithm adds either a regression tree or reproducing kernel Hilbert space (RKHS) regression function to the ensemble of base learners. Intuitively, the idea is that discontinuous trees and continuous RKHS regression functions complement each other, and that this combination allows for better learning of functions that have parts with varying degrees of regularity such as discontinuities and smooth parts. We empirically show that KTBoost significantly outperforms both tree and kernel boosting in terms of predictive accuracy in a comparison on a wide array of data sets.

Online regularized pairwise learning with least squares loss

2019 ◽  
Vol 18 (01) ◽  
pp. 49-78 ◽  
Author(s):  
Cheng Wang ◽  
Ting Hu
Keyword(s):  
Hilbert Space ◽  
Least Squares ◽  
Online Algorithm ◽  
Reproducing Kernel ◽  
Sharp Estimate ◽  
Reproducing Kernel Hilbert Space ◽  
Learning Sequence ◽  
Pairwise Learning ◽  
Regularization Parameters

In this paper, we study online algorithm for pairwise problems generated from the Tikhonov regularization scheme associated with the least squares loss function and a reproducing kernel Hilbert space (RKHS). This work establishes the convergence for the last iterate of the online pairwise algorithm with the polynomially decaying step sizes and varying regularization parameters. We show that the obtained error rate in [Formula: see text]-norm can be nearly optimal in the minimax sense under some mild conditions. Our analysis is achieved by a sharp estimate for the norms of the learning sequence and the characterization of RKHS using its associated integral operators and probability inequalities for random variables with values in a Hilbert space.

Learning rates for regularized least squares ranking algorithm

2017 ◽  
Vol 15 (06) ◽  
pp. 815-836 ◽  
Author(s):  
Yulong Zhao ◽  
Jun Fan ◽  
Lei Shi
Keyword(s):  
Least Squares ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Ranking Algorithm ◽  
Generalization Error ◽  
Ranking Problem ◽  
Regularized Least Squares ◽  
U Statistics ◽  
Learning Rates ◽  
Error Decomposition

The ranking problem aims at learning real-valued functions to order instances, which has attracted great interest in statistical learning theory. In this paper, we consider the regularized least squares ranking algorithm within the framework of reproducing kernel Hilbert space. In particular, we focus on analysis of the generalization error for this ranking algorithm, and improve the existing learning rates by virtue of an error decomposition technique from regression and Hoeffding’s decomposition for U-statistics.

scholarly journals Robust Regression via Heuristic Hard Thresholding

2017 ◽  
Author(s):  
Xuchao Zhang ◽  
Liang Zhao ◽  
Arnold P. Boedihardjo ◽  
Chang-Tien Lu
Keyword(s):  
Least Squares ◽  
Robust Regression ◽  
Convergence Rates ◽  
State Of The Art ◽  
Fundamental Problem ◽  
Regression Coefficients ◽  
Least Squares Regression ◽  
Hard Thresholding ◽  
Robust Least Squares ◽  
Competitive Efficiency

The presence of data noise and corruptions recently invokes increasing attention on Robust Least Squares Regression (RLSR), which addresses the fundamental problem that learns reliable regression coefficients when response variables can be arbitrarily corrupted. Until now, several important challenges still cannot be handled concurrently: 1) exact recovery guarantee of regression coefficients 2) difficulty in estimating the corruption ratio parameter; and 3) scalability to massive dataset. This paper proposes a novel Robust Least squares regression algorithm via Heuristic Hard thresholding (RLHH), that concurrently addresses all the above challenges. Specifically, the algorithm alternately optimizes the regression coefficients and estimates the optimal uncorrupted set via heuristic hard thresholding without corruption ratio parameter until it converges. We also prove that our algorithm benefits from strong guarantees analogous to those of state-of-the-art methods in terms of convergence rates and recovery guarantees. We provide empirical evidence to demonstrate that the effectiveness of our new method is superior to that of existing methods in the recovery of both regression coefficients and uncorrupted sets, with very competitive efficiency.

Distributed learning with partial coefficients regularization

2018 ◽  
Vol 16 (04) ◽  
pp. 1850025 ◽  
Author(s):  
Mengjuan Pang ◽  
Hongwei Sun
Keyword(s):  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Distributed Learning ◽  
Regression Function ◽  
Operator Technique ◽  
Interesting Phenomenon ◽  
Regularization Scheme ◽  
Complexity Of Algorithms ◽  
Learning Rates ◽  
The Individual

We study distributed learning with partial coefficients regularization scheme in a reproducing kernel Hilbert space (RKHS). The algorithm randomly partitions the sample set [Formula: see text] into [Formula: see text] disjoint sample subsets of equal size. In order to reduce the complexity of algorithms, we apply a partial coefficients regularization scheme to each sample subset to produce an output function, and average the individual output functions to get the final global estimator. The error bound in the [Formula: see text]-metric is deduced and the asymptotic convergence for this distributed learning with partial coefficients regularization is proved by the integral operator technique. Satisfactory learning rates are then derived under a standard regularity condition on the regression function, which reveals an interesting phenomenon that when [Formula: see text] and [Formula: see text] is small enough, this distributed learning has the same convergence rate with the algorithm processing the whole data in one single machine.

Sign in / Sign up

Export Citation Format

Share Document