Online Pairwise Learning Algorithms

2016 ◽  
Vol 28 (4) ◽  
pp. 743-777 ◽  
Author(s):  
Yiming Ying ◽  
Ding-Xuan Zhou
Keyword(s):  
Hilbert Space ◽  
Loss Function ◽  
Reproducing Kernel ◽  
Convergence Rates ◽  
Target Function ◽  
Almost Sure Convergence ◽  
Least Square ◽  
Strongly Convex ◽  
Pairwise Learning ◽  
Auc Maximization

Pairwise learning usually refers to a learning task that involves a loss function depending on pairs of examples, among which the most notable ones are bipartite ranking, metric learning, and AUC maximization. In this letter we study an online algorithm for pairwise learning with a least-square loss function in an unconstrained setting of a reproducing kernel Hilbert space (RKHS) that we refer to as the Online Pairwise lEaRning Algorithm (OPERA). In contrast to existing works (Kar, Sriperumbudur, Jain, & Karnick, 2013 ; Wang, Khardon, Pechyony, & Jones, 2012 ), which require that the iterates are restricted to a bounded domain or the loss function is strongly convex, OPERA is associated with a non-strongly convex objective function and learns the target function in an unconstrained RKHS. Specifically, we establish a general theorem that guarantees the almost sure convergence for the last iterate of OPERA without any assumptions on the underlying distribution. Explicit convergence rates are derived under the condition of polynomially decaying step sizes. We also establish an interesting property for a family of widely used kernels in the setting of pairwise learning and illustrate the convergence results using such kernels. Our methodology mainly depends on the characterization of RKHSs using its associated integral operators and probability inequalities for random variables with values in a Hilbert space.

Online regularized pairwise learning with least squares loss

2019 ◽  
Vol 18 (01) ◽  
pp. 49-78 ◽  
Author(s):  
Cheng Wang ◽  
Ting Hu
Keyword(s):  
Hilbert Space ◽  
Least Squares ◽  
Online Algorithm ◽  
Reproducing Kernel ◽  
Sharp Estimate ◽  
Reproducing Kernel Hilbert Space ◽  
Learning Sequence ◽  
Pairwise Learning ◽  
Regularization Parameters

In this paper, we study online algorithm for pairwise problems generated from the Tikhonov regularization scheme associated with the least squares loss function and a reproducing kernel Hilbert space (RKHS). This work establishes the convergence for the last iterate of the online pairwise algorithm with the polynomially decaying step sizes and varying regularization parameters. We show that the obtained error rate in [Formula: see text]-norm can be nearly optimal in the minimax sense under some mild conditions. Our analysis is achieved by a sharp estimate for the norms of the learning sequence and the characterization of RKHS using its associated integral operators and probability inequalities for random variables with values in a Hilbert space.

Online regularized pairwise learning with non-i.i.d. observations

2021 ◽  
pp. 2150041
Author(s):  
Yimo Qin ◽  
Bin Zou ◽  
Jingjing Zeng ◽  
Zhifei Sheng ◽  
Lei Yin
Keyword(s):  
Markov Chain ◽  
Least Squares ◽  
Loss Function ◽  
Convergence Rates ◽  
Optimal Rate ◽  
Ergodic Markov Chain ◽  
Pairwise Learning ◽  
Regularization Parameters ◽  
Mixing Sequence

In this paper, we consider the online regularized pairwise learning (ORPL) algorithm with least squares loss function for non-independently and identically distribution (non-i.i.d.) observations. We first establish new Bennett’s inequalities for [Formula: see text]-mixing sequence, geometrically [Formula: see text]-mixing sequence, [Formula: see text]-geometrically ergodic Markov chain and uniformly ergodic Markov chain. Then we establish the convergence rates for the last iterate of the ORPL algorithm with the polynomially decaying step sizes and varying regularization parameters for non-i.i.d. observations. These established results in this paper extend the previously known results of ORPL from i.i.d. observations to the case of non-i.i.d. observations, and the established result of ORPL for [Formula: see text]-mixing can be nearly optimal rate of ORPL for i.i.d. observations with [Formula: see text]-norm.

Reproducing Kernel Hilbert space and penalized weighted least square in nonparametric regression

2014 ◽  
Vol 8 ◽  
pp. 7289-7300 ◽  
Author(s):  
Adji Achmad Rinaldo Fernandes ◽  
I Nyoman Budiantara ◽  
Bambang Widjanarko Otok ◽  
Suhartono
Keyword(s):  
Hilbert Space ◽  
Nonparametric Regression ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Least Square ◽  
Weighted Least Square

scholarly journals Lebesgue Decomposition of Non-Commutative Measures

2020 ◽  
Author(s):  
Michael T Jury ◽  
Robert T W Martin
Keyword(s):  
Hilbert Space ◽  
Reproducing Kernel ◽  
Fock Space ◽  
Reproducing Kernel Hilbert Space ◽  
Semigroup Algebra ◽  
Space Theory ◽  
Lebesgue Decomposition ◽  
Sesquilinear Forms ◽  
Positive Linear Functionals ◽  
Algebra Theory

Abstract We extend the Lebesgue decomposition of positive measures with respect to Lebesgue measure on the complex unit circle to the non-commutative (NC) multi-variable setting of (positive) NC measures. These are positive linear functionals on a certain self-adjoint subspace of the Cuntz–Toeplitz $C^{\ast }-$algebra, the $C^{\ast }-$algebra of the left creation operators on the full Fock space. This theory is fundamentally connected to the representation theory of the Cuntz and Cuntz–Toeplitz $C^{\ast }-$algebras; any *−representation of the Cuntz–Toeplitz $C^{\ast }-$algebra is obtained (up to unitary equivalence), by applying a Gelfand–Naimark–Segal construction to a positive NC measure. Our approach combines the theory of Lebesgue decomposition of sesquilinear forms in Hilbert space, Lebesgue decomposition of row isometries, free semigroup algebra theory, NC reproducing kernel Hilbert space theory, and NC Hardy space theory.

scholarly journals An attractive numerical algorithm for solving nonlinear Caputo–Fabrizio fractional Abel differential equation in a Hilbert space

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Mohammed Al-Smadi ◽  
Nadir Djeddi ◽  
Shaher Momani ◽  
Shrideh Al-Omari ◽  
Serkan Araci
Keyword(s):  
Differential Equation ◽  
Hilbert Space ◽  
Reproducing Kernel ◽  
Numerical Approach ◽  
Accurate Solution ◽  
Stability And Convergence ◽  
Type Model ◽  
Schmidt Orthogonalization ◽  
And Performance ◽  
Abel Differential Equation

AbstractOur aim in this paper is presenting an attractive numerical approach giving an accurate solution to the nonlinear fractional Abel differential equation based on a reproducing kernel algorithm with model endowed with a Caputo–Fabrizio fractional derivative. By means of such an approach, we utilize the Gram–Schmidt orthogonalization process to create an orthonormal set of bases that leads to an appropriate solution in the Hilbert space $\mathcal{H}^{2}[a,b]$ H 2 [ a , b ] . We investigate and discuss stability and convergence of the proposed method. The n-term series solution converges uniformly to the analytic solution. We present several numerical examples of potential interests to illustrate the reliability, efficacy, and performance of the method under the influence of the Caputo–Fabrizio derivative. The gained results have shown superiority of the reproducing kernel algorithm and its infinite accuracy with a least time and efforts in solving the fractional Abel-type model. Therefore, in this direction, the proposed algorithm is an alternative and systematic tool for analyzing the behavior of many nonlinear temporal fractional differential equations emerging in the fields of engineering, physics, and sciences.

scholarly journals Multi-omics-based prediction of hybrid performance in canola

2021 ◽  
Author(s):  
Dominic Knoch ◽  
Christian R. Werner ◽  
Rhonda C. Meyer ◽  
David Riewe ◽  
Amine Abbadi ◽  
...  
Keyword(s):  
Hilbert Space ◽  
Genetic Markers ◽  
Reproducing Kernel ◽  
Prediction Models ◽  
Agronomic Traits ◽  
Reproducing Kernel Hilbert Space ◽  
Hybrid Performance ◽  
Hybrid Prediction ◽  
Gaussian Kernels ◽  
Breeding Programmes

Abstract Key message Complementing or replacing genetic markers with transcriptomic data and use of reproducing kernel Hilbert space regression based on Gaussian kernels increases hybrid prediction accuracies for complex agronomic traits in canola. In plant breeding, hybrids gained particular importance due to heterosis, the superior performance of offspring compared to their inbred parents. Since the development of new top performing hybrids requires labour-intensive and costly breeding programmes, including testing of large numbers of experimental hybrids, the prediction of hybrid performance is of utmost interest to plant breeders. In this study, we tested the effectiveness of hybrid prediction models in spring-type oilseed rape (Brassica napus L./canola) employing different omics profiles, individually and in combination. To this end, a population of 950 F1 hybrids was evaluated for seed yield and six other agronomically relevant traits in commercial field trials at several locations throughout Europe. A subset of these hybrids was also evaluated in a climatized glasshouse regarding early biomass production. For each of the 477 parental rapeseed lines, 13,201 single nucleotide polymorphisms (SNPs), 154 primary metabolites, and 19,479 transcripts were determined and used as predictive variables. Both, SNP markers and transcripts, effectively predict hybrid performance using (genomic) best linear unbiased prediction models (gBLUP). Compared to models using pure genetic markers, models incorporating transcriptome data resulted in significantly higher prediction accuracies for five out of seven agronomic traits, indicating that transcripts carry important information beyond genomic data. Notably, reproducing kernel Hilbert space regression based on Gaussian kernels significantly exceeded the predictive abilities of gBLUP models for six of the seven agronomic traits, demonstrating its potential for implementation in future canola breeding programmes.

scholarly journals KTBoost: Combined Kernel and Tree Boosting

2021 ◽  
Author(s):  
Fabio Sigrist
Keyword(s):  
Hilbert Space ◽  
Reproducing Kernel ◽  
Predictive Accuracy ◽  
Reproducing Kernel Hilbert Space ◽  
Regression Tree ◽  
Regression Function ◽  
Data Sets ◽  
Regression Functions ◽  
Boosting Algorithm

AbstractWe introduce a novel boosting algorithm called ‘KTBoost’ which combines kernel boosting and tree boosting. In each boosting iteration, the algorithm adds either a regression tree or reproducing kernel Hilbert space (RKHS) regression function to the ensemble of base learners. Intuitively, the idea is that discontinuous trees and continuous RKHS regression functions complement each other, and that this combination allows for better learning of functions that have parts with varying degrees of regularity such as discontinuities and smooth parts. We empirically show that KTBoost significantly outperforms both tree and kernel boosting in terms of predictive accuracy in a comparison on a wide array of data sets.

Sign in / Sign up

Export Citation Format

Share Document