Approximation Analysis of Margin-Based Ranking Algorithm

Natural Extension ◽

Least Square ◽

Regularized Regression ◽

Fast Learning ◽

Learning Rates

The study of regularized learning algorithms is a very important issue and functional data analysis extends classical methods. We establish the learning rates of the least square regularized regression algorithm in reproducing kernel Hilbert space for functional data. With the iteration method, we obtain fast learning rate for functional data. Our result is a natural extension for least square regularized regression algorithm when the dimension of input data is finite.

Kernel Treelets

Advances in Data Science and Adaptive Analysis ◽

10.1142/s2424922x19500062 ◽

2019 ◽

Vol 11 (03n04) ◽

pp. 1950006

Author(s):

Hedi Xia ◽

Hector D. Ceniceros

Keyword(s):

Hierarchical Clustering ◽

Reproducing Kernel ◽

Feature Space ◽

Coefficient Matrix ◽

Data Sets ◽

Clustering Methods ◽

Orthonormal Bases ◽

Data Points ◽

General Data

A new method for hierarchical clustering of data points is presented. It combines treelets, a particular multiresolution decomposition of data, with a mapping on a reproducing kernel Hilbert space. The proposed approach, called kernel treelets (KT), uses this mapping to go from a hierarchical clustering over attributes (the natural output of treelets) to a hierarchical clustering over data. KT effectively substitutes the correlation coefficient matrix used in treelets with a symmetric and positive semi-definite matrix efficiently constructed from a symmetric and positive semi-definite kernel function. Unlike most clustering methods, which require data sets to be numeric, KT can be applied to more general data and yields a multiresolution sequence of orthonormal bases on the data directly in feature space. The effectiveness and potential of KT in clustering analysis are illustrated with some examples.

Learning rates for regularized least squares ranking algorithm

Analysis and Applications ◽

10.1142/s0219530517500063 ◽

2017 ◽

Vol 15 (06) ◽

pp. 815-836 ◽

Cited By ~ 12

Author(s):

Yulong Zhao ◽

Jun Fan ◽

Lei Shi

Keyword(s):

Least Squares ◽

Reproducing Kernel ◽

Ranking Algorithm ◽

Generalization Error ◽

Ranking Problem ◽

Regularized Least Squares ◽

U Statistics ◽

Learning Rates ◽

Error Decomposition

The ranking problem aims at learning real-valued functions to order instances, which has attracted great interest in statistical learning theory. In this paper, we consider the regularized least squares ranking algorithm within the framework of reproducing kernel Hilbert space. In particular, we focus on analysis of the generalization error for this ranking algorithm, and improve the existing learning rates by virtue of an error decomposition technique from regression and Hoeffding’s decomposition for U-statistics.

Reproducing Kernel Hilbert space and penalized weighted least square in nonparametric regression

Applied Mathematical Sciences ◽

10.12988/ams.2014.49759 ◽

2014 ◽

Vol 8 ◽

pp. 7289-7300 ◽

Cited By ~ 2

Author(s):

Adji Achmad Rinaldo Fernandes ◽

I Nyoman Budiantara ◽

Bambang Widjanarko Otok ◽

Suhartono

Keyword(s):

Hilbert Space ◽

Nonparametric Regression ◽

Reproducing Kernel ◽

Least Square ◽

Weighted Least Square

Two-sample statistics based on anisotropic kernels

Information and Inference A Journal of the IMA ◽

10.1093/imaiai/iaz018 ◽

2019 ◽

Vol 9 (3) ◽

pp. 677-719 ◽

Cited By ~ 1

Author(s):

Xiuyuan Cheng ◽

Alexander Cloninger ◽

Ronald R Coifman

Keyword(s):

Reproducing Kernel ◽

Reference Points ◽

Finite Sample ◽

Maximum Mean Discrepancy ◽

Data Points ◽

Consistency Of The Test ◽

Low Dimensional ◽

Anisotropic Kernel ◽

Anisotropic Kernels

Abstract The paper introduces a new kernel-based Maximum Mean Discrepancy (MMD) statistic for measuring the distance between two distributions given finitely many multivariate samples. When the distributions are locally low-dimensional, the proposed test can be made more powerful to distinguish certain alternatives by incorporating local covariance matrices and constructing an anisotropic kernel. The kernel matrix is asymmetric; it computes the affinity between $n$ data points and a set of $n_R$ reference points, where $n_R$ can be drastically smaller than $n$. While the proposed statistic can be viewed as a special class of Reproducing Kernel Hilbert Space MMD, the consistency of the test is proved, under mild assumptions of the kernel, as long as $\|p-q\| \sqrt{n} \to \infty $, and a finite-sample lower bound of the testing power is obtained. Applications to flow cytometry and diffusion MRI datasets are demonstrated, which motivate the proposed approach to compare distributions.

OUTLIER-ROBUST KERNEL HIERARCHICAL-OPTIMIZATION RLS ON A BUDGET WITH AFFINE CONSTRAINTS

10.36227/techrxiv.13110893.v2 ◽

2020 ◽

Author(s):

Konstantinos Slavakis ◽

Masahiro Yukawa

Keyword(s):

Reproducing Kernel ◽

Side Information ◽

Learning Task ◽

Quadratic Loss ◽

Infinite Dimensional ◽

Learning Framework ◽

Affine Constraints ◽

<div>This paper introduces a non-parametric learning framework to combat outliers in online, multi-output, and nonlinear regression tasks. A hierarchical-optimization problem underpins the learning task: Search in a reproducing kernel Hilbert space (RKHS) for a function that minimizes a sample average $\ell_p$-norm ($1 \leq p \leq 2$) error loss on data contaminated by noise and outliers, subject to side information that takes the form of affine constraints defined as the set of minimizers of a quadratic loss on a finite number of faithful data devoid of noise and outliers. To surmount the computational obstacles inflicted by the choice of loss and the potentially infinite dimensional RKHS, approximations of the $\ell_p$-norm loss, as well as a novel twist of the criterion of approximate linear dependency are devised to keep the computational-complexity footprint of the proposed algorithm bounded over time. Numerical tests on datasets showcase the robust behavior of the advocated framework against different types of outliers, under a low computational load, while satisfying at the same time the affine constraints, in contrast to the state-of-the-art methods which are constraint agnostic.</div><div><br></div><div>-------</div><div><br></div><div>© 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.<br></div>

Biologically-Inspired Pulse Signal Processing for Intelligence at the Edge

Frontiers in Artificial Intelligence ◽

10.3389/frai.2021.568384 ◽

2021 ◽

Vol 4 ◽

Author(s):

Kan Li ◽

José C. Príncipe

Keyword(s):

Machine Learning ◽

Energy Efficient ◽

Reproducing Kernel ◽

Pulse Signal ◽

Learning Task ◽

Keyword Spotting ◽

Data Types ◽

Biologically Inspired ◽

Neural Network Approach

There is an ever-growing mismatch between the proliferation of data-intensive, power-hungry deep learning solutions in the machine learning (ML) community and the need for agile, portable solutions in resource-constrained devices, particularly for intelligence at the edge. In this paper, we present a fundamentally novel approach that leverages data-driven intelligence with biologically-inspired efficiency. The proposed Sparse Embodiment Neural-Statistical Architecture (SENSA) decomposes the learning task into two distinct phases: a training phase and a hardware embedment phase where prototypes are extracted from the trained network and used to construct fast, sparse embodiment for hardware deployment at the edge. Specifically, we propose the Sparse Pulse Automata via Reproducing Kernel (SPARK) method, which first constructs a learning machine in the form of a dynamical system using energy-efficient spike or pulse trains, commonly used in neuroscience and neuromorphic engineering, then extracts a rule-based solution in the form of automata or lookup tables for rapid deployment in edge computing platforms. We propose to use the theoretically-grounded unifying framework of the Reproducing Kernel Hilbert Space (RKHS) to provide interpretable, nonlinear, and nonparametric solutions, compared to the typical neural network approach. In kernel methods, the explicit representation of the data is of secondary nature, allowing the same algorithm to be used for different data types without altering the learning rules. To showcase SPARK’s capabilities, we carried out the first proof-of-concept demonstration on the task of isolated-word automatic speech recognition (ASR) or keyword spotting, benchmarked on the TI-46 digit corpus. Together, these energy-efficient and resource-conscious techniques will bring advanced machine learning solutions closer to the edge.

OUTLIER-ROBUST KERNEL HIERARCHICAL-OPTIMIZATION RLS ON A BUDGET WITH AFFINE CONSTRAINTS

10.36227/techrxiv.13110893 ◽

2020 ◽

Author(s):

Konstantinos Slavakis ◽

Masahiro Yukawa

Keyword(s):

Reproducing Kernel ◽

Side Information ◽

Learning Task ◽

Quadratic Loss ◽

Infinite Dimensional ◽

Learning Framework ◽

Affine Constraints ◽

<div>This paper introduces a non-parametric learning framework to combat outliers in online, multi-output, and nonlinear regression tasks. A hierarchical-optimization problem underpins the learning task: Search in a reproducing kernel Hilbert space (RKHS) for a function that minimizes a sample average $\ell_p$-norm ($1 \leq p \leq 2$) error loss on data contaminated by noise and outliers, subject to side information that takes the form of affine constraints defined as the set of minimizers of a quadratic loss on a finite number of faithful data devoid of noise and outliers. To surmount the computational obstacles inflicted by the choice of loss and the potentially infinite dimensional RKHS, approximations of the $\ell_p$-norm loss, as well as a novel twist of the criterion of approximate linear dependency are devised to keep the computational-complexity footprint of the proposed algorithm bounded over time. Numerical tests on datasets showcase the robust behavior of the advocated framework against different types of outliers, under a low computational load, while satisfying at the same time the affine constraints, in contrast to the state-of-the-art methods which are constraint agnostic.</div><div><br></div><div>-------</div><div><br></div><div>© 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.<br></div>

OUTLIER-ROBUST KERNEL HIERARCHICAL-OPTIMIZATION RLS ON A BUDGET WITH AFFINE CONSTRAINTS

10.36227/techrxiv.13110893.v3 ◽

2021 ◽

Author(s):

Konstantinos Slavakis ◽

Masahiro Yukawa

Keyword(s):

Reproducing Kernel ◽

Side Information ◽

Learning Task ◽

Quadratic Loss ◽

Infinite Dimensional ◽

Learning Framework ◽

Affine Constraints ◽

<div>This paper introduces a non-parametric learning framework to combat outliers in online, multi-output, and nonlinear regression tasks. A hierarchical-optimization problem underpins the learning task: Search in a reproducing kernel Hilbert space (RKHS) for a function that minimizes a sample average $\ell_p$-norm ($1 \leq p \leq 2$) error loss on data contaminated by noise and outliers, subject to side information that takes the form of affine constraints defined as the set of minimizers of a quadratic loss on a finite number of faithful data devoid of noise and outliers. To surmount the computational obstacles inflicted by the choice of loss and the potentially infinite dimensional RKHS, approximations of the $\ell_p$-norm loss, as well as a novel twist of the criterion of approximate linear dependency are devised to keep the computational-complexity footprint of the proposed algorithm bounded over time. Numerical tests on datasets showcase the robust behavior of the advocated framework against different types of outliers, under a low computational load, while satisfying at the same time the affine constraints, in contrast to the state-of-the-art methods which are constraint agnostic.</div><div><br></div><div>-------</div><div><br></div><div>© 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.<br></div>

Outlier-Robust Kernel Hierarchical-Optimization RLS On A Budget With Affine Constraints

10.36227/techrxiv.13110893.v1 ◽

2020 ◽

Author(s):

Konstantinos Slavakis ◽

Masahiro Yukawa

Keyword(s):

Reproducing Kernel ◽

Side Information ◽

Learning Task ◽

Quadratic Loss ◽

Infinite Dimensional ◽

Learning Framework ◽

Affine Constraints ◽

<div>This paper introduces a non-parametric learning framework to combat outliers in online, multi-output, and nonlinear regression tasks. A hierarchical-optimization problem underpins the learning task: Search in a reproducing kernel Hilbert space (RKHS) for a function that minimizes a sample average $\ell_p$-norm ($1 \leq p \leq 2$) error loss on data contaminated by noise and outliers, subject to side information that takes the form of affine constraints defined as the set of minimizers of a quadratic loss on a finite number of faithful data devoid of noise and outliers. To surmount the computational obstacles inflicted by the choice of loss and the potentially infinite dimensional RKHS, approximations of the $\ell_p$-norm loss, as well as a novel twist of the criterion of approximate linear dependency are devised to keep the computational-complexity footprint of the proposed algorithm bounded over time. Numerical tests on datasets showcase the robust behavior of the advocated framework against different types of outliers, under a low computational load, while satisfying at the same time the affine constraints, in contrast to the state-of-the-art methods which are constraint agnostic.</div><div><br></div><div>-------</div><div><br></div><div>© 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.<br></div>