scholarly journals Gaussian Processes for Regression and Optimisation

2021 ◽  
Author(s):  
◽  
Phillip Boyle
Keyword(s):  
Gaussian Process ◽  
Gaussian Processes ◽  
Model Comparison ◽  
Classical Method ◽  
Gaussian Process Regression ◽  
Training Data ◽  
Linear Filter ◽  
Efficient Computation ◽  
Linear Filters ◽  
Reduced Rank

<p>Gaussian processes have proved to be useful and powerful constructs for the purposes of regression. The classical method proceeds by parameterising a covariance function, and then infers the parameters given the training data. In this thesis, the classical approach is augmented by interpreting Gaussian processes as the outputs of linear filters excited by white noise. This enables a straightforward definition of dependent Gaussian processes as the outputs of a multiple output linear filter excited by multiple noise sources. We show how dependent Gaussian processes defined in this way can also be used for the purposes of system identification. Onewell known problem with Gaussian process regression is that the computational complexity scales poorly with the amount of training data. We review one approximate solution that alleviates this problem, namely reduced rank Gaussian processes. We then show how the reduced rank approximation can be applied to allow for the efficient computation of dependent Gaussian processes. We then examine the application of Gaussian processes to the solution of other machine learning problems. To do so, we review methods for the parameterisation of full covariance matrices. Furthermore, we discuss how improvements can be made by marginalising over alternative models, and introduce methods to perform these computations efficiently. In particular, we introduce sequential annealed importance sampling as a method for calculating model evidence in an on-line fashion as new data arrives. Gaussian process regression can also be applied to optimisation. An algorithm is described that uses model comparison between multiple models to find the optimum of a function while taking as few samples as possible. This algorithm shows impressive performance on the standard control problem of double pole balancing. Finally, we describe how Gaussian processes can be used to efficiently estimate gradients of noisy functions, and numerically estimate integrals.</p>

scholarly journals Gaussian Processes for Regression and Optimisation

2021 ◽  
Author(s):  
◽  
Phillip Boyle
Keyword(s):  
Gaussian Process ◽  
Gaussian Processes ◽  
Model Comparison ◽  
Classical Method ◽  
Gaussian Process Regression ◽  
Training Data ◽  
Linear Filter ◽  
Efficient Computation ◽  
Linear Filters ◽  
Reduced Rank

<p>Gaussian processes have proved to be useful and powerful constructs for the purposes of regression. The classical method proceeds by parameterising a covariance function, and then infers the parameters given the training data. In this thesis, the classical approach is augmented by interpreting Gaussian processes as the outputs of linear filters excited by white noise. This enables a straightforward definition of dependent Gaussian processes as the outputs of a multiple output linear filter excited by multiple noise sources. We show how dependent Gaussian processes defined in this way can also be used for the purposes of system identification. Onewell known problem with Gaussian process regression is that the computational complexity scales poorly with the amount of training data. We review one approximate solution that alleviates this problem, namely reduced rank Gaussian processes. We then show how the reduced rank approximation can be applied to allow for the efficient computation of dependent Gaussian processes. We then examine the application of Gaussian processes to the solution of other machine learning problems. To do so, we review methods for the parameterisation of full covariance matrices. Furthermore, we discuss how improvements can be made by marginalising over alternative models, and introduce methods to perform these computations efficiently. In particular, we introduce sequential annealed importance sampling as a method for calculating model evidence in an on-line fashion as new data arrives. Gaussian process regression can also be applied to optimisation. An algorithm is described that uses model comparison between multiple models to find the optimum of a function while taking as few samples as possible. This algorithm shows impressive performance on the standard control problem of double pole balancing. Finally, we describe how Gaussian processes can be used to efficiently estimate gradients of noisy functions, and numerically estimate integrals.</p>

scholarly journals Vision-Based Satellite Recognition and Pose Estimation Using Gaussian Process Regression

2019 ◽  
Vol 2019 ◽  
pp. 1-20 ◽  
Author(s):  
Haopeng Zhang ◽  
Cong Zhang ◽  
Zhiguo Jiang ◽  
Yuan Yao ◽  
Gang Meng
Keyword(s):  
Gaussian Process ◽  
Pose Estimation ◽  
Gaussian Process Regression ◽  
Regression Function ◽  
Mean Value ◽  
Training Data ◽  
Simulated Image ◽  
Lighting Conditions ◽  
Mean Function ◽  
Imaging Sensors

In this paper, we address the problem of vision-based satellite recognition and pose estimation, which is to recognize the satellite from multiviews and estimate the relative poses using imaging sensors. We propose a vision-based method to solve these two problems using Gaussian process regression (GPR). Assuming that the regression function mapping from the image (or feature) of the target satellite to its category or pose follows a Gaussian process (GP) properly parameterized by a mean function and a covariance function, the predictive equations can be easily obtained by a maximum-likelihood approach when training data are given. These explicit formulations can not only offer the category or estimated pose by the mean value of the predicted output but also give its uncertainty by the variance which makes the predicted result convincing and applicable in practice. Besides, we also introduce a manifold constraint to the output of the GPR model to improve its performance for satellite pose estimation. Extensive experiments are performed on two simulated image datasets containing satellite images of 1D and 2D pose variations, as well as different noises and lighting conditions. Experimental results validate the effectiveness and robustness of our approach.

scholarly journals Detecting periodicities with Gaussian processes

2016 ◽  
Vol 2 ◽  
pp. e50 ◽  
Author(s):  
Nicolas Durrande ◽  
James Hensman ◽  
Magnus Rattray ◽  
Neil D. Lawrence
Keyword(s):  
Hilbert Space ◽  
Gaussian Process ◽  
Gaussian Processes ◽  
Covariance Function ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Gaussian Process Regression ◽  
Inner Product ◽  
Input Output ◽  
Periodic Data

We consider the problem of detecting and quantifying the periodic component of a function given noise-corrupted observations of a limited number of input/output tuples. Our approach is based on Gaussian process regression, which provides a flexible non-parametric framework for modelling periodic data. We introduce a novel decomposition of the covariance function as the sum of periodic and aperiodic kernels. This decomposition allows for the creation of sub-models which capture the periodic nature of the signal and its complement. To quantify the periodicity of the signal, we derive a periodicity ratio which reflects the uncertainty in the fitted sub-models. Although the method can be applied to many kernels, we give a special emphasis to the Matérn family, from the expression of the reproducing kernel Hilbert space inner product to the implementation of the associated periodic kernels in a Gaussian process toolkit. The proposed method is illustrated by considering the detection of periodically expressed genes in thearabidopsisgenome.

scholarly journals Searching for rewards in graph-structured spaces

2019 ◽  
Author(s):  
Eric Schulz ◽  
Charley M Wu
Keyword(s):  
Reinforcement Learning ◽  
Regression Model ◽  
Gaussian Process ◽  
Predictive Model ◽  
Human Behavior ◽  
Model Comparison ◽  
Gaussian Process Regression ◽  
Diffusion Kernel ◽  
Function Learning ◽  
Psychological Models

How do people generalize and explore structured spaces? We study human behavior on a multi-armed bandit task, where rewards are influenced by the connectivity structure of a graph. A detailed predictive model comparison shows that a Gaussian Process regression model using a diffusion kernel is able to best describe participant choices, and also predict judgments about expected reward and confidence. This model unifies psychological models of function learning with the Successor Representation used in reinforcement learning, thereby building a bridge between different models of generalization.

scholarly journals Student-t Process Regression with Student-t Likelihood

2017 ◽  
Author(s):  
Qingtao Tang ◽  
Li Niu ◽  
Yisen Wang ◽  
Tao Dai ◽  
Wangpeng An ◽  
...  
Keyword(s):  
Gaussian Process ◽  
Gaussian Noise ◽  
Bayesian Method ◽  
Gaussian Process Regression ◽  
Training Data ◽  
Special Cases ◽  
The Common

Gaussian Process Regression (GPR) is a powerful Bayesian method. However, the performance of GPR can be significantly degraded when the training data are contaminated by outliers, including target outliers and input outliers. Although there are some variants of GPR (e.g., GPR with Student-t likelihood (GPRT)) aiming to handle outliers, most of the variants focus on handling the target outliers while little effort has been done to deal with the input outliers. In contrast, in this work, we aim to handle both the target outliers and the input outliers at the same time. Specifically, we replace the Gaussian noise in GPR with independent Student-t noise to cope with the target outliers. Moreover, to enhance the robustness w.r.t. the input outliers, we use a Student-t Process prior instead of the common Gaussian Process prior, leading to Student-t Process Regression with Student-t Likelihood (TPRT). We theoretically show that TPRT is more robust to both input and target outliers than GPR and GPRT, and prove that both GPR and GPRT are special cases of TPRT. Various experiments demonstrate that TPRT outperforms GPR and its variants on both synthetic and real datasets.

scholarly journals Linking Gaussian process regression with data-driven manifold embeddings for nonlinear data fusion

2019 ◽  
Vol 9 (3) ◽  
pp. 20180083 ◽  
Author(s):  
Seungjoon Lee ◽  
Felix Dietrich ◽  
George E. Karniadakis ◽  
Ioannis G. Kevrekidis
Keyword(s):  
Gaussian Process ◽  
Information Fusion ◽  
Gaussian Process Regression ◽  
Statistical Modelling ◽  
Training Data ◽  
High Fidelity ◽  
Several Variables ◽  
Mathematical Algorithms ◽  
Data Points ◽  
Fidelity Model

In statistical modelling with Gaussian process regression, it has been shown that combining (few) high-fidelity data with (many) low-fidelity data can enhance prediction accuracy, compared to prediction based on the few high-fidelity data only. Such information fusion techniques for multi-fidelity data commonly approach the high-fidelity model f h ( t ) as a function of two variables ( t , s ), and then use f l ( t ) as the s data. More generally, the high-fidelity model can be written as a function of several variables ( t , s 1 , s 2 ….); the low-fidelity model f l and, say, some of its derivatives can then be substituted for these variables. In this paper, we will explore mathematical algorithms for multi-fidelity information fusion that use such an approach towards improving the representation of the high-fidelity function with only a few training data points. Given that f h may not be a simple function—and sometimes not even a function—of f l , we demonstrate that using additional functions of t , such as derivatives or shifts of f l , can drastically improve the approximation of f h through Gaussian processes. We also point out a connection with ‘embedology’ techniques from topology and dynamical systems. Our illustrative examples range from instructive caricatures to computational biology models, such as Hodgkin–Huxley neural oscillations.

A Bayesian Committee Machine

2000 ◽  
Vol 12 (11) ◽  
pp. 2719-2741 ◽  
Author(s):  
Volker Tresp
Keyword(s):  
Gaussian Process ◽  
Degrees Of Freedom ◽  
Gaussian Process Regression ◽  
Smoothing Splines ◽  
Training Data ◽  
Data Sets ◽  
Committee Machine ◽  
Novel Approach ◽  
Potential Applications ◽  
Test Points

The Bayesian committee machine (BCM) is a novel approach to combining estimators that were trained on different data sets. Although the BCM can be applied to the combination of any kind of estimators, the main foci are gaussian process regression and related systems such as regularization networks and smoothing splines for which the degrees of freedom increase with the number of training data. Somewhat surprisingly, we find that the performance of the BCM improves if several test points are queried at the same time and is optimal if the number of test points is at least as large as the degrees of freedom of the estimator. The BCM also provides a new solution for on-line learning with potential applications to data mining. We apply the BCM to systems with fixed basis functions and discuss its relationship to gaussian process regression. Finally, we show how the ideas behind the BCM can be applied in a non-Bayesian setting to extend the input-dependent combination of estimators.

scholarly journals Detecting periodicities with Gaussian processes

2016 ◽  
Author(s):  
Nicolas Durrande ◽  
James Hensman ◽  
Magnus Rattray ◽  
Neil D Lawrence
Keyword(s):  
Hilbert Space ◽  
Gaussian Process ◽  
Gaussian Processes ◽  
Covariance Function ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Gaussian Process Regression ◽  
Inner Product ◽  
Input Output ◽  
Periodic Data

We consider the problem of detecting and quantifying the periodic component of a function given noise-corrupted observations of a limited number of input/output tuples. Our approach is based on Gaussian process regression which provides a flexible non-parametric framework for modelling periodic data. We introduce a novel decomposition of the covariance function as the sum of periodic and aperiodic kernels. This decomposition allows for the creation of sub-models which capture the periodic nature of the signal and its complement. To quantify the periodicity of the signal, we derive a periodicity ratio which reflects the uncertainty in the fitted sub-models. Although the method can be applied to many kernels, we give a special emphasis to the Matérn family, from the expression of the reproducing kernel Hilbert space inner product to the implementation of the associated periodic kernels in a Gaussian process toolkit. The proposed method is illustrated by considering the detection of periodically expressed genes in the arabidopsis genome.

scholarly journals Detecting periodicities with Gaussian processes

2016 ◽  
Author(s):  
Nicolas Durrande ◽  
James Hensman ◽  
Magnus Rattray ◽  
Neil D Lawrence
Keyword(s):  
Hilbert Space ◽  
Gaussian Process ◽  
Gaussian Processes ◽  
Covariance Function ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Gaussian Process Regression ◽  
Inner Product ◽  
Input Output ◽  
Periodic Data

We consider the problem of detecting and quantifying the periodic component of a function given noise-corrupted observations of a limited number of input/output tuples. Our approach is based on Gaussian process regression which provides a flexible non-parametric framework for modelling periodic data. We introduce a novel decomposition of the covariance function as the sum of periodic and aperiodic kernels. This decomposition allows for the creation of sub-models which capture the periodic nature of the signal and its complement. To quantify the periodicity of the signal, we derive a periodicity ratio which reflects the uncertainty in the fitted sub-models. Although the method can be applied to many kernels, we give a special emphasis to the Matérn family, from the expression of the reproducing kernel Hilbert space inner product to the implementation of the associated periodic kernels in a Gaussian process toolkit. The proposed method is illustrated by considering the detection of periodically expressed genes in the arabidopsis genome.

scholarly journals The pitfalls of using Gaussian Process Regression for normative modeling

PLoS ONE ◽  
2021 ◽  
Vol 16 (9) ◽  
pp. e0252108
Author(s):  
Bohan Xu ◽  
Rayus Kuplicki ◽  
Sandip Sen ◽  
Martin P. Paulus
Keyword(s):  
Gaussian Process ◽  
Gaussian Processes ◽  
Gaussian Process Regression ◽  
Input Domain

Normative modeling, a group of methods used to quantify an individual’s deviation from some expected trajectory relative to observed variability around that trajectory, has been used to characterize subject heterogeneity. Gaussian Processes Regression includes an estimate of variable uncertainty across the input domain, which at face value makes it an attractive method to normalize the cohort heterogeneity where the deviation between predicted value and true observation is divided by the derived uncertainty directly from Gaussian Processes Regression. However, we show that the uncertainty directly from Gaussian Processes Regression is irrelevant to the cohort heterogeneity in general.

Sign in / Sign up

Export Citation Format

Share Document