Gaussian Processes for Regression and Optimisation

<p>Gaussian processes have proved to be useful and powerful constructs for the purposes of regression. The classical method proceeds by parameterising a covariance function, and then infers the parameters given the training data. In this thesis, the classical approach is augmented by interpreting Gaussian processes as the outputs of linear filters excited by white noise. This enables a straightforward definition of dependent Gaussian processes as the outputs of a multiple output linear filter excited by multiple noise sources. We show how dependent Gaussian processes defined in this way can also be used for the purposes of system identification. Onewell known problem with Gaussian process regression is that the computational complexity scales poorly with the amount of training data. We review one approximate solution that alleviates this problem, namely reduced rank Gaussian processes. We then show how the reduced rank approximation can be applied to allow for the efficient computation of dependent Gaussian processes. We then examine the application of Gaussian processes to the solution of other machine learning problems. To do so, we review methods for the parameterisation of full covariance matrices. Furthermore, we discuss how improvements can be made by marginalising over alternative models, and introduce methods to perform these computations efficiently. In particular, we introduce sequential annealed importance sampling as a method for calculating model evidence in an on-line fashion as new data arrives. Gaussian process regression can also be applied to optimisation. An algorithm is described that uses model comparison between multiple models to find the optimum of a function while taking as few samples as possible. This algorithm shows impressive performance on the standard control problem of double pole balancing. Finally, we describe how Gaussian processes can be used to efficiently estimate gradients of noisy functions, and numerically estimate integrals.</p>

Gaussian Processes for Regression and Optimisation

<p>Gaussian processes have proved to be useful and powerful constructs for the purposes of regression. The classical method proceeds by parameterising a covariance function, and then infers the parameters given the training data. In this thesis, the classical approach is augmented by interpreting Gaussian processes as the outputs of linear filters excited by white noise. This enables a straightforward definition of dependent Gaussian processes as the outputs of a multiple output linear filter excited by multiple noise sources. We show how dependent Gaussian processes defined in this way can also be used for the purposes of system identification. Onewell known problem with Gaussian process regression is that the computational complexity scales poorly with the amount of training data. We review one approximate solution that alleviates this problem, namely reduced rank Gaussian processes. We then show how the reduced rank approximation can be applied to allow for the efficient computation of dependent Gaussian processes. We then examine the application of Gaussian processes to the solution of other machine learning problems. To do so, we review methods for the parameterisation of full covariance matrices. Furthermore, we discuss how improvements can be made by marginalising over alternative models, and introduce methods to perform these computations efficiently. In particular, we introduce sequential annealed importance sampling as a method for calculating model evidence in an on-line fashion as new data arrives. Gaussian process regression can also be applied to optimisation. An algorithm is described that uses model comparison between multiple models to find the optimum of a function while taking as few samples as possible. This algorithm shows impressive performance on the standard control problem of double pole balancing. Finally, we describe how Gaussian processes can be used to efficiently estimate gradients of noisy functions, and numerically estimate integrals.</p>

Reduced-cost construction of Jacobian matrices for high-resolution inversions of satellite observations of atmospheric composition

Global high-resolution observations of atmospheric composition from satellites can greatly improve our understanding of surface emissions through inverse analyses. Variational inverse methods can optimize surface emissions at any resolution but do not readily quantify the error and information content of the posterior solution. The information content of satellite data may be much lower than its coverage would suggest because of failed retrievals, instrument noise, and error correlations that propagate through the inversion. Analytical solution of the inverse problem provides closed-form characterization of posterior error statistics and information content but requires the construction of the Jacobian matrix that relates emissions to atmospheric concentrations. Building the Jacobian matrix is computationally expensive at high resolution because it involves perturbing each emission element, typically individual grid cells, in the atmospheric transport model used as the forward model for the inversion. We propose and analyze two methods, reduced dimension and reduced rank, to construct the Jacobian matrix at greatly decreased computational cost while retaining information content. Both methods are two-step iterative procedures that begin from an initial native-resolution estimate of the Jacobian matrix constructed at no computational cost by assuming that atmospheric concentrations are most sensitive to local emissions. The reduced-dimension method uses this estimate to construct a Jacobian matrix on a multiscale grid that maintains a high resolution in areas with high information content and aggregates grid cells elsewhere. The reduced-rank method constructs the Jacobian matrix at native resolution by perturbing the leading patterns of information content given by the initial estimate. We demonstrate both methods in an analytical Bayesian inversion of Greenhouse Gases Observing Satellite (GOSAT) methane data with augmented information content over North America in July 2009. We show that both methods reproduce the results of the native-resolution inversion while achieving a factor of 4 improvement in computational performance. The reduced-dimension method produces an exact solution at a lower spatial resolution, while the reduced-rank method solves the inversion at native resolution in areas of high information content and defaults to the prior estimate elsewhere.

Exploring the landscape of CHL strings on Td

Compactifications of the heterotic string on special Td/ℤ2 orbifolds realize a landscape of string models with 16 supercharges and a gauge group on the left-moving sector of reduced rank d + 8. The momenta of untwisted and twisted states span a lattice known as the Mikhailov lattice II(d), which is not self-dual for d > 1. By using computer algorithms which exploit the properties of lattice embeddings, we perform a systematic exploration of the moduli space for d ≤ 2, and give a list of maximally enhanced points where the U(1)d+8 enhances to a rank d + 8 non-Abelian gauge group. For d = 1, these groups are simply-laced and simply-connected, and in fact can be obtained from the Dynkin diagram of E10. For d = 2 there are also symplectic and doubly-connected groups. For the latter we find the precise form of their fundamental groups from embeddings of lattices into the dual of II(2). Our results easily generalize to d > 2.

GH Biplot in Reduced-Rank Regression based on Partial Least Squares

One of the challenges facing statisticians is to provide tools to enable researchers to interpret and present their data and conclusions in ways easily understood by the scientific community. One of the tools available for this purpose is a multivariate graphical representation called reduced rank regression biplot. This biplot describes how to construct a graphical representation in nonsymmetric contexts such as approximations by least squares in multivariate linear regression models of reduced rank. However multicollinearity invalidates the interpretation of a regression coefficient as the conditional effect of a regressor, given the values of the other regressors, and hence makes biplots of regression coefficients useless. So it was, in the search to overcome this problem, Alvarez and Griffin presented a procedure for coefficient estimation in a multivariate regression model of reduced rank in the presence of multicollinearity based on PLS (Partial Least Squares) and generalized singular value decomposition. Based on these same procedures, a biplot construction is now presented for a multivariate regression model of reduced rank in the presence of multicollinearity. This procedure, called PLSSVD GH biplot, provides a useful data analysis tool which allows the visual appraisal of the structure of the dependent and independent variables. This paper defines the procedure and shows several of its properties. It also provides an implementation of the routines in R and presents a real life application involving data from the FAO food database to illustrate the procedure and its properties.

Dietary patterns related to zinc and polyunsaturated fatty acids intake are associated with serum linoleic/dihomo-γ-linolenic ratio in NHANES males and females

Identifying dietary patterns that contribute to zinc (Zn) and fatty acids intake and their biomarkers that may have an impact on health of males and females. The present study was designed to (a) extract dietary patterns with foods that explain the variation of Zn and PUFAs intake in adult men and women; and (b) evaluate the association between the extracted dietary patterns with circulating levels of serum dihomo-γ-linolenic fatty acid (DGLA) or serum linoleic/dihomo-γ-linolenic (LA/DGLA) ratio in males and females. We used reduced rank regression (RRR) to extract the dietary patterns separated by sex in the NHANES 2011–2012 data. A dietary pattern with foods rich in Zn (1st quintile = 8.67 mg/day; 5th quintile = 11.11 mg/day) and poor in PUFAs (5th quintile = 15.28 g/day; 1st quintile = 18.03 g/day) was found in females (S-FDP2) and the same pattern, with foods poor in PUFAs (5th quintile = 17.6 g/day; 1st quintile = 20.7 g/day) and rich in Zn (1st quintile = 10.4 mg/day; 5th quintile = 12.9 mg/day) (S-MDP2), was found in males. The dietary patterns with foods rich in Zn and poor in PUFAs were negatively associated with serum LA/DGLA ratio. This is the first study to associate the LA/DGLA ratio with Zn and PUFAs related dietary patterns in males and females.