scholarly journals Bayesian3 Active Learning for the Gaussian Process Emulator Using Information Theory

Entropy ◽  
2020 ◽  
Vol 22 (8) ◽  
pp. 890
Author(s):  
Sergey Oladyshkin ◽  
Farid Mohammadi ◽  
Ilja Kroeker ◽  
Wolfgang Nowak
Keyword(s):  
Bayesian Inference ◽  
Active Learning ◽  
Gaussian Process ◽  
Relative Entropy ◽  
Bayesian Model ◽  
Information Gain ◽  
Measurement Data ◽  
Superior Performance ◽  
Evidence Based ◽  
Model Evidence

Gaussian process emulators (GPE) are a machine learning approach that replicates computational demanding models using training runs of that model. Constructing such a surrogate is very challenging and, in the context of Bayesian inference, the training runs should be well invested. The current paper offers a fully Bayesian view on GPEs for Bayesian inference accompanied by Bayesian active learning (BAL). We introduce three BAL strategies that adaptively identify training sets for the GPE using information-theoretic arguments. The first strategy relies on Bayesian model evidence that indicates the GPE’s quality of matching the measurement data, the second strategy is based on relative entropy that indicates the relative information gain for the GPE, and the third is founded on information entropy that indicates the missing information in the GPE. We illustrate the performance of our three strategies using analytical- and carbon-dioxide benchmarks. The paper shows evidence of convergence against a reference solution and demonstrates quantification of post-calibration uncertainty by comparing the introduced three strategies. We conclude that Bayesian model evidence-based and relative entropy-based strategies outperform the entropy-based strategy because the latter can be misleading during the BAL. The relative entropy-based strategy demonstrates superior performance to the Bayesian model evidence-based strategy.

scholarly journals The Connection between Bayesian Inference and Information Theory for Model Selection, Information Gain and Experimental Design

Entropy ◽  
2019 ◽  
Vol 21 (11) ◽  
pp. 1081 ◽  
Author(s):  
Sergey Oladyshkin ◽  
Wolfgang Nowak
Keyword(s):  
Information Theory ◽  
Monte Carlo ◽  
Bayesian Inference ◽  
Experimental Design ◽  
Model Selection ◽  
Relative Entropy ◽  
Information Entropy ◽  
Bayesian Model ◽  
Information Gain ◽  
Information Criteria

We show a link between Bayesian inference and information theory that is useful for model selection, assessment of information entropy and experimental design. We align Bayesian model evidence (BME) with relative entropy and cross entropy in order to simplify computations using prior-based (Monte Carlo) or posterior-based (Markov chain Monte Carlo) BME estimates. On the one hand, we demonstrate how Bayesian model selection can profit from information theory to estimate BME values via posterior-based techniques. Hence, we use various assumptions including relations to several information criteria. On the other hand, we demonstrate how relative entropy can profit from BME to assess information entropy during Bayesian updating and to assess utility in Bayesian experimental design. Specifically, we emphasize that relative entropy can be computed avoiding unnecessary multidimensional integration from both prior and posterior-based sampling techniques. Prior-based computation does not require any assumptions, however posterior-based estimates require at least one assumption. We illustrate the performance of the discussed estimates of BME, information entropy and experiment utility using a transparent, non-linear example. The multivariate Gaussian posterior estimate includes least assumptions and shows the best performance for BME estimation, information entropy and experiment utility from posterior-based sampling.

Bayesian Active Learning of Neural Firing Rate Maps with Transformed Gaussian Process Priors

2014 ◽  
Vol 26 (8) ◽  
pp. 1519-1541 ◽  
Author(s):  
Mijung Park ◽  
J. Patrick Weller ◽  
Gregory D. Horwitz ◽  
Jonathan W. Pillow
Keyword(s):  
Active Learning ◽  
Gaussian Process ◽  
Firing Rate ◽  
Utility Function ◽  
Information Gain ◽  
Tuning Curve ◽  
Strong Dependence ◽  
Learning Performance ◽  
Stimulus Selection ◽  
Neural Firing

A firing rate map, also known as a tuning curve, describes the nonlinear relationship between a neuron's spike rate and a low-dimensional stimulus (e.g., orientation, head direction, contrast, color). Here we investigate Bayesian active learning methods for estimating firing rate maps in closed-loop neurophysiology experiments. These methods can accelerate the characterization of such maps through the intelligent, adaptive selection of stimuli. Specifically, we explore the manner in which the prior and utility function used in Bayesian active learning affect stimulus selection and performance. Our approach relies on a flexible model that involves a nonlinearly transformed gaussian process (GP) prior over maps and conditionally Poisson spiking. We show that infomax learning, which selects stimuli to maximize the information gain about the firing rate map, exhibits strong dependence on the seemingly innocuous choice of nonlinear transformation function. We derive an alternate utility function that selects stimuli to minimize the average posterior variance of the firing rate map and analyze the surprising relationship between prior parameterization, stimulus selection, and active learning performance in GP-Poisson models. We apply these methods to color tuning measurements of neurons in macaque primary visual cortex.

scholarly journals Bayesian inference for the information gain model

2011 ◽  
Vol 43 (2) ◽  
pp. 297-309 ◽  
Author(s):  
Sven Stringer ◽  
Denny Borsboom ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Bayesian Inference ◽  
Information Gain

scholarly journals Adaptive Gaussian Process Approximation for Bayesian Inference with Expensive Likelihood Functions

2018 ◽  
Vol 30 (11) ◽  
pp. 3072-3094 ◽  
Author(s):  
Hongqiao Wang ◽  
Jinglai Li
Keyword(s):  
Bayesian Inference ◽  
Gaussian Process ◽  
Adaptive Algorithm ◽  
Posterior Density ◽  
Unknown Parameters ◽  
Likelihood Functions ◽  
Inference Problems ◽  
Active Learning Method ◽  
Computationally Intensive ◽  
Process Approximation

We consider Bayesian inference problems with computationally intensive likelihood functions. We propose a Gaussian process (GP)–based method to approximate the joint distribution of the unknown parameters and the data, built on recent work (Kandasamy, Schneider, & Póczos, 2015 ). In particular, we write the joint density approximately as a product of an approximate posterior density and an exponentiated GP surrogate. We then provide an adaptive algorithm to construct such an approximation, where an active learning method is used to choose the design points. With numerical examples, we illustrate that the proposed method has competitive performance against existing approaches for Bayesian computation.

scholarly journals Bayesian inference across multiple models suggests a strong increase in lethality of COVID-19 in late 2020 in the UK

PLoS ONE ◽  
2021 ◽  
Vol 16 (11) ◽  
pp. e0258968
Author(s):  
Patrick Pietzonka ◽  
Erik Brorson ◽  
William Bankes ◽  
Michael E. Cates ◽  
Robert L. Jack ◽  
...  
Keyword(s):  
Bayesian Inference ◽  
Time Window ◽  
Fold Increase ◽  
Strong Increase ◽  
Mortality Data ◽  
New Variant ◽  
Model Evidence ◽  
Age Structured ◽  
The Uk ◽  
Inference Methods

We apply Bayesian inference methods to a suite of distinct compartmental models of generalised SEIR type, in which diagnosis and quarantine are included via extra compartments. We investigate the evidence for a change in lethality of COVID-19 in late autumn 2020 in the UK, using age-structured, weekly national aggregate data for cases and mortalities. Models that allow a (step-like or graded) change in infection fatality rate (IFR) have consistently higher model evidence than those without. Moreover, they all infer a close to two-fold increase in IFR. This value lies well above most previously available estimates. However, the same models consistently infer that, most probably, the increase in IFR preceded the time window during which variant B.1.1.7 (alpha) became the dominant strain in the UK. Therefore, according to our models, the caseload and mortality data do not offer unequivocal evidence for higher lethality of a new variant. We compare these results for the UK with similar models for Germany and France, which also show increases in inferred IFR during the same period, despite the even later arrival of new variants in those countries. We argue that while the new variant(s) may be one contributing cause of a large increase in IFR in the UK in autumn 2020, other factors, such as seasonality, or pressure on health services, are likely to also have contributed.

scholarly journals Active Learning in the Geometric Block Model

2020 ◽  
Vol 34 (04) ◽  
pp. 3641-3648 ◽  
Author(s):  
Eli Chien ◽  
Antonia Tulino ◽  
Jaime Llorca
Keyword(s):  
Active Learning ◽  
Community Detection ◽  
Random Graphs ◽  
State Of The Art ◽  
Superior Performance ◽  
Model Parameters ◽  
Block Model ◽  
Stochastic Block Model ◽  
Synthetic Datasets ◽  
Community Detection Problems

The geometric block model is a recently proposed generative model for random graphs that is able to capture the inherent geometric properties of many community detection problems, providing more accurate characterizations of practical community structures compared with the popular stochastic block model. Galhotra et al. recently proposed a motif-counting algorithm for unsupervised community detection in the geometric block model that is proved to be near-optimal. They also characterized the regimes of the model parameters for which the proposed algorithm can achieve exact recovery. In this work, we initiate the study of active learning in the geometric block model. That is, we are interested in the problem of exactly recovering the community structure of random graphs following the geometric block model under arbitrary model parameters, by possibly querying the labels of a limited number of chosen nodes. We propose two active learning algorithms that combine the use of motif-counting with two different label query policies. Our main contribution is to show that sampling the labels of a vanishingly small fraction of nodes (sub-linear in the total number of nodes) is sufficient to achieve exact recovery in the regimes under which the state-of-the-art unsupervised method fails. We validate the superior performance of our algorithms via numerical simulations on both real and synthetic datasets.

scholarly journals Efficient Bayesian inference of fully stochastic epidemiological models with applications to COVID-19

2021 ◽  
Vol 8 (8) ◽  
pp. 211065
Author(s):  
Yuting I. Li ◽  
Günther Turk ◽  
Paul B. Rohrbach ◽  
Patrick Pietzonka ◽  
Julian Kappler ◽  
...  
Keyword(s):  
Bayesian Inference ◽  
Information Matrix ◽  
Monte Carlo Sampling ◽  
Epidemiological Models ◽  
Large Populations ◽  
Model Evidence ◽  
Age Structured ◽  
The Uk ◽  
Functional Central Limit

Epidemiological forecasts are beset by uncertainties about the underlying epidemiological processes, and the surveillance process through which data are acquired. We present a Bayesian inference methodology that quantifies these uncertainties, for epidemics that are modelled by (possibly) non-stationary, continuous-time, Markov population processes. The efficiency of the method derives from a functional central limit theorem approximation of the likelihood, valid for large populations. We demonstrate the methodology by analysing the early stages of the COVID-19 pandemic in the UK, based on age-structured data for the number of deaths. This includes maximum a posteriori estimates, Markov chain Monte Carlo sampling of the posterior, computation of the model evidence, and the determination of parameter sensitivities via the Fisher information matrix. Our methodology is implemented in PyRoss, an open-source platform for analysis of epidemiological compartment models.

scholarly journals A Bayesian Account of Memory for Text

2019 ◽  
Author(s):  
Mark Andrews
Keyword(s):  
Bayesian Inference ◽  
Strong Evidence ◽  
Bayesian Model ◽  
Background Knowledge ◽  
Predictive Inference ◽  
Memory Representation ◽  
British English ◽  
Text Memory ◽  
Memory Experiment ◽  
Memory For Text

The study of memory for texts has had an long tradition of research in psychology. According to most general accounts, the recognition or recall of items in a text is based on querying a memory representation that is built up on the basis of background knowledge. The objective of this paper is to describe and thoroughly test a Bayesian model of these general accounts. In particular, we present a model that describes how we use our background knowledge to form memories in terms of Bayesian inference of statistical patterns in the text, followed by posterior predictive inference of the words that are typical of those inferred patterns. This provides us with precise predictions about which words will be remembered, whether veridically or erroneously, from any given text. We tested these predictions using behavioural data from a memory experiment using a large sample of randomly chosen texts from a representative corpus of British English. The results show that the probability of remembering any given word in the text, whether falsely or veridically, is well predicted by the Bayesian model. Moreover, compared to nontrivial alternative models of text memory, by every measure used in the analyses, the predictions of the Bayesian model were superior, often overwhelmingly so. We conclude that these results provide strong evidence in favour of the Bayesian account of text memory that we have presented in this paper.

Sign in / Sign up

Export Citation Format

Share Document