scholarly journals INFORMATION BOTTLENECKS, CAUSAL STATES, AND STATISTICAL RELEVANCE BASES: HOW TO REPRESENT RELEVANT INFORMATION IN MEMORYLESS TRANSDUCTION

2002 ◽  
Vol 05 (01) ◽  
pp. 91-95 ◽  
Author(s):  
COSMA ROHILLA SHALIZI ◽  
JAMES P. CRUTCHFIELD
Keyword(s):  
Computational Mechanics ◽  
Hidden Variables ◽  
Relevant Information ◽  
Natural World ◽  
Information Bottleneck ◽  
Statistical Relevance ◽  
Causal States

Discovering relevant, but possibly hidden, variables is a key step in constructing useful and predictive theories about the natural world. This brief note explains the connections between three approaches to this problem: the recently introduced information-bottleneck method, the computational mechanics approach to inferring optimal models, and Salmon's statistical relevance basis.

scholarly journals Information Bottleneck for Estimating Treatment Effects with Systematically Missing Covariates

Entropy ◽  
2020 ◽  
Vol 22 (4) ◽  
pp. 389
Author(s):  
Sonali Parbhoo ◽  
Mario Wieser ◽  
Aleksander Wieczorek ◽  
Volker Roth
Keyword(s):  
Treatment Effects ◽  
State Of The Art ◽  
Relevant Information ◽  
High Dimensional ◽  
Second Step ◽  
Test Time ◽  
Healthcare Applications ◽  
Information Bottleneck ◽  
Covariate Information ◽  
Low Dimensional

Estimating the effects of an intervention from high-dimensional observational data is a challenging problem due to the existence of confounding. The task is often further complicated in healthcare applications where a set of observations may be entirely missing for certain patients at test time, thereby prohibiting accurate inference. In this paper, we address this issue using an approach based on the information bottleneck to reason about the effects of interventions. To this end, we first train an information bottleneck to perform a low-dimensional compression of covariates by explicitly considering the relevance of information for treatment effects. As a second step, we subsequently use the compressed covariates to perform a transfer of relevant information to cases where data are missing during testing. In doing so, we can reliably and accurately estimate treatment effects even in the absence of a full set of covariate information at test time. Our results on two causal inference benchmarks and a real application for treating sepsis show that our method achieves state-of-the-art performance, without compromising interpretability.

scholarly journals Genetic Algorithms to Maximize the Relevant Mutual Information in Communication Receivers

Electronics ◽  
2021 ◽  
Vol 10 (12) ◽  
pp. 1434
Author(s):  
Jan Lewandowsky ◽  
Sumedh Jitendra Dongare ◽  
Rocío Martín Lima ◽  
Marc Adrat ◽  
Matthias Schrammen ◽  
...  
Keyword(s):  
Genetic Algorithms ◽  
Mutual Information ◽  
Relevant Information ◽  
Channel Decoding ◽  
Channel Output ◽  
Apply Genetic ◽  
Information Bottleneck ◽  
Parametrized Systems ◽  
Receiver Algorithms ◽  
Coarse Quantization

The preservation of relevant mutual information under compression is the fundamental challenge of the information bottleneck method. It has many applications in machine learning and in communications. The recent literature describes successful applications of this concept in quantized detection and channel decoding schemes. The focal idea is to build receiver algorithms intended to preserve the maximum possible amount of relevant information, despite very coarse quantization. The existent literature shows that the resulting quantized receiver algorithms can achieve performance very close to that of conventional high-precision systems. Moreover, all demanding signal processing operations get replaced with lookup operations in the considered system design. In this paper, we develop the idea of maximizing the preserved relevant information in communication receivers further by considering parametrized systems. Such systems can help overcome the need of lookup tables in cases where their huge sizes make them impractical. We propose to apply genetic algorithms which are inspired from the natural evolution of the species for the problem of parameter optimization. We exemplarily investigate receiver-sided channel output quantization and demodulation to illustrate the notable performance and the flexibility of the proposed concept.

Evaluation of sewer infiltration/inflow using COD mass flux method: case study in Prague

2012 ◽  
Vol 66 (3) ◽  
pp. 673-680 ◽  
Author(s):  
V. Bareš ◽  
D. Stránský ◽  
P. Sýkora
Keyword(s):  
Oxygen Demand ◽  
Absorption Spectrometry ◽  
Water Quality Monitoring ◽  
Relevant Information ◽  
Data Series ◽  
Sewer System ◽  
Statistical Relevance ◽  
The City

The combined sewer system of the City of Prague, similar to other relevant locations, is strongly affected by infiltrating waters. The knowledge of their volume is one of the important parameters with respect to sewer system operation, maintenance and reconstruction. The methodology is based on the variation of diurnal chemical oxygen demand (COD) load and continuous water quality monitoring using in-line absorption spectrometry. This approach allows the identification of individual components of the diurnal wastewater hydrograph, in particular the contribution of parasitic water flow to wastewater discharge. The statistical relevance was tested using Monte Carlo simulations on a 7-year data series of flow rate. The results show that application of this methodology provides specific relevant information about individual sub-catchments within an entire sewer system, particularly in terms of absolute and relative values of I/I and structural state indicators. Processing of long-term data series gives clear information about the significance of the monitoring period length for the relevance of obtained results.

scholarly journals Markov Information Bottleneck to Improve Information Flow in Stochastic Neural Networks

Entropy ◽  
2019 ◽  
Vol 21 (10) ◽  
pp. 976 ◽  
Author(s):  
Thanh Tang Nguyen ◽  
Jaesik Choi
Keyword(s):  
Neural Networks ◽  
Information Flow ◽  
Latent Variables ◽  
Rate Distortion ◽  
Relevant Information ◽  
Variational Approximation ◽  
Stochastic Neural Networks ◽  
Rate Distortion Theory ◽  
Information Bottleneck ◽  
Hidden Layer

While rate distortion theory compresses data under a distortion constraint, information bottleneck (IB) generalizes rate distortion theory to learning problems by replacing a distortion constraint with a constraint of relevant information. In this work, we further extend IB to multiple Markov bottlenecks (i.e., latent variables that form a Markov chain), namely Markov information bottleneck (MIB), which particularly fits better in the context of stochastic neural networks (SNNs) than the original IB. We show that Markov bottlenecks cannot simultaneously achieve their information optimality in a non-collapse MIB, and thus devise an optimality compromise. With MIB, we take the novel perspective that each layer of an SNN is a bottleneck whose learning goal is to encode relevant information in a compressed form from the data. The inference from a hidden layer to the output layer is then interpreted as a variational approximation to the layer’s decoding of relevant information in the MIB. As a consequence of this perspective, the maximum likelihood estimate (MLE) principle in the context of SNNs becomes a special case of the variational MIB. We show that, compared to MLE, the variational MIB can encourage better information flow in SNNs in both principle and practice, and empirically improve performance in classification, adversarial robustness, and multi-modal learning in MNIST.

A Spiking Neuron as Information Bottleneck

2010 ◽  
Vol 22 (8) ◽  
pp. 1961-1992 ◽  
Author(s):  
Lars Buesing ◽  
Wolfgang Maass
Keyword(s):  
Spike Train ◽  
Contextual Information ◽  
Learning Rule ◽  
Relevant Information ◽  
Spiking Neurons ◽  
Single Output ◽  
Information Bottleneck ◽  
Plausible Model ◽  
Output Spike ◽  
Input Spike

Neurons receive thousands of presynaptic input spike trains while emitting a single output spike train. This drastic dimensionality reduction suggests considering a neuron as a bottleneck for information transmission. Extending recent results, we propose a simple learning rule for the weights of spiking neurons derived from the information bottleneck (IB) framework that minimizes the loss of relevant information transmitted in the output spike train. In the IB framework, relevance of information is defined with respect to contextual information, the latter entering the proposed learning rule as a “third” factor besides pre- and postsynaptic activities. This renders the theoretically motivated learning rule a plausible model for experimentally observed synaptic plasticity phenomena involving three factors. Furthermore, we show that the proposed IB learning rule allows spiking neurons to learn a predictive code, that is, to extract those parts of their input that are predictive for future input.

What sentimentalists should say about emotion

2019 ◽  
Vol 42 ◽  
Author(s):  
Charlie Kurth
Keyword(s):  
Moral Judgment ◽  
Recent Work ◽  
Relevant Information ◽  
Moral Cognition ◽  
Multilevel Structure

Abstract Recent work by emotion researchers indicates that emotions have a multilevel structure. Sophisticated sentimentalists should take note of this work – for it better enables them to defend a substantive role for emotion in moral cognition. Contra May's rationalist criticisms, emotions are not only able to carry morally relevant information, but can also substantially influence moral judgment and reasoning.

Automatic analysis of Kikuchi diffraction patterns

1994 ◽  
Vol 52 ◽  
pp. 900-901
Author(s):  
H. Weiland ◽  
D. P. Field
Keyword(s):  
Electron Beam ◽  
Electron Microscope ◽  
Spatial Resolution ◽  
Relevant Information ◽  
Crystalline Materials ◽  
Large Size ◽  
Transmission Electron ◽  
New Type ◽  
Diffraction Patterns ◽  
Orientation Determination

Recent advances in the automatic indexing of backscatter Kikuchi diffraction patterns on the scanning electron microscope (SEM) has resulted in the development of a new type of microscopy. The ability to obtain statistically relevant information on the spatial distribution of crystallite orientations is giving rise to new insight into polycrystalline microstructures and their relation to materials properties. A limitation of the technique in the SEM is that the spatial resolution of the measurement is restricted by the relatively large size of the electron beam in relation to various microstructural features. Typically the spatial resolution in the SEM is limited to about half a micron or greater. Heavily worked structures exhibit microstructural features much finer than this and require resolution on the order of nanometers for accurate characterization. Transmission electron microscope (TEM) techniques offer sufficient resolution to investigate heavily worked crystalline materials.Crystal lattice orientation determination from Kikuchi diffraction patterns in the TEM (Figure 1) requires knowledge of the relative positions of at least three non-parallel Kikuchi line pairs in relation to the crystallite and the electron beam.

Slow Wave Activity Related to Working Memory Maintenance in the N-Back Task

2016 ◽  
Vol 30 (4) ◽  
pp. 141-154 ◽  
Author(s):  
Kira Bailey ◽  
Gregory Mlynarczyk ◽  
Robert West
Keyword(s):  
Working Memory ◽  
Slow Wave ◽  
Time Course ◽  
Relevant Information ◽  
Wave Activity ◽  
Slow Wave Activity ◽  
Response Accuracy ◽  
Functional Neuroanatomy ◽  
Stimulus Interval ◽  
Back Task

Abstract. Working memory supports our ability to maintain goal-relevant information that guides cognition in the face of distraction or competing tasks. The N-back task has been widely used in cognitive neuroscience to examine the functional neuroanatomy of working memory. Fewer studies have capitalized on the temporal resolution of event-related brain potentials (ERPs) to examine the time course of neural activity in the N-back task. The primary goal of the current study was to characterize slow wave activity observed in the response-to-stimulus interval in the N-back task that may be related to maintenance of information between trials in the task. In three experiments, we examined the effects of N-back load, interference, and response accuracy on the amplitude of the P3b following stimulus onset and slow wave activity elicited in the response-to-stimulus interval. Consistent with previous research, the amplitude of the P3b decreased as N-back load increased. Slow wave activity over the frontal and posterior regions of the scalp was sensitive to N-back load and was insensitive to interference or response accuracy. Together these findings lead to the suggestion that slow wave activity observed in the response-to-stimulus interval is related to the maintenance of information between trials in the 1-back task.

Spatial Representations as an Emergent Feature of Perceptual Processing

2010 ◽  
Vol 24 (3) ◽  
pp. 161-172 ◽  
Author(s):  
Edmund Wascher ◽  
C. Beste
Keyword(s):  
Common Ground ◽  
Event Related Potentials ◽  
Vertical Axis ◽  
Relevant Information ◽  
Sources Of Information ◽  
Integrated Network ◽  
Perceptual Maps ◽  
Emergent Feature ◽  
Related Potentials ◽  
Selection Of

Spatial selection of relevant information has been proposed to reflect an emergent feature of stimulus processing within an integrated network of perceptual areas. Stimulus-based and intention-based sources of information might converge in a common stage when spatial maps are generated. This approach appears to be inconsistent with the assumption of distinct mechanisms for stimulus-driven and top-down controlled attention. In two experiments, the common ground of stimulus-driven and intention-based attention was tested by means of event-related potentials (ERPs) in the human EEG. In both experiments, the processing of a single transient was compared to the selection of a physically comparable stimulus among distractors. While single transients evoked a spatially sensitive N1, the extraction of relevant information out of a more complex display was reflected in an N2pc. The high similarity of the spatial portion of these two components (Experiment 1), and the replication of this finding for the vertical axis (Experiment 2) indicate that these two ERP components might both reflect the spatial representation of relevant information as derived from the organization of perceptual maps, just at different points in time.

Power-Enhanced Funnel Plots for Meta-Analysis

2020 ◽  
Vol 228 (1) ◽  
pp. 43-49 ◽  
Author(s):  
Michael Kossmeier ◽  
Ulrich S. Tran ◽  
Martin Voracek
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Relevant Information ◽  
Selective Reporting ◽  
Graphical Display ◽  
Funnel Plot ◽  
Evidential Value ◽  
Graphical Displays ◽  
Funnel Plots ◽  
Meta Analyses

Abstract. Currently, dedicated graphical displays to depict study-level statistical power in the context of meta-analysis are unavailable. Here, we introduce the sunset (power-enhanced) funnel plot to visualize this relevant information for assessing the credibility, or evidential value, of a set of studies. The sunset funnel plot highlights the statistical power of primary studies to detect an underlying true effect of interest in the well-known funnel display with color-coded power regions and a second power axis. This graphical display allows meta-analysts to incorporate power considerations into classic funnel plot assessments of small-study effects. Nominally significant, but low-powered, studies might be seen as less credible and as more likely being affected by selective reporting. We exemplify the application of the sunset funnel plot with two published meta-analyses from medicine and psychology. Software to create this variation of the funnel plot is provided via a tailored R function. In conclusion, the sunset (power-enhanced) funnel plot is a novel and useful graphical display to critically examine and to present study-level power in the context of meta-analysis.

Sign in / Sign up

Export Citation Format

Share Document