Local Synaptic Learning Rules Suffice to Maximize Mutual Information in a Linear Network

1992 ◽  
Vol 4 (5) ◽  
pp. 691-702 ◽  
Author(s):  
Ralph Linsker
Keyword(s):  
Mutual Information ◽  
Input Signal ◽  
Learning Rule ◽  
Linear Network ◽  
Information Theoretic ◽  
Network Function ◽  
Learning Rules ◽  
Lateral Connection ◽  
Stochastic Gradient Ascent ◽  
Synaptic Learning

A network that develops to maximize the mutual information between its output and the signal portion of its input (which is admixed with noise) is useful for extracting salient input features, and may provide a model for aspects of biological neural network function. I describe a local synaptic Learning rule that performs stochastic gradient ascent in this information-theoretic quantity, for the case in which the input-output mapping is linear and the input signal and noise are multivariate gaussian. Feedforward connection strengths are modified by a Hebbian rule during a "learning" phase in which examples of input signal plus noise are presented to the network, and by an anti-Hebbian rule during an "unlearning" phase in which examples of noise alone are presented. Each recurrent lateral connection has two values of connection strength, one for each phase; these values are updated by an anti-Hebbian rule.

On an Unsupervised Learning Rule for Scalar Quantization following the Maximum Entropy Principle

1993 ◽  
Vol 5 (6) ◽  
pp. 939-953 ◽  
Author(s):  
Marc M. Van Hulle ◽  
Dominique Martinez
Keyword(s):  
Unsupervised Learning ◽  
Maximum Entropy Principle ◽  
Learning Rule ◽  
Probability Density Functions ◽  
Analog To Digital ◽  
Information Theoretic ◽  
Scalar Quantization ◽  
Learning Rules ◽  
Average Mean Square Error ◽  
Entropy Principle

A novel unsupervised learning rule, called Boundary Adaptation Rule (BAR), is introduced for scalar quantization. It is shown that the rule maximizes information-theoretic entropy and thus yields equiprobable quantizations of univariate probability density functions. It is shown by simulations that BAR outperforms other unsupervised competitive learning rules in generating equiprobable quantizations. It is also shown that our rule can do better or worse than the Lloyd I algorithm in minimizing average mean square error, depending on the input distribution. Finally, an application to adaptive non-uniform analog to digital (A/D) conversion is considered.

How to Generate Ordered Maps by Maximizing the Mutual Information between Input and Output Signals

1989 ◽  
Vol 1 (3) ◽  
pp. 402-411 ◽  
Author(s):  
Ralph Linsker
Keyword(s):  
Mutual Information ◽  
Learning Rule ◽  
Topographic Map ◽  
Lateral Interactions ◽  
Magnification Factor ◽  
Average Mutual Information ◽  
Information Theoretic ◽  
Input And Output ◽  
Analytic Expression ◽  
Local Map

A learning rule that performs gradient ascent in the average mutual information between input and an output signal is derived for a system having feedforward and lateral interactions. Several processes emerge as components of this learning rule: Hebb-like modification, and cooperation and competition among processing nodes. Topographic map formation is demonstrated using the learning rule. An analytic expression relating the average mutual information to the response properties of nodes and their geometric arrangement is derived in certain cases. This yields a relation between the local map magnification factor and the probability distribution in the input space. The results provide new links between unsupervised learning and information-theoretic optimization in a system whose properties are biologically motivated.

Recurrent Information Optimization with Local, Metaplastic Synaptic Dynamics

2017 ◽  
Vol 29 (9) ◽  
pp. 2528-2552 ◽  
Author(s):  
Sensen Liu ◽  
ShiNung Ching
Keyword(s):  
Learning Rule ◽  
Recurrent Networks ◽  
Local Policy ◽  
Optimal Learning ◽  
Information Theoretic ◽  
Pairwise Interactions ◽  
Synaptic Dynamics ◽  
Information Optimization ◽  
Dependent Tasks ◽  
Synaptic Learning

We consider the problem of optimizing information-theoretic quantities in recurrent networks via synaptic learning. In contrast to feedforward networks, the recurrence presents a key challenge insofar as an optimal learning rule must aggregate the joint distribution of the whole network. This challenge, in particular, makes a local policy (i.e., one that depends on only pairwise interactions) difficult. Here, we report a local metaplastic learning rule that performs approximate optimization by estimating whole-network statistics through the use of several slow, nested dynamical variables. These dynamics provide the rule with both anti-Hebbian and Hebbian components, thus allowing for decorrelating and correlating learning regimes that can occur when either is favorable for optimality. We demonstrate the performance of the synthesized rule in comparison to classical BCM dynamics and use the networks to conduct history-dependent tasks that highlight the advantages of recurrence. Finally, we show the consistency of the resultant learned networks with notions of criticality, including balanced ratios of excitation and inhibition.

scholarly journals Do galactic bars depend on environment?: an information theoretic analysis of Galaxy Zoo 2

2020 ◽  
Vol 501 (1) ◽  
pp. 994-1001
Author(s):  
Suman Sarkar ◽  
Biswajit Pandey ◽  
Snehasish Bhattacharjee
Keyword(s):  
Spatial Distribution ◽  
Mutual Information ◽  
Local Density ◽  
Statistical Significance ◽  
Distribution Functions ◽  
Cumulative Distribution ◽  
Host Galaxy ◽  
Data Sets ◽  
Data Set ◽  
Information Theoretic

ABSTRACT We use an information theoretic framework to analyse data from the Galaxy Zoo 2 project and study if there are any statistically significant correlations between the presence of bars in spiral galaxies and their environment. We measure the mutual information between the barredness of galaxies and their environments in a volume limited sample (Mr ≤ −21) and compare it with the same in data sets where (i) the bar/unbar classifications are randomized and (ii) the spatial distribution of galaxies are shuffled on different length scales. We assess the statistical significance of the differences in the mutual information using a t-test and find that both randomization of morphological classifications and shuffling of spatial distribution do not alter the mutual information in a statistically significant way. The non-zero mutual information between the barredness and environment arises due to the finite and discrete nature of the data set that can be entirely explained by mock Poisson distributions. We also separately compare the cumulative distribution functions of the barred and unbarred galaxies as a function of their local density. Using a Kolmogorov–Smirnov test, we find that the null hypothesis cannot be rejected even at $75{{\ \rm per\ cent}}$ confidence level. Our analysis indicates that environments do not play a significant role in the formation of a bar, which is largely determined by the internal processes of the host galaxy.

scholarly journals Sequential neuromodulation of Hebbian plasticity offers mechanism for effective reward-based navigation

eLife ◽  
2017 ◽  
Vol 6 ◽  
Author(s):  
Zuzanna Brzosko ◽  
Sara Zannone ◽  
Wolfram Schultz ◽  
Claudia Clopath ◽  
Ole Paulsen
Keyword(s):  
Time Window ◽  
Hippocampal Slices ◽  
Learning Rule ◽  
Spike Timing ◽  
Synaptic Potentiation ◽  
Behavioral Learning ◽  
Hebbian Plasticity ◽  
Subsequent Application ◽  
Behavioral States ◽  
Synaptic Learning

Spike timing-dependent plasticity (STDP) is under neuromodulatory control, which is correlated with distinct behavioral states. Previously, we reported that dopamine, a reward signal, broadens the time window for synaptic potentiation and modulates the outcome of hippocampal STDP even when applied after the plasticity induction protocol (Brzosko et al., 2015). Here, we demonstrate that sequential neuromodulation of STDP by acetylcholine and dopamine offers an efficacious model of reward-based navigation. Specifically, our experimental data in mouse hippocampal slices show that acetylcholine biases STDP toward synaptic depression, whilst subsequent application of dopamine converts this depression into potentiation. Incorporating this bidirectional neuromodulation-enabled correlational synaptic learning rule into a computational model yields effective navigation toward changing reward locations, as in natural foraging behavior. Thus, temporally sequenced neuromodulation of STDP enables associations to be made between actions and outcomes and also provides a possible mechanism for aligning the time scales of cellular and behavioral learning.

scholarly journals Bulk private curves require large conditional mutual information

2021 ◽  
Vol 2021 (9) ◽  
Author(s):  
Alex May
Keyword(s):  
Mutual Information ◽  
Boundary Region ◽  
Theoretic Approach ◽  
Strong Correlations ◽  
Conditional Mutual Information ◽  
Information Theoretic ◽  
Causal Curve ◽  
Resource Requirements ◽  
Theoretic Argument ◽  
Information Theoretic Approach

Abstract We prove a theorem showing that the existence of “private” curves in the bulk of AdS implies two regions of the dual CFT share strong correlations. A private curve is a causal curve which avoids the entanglement wedge of a specified boundary region $$ \mathcal{U} $$ U . The implied correlation is measured by the conditional mutual information $$ I\left({\mathcal{V}}_1:\left.{\mathcal{V}}_2\right|\mathcal{U}\right) $$ I V 1 : V 2 U , which is O(1/GN) when a private causal curve exists. The regions $$ {\mathcal{V}}_1 $$ V 1 and $$ {\mathcal{V}}_2 $$ V 2 are specified by the endpoints of the causal curve and the placement of the region $$ \mathcal{U} $$ U . This gives a causal perspective on the conditional mutual information in AdS/CFT, analogous to the causal perspective on the mutual information given by earlier work on the connected wedge theorem. We give an information theoretic argument for our theorem, along with a bulk geometric proof. In the geometric perspective, the theorem follows from the maximin formula and entanglement wedge nesting. In the information theoretic approach, the theorem follows from resource requirements for sending private messages over a public quantum channel.

scholarly journals Unsupervised Learning via Total Correlation Explanation

2017 ◽  
Author(s):  
Greg Ver Steeg
Keyword(s):  
Mutual Information ◽  
Unsupervised Learning ◽  
Supervised Learning ◽  
Human Behavior ◽  
Learning Problems ◽  
Information Theoretic ◽  
Sensory Environment ◽  
Total Correlation ◽  
Environment Dependence ◽  
Multivariate Mutual Information

Learning by children and animals occurs effortlessly and largely without obvious supervision. Successes in automating supervised learning have not translated to the more ambiguous realm of unsupervised learning where goals and labels are not provided. Barlow (1961) suggested that the signal that brains leverage for unsupervised learning is dependence, or redundancy, in the sensory environment. Dependence can be characterized using the information-theoretic multivariate mutual information measure called total correlation. The principle of Total Cor-relation Ex-planation (CorEx) is to learn representations of data that "explain" as much dependence in the data as possible. We review some manifestations of this principle along with successes in unsupervised learning problems across diverse domains including human behavior, biology, and language.

scholarly journals An Information-Theoretic Account of Semantic Interference in Word Production

2021 ◽  
Vol 12 ◽  
Author(s):  
Richard Futrell
Keyword(s):  
Mutual Information ◽  
Semantic Similarity ◽  
Rate Distortion ◽  
Word Production ◽  
Interference Effects ◽  
Semantic Interference ◽  
Information Theoretic ◽  
Human Data ◽  
Level Model ◽  
Computational Level

I present a computational-level model of semantic interference effects in online word production within a rate–distortion framework. I consider a bounded-rational agent trying to produce words. The agent's action policy is determined by maximizing accuracy in production subject to computational constraints. These computational constraints are formalized using mutual information. I show that semantic similarity-based interference among words falls out naturally from this setup, and I present a series of simulations showing that the model captures some of the key empirical patterns observed in Stroop and Picture–Word Interference paradigms, including comparisons to human data from previous experiments.

Sign in / Sign up

Export Citation Format

Share Document