scholarly journals Mutual Information, Fisher Information, and Population Coding

1998 ◽  
Vol 10 (7) ◽  
pp. 1731-1757 ◽  
Author(s):  
Nicolas Brunel ◽  
Jean-Pierre Nadal
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Fisher Information ◽  
Population Coding ◽  
Angular Variable ◽  
Large Array ◽  
Direct Link ◽  
Information Theoretic ◽  
Tuning Curves ◽  
Standard Framework

In the context of parameter estimation and model selection, it is only quite recently that a direct link between the Fisher information and information-theoretic quantities has been exhibited. We give an interpretation of this link within the standard framework of information theory. We show that in the context of population coding, the mutual information between the activity of a large array of neurons and a stimulus to which the neurons are tuned is naturally related to the Fisher information. In the light of this result, we consider the optimization of the tuning curves parameters in the case of neurons responding to a stimulus represented by an angular variable.

scholarly journals Optimal Short-Term Population Coding: When Fisher Information Fails

2002 ◽  
Vol 14 (10) ◽  
pp. 2317-2351 ◽  
Author(s):  
M. Bethge ◽  
D. Rotermund ◽  
K. Pawelzik
Keyword(s):  
Fisher Information ◽  
Dynamic Range ◽  
Mean Squared Error ◽  
Average Rate ◽  
Neuronal Response ◽  
Minimum Mean Square Error ◽  
Population Coding ◽  
Neuronal Firing ◽  
Tuning Curves ◽  
Efficient Coding

Efficient coding has been proposed as a first principle explaining neuronal response properties in the central nervous system. The shape of optimal codes, however, strongly depends on the natural limitations of the particular physical system. Here we investigate how optimal neuronal encoding strategies are influenced by the finite number of neurons N (place constraint), the limited decoding time window length T (time constraint), the maximum neuronal firing rate fmax (power constraint), and the maximal average rate fmax (energy constraint). While Fisher information provides a general lower bound for the mean squared error of unbiased signal reconstruction, its use to characterize the coding precision is limited. Analyzing simple examples, we illustrate some typical pitfalls and thereby show that Fisher information provides a valid measure for the precision of a code only if the dynamic range (fmin T, fmax T) is sufficiently large. In particular, we demonstrate that the optimal width of gaussian tuning curves depends on the available decoding time T. Within the broader class of unimodal tuning functions, it turns out that the shape of a Fisher-optimal coding scheme is not unique. We solve this ambiguity by taking the minimum mean square error into account, which leads to flat tuning curves. The tuning width, however, remains to be determined by energy constraints rather than by the principle of efficient coding.

scholarly journals Fisher and Shannon Information in Finite Neural Populations

2012 ◽  
Vol 24 (7) ◽  
pp. 1740-1780 ◽  
Author(s):  
Stuart Yarrow ◽  
Edward Challis ◽  
Peggy Seriès
Keyword(s):  
Mutual Information ◽  
Fisher Information ◽  
Tuning Curve ◽  
Finite Size ◽  
Research Groups ◽  
Information Theoretic ◽  
Statistical Measures ◽  
Neural Populations ◽  
Large Populations ◽  
The Relationship

The precision of the neural code is commonly investigated using two families of statistical measures: Shannon mutual information and derived quantities when investigating very small populations of neurons and Fisher information when studying large populations. These statistical tools are no longer the preserve of theorists and are being applied by experimental research groups in the analysis of empirical data. Although the relationship between information-theoretic and Fisher-based measures in the limit of infinite populations is relatively well understood, how these measures compare in finite-size populations has not yet been systematically explored. We aim to close this gap. We are particularly interested in understanding which stimuli are best encoded by a given neuron within a population and how this depends on the chosen measure. We use a novel Monte Carlo approach to compute a stimulus-specific decomposition of the mutual information (the SSI) for populations of up to 256 neurons and show that Fisher information can be used to accurately estimate both mutual information and SSI for populations of the order of 100 neurons, even in the presence of biologically realistic variability, noise correlations, and experimentally relevant integration times. According to both measures, the stimuli that are best encoded are those falling at the flanks of the neuron's tuning curve. In populations of fewer than around 50 neurons, however, Fisher information can be misleading.

scholarly journals Information-Theoretic Bounds and Approximations in Neural Population Coding

2018 ◽  
Vol 30 (4) ◽  
pp. 885-944 ◽  
Author(s):  
Wentao Huang ◽  
Kechen Zhang
Keyword(s):  
Mutual Information ◽  
Numerical Solutions ◽  
Population Coding ◽  
Neural Population ◽  
Asymptotic Formulas ◽  
High Dimensional ◽  
Information Theoretic ◽  
Special Cases ◽  
Neural Populations ◽  
Approximation Formulas

While Shannon's mutual information has widespread applications in many disciplines, for practical applications it is often difficult to calculate its value accurately for high-dimensional variables because of the curse of dimensionality. This article focuses on effective approximation methods for evaluating mutual information in the context of neural population coding. For large but finite neural populations, we derive several information-theoretic asymptotic bounds and approximation formulas that remain valid in high-dimensional spaces. We prove that optimizing the population density distribution based on these approximation formulas is a convex optimization problem that allows efficient numerical solutions. Numerical simulation results confirmed that our asymptotic formulas were highly accurate for approximating mutual information for large neural populations. In special cases, the approximation formulas are exactly equal to the true mutual information. We also discuss techniques of variable transformation and dimensionality reduction to facilitate computation of the approximations.

scholarly journals Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

Entropy ◽  
2019 ◽  
Vol 21 (3) ◽  
pp. 243
Author(s):  
Wentao Huang ◽  
Kechen Zhang
Keyword(s):  
Mutual Information ◽  
Fisher Information ◽  
Population Coding ◽  
Neural Population ◽  
Asymptotic Formulas ◽  
Continuous Variables ◽  
Discrete Variables ◽  
Leibler Divergence ◽  
Approximation Formulas ◽  
Consistent Performance

Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback-Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.

Mutual Information, Fisher Information, and Efficient Coding

2016 ◽  
Vol 28 (2) ◽  
pp. 305-326 ◽  
Author(s):  
Xue-Xin Wei ◽  
Alan A. Stocker
Keyword(s):  
Mutual Information ◽  
Fisher Information ◽  
Neural Coding ◽  
Population Coding ◽  
Neural Population ◽  
Small Noise ◽  
Efficient Coding ◽  
Coding Efficiency ◽  
Stimulus Variable ◽  
Neural Populations

Fisher information is generally believed to represent a lower bound on mutual information (Brunel & Nadal, 1998 ), a result that is frequently used in the assessment of neural coding efficiency. However, we demonstrate that the relation between these two quantities is more nuanced than previously thought. For example, we find that in the small noise regime, Fisher information actually provides an upper bound on mutual information. Generally our results show that it is more appropriate to consider Fisher information as an approximation rather than a bound on mutual information. We analytically derive the correspondence between the two quantities and the conditions under which the approximation is good. Our results have implications for neural coding theories and the link between neural population coding and psychophysically measurable behavior. Specifically, they allow us to formulate the efficient coding problem of maximizing mutual information between a stimulus variable and the response of a neural population in terms of Fisher information. We derive a signature of efficient coding expressed as the correspondence between the population Fisher information and the distribution of the stimulus variable. The signature is more general than previously proposed solutions that rely on specific assumptions about the neural tuning characteristics. We demonstrate that it can explain measured tuning characteristics of cortical neural populations that do not agree with previous models of efficient coding.

scholarly journals Neural Population Coding and Approximations of Mutual Information for Discrete Variables

2018 ◽  
Author(s):  
Wentao Huang ◽  
Kechen Zhang
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Population Coding ◽  
Neural Population ◽  
Asymptotic Formulas ◽  
Continuous Variables ◽  
Discrete Variables ◽  
Practical Applications ◽  
Leibler Divergence ◽  
Approximation Formulas

Information theory is widely used in various disciplines, and effective calculation of Shannon mutual information is typically not an easy task for many practical applications, including problems of neural population coding in computational and theoretical neuroscience. Asymptotic formulas based on Fisher information may provide accurate approximations to mutual information but this approach is restricted to continuous variables because the calculation requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations based on Kullback-Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding, and these asymptotic formulas hold true for discrete variables as there is no requirement for differentiability. In particular, one of our approximation formulas has consistent performance and good accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating mutual information between the discrete variables or stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.

scholarly journals Limitations to Estimating Mutual Information in Large Neural Populations

Entropy ◽  
2020 ◽  
Vol 22 (4) ◽  
pp. 490
Author(s):  
Jan Mölter ◽  
Geoffrey J. Goodhill
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Neural Activity ◽  
Neural Population ◽  
Population Activity ◽  
Direct Estimate ◽  
Sensory Stimuli ◽  
Information Theoretic ◽  
Finite Samples ◽  
Neural Populations

Information theory provides a powerful framework to analyse the representation of sensory stimuli in neural population activity. However, estimating the quantities involved such as entropy and mutual information from finite samples is notoriously hard and any direct estimate is known to be heavily biased. This is especially true when considering large neural populations. We study a simple model of sensory processing and show through a combinatorial argument that, with high probability, for large neural populations any finite number of samples of neural activity in response to a set of stimuli is mutually distinct. As a consequence, the mutual information when estimated directly from empirical histograms will be equal to the stimulus entropy. Importantly, this is the case irrespective of the precise relation between stimulus and neural activity and corresponds to a maximal bias. This argument is general and applies to any application of information theory, where the state space is large and one relies on empirical histograms. Overall, this work highlights the need for alternative approaches for an information theoretic analysis when dealing with large neural populations.

Quantifying Stimulus Discriminability: A Comparison of Information Theory and Ideal Observer Analysis

2005 ◽  
Vol 17 (4) ◽  
pp. 741-778 ◽  
Author(s):  
Eric E. Thomson ◽  
William B. Kristan
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Conditional Entropy ◽  
Ideal Observer ◽  
Upper And Lower Bounds ◽  
Specific Information ◽  
Information Theoretic ◽  
Information Measures ◽  
Improved Performance ◽  
Ideal Observer Analysis

Performance in sensory discrimination tasks is commonly quantified using either information theory or ideal observer analysis. These two quantitative frameworks are often assumed to be equivalent. For example, higher mutual information is said to correspond to improved performance of an ideal observer in a stimulus estimation task. To the contrary, drawing on and extending previous results, we show that five information-theoretic quantities (entropy, response-conditional entropy, specific information, equivocation, and mutual information) violate this assumption. More positively, we show how these information measures can be used to calculate upper and lower bounds on ideal observer performance, and vice versa. The results show that the mathematical resources of ideal observer analysis are preferable to information theory for evaluating performance in a stimulus discrimination task. We also discuss the applicability of information theory to questions that ideal observer analysis cannot address.

scholarly journals Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

2019 ◽  
Author(s):  
Wentao Huang ◽  
Kechen Zhang
Keyword(s):  
Mutual Information ◽  
Fisher Information ◽  
Population Coding ◽  
Neural Population ◽  
Asymptotic Formulas ◽  
Continuous Variables ◽  
Discrete Variables ◽  
Leibler Divergence ◽  
Approximation Formulas ◽  
Consistent Performance

Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback--Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.

scholarly journals Comparative performance of mutual information and transfer entropy for analyzing the balance of information flow and energy consumption at synapses

2020 ◽  
Author(s):  
Mireille Conrad ◽  
Renaud B Jolivet
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Information Flow ◽  
Experimental Finding ◽  
Brain Regions ◽  
Transfer Entropy ◽  
Information Flows ◽  
Experimental Conditions ◽  
Information Theoretic ◽  
Thalamic Relay

AbstractInformation theory has become an essential tool of modern neuroscience. It can however be difficult to apply in experimental contexts when acquisition of very large datasets is prohibitive. Here, we compare the relative performance of two information theoretic measures, mutual information and transfer entropy, for the analysis of information flow and energetic consumption at synapses. We show that transfer entropy outperforms mutual information in terms of reliability of estimates for small datasets. However, we also show that a detailed understanding of the underlying neuronal biophysics is essential for properly interpreting the results obtained with transfer entropy. We conclude that when time and experimental conditions permit, mutual information might provide an easier to interpret alternative. Finally, we apply both measures to the study of energetic optimality of information flow at thalamic relay synapses in the visual pathway. We show that both measures recapitulate the experimental finding that these synapses are tuned to optimally balance information flowing through them with the energetic consumption associated with that synaptic and neuronal activity. Our results highlight the importance of conducting systematic computational studies prior to applying information theoretic tools to experimental data.Author summaryInformation theory has become an essential tool of modern neuroscience. It is being routinely used to evaluate how much information flows from external stimuli to various brain regions or individual neurons. It is also used to evaluate how information flows between brain regions, between neurons, across synapses, or in neural networks. Information theory offers multiple measures to do that. Two of the most popular are mutual information and transfer entropy. While these measures are related to each other, they differ in one important aspect: transfer entropy reports a directional flow of information, as mutual information does not. Here, we proceed to a systematic evaluation of their respective performances and trade-offs from the perspective of an experimentalist looking to apply these measures to binarized spike trains. We show that transfer entropy might be a better choice than mutual information when time for experimental data collection is limited, as it appears less affected by systematic biases induced by a relative lack of data. Transmission delays and integration properties of the output neuron can however complicate this picture, and we provide an example of the effect this has on both measures. We conclude that when time and experimental conditions permit, mutual information – especially when estimated using a method referred to as the ‘direct’ method – might provide an easier to interpret alternative. Finally, we apply both measures in the biophysical context of evaluating the energetic optimality of information flow at thalamic relay synapses in the visual pathway. We show that both measures capture the original experimental finding that those synapses are tuned to optimally balance information flowing through them with the concomitant energetic consumption associated with that synaptic and neuronal activity.

Sign in / Sign up

Export Citation Format

Share Document