Information Theory: A Swiss Army Knife for system characterization, learning and prediction

2021 ◽  
Author(s):  
Uwe Ehret
Keyword(s):  
Information Theory ◽  
Information Content ◽  
Earth Science ◽  
Probability Distributions ◽  
Limited Data ◽  
Information Measures ◽  
Dynamical Complexity ◽  
System Characterization ◽  
Combine Information ◽  
Well Models

<p>In this contribution, I will – with examples from hydrology - make the case for information theory as a general language and framework for i) characterizing systems, ii) quantifying the information content in data, iii) evaluating how well models can learn from data, and iv) measuring how well models do in prediction. In particular, I will discuss how information measures can be used to characterize systems by the state space volume they occupy, their dynamical complexity, and their distance from equilibrium. Likewise, I will discuss how we can measure the information content of data through systematic perturbations, and how much information a model absorbs (or ignores) from data during learning. This can help building hybrid models that optimally combine information in data and general knowledge from physical and other laws, which is currently among the key challenges in machine learning applied to earth science problems.</p><p>While I will try my best to convince everybody of taking an information perspective henceforth, I will also name the related challenges: Data demands, binning choices, estimation of probability distributions from limited data, and issues with excessive data dimensionality.</p>

Information Theory: Some Concepts and Measures

1977 ◽  
Vol 9 (4) ◽  
pp. 395-417 ◽  
Author(s):  
J A Walsh ◽  
M J Webber
Keyword(s):  
Information Theory ◽  
Spatial Analysis ◽  
Sample Size ◽  
Information Content ◽  
Spatial Distributions ◽  
Information Measures ◽  
Entropy Information ◽  
Spatial Problems

The concepts of entropy and of information are increasingly used in spatial analysis. This paper analyses these ideas in order to show how measures of spatial distributions may be constructed from them. First, the information content of messages is examined and related to the notion of uncertainty. Then three information measures, due to Shannon, Brillouin, and Good, are derived and shown to be appropriate in analysing different spatial problems; in particular, the Shannon and Brillouin measures are extensively compared and the effects of sample size on them are investigated. The paper also develops appropriate multivariate analogues of the information measures. Finally, some comments are made on the relations between the concepts of entropy, information, and order.

scholarly journals Information dynamics: temporal behavior of uncertainty measures

Open Physics ◽  
2008 ◽  
Vol 6 (1) ◽  
Author(s):  
Piotr Garbaczewski
Keyword(s):  
Information Theory ◽  
Systematic Study ◽  
Probability Distributions ◽  
Shannon Information ◽  
Temporal Behavior ◽  
Dynamical Processes ◽  
Information Dynamics ◽  
Information Measures ◽  
Uncertainty Measures ◽  
Probability Densities

AbstractWe carry out a systematic study of uncertainty measures that are generic to dynamical processes of varied origins, provided they induce suitable continuous probability distributions. The major technical tools are the information theory methods and inequalities satisfied by Fisher and Shannon information measures. We focus on the compatibility of these inequalities with the prescribed (deterministic, random or quantum) temporal behavior of pertinent probability densities.

STATISTICAL MEASURES AS MEASURES OF DIVERSITY

2010 ◽  
Vol 03 (02) ◽  
pp. 173-185 ◽  
Author(s):  
OM PARKASH ◽  
A. K. THUKRAL
Keyword(s):  
Information Theory ◽  
Data Analysis ◽  
Information Content ◽  
Biological Data ◽  
Central Tendency ◽  
Information Measures ◽  
Interdisciplinary Field ◽  
Statistical Measures ◽  
Data Statistics

Two fields of research have found tremendous applicability in the analysis of biological data-statistics and information theory. Statistics is extensively used for the measurement of central tendency, dispersion, comparison and covariation. Measures of information are used to study diversity and equitability. These two fields have been used independent of each other for data analysis. In this communication, we develop the link between the two and prove that statistical measures can be used as information measures. Our study will be a new interdisciplinary field of research and it will be possible to describe information content of a system from its statistics.

scholarly journals New Inequalities between Information Measures of Network Information Content

2013 ◽  
Vol 2013 ◽  
pp. 1-3 ◽  
Author(s):  
Pantelimon-George Popescu ◽  
Florin Pop ◽  
Alexandru Herişanu ◽  
Nicolae Ţăpuş
Keyword(s):  
Information Processing ◽  
Complex Systems ◽  
Complex Networks ◽  
Information Content ◽  
Discrete Case ◽  
Network Information ◽  
Information Inequalities ◽  
Information Measures ◽  
New Inequalities

We refine a classical logarithmic inequality using a discrete case of Bernoulli inequality, and then we refine furthermore two information inequalities between information measures for graphs, based on information functionals, presented by Dehmer and Mowshowitz in (2010) as Theorems 4.7 and 4.8. The inequalities refer to entropy-based measures of network information content and have a great impact for information processing in complex networks (a subarea of research in modeling of complex systems).

scholarly journals What’s Worth Talking About? Information Theory Reveals How Children Balance Informativeness and Ease of Production

2017 ◽  
Vol 28 (7) ◽  
pp. 954-966 ◽  
Author(s):  
Colin Bannard ◽  
Marla Rosner ◽  
Danielle Matthews
Keyword(s):  
Information Theory ◽  
Information Content ◽  
Low Frequency ◽  
Initial Experiment ◽  
Information Theoretic ◽  
Information Theoretic Measures

Of all the things a person could say in a given situation, what determines what is worth saying? Greenfield’s principle of informativeness states that right from the onset of language, humans selectively comment on whatever they find unexpected. In this article, we quantify this tendency using information-theoretic measures and report on a study in which we tested the counterintuitive prediction that children will produce words that have a low frequency given the context, because these will be most informative. Using corpora of child-directed speech, we identified adjectives that varied in how informative (i.e., unexpected) they were given the noun they modified. In an initial experiment ( N = 31) and in a replication ( N = 13), 3-year-olds heard an experimenter use these adjectives to describe pictures. The children’s task was then to describe the pictures to another person. As the information content of the experimenter’s adjective increased, so did children’s tendency to comment on the feature that adjective had encoded. Furthermore, our analyses suggest that children balance informativeness with a competing drive to ease production.

scholarly journals Quantifying Legal Entropy

2021 ◽  
Vol 9 ◽  
Author(s):  
Ted Sichelman
Keyword(s):  
Information Theory ◽  
Information Content ◽  
Legal System ◽  
Legal Systems ◽  
Information Costs ◽  
Precise Quantification ◽  
The Law ◽  
Legal Contexts ◽  
Definition Of ◽  
Quantitative Definition

Many scholars have employed the term “entropy” in the context of law and legal systems to roughly refer to the amount of “uncertainty” present in a given law, doctrine, or legal system. Just a few of these scholars have attempted to formulate a quantitative definition of legal entropy, and none have provided a precise formula usable across a variety of legal contexts. Here, relying upon Claude Shannon's definition of entropy in the context of information theory, I provide a quantitative formalization of entropy in delineating, interpreting, and applying the law. In addition to offering a precise quantification of uncertainty and the information content of the law, the approach offered here provides other benefits. For example, it offers a more comprehensive account of the uses and limits of “modularity” in the law—namely, using the terminology of Henry Smith, the use of legal “boundaries” (be they spatial or intangible) that “economize on information costs” by “hiding” classes of information “behind” those boundaries. In general, much of the “work” performed by the legal system is to reduce legal entropy by delineating, interpreting, and applying the law, a process that can in principle be quantified.

Introduction to Information Theory

2014 ◽  
Author(s):  
M. Vidyasagar
Keyword(s):  
Information Theory ◽  
Probability Distribution ◽  
Relative Entropy ◽  
Probability Distributions ◽  
Random Variables ◽  
Conditional Entropy ◽  
Random Variable ◽  
Entropy Function ◽  
Concave Functions ◽  
Leibler Divergence

This chapter provides an introduction to some elementary aspects of information theory, including entropy in its various forms. Entropy refers to the level of uncertainty associated with a random variable (or more precisely, the probability distribution of the random variable). When there are two or more random variables, it is worthwhile to study the conditional entropy of one random variable with respect to another. The last concept is relative entropy, also known as the Kullback–Leibler divergence, which measures the “disparity” between two probability distributions. The chapter first considers convex and concave functions before discussing the properties of the entropy function, conditional entropy, uniqueness of the entropy function, and the Kullback–Leibler divergence.

Information Measures

2015 ◽  
pp. 129-142
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Order Logic ◽  
Unified Framework ◽  
Higher Order Logic ◽  
Absolutely Continuous ◽  
Information Measures ◽  
Finite Spaces ◽  
Nikodym Derivative ◽  
Lebesgue Integration

This chapter presents a higher-order-logic formalization of the main concepts of information theory (Cover & Thomas, 1991), such as the Shannon entropy and mutual information, using the formalization of the foundational theories of measure, Lebesgue integration, and probability. The main results of the chapter include the formalizations of the Radon-Nikodym derivative and the Kullback-Leibler (KL) divergence (Coble, 2010). The latter provides a unified framework based on which most of the commonly used measures of information can be defined. The chapter then provides the general definitions that are valid for both discrete and continuous cases and then proves the corresponding reduced expressions where the measures considered are absolutely continuous over finite spaces.

Sign in / Sign up

Export Citation Format

Share Document