Information Theory: Some Concepts and Measures

1977 ◽  
Vol 9 (4) ◽  
pp. 395-417 ◽  
Author(s):  
J A Walsh ◽  
M J Webber
Keyword(s):  
Information Theory ◽  
Spatial Analysis ◽  
Sample Size ◽  
Information Content ◽  
Spatial Distributions ◽  
Information Measures ◽  
Entropy Information ◽  
Spatial Problems

The concepts of entropy and of information are increasingly used in spatial analysis. This paper analyses these ideas in order to show how measures of spatial distributions may be constructed from them. First, the information content of messages is examined and related to the notion of uncertainty. Then three information measures, due to Shannon, Brillouin, and Good, are derived and shown to be appropriate in analysing different spatial problems; in particular, the Shannon and Brillouin measures are extensively compared and the effects of sample size on them are investigated. The paper also develops appropriate multivariate analogues of the information measures. Finally, some comments are made on the relations between the concepts of entropy, information, and order.

Information Theory: A Swiss Army Knife for system characterization, learning and prediction

2021 ◽  
Author(s):  
Uwe Ehret
Keyword(s):  
Information Theory ◽  
Information Content ◽  
Earth Science ◽  
Probability Distributions ◽  
Limited Data ◽  
Information Measures ◽  
Dynamical Complexity ◽  
System Characterization ◽  
Combine Information ◽  
Well Models

<p>In this contribution, I will – with examples from hydrology - make the case for information theory as a general language and framework for i) characterizing systems, ii) quantifying the information content in data, iii) evaluating how well models can learn from data, and iv) measuring how well models do in prediction. In particular, I will discuss how information measures can be used to characterize systems by the state space volume they occupy, their dynamical complexity, and their distance from equilibrium. Likewise, I will discuss how we can measure the information content of data through systematic perturbations, and how much information a model absorbs (or ignores) from data during learning. This can help building hybrid models that optimally combine information in data and general knowledge from physical and other laws, which is currently among the key challenges in machine learning applied to earth science problems.</p><p>While I will try my best to convince everybody of taking an information perspective henceforth, I will also name the related challenges: Data demands, binning choices, estimation of probability distributions from limited data, and issues with excessive data dimensionality.</p>

scholarly journals INFORMATION MEASURES FOR RECORD RANKED SET SAMPLES

2016 ◽  
Vol 38 (2) ◽  
pp. 554 ◽  
Author(s):  
Saeid Tahmasebi ◽  
Mahmoud Afshari ◽  
Maryam Eskandarzadeh
Keyword(s):  
Sample Size ◽  
Information Content ◽  
Shannon Entropy ◽  
Ranked Set Sampling ◽  
Sampling Scheme ◽  
Distribution Free ◽  
Information Measures ◽  
Gamma Distributions ◽  
Random Samples ◽  
The Difference

Salehi and Ahmadi (2014) introduced a new sampling scheme for generating record-breaking data called record ranked set sampling. In this paper, we consider the uncertainty and information content of record ranked set samples (RRSS) in terms of Shannon entropy, Rényi and Kullback-Leibler (KL) information measures. We show that the difference between the Shannon entropy of RRSS and the simple random samples (SRS) is depends on the parent distribution F. We also compare the information content of RRSS with a SRS data in the uniform, exponential, Weibull, Pareto, and gamma distributions. We obtain similar results for RRSS under the Rényi information. Finally, we show that the KL information between the distribution of SRS and distribution of RRSS is distribution-free and increases as the sample size increases.

STATISTICAL MEASURES AS MEASURES OF DIVERSITY

2010 ◽  
Vol 03 (02) ◽  
pp. 173-185 ◽  
Author(s):  
OM PARKASH ◽  
A. K. THUKRAL
Keyword(s):  
Information Theory ◽  
Data Analysis ◽  
Information Content ◽  
Biological Data ◽  
Central Tendency ◽  
Information Measures ◽  
Interdisciplinary Field ◽  
Statistical Measures ◽  
Data Statistics

Two fields of research have found tremendous applicability in the analysis of biological data-statistics and information theory. Statistics is extensively used for the measurement of central tendency, dispersion, comparison and covariation. Measures of information are used to study diversity and equitability. These two fields have been used independent of each other for data analysis. In this communication, we develop the link between the two and prove that statistical measures can be used as information measures. Our study will be a new interdisciplinary field of research and it will be possible to describe information content of a system from its statistics.

scholarly journals New Inequalities between Information Measures of Network Information Content

2013 ◽  
Vol 2013 ◽  
pp. 1-3 ◽  
Author(s):  
Pantelimon-George Popescu ◽  
Florin Pop ◽  
Alexandru Herişanu ◽  
Nicolae Ţăpuş
Keyword(s):  
Information Processing ◽  
Complex Systems ◽  
Complex Networks ◽  
Information Content ◽  
Discrete Case ◽  
Network Information ◽  
Information Inequalities ◽  
Information Measures ◽  
New Inequalities

We refine a classical logarithmic inequality using a discrete case of Bernoulli inequality, and then we refine furthermore two information inequalities between information measures for graphs, based on information functionals, presented by Dehmer and Mowshowitz in (2010) as Theorems 4.7 and 4.8. The inequalities refer to entropy-based measures of network information content and have a great impact for information processing in complex networks (a subarea of research in modeling of complex systems).

scholarly journals What’s Worth Talking About? Information Theory Reveals How Children Balance Informativeness and Ease of Production

2017 ◽  
Vol 28 (7) ◽  
pp. 954-966 ◽  
Author(s):  
Colin Bannard ◽  
Marla Rosner ◽  
Danielle Matthews
Keyword(s):  
Information Theory ◽  
Information Content ◽  
Low Frequency ◽  
Initial Experiment ◽  
Information Theoretic ◽  
Information Theoretic Measures

Of all the things a person could say in a given situation, what determines what is worth saying? Greenfield’s principle of informativeness states that right from the onset of language, humans selectively comment on whatever they find unexpected. In this article, we quantify this tendency using information-theoretic measures and report on a study in which we tested the counterintuitive prediction that children will produce words that have a low frequency given the context, because these will be most informative. Using corpora of child-directed speech, we identified adjectives that varied in how informative (i.e., unexpected) they were given the noun they modified. In an initial experiment ( N = 31) and in a replication ( N = 13), 3-year-olds heard an experimenter use these adjectives to describe pictures. The children’s task was then to describe the pictures to another person. As the information content of the experimenter’s adjective increased, so did children’s tendency to comment on the feature that adjective had encoded. Furthermore, our analyses suggest that children balance informativeness with a competing drive to ease production.

scholarly journals Quantifying Legal Entropy

2021 ◽  
Vol 9 ◽  
Author(s):  
Ted Sichelman
Keyword(s):  
Information Theory ◽  
Information Content ◽  
Legal System ◽  
Legal Systems ◽  
Information Costs ◽  
Precise Quantification ◽  
The Law ◽  
Legal Contexts ◽  
Definition Of ◽  
Quantitative Definition

Many scholars have employed the term “entropy” in the context of law and legal systems to roughly refer to the amount of “uncertainty” present in a given law, doctrine, or legal system. Just a few of these scholars have attempted to formulate a quantitative definition of legal entropy, and none have provided a precise formula usable across a variety of legal contexts. Here, relying upon Claude Shannon's definition of entropy in the context of information theory, I provide a quantitative formalization of entropy in delineating, interpreting, and applying the law. In addition to offering a precise quantification of uncertainty and the information content of the law, the approach offered here provides other benefits. For example, it offers a more comprehensive account of the uses and limits of “modularity” in the law—namely, using the terminology of Henry Smith, the use of legal “boundaries” (be they spatial or intangible) that “economize on information costs” by “hiding” classes of information “behind” those boundaries. In general, much of the “work” performed by the legal system is to reduce legal entropy by delineating, interpreting, and applying the law, a process that can in principle be quantified.

Information Measures

2015 ◽  
pp. 129-142
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Order Logic ◽  
Unified Framework ◽  
Higher Order Logic ◽  
Absolutely Continuous ◽  
Information Measures ◽  
Finite Spaces ◽  
Nikodym Derivative ◽  
Lebesgue Integration

This chapter presents a higher-order-logic formalization of the main concepts of information theory (Cover & Thomas, 1991), such as the Shannon entropy and mutual information, using the formalization of the foundational theories of measure, Lebesgue integration, and probability. The main results of the chapter include the formalizations of the Radon-Nikodym derivative and the Kullback-Leibler (KL) divergence (Coble, 2010). The latter provides a unified framework based on which most of the commonly used measures of information can be defined. The chapter then provides the general definitions that are valid for both discrete and continuous cases and then proves the corresponding reduced expressions where the measures considered are absolutely continuous over finite spaces.

scholarly journals Zipf’s Law, unbounded complexity and open-ended evolution

2018 ◽  
Vol 15 (149) ◽  
pp. 20180395 ◽  
Author(s):  
Bernat Corominas-Murtra ◽  
Luís F. Seoane ◽  
Ricard Solé
Keyword(s):  
Information Theory ◽  
Information Content ◽  
Multiple Scales ◽  
Evolutionary Process ◽  
Zipf’S Law ◽  
Shannon Information ◽  
Zipf's Law ◽  
Paradoxical Situation ◽  
Algorithmic Information ◽  
Standard Information

A major problem for evolutionary theory is understanding the so-called open-ended nature of evolutionary change, from its definition to its origins. Open-ended evolution (OEE) refers to the unbounded increase in complexity that seems to characterize evolution on multiple scales. This property seems to be a characteristic feature of biological and technological evolution and is strongly tied to the generative potential associated with combinatorics, which allows the system to grow and expand their available state spaces. Interestingly, many complex systems presumably displaying OEE, from language to proteins, share a common statistical property: the presence of Zipf’s Law. Given an inventory of basic items (such as words or protein domains) required to build more complex structures (sentences or proteins) Zipf’s Law tells us that most of these elements are rare whereas a few of them are extremely common. Using algorithmic information theory, in this paper we provide a fundamental definition for open-endedness, which can be understood as postulates . Its statistical counterpart, based on standard Shannon information theory, has the structure of a variational problem which is shown to lead to Zipf’s Law as the expected consequence of an evolutionary process displaying OEE. We further explore the problem of information conservation through an OEE process and we conclude that statistical information (standard Shannon information) is not conserved, resulting in the paradoxical situation in which the increase of information content has the effect of erasing itself. We prove that this paradox is solved if we consider non-statistical forms of information. This last result implies that standard information theory may not be a suitable theoretical framework to explore the persistence and increase of the information content in OEE systems.

Sign in / Sign up

Export Citation Format

Share Document