The Metrics of Info-Metrics

2017 ◽  
Author(s):  
Amos Golan
Keyword(s):  
Maximum Entropy ◽  
Maximum Entropy Principle ◽  
Complete List ◽  
Entropy Measure ◽  
Limited Information ◽  
Quantitative Metrics ◽  
Entropy Principle

In this chapter I present the key ideas and develop the essential quantitative metrics needed for modeling and inference with limited information. I provide the necessary tools to study the traditional maximum-entropy principle, which is the cornerstone for info-metrics. The chapter starts by defining the primary notions of information and entropy as they are related to probabilities and uncertainty. The unique properties of the entropy are explained. The derivations and discussion are extended to multivariable entropies and informational quantities. For completeness, I also discuss the complete list of the Shannon-Khinchin axioms behind the entropy measure. An additional derivation of information and entropy, due to the independently developed work of Wiener, is provided as well.

scholarly journals A LINK BETWEEN THE MAXIMUM ENTROPY APPROACH AND THE VARIATIONAL ENTROPY FORM

2011 ◽  
Vol 25 (22) ◽  
pp. 1821-1828 ◽  
Author(s):  
E. V. VAKARIN ◽  
J. P. BADIALI
Keyword(s):  
Probability Distribution ◽  
Maximum Entropy ◽  
Variational Formulation ◽  
Variational Approach ◽  
Maximum Entropy Principle ◽  
Random Variable ◽  
Entropy Measure ◽  
Universal Relation ◽  
Amount Of Information ◽  
Entropy Principle

The maximum entropy approach operating with quite general entropy measure and constraint is considered. It is demonstrated that for a conditional or parametrized probability distribution f(x|μ), there is a "universal" relation among the entropy rate and the functions appearing in the constraint. This relation allows one to translate the specificities of the observed behavior θ(μ) into the amount of information on the relevant random variable x at different values of the parameter μ. It is shown that the recently proposed variational formulation of the entropic functional can be obtained as a consequence of this relation, that is from the maximum entropy principle. This resolves certain puzzling points that appeared in the variational approach.

scholarly journals Inferring metabolic fluxes in nutrient-limited continuous cultures: A Maximum Entropy Approach with minimum information

2021 ◽  
Author(s):  
Jose Pereiro ◽  
Jorge Fernandez-de-Cossio-Diaz ◽  
Roberto Mulet
Keyword(s):  
Maximum Entropy ◽  
Maximum Entropy Principle ◽  
Limited Information ◽  
Continuous Cultures ◽  
Mean Values ◽  
E Coli ◽  
Metabolic Fluxes ◽  
Limiting Nutrient ◽  
Entropy Principle ◽  
Balance Analysis

We propose a new scheme to infer the metabolic fluxes of cell cultures in a chemostat. Our approach is based on the Maximum Entropy Principle and exploits the understanding of the chemostat dynamics and its connection with the actual metabolism of cells. We show that, in continuous cultures with limiting nutrients, the inference can be done with limited information about the culture: the dilution rate of the chemostat, the concentration in the feed media of the limiting nutrient and the cell concentration at steady state. Also, we remark that our technique provides information, not only about the mean values of the fluxes in the culture, but also its heterogeneity. We first present these results studying a computational model of a chemostat. Having control of this model we can test precisely the quality of the inference, and also unveil the mechanisms behind the success of our approach. Then, we apply our method to E. coli experimental data from the literature and show that it outperforms alternative formulations that rest on a Flux Balance Analysis framework.

An entropy concentration theorem: applications in artificial intelligence and descriptive statistics

1990 ◽  
Vol 27 (2) ◽  
pp. 303-313 ◽  
Author(s):  
Claudine Robert
Keyword(s):  
Artificial Intelligence ◽  
Maximum Entropy ◽  
Large Deviation ◽  
Maximum Entropy Principle ◽  
Statistical Modelling ◽  
Linear Constraints ◽  
Descriptive Statistics ◽  
Entropy Principle ◽  
Maximum Entropy Distribution ◽  
Mathematical Justification

The maximum entropy principle is used to model uncertainty by a maximum entropy distribution, subject to some appropriate linear constraints. We give an entropy concentration theorem (whose demonstration is based on large deviation techniques) which is a mathematical justification of this statistical modelling principle. Then we indicate how it can be used in artificial intelligence, and how relevant prior knowledge is provided by some classical descriptive statistical methods. It appears furthermore that the maximum entropy principle yields to a natural binding between descriptive methods and some statistical structures.

SINE ENTROPY FOR UNCERTAIN VARIABLES

2013 ◽  
Vol 21 (05) ◽  
pp. 743-753 ◽  
Author(s):  
KAI YAO ◽  
JINWU GAO ◽  
WEI DAI
Keyword(s):  
Maximum Entropy ◽  
Uncertainty Theory ◽  
Maximum Entropy Principle ◽  
Expected Value ◽  
Uncertain Variables ◽  
Entropy Principle

Entropy is a measure of the uncertainty associated with a variable whose value cannot be exactly predicated. In uncertainty theory, it has been quantified so far by logarithmic entropy. However, logarithmic entropy sometimes fails to measure the uncertainty. This paper will propose another type of entropy named sine entropy as a supplement, and explore its properties. After that, the maximum entropy principle will be introduced, and the arc-cosine distributed variables will be proved to have the maximum sine entropy with given expected value and variance.

Sign in / Sign up

Export Citation Format

Share Document