Variational Contrastive Log Ratio Upper Bound of Mutual Information for Training Generative Models

2021 ◽  
Author(s):  
Marshal Arijona Sinaga ◽  
Machmud Roby Alhamidi ◽  
Muhammad Febrian Rachmadi ◽  
Wisnu Jatmiko
Keyword(s):  
Mutual Information ◽  
Upper Bound ◽  
Generative Models ◽  
Log Ratio

scholarly journals Mutual information decay for factors of i.i.d.

2018 ◽  
Vol 39 (11) ◽  
pp. 3015-3030
Author(s):  
BALÁZS GERENCSÉR ◽  
VIKTOR HARANGI
Keyword(s):  
Mutual Information ◽  
Upper Bound ◽  
Upper Bounds ◽  
Interesting Phenomenon ◽  
Regular Tree ◽  
Normalized Mutual Information ◽  
Distributed Processes ◽  
Independent And Identically Distributed

This paper is concerned with factors of independent and identically distributed processes on the $d$ -regular tree for $d\geq 3$ . We study the mutual information of values on two given vertices. If the vertices are neighbors (i.e. their distance is $1$ ), then a known inequality between the entropy of a vertex and the entropy of an edge provides an upper bound for the (normalized) mutual information. In this paper we obtain upper bounds for vertices at an arbitrary distance $k$ , of order $(d-1)^{-k/2}$ . Although these bounds are sharp, we also show that an interesting phenomenon occurs here: for any fixed process, the rate of mutual information decay is much faster, essentially of order $(d-1)^{-k}$ .

scholarly journals Information Bottleneck Analysis by a Conditional Mutual Information Bound

Entropy ◽  
2021 ◽  
Vol 23 (8) ◽  
pp. 974
Author(s):  
Taro Tezuka ◽  
Shizuma Namekawa
Keyword(s):  
Mutual Information ◽  
Supervised Learning ◽  
Upper Bound ◽  
Latent Variables ◽  
Upper Bounds ◽  
Conditional Mutual Information ◽  
Information Bottleneck ◽  
Input Matching ◽  
Bottleneck Analysis ◽  
Information Bound

Task-nuisance decomposition describes why the information bottleneck loss I(z;x)−βI(z;y) is a suitable objective for supervised learning. The true category y is predicted for input x using latent variables z. When n is a nuisance independent from y, I(z;n) can be decreased by reducing I(z;x) since the latter upper bounds the former. We extend this framework by demonstrating that conditional mutual information I(z;x|y) provides an alternative upper bound for I(z;n). This bound is applicable even if z is not a sufficient representation of x, that is, I(z;y)≠I(x;y). We used mutual information neural estimation (MINE) to estimate I(z;x|y). Experiments demonstrated that I(z;x|y) is smaller than I(z;x) for layers closer to the input, matching the claim that the former is a tighter bound than the latter. Because of this difference, the information plane differs when I(z;x|y) is used instead of I(z;x).

scholarly journals A Two-Moment Inequality with Applications to Rényi Entropy and Mutual Information

Entropy ◽  
2020 ◽  
Vol 22 (11) ◽  
pp. 1244
Author(s):  
Galen Reeves
Keyword(s):  
Mutual Information ◽  
Upper Bound ◽  
Renyi Entropy ◽  
Rényi Entropy ◽  
Conditional Density ◽  
Moment Inequality ◽  
Single Moment ◽  
Variance Decompositions ◽  
Moment Constraint ◽  
Maximum Entropy Distributions

This paper explores some applications of a two-moment inequality for the integral of the rth power of a function, where 0<r<1. The first contribution is an upper bound on the Rényi entropy of a random vector in terms of the two different moments. When one of the moments is the zeroth moment, these bounds recover previous results based on maximum entropy distributions under a single moment constraint. More generally, evaluation of the bound with two carefully chosen nonzero moments can lead to significant improvements with a modest increase in complexity. The second contribution is a method for upper bounding mutual information in terms of certain integrals with respect to the variance of the conditional density. The bounds have a number of useful properties arising from the connection with variance decompositions.

scholarly journals A Tight Upper Bound on Mutual Information

2019 ◽  
Author(s):  
Michal Hledik ◽  
Thomas R. Sokolowski ◽  
Gasper Tkacik
Keyword(s):  
Mutual Information ◽  
Upper Bound

scholarly journals Mutual Information Loss in Pyramidal Image Processing

Information ◽  
2020 ◽  
Vol 11 (6) ◽  
pp. 322 ◽  
Author(s):  
Jerry Gibson ◽  
Hoontaek Oh
Keyword(s):  
Image Analysis ◽  
Mutual Information ◽  
Probability Distributions ◽  
Reconstruction Error ◽  
Information Loss ◽  
Gaussian Pyramid ◽  
The Difference ◽  
Two Stages ◽  
Log Ratio

Gaussian and Laplacian pyramids have long been important for image analysis and compression. More recently, multiresolution pyramids have become an important component of machine learning and deep learning for image analysis and image recognition. Constructing Gaussian and Laplacian pyramids consists of a series of filtering, decimation, and differencing operations, and the quality indicator is usually mean squared reconstruction error in comparison to the original image. We present a new characterization of the information loss in a Gaussian pyramid in terms of the change in mutual information. More specifically, we show that one half the log ratio of entropy powers between two stages in a Gaussian pyramid is equal to the difference in mutual information between these two stages. We show that this relationship holds for a wide variety of probability distributions and present several examples of analyzing Gaussian and Laplacian pyramids for different images.

scholarly journals Mutual Information, the Linear Prediction Model, and CELP Voice Codecs

2019 ◽  
Author(s):  
Jerry Gibson
Keyword(s):  
Mutual Information ◽  
Linear Prediction ◽  
Rate Distortion ◽  
Source Model ◽  
Rate Distortion Theory ◽  
Input Source ◽  
Distortion Measure ◽  
Adaptive Codebook ◽  
Log Ratio ◽  
Linear Prediction Model

We write the mutual information between an input speech utterance and its reconstruction by a Code-Excited Linear Prediction (CELP) codec in terms of the mutual information between the input speech and the contributions due to the short term predictor, the adaptive codebook, and the fixed codebook. We then show that a recently introduced quantity, the log ratio of entropy powers, can be used to estimate these mutual informations in terms of bits/sample. A key result is that for many common distributions and for Gaussian autoregressive processes, the entropy powers in the ratio can be replaced by the corresponding minimum mean squared errors. We provide examples of estimating CELP codec performance using the new results and compare to the performance of the AMR codec and other CELP codecs. Similar to rate distortion theory, this method only needs the input source model and the appropriate distortion measure.

scholarly journals Matrix Liberation Process II: Relation to Orbital Free Entropy

2020 ◽  
pp. 1-49
Author(s):  
Yoshimichi Ueda
Keyword(s):  
Mutual Information ◽  
Upper Bound ◽  
Large Deviation Principle ◽  
Large Deviation ◽  
Rate Function ◽  
New Approach ◽  
Free Entropy ◽  
The Matrix ◽  
Orbital Free ◽  
Definition Of

Abstract We investigate the concept of orbital free entropy from the viewpoint of the matrix liberation process. We will show that many basic questions around the definition of orbital free entropy are reduced to the question of full large deviation principle for the matrix liberation process. We will also obtain a large deviation upper bound for a certain family of random matrices that is essential to define the orbital free entropy. The resulting rate function is made up into a new approach to free mutual information.

Sign in / Sign up

Export Citation Format

Share Document