scholarly journals Information Bottleneck Analysis by a Conditional Mutual Information Bound

Entropy ◽  
2021 ◽  
Vol 23 (8) ◽  
pp. 974
Author(s):  
Taro Tezuka ◽  
Shizuma Namekawa
Keyword(s):  
Mutual Information ◽  
Supervised Learning ◽  
Upper Bound ◽  
Latent Variables ◽  
Upper Bounds ◽  
Conditional Mutual Information ◽  
Information Bottleneck ◽  
Input Matching ◽  
Bottleneck Analysis ◽  
Information Bound

Task-nuisance decomposition describes why the information bottleneck loss I(z;x)−βI(z;y) is a suitable objective for supervised learning. The true category y is predicted for input x using latent variables z. When n is a nuisance independent from y, I(z;n) can be decreased by reducing I(z;x) since the latter upper bounds the former. We extend this framework by demonstrating that conditional mutual information I(z;x|y) provides an alternative upper bound for I(z;n). This bound is applicable even if z is not a sufficient representation of x, that is, I(z;y)≠I(x;y). We used mutual information neural estimation (MINE) to estimate I(z;x|y). Experiments demonstrated that I(z;x|y) is smaller than I(z;x) for layers closer to the input, matching the claim that the former is a tighter bound than the latter. Because of this difference, the information plane differs when I(z;x|y) is used instead of I(z;x).

scholarly journals Mutual information decay for factors of i.i.d.

2018 ◽  
Vol 39 (11) ◽  
pp. 3015-3030
Author(s):  
BALÁZS GERENCSÉR ◽  
VIKTOR HARANGI
Keyword(s):  
Mutual Information ◽  
Upper Bound ◽  
Upper Bounds ◽  
Interesting Phenomenon ◽  
Regular Tree ◽  
Normalized Mutual Information ◽  
Distributed Processes ◽  
Independent And Identically Distributed

This paper is concerned with factors of independent and identically distributed processes on the $d$ -regular tree for $d\geq 3$ . We study the mutual information of values on two given vertices. If the vertices are neighbors (i.e. their distance is $1$ ), then a known inequality between the entropy of a vertex and the entropy of an edge provides an upper bound for the (normalized) mutual information. In this paper we obtain upper bounds for vertices at an arbitrary distance $k$ , of order $(d-1)^{-k/2}$ . Although these bounds are sharp, we also show that an interesting phenomenon occurs here: for any fixed process, the rate of mutual information decay is much faster, essentially of order $(d-1)^{-k}$ .

scholarly journals Target space entanglement in quantum mechanics of fermions and matrices

2021 ◽  
Vol 2021 (8) ◽  
Author(s):  
Sotaro Sugishita
Keyword(s):  
Mutual Information ◽  
Ground State ◽  
Entanglement Entropy ◽  
Algebraic Approach ◽  
Upper Bounds ◽  
Wave Functions ◽  
Target Space ◽  
Single Interval ◽  
Area Law ◽  
Formula Expression

Abstract We consider entanglement of first-quantized identical particles by adopting an algebraic approach. In particular, we investigate fermions whose wave functions are given by the Slater determinants, as for singlet sectors of one-matrix models. We show that the upper bounds of the general Rényi entropies are N log 2 for N particles or an N × N matrix. We compute the target space entanglement entropy and the mutual information in a free one-matrix model. We confirm the area law: the single-interval entropy for the ground state scales as $$ \frac{1}{3} $$ 1 3 log N in the large N model. We obtain an analytical $$ \mathcal{O}\left({N}^0\right) $$ O N 0 expression of the mutual information for two intervals in the large N expansion.

Upper bounds on the energy dissipation in turbulent precession

1996 ◽  
Vol 321 ◽  
pp. 335-370 ◽  
Author(s):  
R. R. Kerswell
Keyword(s):  
Experimental Data ◽  
Energy Dissipation ◽  
Viscous Dissipation ◽  
Upper Bound ◽  
Dissipation Rate ◽  
Oblate Spheroid ◽  
Upper Bounds ◽  
Precession Rate ◽  
Long Cylinder

Rigorous upper bounds on the viscous dissipation rate are identified for two commonly studied precessing fluid-filled configurations: an oblate spheroid and a long cylinder. The latter represents an interesting new application of the upper-bounding techniques developed by Howard and Busse. A novel ‘background’ method recently introduced by Doering & Constantin is also used to deduce in both instances an upper bound which is independent of the fluid's viscosity and the forcing precession rate. Experimental data provide some evidence that the observed viscous dissipation rate mirrors this behaviour at sufficiently high precessional forcing. Implications are then discussed for the Earth's precessional response.

scholarly journals The Essential Dimension of Stacks of Parabolic Vector Bundles over Curves

2012 ◽  
Vol 10 (3) ◽  
pp. 455-488 ◽  
Author(s):  
Indranil Biswas ◽  
Ajneet Dhillon ◽  
Nicole Lemire
Keyword(s):  
Lower Bounds ◽  
Upper Bound ◽  
Vector Bundles ◽  
Upper Bounds ◽  
Essential Dimension ◽  
Parabolic Structure ◽  
Moduli Stack

AbstractWe find upper bounds on the essential dimension of the moduli stack of parabolic vector bundles over a curve. When there is no parabolic structure, we improve the known upper bound on the essential dimension of the usual moduli stack. Our calculations also give lower bounds on the essential dimension of the semistable locus inside the moduli stack of vector bundles of rank r and degree d without parabolic structure.

ances and covariances obtained from REML are normally distributed with expectation vector and variance-covariance matrix equal to the fol-low  ing, r  espectiv    ely,   When σˆ > 0.04, let νˆ = δˆ + σˆ + σˆ − 2ωˆ + σˆ − (1 + c (7.6) be an estimate for the (7.3) reference-scaled metric in accordance with FDA Guidance (2001) and using a REML UN model. Then (Patter-son, 2003; Patterson and Jones, 2002b), this estimate is asymptotically normally distributed and unbiased with E[νˆ ] = δ +σ − (1 + c and Var[νˆ ] = 4σ + l + 4l + (1 + c ) (l )+ 2l −2(1+c − 2(1+c +4(1+c −2(1+c . Similarly, for the constant-scaled metric, when σˆ ≤ 0.04, νˆ = δˆ + σˆ + σˆ − 2ωˆ + σˆ − σˆ − 0.04(c ) (7.7) E[νˆ ] = δ +σ − 0.04(c ) Var[νˆ ] = 4σ + l + 4l + 2l − 2l − 4l + 4l − 2l . The required asymptotic upper bound √ of the 90% confidence interval can √ then be calculated as νˆ + 1.645× V̂ ar[νˆ ] or νˆ + 1.645× V̂ ar[νˆ ], where the variances are obtained by ‘plugging in’ the estimated values of the variances and covariances obtained from SAS proc mixed into the formulae for Var[νˆ ] or Var[νˆ ]. The necessary SAS code to do this is given in Appendix B. The output reveals that σˆ = 0.0714 and the upper bound is−0.060 for log(AUC). For log(Cmax), σˆ = 0.1060 and the upper bound is −0.055. As both of these upper bounds are below zero, IBE can be claimed.

2003 ◽  
pp. 367-367
Keyword(s):  
Confidence Interval ◽  
Covariance Matrix ◽  
Upper Bound ◽  
Upper Bounds ◽  
Fda Guidance ◽  
Asymptotic Upper Bound ◽  
Variance Covariance Matrix ◽  
Normally Distributed

scholarly journals Bulk private curves require large conditional mutual information

2021 ◽  
Vol 2021 (9) ◽  
Author(s):  
Alex May
Keyword(s):  
Mutual Information ◽  
Boundary Region ◽  
Theoretic Approach ◽  
Strong Correlations ◽  
Conditional Mutual Information ◽  
Information Theoretic ◽  
Causal Curve ◽  
Resource Requirements ◽  
Theoretic Argument ◽  
Information Theoretic Approach

Abstract We prove a theorem showing that the existence of “private” curves in the bulk of AdS implies two regions of the dual CFT share strong correlations. A private curve is a causal curve which avoids the entanglement wedge of a specified boundary region $$ \mathcal{U} $$ U . The implied correlation is measured by the conditional mutual information $$ I\left({\mathcal{V}}_1:\left.{\mathcal{V}}_2\right|\mathcal{U}\right) $$ I V 1 : V 2 U , which is O(1/GN) when a private causal curve exists. The regions $$ {\mathcal{V}}_1 $$ V 1 and $$ {\mathcal{V}}_2 $$ V 2 are specified by the endpoints of the causal curve and the placement of the region $$ \mathcal{U} $$ U . This gives a causal perspective on the conditional mutual information in AdS/CFT, analogous to the causal perspective on the mutual information given by earlier work on the connected wedge theorem. We give an information theoretic argument for our theorem, along with a bulk geometric proof. In the geometric perspective, the theorem follows from the maximin formula and entanglement wedge nesting. In the information theoretic approach, the theorem follows from resource requirements for sending private messages over a public quantum channel.

scholarly journals Unsupervised Learning via Total Correlation Explanation

2017 ◽  
Author(s):  
Greg Ver Steeg
Keyword(s):  
Mutual Information ◽  
Unsupervised Learning ◽  
Supervised Learning ◽  
Human Behavior ◽  
Learning Problems ◽  
Information Theoretic ◽  
Sensory Environment ◽  
Total Correlation ◽  
Environment Dependence ◽  
Multivariate Mutual Information

Learning by children and animals occurs effortlessly and largely without obvious supervision. Successes in automating supervised learning have not translated to the more ambiguous realm of unsupervised learning where goals and labels are not provided. Barlow (1961) suggested that the signal that brains leverage for unsupervised learning is dependence, or redundancy, in the sensory environment. Dependence can be characterized using the information-theoretic multivariate mutual information measure called total correlation. The principle of Total Cor-relation Ex-planation (CorEx) is to learn representations of data that "explain" as much dependence in the data as possible. We review some manifestations of this principle along with successes in unsupervised learning problems across diverse domains including human behavior, biology, and language.

Sign in / Sign up

Export Citation Format

Share Document