Bits and pieces: understanding information decomposition from part-whole relationships and formal logic

Partial information decomposition (PID) seeks to decompose the multivariate mutual information that a set of source variables contains about a target variable into basic pieces, the so-called ‘atoms of information’. Each atom describes a distinct way in which the sources may contain information about the target. For instance, some information may be contained uniquely in a particular source, some information may be shared by multiple sources and some information may only become accessible synergistically if multiple sources are combined. In this paper, we show that the entire theory of PID can be derived, firstly, from considerations of part-whole relationships between information atoms and mutual information terms, and secondly, based on a hierarchy of logical constraints describing how a given information atom can be accessed. In this way, the idea of a PID is developed on the basis of two of the most elementary relationships in nature: the part-whole relationship and the relation of logical implication. This unifying perspective provides insights into pressing questions in the field such as the possibility of constructing a PID based on concepts other than redundant information in the general n-sources case. Additionally, it admits of a particularly accessible exposition of PID theory.

Ensemble Linear Subspace Analysis of High-Dimensional Data

Regression models provide prediction frameworks for multivariate mutual information analysis that uses information concepts when choosing covariates (also called features) that are important for analysis and prediction. We consider a high dimensional regression framework where the number of covariates (p) exceed the sample size (n). Recent work in high dimensional regression analysis has embraced an ensemble subspace approach that consists of selecting random subsets of covariates with fewer than p covariates, doing statistical analysis on each subset, and then merging the results from the subsets. We examine conditions under which penalty methods such as Lasso perform better when used in the ensemble approach by computing mean squared prediction errors for simulations and a real data example. Linear models with both random and fixed designs are considered. We examine two versions of penalty methods: one where the tuning parameter is selected by cross-validation; and one where the final predictor is a trimmed average of individual predictors corresponding to the members of a set of fixed tuning parameters. We find that the ensemble approach improves on penalty methods for several important real data and model scenarios. The improvement occurs when covariates are strongly associated with the response, when the complexity of the model is high. In such cases, the trimmed average version of ensemble Lasso is often the best predictor.

Discovering Higher-Order Interactions Through Neural Information Decomposition

If regularity in data takes the form of higher-order functions among groups of variables, models which are biased towards lower-order functions may easily mistake the data for noise. To distinguish whether this is the case, one must be able to quantify the contribution of different orders of dependence to the total information. Recent work in information theory attempts to do this through measures of multivariate mutual information (MMI) and information decomposition (ID). Despite substantial theoretical progress, practical issues related to tractability and learnability of higher-order functions are still largely unaddressed. In this work, we introduce a new approach to information decomposition—termed Neural Information Decomposition (NID)—which is both theoretically grounded, and can be efficiently estimated in practice using neural networks. We show on synthetic data that NID can learn to distinguish higher-order functions from noise, while many unsupervised probability models cannot. Additionally, we demonstrate the usefulness of this framework as a tool for exploring biological and artificial neural networks.

Interpretation of multi-scale permeability data through an information theory perspective

We employ elements of information theory to quantify (i) the information content related to data collected at given measurement scales within the same porous medium domain and (ii) the relationships among information contents of datasets associated with differing scales. We focus on gas permeability data collected over Berea Sandstone and Topopah Spring Tuff blocks, considering four measurement scales. We quantify the way information is shared across these scales through (i) the Shannon entropy of the data associated with each support scale, (ii) mutual information shared between data taken at increasing support scales, and (iii) multivariate mutual information shared within triplets of datasets, each associated with a given scale. We also assess the level of uniqueness, redundancy and synergy (rendering, i.e., information partitioning) of information content that the data associated with the intermediate and largest scales provide with respect to the information embedded in the data collected at the smallest support scale in a triplet. Highlights. Information theory allows characterization of the information content of permeability data related to differing measurement scales. An increase in the measurement scale is associated with quantifiable loss of information about permeability. Redundant, unique and synergetic contributions of information are evaluated for triplets of permeability datasets, each taken at a given scale.

Generalised Measures of Multivariate Information Content

The entropy of a pair of random variables is commonly depicted using a Venn diagram. This representation is potentially misleading, however, since the multivariate mutual information can be negative. This paper presents new measures of multivariate information content that can be accurately depicted using Venn diagrams for any number of random variables. These measures complement the existing measures of multivariate mutual information and are constructed by considering the algebraic structure of information sharing. It is shown that the distinct ways in which a set of marginal observers can share their information with a non-observing third party corresponds to the elements of a free distributive lattice. The redundancy lattice from partial information decomposition is then subsequently and independently derived by combining the algebraic structures of joint and shared information content.

Interpretation of Multi-scale Permeability Data through an Information Theory Perspective

We employ elements of Information Theory to quantify (i) the information content related to data collected at given measurement scales within the same porous medium domain, and (ii) the relationships among Information contents of datasets associated with differing scales. We focus on gas permeability data collected over a Berea Sandstone and a Topopah Spring Tuff blocks, considering four measurement scales. We quantify the way information is shared across these scales through (i) the Shannon entropy of the data associated with each support scale, (ii) mutual information shared between data taken at increasing support scales, and (iii) multivariate mutual information shared within triplets of datasets, each associated with a given scale. We also assess the level of uniqueness, redundancy and synergy (rendering, i.e., the information partitioning) of information content that the data associated with the intermediate and largest scales provide with respect to the information embedded in the data collected at the smallest support scale in a triplet.