Probabilistic Autoencoder Using Fisher Information

Neural networks play a growing role in many scientific disciplines, including physics. Variational autoencoders (VAEs) are neural networks that are able to represent the essential information of a high dimensional data set in a low dimensional latent space, which have a probabilistic interpretation. In particular, the so-called encoder network, the first part of the VAE, which maps its input onto a position in latent space, additionally provides uncertainty information in terms of variance around this position. In this work, an extension to the autoencoder architecture is introduced, the FisherNet. In this architecture, the latent space uncertainty is not generated using an additional information channel in the encoder but derived from the decoder by means of the Fisher information metric. This architecture has advantages from a theoretical point of view as it provides a direct uncertainty quantification derived from the model and also accounts for uncertainty cross-correlations. We can show experimentally that the FisherNet produces more accurate data reconstructions than a comparable VAE and its learning performance also apparently scales better with the number of latent space dimensions.

Space from entanglement: An information-geometric perspective

In this paper, we study the construction of classical geometry from the quantum entanglement structure by using information geometry. In the information geometry of classical spacetime, the Fisher information metric is related to a blurred metric of a classical physical space. We first show that a local information metric can be obtained from the entanglement contour in a local subregion. This local information metric measures the fine structure of entanglement spectra inside the subregion, which suggests a quantum origin of the information-geometric blurred space. We study both the continuous and the classical limits of the quantum-originated blurred space by using the techniques from the statistical sampling algorithms, the sampling theory of spacetime and the projective limit. A scheme for going from a blurred space with quantum features to a classical geometry is also explored.

The Fundamental Theorem of Natural Selection

Suppose we have n different types of self-replicating entity, with the population Pi of the ith type changing at a rate equal to Pi times the fitness fi of that type. Suppose the fitness fi is any continuous function of all the populations P1,⋯,Pn. Let pi be the fraction of replicators that are of the ith type. Then p=(p1,⋯,pn) is a time-dependent probability distribution, and we prove that its speed as measured by the Fisher information metric equals the variance in fitness. In rough terms, this says that the speed at which information is updated through natural selection equals the variance in fitness. This result can be seen as a modified version of Fisher’s fundamental theorem of natural selection. We compare it to Fisher’s original result as interpreted by Price, Ewens and Edwards.

Geometric Variational Inference

Efficiently accessing the information contained in non-linear and high dimensional probability distributions remains a core challenge in modern statistics. Traditionally, estimators that go beyond point estimates are either categorized as Variational Inference (VI) or Markov-Chain Monte-Carlo (MCMC) techniques. While MCMC methods that utilize the geometric properties of continuous probability distributions to increase their efficiency have been proposed, VI methods rarely use the geometry. This work aims to fill this gap and proposes geometric Variational Inference (geoVI), a method based on Riemannian geometry and the Fisher information metric. It is used to construct a coordinate transformation that relates the Riemannian manifold associated with the metric to Euclidean space. The distribution, expressed in the coordinate system induced by the transformation, takes a particularly simple form that allows for an accurate variational approximation by a normal distribution. Furthermore, the algorithmic structure allows for an efficient implementation of geoVI which is demonstrated on multiple examples, ranging from low-dimensional illustrative ones to non-linear, hierarchical Bayesian inverse problems in thousands of dimensions.

Pathological Spectra of the Fisher Information Metric and Its Variants in Deep Neural Networks

The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions. Focusing on the FIM and its variants in deep neural networks (DNNs), we reveal their characteristic scale dependence on the network width, depth, and sample size when the network has random weights and is sufficiently wide. This study covers two widely used FIMs for regression with linear output and for classification with softmax output. Both FIMs asymptotically show pathological eigenvalue spectra in the sense that a small number of eigenvalues become large outliers depending on the width or sample size, while the others are much smaller. It implies that the local shape of the parameter space or loss landscape is very sharp in a few specific directions while almost flat in the other directions. In particular, the softmax output disperses the outliers and makes a tail of the eigenvalue density spread from the bulk. We also show that pathological spectra appear in other variants of FIMs: one is the neural tangent kernel; another is a metric for the input signal and feature space that arises from feedforward signal propagation. Thus, we provide a unified perspective on the FIM and its variants that will lead to more quantitative understanding of learning in large-scale DNNs.

Appearance of Thermal Time

In this paper a viewpoint that time is an informational and thermal entity is presented. We consider a model for a simple relaxation process for which a relationship among event, time and temperature is mathematically formulated. It is then explicitly illustrated that temperature and time are statistically inferred through measurement of events. The probability distribution of the events thus provides an intrinsic correlation between temperature and time, which can relevantly be expressed in terms of the Fisher information metric. The two-dimensional differential geometry of temperature and time then leads us to a finding of a simple equation for the scalar curvature, $$R = -1$$ R = - 1 , in this case of relaxation process. This basic equation, in turn, may be regarded as characterizing a nonequilibrium dynamical process and having a solution given by the Fisher information metric. The time can then be interpreted so as to appear in a thermal way.

Information geometry for phylogenetic trees

We propose a new space of phylogenetic trees which we call wald space. The motivation is to develop a space suitable for statistical analysis of phylogenies, but with a geometry based on more biologically principled assumptions than existing spaces: in wald space, trees are close if they induce similar distributions on genetic sequence data. As a point set, wald space contains the previously developed Billera–Holmes–Vogtmann (BHV) tree space; it also contains disconnected forests, like the edge-product (EP) space but without certain singularities of the EP space. We investigate two related geometries on wald space. The first is the geometry of the Fisher information metric of character distributions induced by the two-state symmetric Markov substitution process on each tree. Infinitesimally, the metric is proportional to the Kullback–Leibler divergence, or equivalently, as we show, to any f-divergence. The second geometry is obtained analogously but using a related continuous-valued Gaussian process on each tree, and it can be viewed as the trace metric of the affine-invariant metric for covariance matrices. We derive a gradient descent algorithm to project from the ambient space of covariance matrices to wald space. For both geometries we derive computational methods to compute geodesics in polynomial time and show numerically that the two information geometries (discrete and continuous) are very similar. In particular, geodesics are approximated extrinsically. Comparison with the BHV geometry shows that our canonical and biologically motivated space is substantially different.

Purification complexity without purifications

We generalize the Fubini-Study method for pure-state complexity to generic quantum states by taking Bures metric or quantum Fisher information metric (QFIM) on the space of density matrices as the complexity measure. Due to Uhlmann’s theorem, we show that the mixed-state complexity exactly equals the purification complexity measured by the Fubini-Study metric for purified states but without explicitly applying any purification. We also find the purification complexity is non-increasing under any trace-preserving quantum operations. We also study the mixed Gaussian states as an example to explicitly illustrate our conclusions for purification complexity.

One Bit is All It Takes: A Devastating Timing Attack on BLISS’s Non-Constant Time Sign Flips

As one of the most efficient lattice-based signature schemes, and one of the only ones to have seen deployment beyond an academic setting (e.g., as part of the VPN software suite strongSwan), BLISS has attracted a significant amount of attention in terms of its implementation security, and side-channel vulnerabilities of several parts of its signing algorithm have been identified in previous works. In this paper, we present an even simpler timing attack against it. The bimodal Gaussian distribution that BLISS is named after is achieved using a random sign flip during signature generation, and neither the original implementation of BLISS nor strongSwan ensure that this sign flip is carried out in constant time. It is therefore possible to recover the corresponding sign through side-channel leakage (using, e.g., cache attacks or branch tracing). We show that obtaining this single bit of leakage (for a moderate number of signatures) is in fact sufficient for a full key recovery attack. The recovery is carried out using a maximum likelihood estimation on the space of parameters, which can be seen as a statistical manifold. The analysis of the attack thus reduces to the computation of the Fisher information metric.