Information geometry connecting Wasserstein distance and Kullback–Leibler divergence via the entropy-relaxed transportation problem

We propose a new divergence on the manifold of probability distributions, building on the entropic regularization of optimal transportation problems. As Cuturi ( 2013 ) showed, regularizing the optimal transport problem with an entropic term is known to bring several computational benefits. However, because of that regularization, the resulting approximation of the optimal transport cost does not define a proper distance or divergence between probability distributions. We recently tried to introduce a family of divergences connecting the Wasserstein distance and the Kullback-Leibler divergence from an information geometry point of view (see Amari, Karakida, & Oizumi, 2018 ). However, that proposal was not able to retain key intuitive aspects of the Wasserstein geometry, such as translation invariance, which plays a key role when used in the more general problem of computing optimal transport barycenters. The divergence we propose in this work is able to retain such properties and admits an intuitive interpretation.

Download Full-text

On Voronoi Diagrams on the Information-Geometric Cauchy Manifolds

Entropy ◽

10.3390/e22070713 ◽

2020 ◽

Vol 22 (7) ◽

pp. 713 ◽

Cited By ~ 1

Author(s):

Frank Nielsen

Keyword(s):

Voronoi Diagram ◽

Voronoi Diagrams ◽

Information Geometry ◽

Square Root ◽

Chi Square ◽

Scale Parameters ◽

Leibler Divergence ◽

Finite Set ◽

Metric Distance ◽

Rao Distance

We study the Voronoi diagrams of a finite set of Cauchy distributions and their dual complexes from the viewpoint of information geometry by considering the Fisher-Rao distance, the Kullback-Leibler divergence, the chi square divergence, and a flat divergence derived from Tsallis entropy related to the conformal flattening of the Fisher-Rao geometry. We prove that the Voronoi diagrams of the Fisher-Rao distance, the chi square divergence, and the Kullback-Leibler divergences all coincide with a hyperbolic Voronoi diagram on the corresponding Cauchy location-scale parameters, and that the dual Cauchy hyperbolic Delaunay complexes are Fisher orthogonal to the Cauchy hyperbolic Voronoi diagrams. The dual Voronoi diagrams with respect to the dual flat divergences amount to dual Bregman Voronoi diagrams, and their dual complexes are regular triangulations. The primal Bregman Voronoi diagram is the Euclidean Voronoi diagram and the dual Bregman Voronoi diagram coincides with the Cauchy hyperbolic Voronoi diagram. In addition, we prove that the square root of the Kullback-Leibler divergence between Cauchy distributions yields a metric distance which is Hilbertian for the Cauchy scale families.

Download Full-text

Information-Geometric Measure for Neural Spikes

Neural Computation ◽

10.1162/08997660260293238 ◽

2002 ◽

Vol 14 (10) ◽

pp. 2269-2316 ◽

Cited By ~ 83

Author(s):

Hiroyuki Nakahara ◽

Shun-ichi Amari

Keyword(s):

Information Geometry ◽

Higher Order ◽

Neural Firing ◽

Firing Patterns ◽

Neural Spikes ◽

Geometric Measure ◽

Leibler Divergence ◽

Novel Method ◽

Account Information ◽

Spike Firing

This study introduces information-geometric measures to analyze neural firing patterns by taking not only the second-order but also higher-order interactions among neurons into account. Information geometry provides useful tools and concepts for this purpose, including the orthogonality of coordinate parameters and the Pythagoras relation in the Kullback-Leibler divergence. Based on this orthogonality, we show a novel method for analyzing spike firing patterns by decomposing the interactions of neurons of various orders. As a result, purely pairwise, triple-wise, and higher-order interactions are singled out. We also demonstrate the benefits of our proposal by using several examples.

Download Full-text

Information geometry of divergence functions

Bulletin of the Polish Academy of Sciences Technical Sciences ◽

10.2478/v10175-010-0019-1 ◽

2010 ◽

Vol 58 (1) ◽

pp. 183-195 ◽

Cited By ~ 34

Author(s):

S. Amari ◽

A. Cichocki

Keyword(s):

Probability Distributions ◽

Flat Space ◽

Information Geometry ◽

Dual Pair ◽

Bregman Divergence ◽

Bregman Divergences ◽

Leibler Divergence ◽

Fisher Information Metric ◽

Information Metric ◽

Dually Flat Space

Information geometry of divergence functionsMeasures of divergence between two points play a key role in many engineering problems. One such measure is a distance function, but there are many important measures which do not satisfy the properties of the distance. The Bregman divergence, Kullback-Leibler divergence andf-divergence are such measures. In the present article, we study the differential-geometrical structure of a manifold induced by a divergence function. It consists of a Riemannian metric, and a pair of dually coupled affine connections, which are studied in information geometry. The class of Bregman divergences are characterized by a dually flat structure, which is originated from the Legendre duality. A dually flat space admits a generalized Pythagorean theorem. The class off-divergences, defined on a manifold of probability distributions, is characterized by information monotonicity, and the Kullback-Leibler divergence belongs to the intersection of both classes. Thef-divergence always gives the α-geometry, which consists of the Fisher information metric and a dual pair of ±α-connections. The α-divergence is a special class off-divergences. This is unique, sitting at the intersection of thef-divergence and Bregman divergence classes in a manifold of positive measures. The geometry derived from the Tsallisq-entropy and related divergences are also addressed.

Download Full-text

Extended Laplace principle for empirical measures of a Markov chain

Advances in Applied Probability ◽

10.1017/apr.2019.6 ◽

2019 ◽

Vol 51 (01) ◽

pp. 136-167 ◽

Cited By ~ 4

Author(s):

Stephan Eckstein

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Large Deviations ◽

Large Deviation ◽

Dual Pair ◽

Wasserstein Distance ◽

Large Deviations Principle ◽

Empirical Measures ◽

Leibler Divergence ◽

Laplace Principle

AbstractWe consider discrete-time Markov chains with Polish state space. The large deviations principle for empirical measures of a Markov chain can equivalently be stated in Laplace principle form, which builds on the convex dual pair of relative entropy (or Kullback– Leibler divergence) and cumulant generating functional f ↦ ln ʃ exp (f). Following the approach by Lacker (2016) in the independent and identically distributed case, we generalize the Laplace principle to a greater class of convex dual pairs. We present in depth one application arising from this extension, which includes large deviation results and a weak law of large numbers for certain robust Markov chains—similar to Markov set chains—where we model robustness via the first Wasserstein distance. The setting and proof of the extended Laplace principle are based on the weak convergence approach to large deviations by Dupuis and Ellis (2011).

Download Full-text

On the Geodesic Distance in Shapes K-means Clustering

Entropy ◽

10.3390/e20090647 ◽

2018 ◽

Vol 20 (9) ◽

pp. 647 ◽

Cited By ~ 5

Author(s):

Stefano Gattone ◽

Angela De Sanctis ◽

Stéphane Puechmorel ◽

Florence Nicol

Keyword(s):

Complex Shape ◽

Geodesic Distance ◽

Transportation Cost ◽

Information Geometry ◽

Wasserstein Distance ◽

Discriminative Power ◽

Statistical Manifold ◽

Probability Densities ◽

Different Shapes ◽

Rotationally Invariant

In this paper, the problem of clustering rotationally invariant shapes is studied and a solution using Information Geometry tools is provided. Landmarks of a complex shape are defined as probability densities in a statistical manifold. Then, in the setting of shapes clustering through a K-means algorithm, the discriminative power of two different shapes distances are evaluated. The first, derived from Fisher–Rao metric, is related with the minimization of information in the Fisher sense and the other is derived from the Wasserstein distance which measures the minimal transportation cost. A modification of the K-means algorithm is also proposed which allows the variances to vary not only among the landmarks but also among the clusters.

Download Full-text

Solving General Elliptical Mixture Models through an Approximate Wasserstein Manifold

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i04.5897 ◽

2020 ◽

Vol 34 (04) ◽

pp. 4658-4666

Author(s):

Shengxi Li ◽

Zeyang Yu ◽

Min Xiang ◽

Danilo Mandic

Keyword(s):

Mixture Models ◽

Finite Mixture Models ◽

Stable Solution ◽

Adaptive Method ◽

Wasserstein Distance ◽

Finite Mixture ◽

Constraint Solving ◽

Statistical Manifold ◽

Excellent Performance ◽

Leibler Divergence

We address the estimation problem for general finite mixture models, with a particular focus on the elliptical mixture models (EMMs). Compared to the widely adopted Kullback–Leibler divergence, we show that the Wasserstein distance provides a more desirable optimisation space. We thus provide a stable solution to the EMMs that is both robust to initialisations and reaches a superior optimum by adaptively optimising along a manifold of an approximate Wasserstein distance. To this end, we first provide a unifying account of computable and identifiable EMMs, which serves as a basis to rigorously address the underpinning optimisation problem. Due to a probability constraint, solving this problem is extremely cumbersome and unstable, especially under the Wasserstein distance. To relieve this issue, we introduce an efficient optimisation method on a statistical manifold defined under an approximate Wasserstein distance, which allows for explicit metrics and computable operations, thus significantly stabilising and improving the EMM estimation. We further propose an adaptive method to accelerate the convergence. Experimental results demonstrate the excellent performance of the proposed EMM solver.

Download Full-text

Complex-Valued Boltzmann Manifold

Complex-Valued Neural Networks ◽

10.4018/978-1-60566-214-5.ch001 ◽

2009 ◽

pp. 1-26 ◽

Cited By ~ 1

Author(s):

Masaki Kobayashi

Keyword(s):

Learning Algorithm ◽

Information Geometry ◽

Exponential Families ◽

Learning Models ◽

Boltzmann Machines ◽

Effective Learning ◽

Stochastic Learning ◽

Leibler Divergence ◽

Hidden Neurons ◽

Complex Valued

Information geometry is one of the most effective tools to investigate stochastic learning models. In it, stochastic learning models are regarded as manifolds in the view of differential geometry. Amari applied it to Boltzmann Machines, which is one of the stochastic learning models. The purpose of this chapter is to apply information geometry to complex-valued Boltzmann Machines. First, we construct the complex-valued Boltzmann Machines. Next, the author describes information geometry. The author will know some important notions of information geometry, exponential families, mixture families, Kullback-Leibler divergence, connections, geodesics, Fisher metrics, potential functions and so on. Finally, they apply information geometry to complex-valued Boltzmann Machines. They will investigate the structure of complex-valued Boltzmann manifold and know the notions of the connections and Fisher metric. Moreover we will get an effective learning algorithm, what is called em algorithm, for complex-valued Boltzmann machines with hidden neurons.

Download Full-text