scholarly journals The Poincaré-Shannon Machine: Statistical Physics and Machine Learning Aspects of Information Cohomology

Entropy ◽  
2019 ◽  
Vol 21 (9) ◽  
pp. 881 ◽  
Author(s):  
Pierre Baudot
Keyword(s):  
Machine Learning ◽  
Free Energy ◽  
Network Architecture ◽  
Statistical Physics ◽  
Mean Field ◽  
Minimum Free Energy ◽  
Energy Functional ◽  
Local Minima ◽  
Mean Field Model ◽  
Energy Principle

Previous works established that entropy is characterized uniquely as the first cohomology class in a topos and described some of its applications to the unsupervised classification of gene expression modules or cell types. These studies raised important questions regarding the statistical meaning of the resulting cohomology of information and its interpretation or consequences with respect to usual data analysis and statistical physics. This paper aims to present the computational methods of information cohomology and to propose its interpretations in terms of statistical physics and machine learning. In order to further underline the cohomological nature of information functions and chain rules, the computation of the cohomology in low degrees is detailed to show more directly that the k multivariate mutual information ( I k ) are ( k - 1 ) -coboundaries. The ( k - 1 ) -cocycles condition corresponds to I k = 0 , which generalizes statistical independence to arbitrary degree k. Hence, the cohomology can be interpreted as quantifying the statistical dependences and the obstruction to factorization. I develop the computationally tractable subcase of simplicial information cohomology represented by entropy H k and information I k landscapes and their respective paths, allowing investigation of Shannon’s information in the multivariate case without the assumptions of independence or of identically distributed variables. I give an interpretation of this cohomology in terms of phase transitions in a model of k-body interactions, holding both for statistical physics without mean field approximations and for data points. The I 1 components define a self-internal energy functional U k and ( - 1 ) k I k , k ≥ 2 components define the contribution to a free energy functional G k (the total correlation) of the k-body interactions. A basic mean field model is developed and computed on genetic data reproducing usual free energy landscapes with phase transition, sustaining the analogy of clustering with condensation. The set of information paths in simplicial structures is in bijection with the symmetric group and random processes, providing a trivial topological expression of the second law of thermodynamics. The local minima of free energy, related to conditional information negativity and conditional independence, characterize a minimum free energy complex. This complex formalizes the minimum free-energy principle in topology, provides a definition of a complex system and characterizes a multiplicity of local minima that quantifies the diversity observed in biology. I give an interpretation of this complex in terms of unsupervised deep learning where the neural network architecture is given by the chain complex and conclude by discussing future supervised applications.

scholarly journals Topological Information Data Analysis: Poincare-Shannon Machine and Statistical Physic of Finite Heterogeneous Systems

2018 ◽  
Author(s):  
Pierre Baudot ◽  
Monica Tapia ◽  
Jean-Marc Goaillard
Keyword(s):  
Statistical Physic ◽  
Free Energy ◽  
Statistical Tests ◽  
Arbitrary Dimension ◽  
Minimum Free Energy ◽  
Energy Functional ◽  
Information Storage ◽  
Local Minima ◽  
Energy Principle ◽  
Data Set

This paper establishes methods that quantify the structure of statistical interactions within a given data set using the characterization of information theory in cohomology by finite methods, and provides their expression in terms of statistical physic and machine learning. Following [1–3], we show directly that k multivariate mutual-informations (Ik) are k-coboundaries. The k-cocycles are given by Ik = 0, which generalize statistical independence to arbitrary dimension k. The topological approach allows to investigate Shannon’s information in the multivariate case without the assumptions of independent identically distributed variables. We develop the computationally tractable subcase of simplicial information cohomology represented by entropy Hk and information Ik landscapes. The I1 component defines a self-internal energy functional Uk, and (−1)k Ik,k≥2 components define the contribution to a free energy functional Gk of the k-body interactions. The set of information paths in simplicial structures is in bijection with the symmetric group and random processes, provides a topological expression of the 2nd law and points toward a discrete Noether theorem (1st law). The local minima of free-energy, related to conditional information negativity and the non-Shannonian cone of Yeung [4], characterize a minimum free energy complex. This complex formalizes the minimum free-energy principle in topology, provides a definition of a complex system, and characterizes a multiplicity of local minima that quantifies the diversity observed in biology. Finite data size effects and estimation bias severely constrain the effective computation of the information topology on data, and we provide simple statistical tests for the undersampling bias and for the k-dependences following [5]. We give an example of application of these methods to genetic expression and cell-type classification. The maximal positive Ik identifies the variables that co-vary the most in the population, whereas the minimal negative Ik identifies clusters and the variables that differentiate-segregate the most. The methods unravel biologically relevant I10 with a sample size of 41. It establishes generic methods to quantify the epigenetic information storage and a unified epigenetic unsupervised learning formalism.

scholarly journals FREE ENERGY IN THE GENERALIZED SHERRINGTON–KIRKPATRICK MEAN FIELD MODEL

2005 ◽  
Vol 17 (07) ◽  
pp. 793-857 ◽  
Author(s):  
DMITRY PANCHENKO
Keyword(s):  
Free Energy ◽  
Real Line ◽  
Prior Distribution ◽  
Field Model ◽  
Mean Field ◽  
Rigorous Proof ◽  
Bounded Subset ◽  
Mean Field Model ◽  
The Real ◽  
Parisi Formula

In [11], Talagrand gave a rigorous proof of the Parisi formula in the classical Sherrington–Kirkpatrick (SK) model. In this paper, we build upon the methodology developed in [11] and extend Talagrand's result to the class of SK type models in which the spins have arbitrary prior distribution on a bounded subset of the real line.

scholarly journals Artificial Swarm Intelligence and Cooperative Robotic Systems

2019 ◽  
Author(s):  
Iaroslav Omelianenko
Keyword(s):  
Free Energy ◽  
Evolutionary Algorithms ◽  
Swarm Intelligence ◽  
Intelligent Agents ◽  
Statistical Physics ◽  
Optimization Method ◽  
Modular System ◽  
Energy Principle ◽  
Free Energy Principle ◽  
Robotic Swarm

In this paper, we look at how Artificial Swarm Intelligence can evolve using evolutionary algorithms that try to minimize the sensory surprise of the system. We will show how to apply the free-energy principle, borrowed from statistical physics, to quantitatively describe the optimization method (sensory surprise minimization), which can be used to support lifelong learning. We provide our ideas about how to combine this optimization method with evolutionary algorithms in order to boost the development of specialized Artificial Neural Networks, which define the proprioceptive configuration of particular robotic units that are part of a swarm. We consider how optimization of the free-energy can promote the homeostasis of the swarm system, i.e. ensures that the system remains within its sensory boundaries throughout its active lifetime. We will show how complex distributed cognitive systems can be build in the form of hierarchical modular system, which consists of specialized micro-intelligent agents connected through information channels. We will also consider the co-evolution of various robotic swarm units, which can result in development of proprioception and a comprehensive awareness of the properties of the environment. And finally, we will give a brief outline of how this system can be implemented in practice and of our progress in this area.

The Role of the Diffusion Mechanism in Models of the Evolution of Microstructure During Phase Separation

1998 ◽  
Vol 529 ◽  
Author(s):  
T.T. Rautialnen ◽  
A.P. Sutton
Keyword(s):  
Monte Carlo ◽  
Phase Separation ◽  
Field Model ◽  
Diffusion Mechanism ◽  
Monte Carlo Model ◽  
Mean Field ◽  
Energy Functional ◽  
Square Lattice ◽  
Mean Field Model ◽  
Evolution Of Microstructure

AbstractWe have studied phase separation and subsequent coarsening of the microstructure in a two-dimensional square lattice using a stochastic Monte Carlo model and a deterministic mean field model. The differences and similarities between these approaches are discussed. We have found that a realistic diffusion mechanism through a vacancy motion in Monte Carlo simulations is cruicial in producing different coarsening mechanisms over a range of temperatures. This cannot be captured by the mean field model, in which the transformation is governed by the minimization of a free energy functional.

scholarly journals Mechanism for the Universal Pattern of Activity in Developing Neuronal Networks

2010 ◽  
Vol 103 (4) ◽  
pp. 2208-2221 ◽  
Author(s):  
Joël Tabak ◽  
Michael Mascagni ◽  
Richard Bertram
Keyword(s):  
Network Architecture ◽  
Mean Field ◽  
Cell Types ◽  
Synaptic Depression ◽  
General Mechanism ◽  
Silent Interval ◽  
Mean Field Model ◽  
Correlation Pattern ◽  
The Right ◽  
Episodic Activity

Spontaneous episodic activity is a fundamental mode of operation of developing networks. Surprisingly, the duration of an episode of activity correlates with the length of the silent interval that precedes it, but not with the interval that follows. Here we use a modeling approach to explain this characteristic, but thus far unexplained, feature of developing networks. Because the correlation pattern is observed in networks with different structures and components, a satisfactory model needs to generate the right pattern of activity regardless of the details of network architecture or individual cell properties. We thus developed simple models incorporating excitatory coupling between heterogeneous neurons and activity-dependent synaptic depression. These models robustly generated episodic activity with the correct correlation pattern. The correlation pattern resulted from episodes being triggered at random levels of recovery from depression while they terminated around the same level of depression. To explain this fundamental difference between episode onset and termination, we used a mean field model, where only average activity and average level of recovery from synaptic depression are considered. In this model, episode onset is highly sensitive to inputs. Thus noise resulting from random coincidences in the spike times of individual neurons led to the high variability at episode onset and to the observed correlation pattern. This work further shows that networks with widely different architectures, different cell types, and different functions all operate according to the same general mechanism early in their development.

scholarly journals Influence of Dislocations on the Spatial Variation of Microstructure in Martensites

2011 ◽  
Vol 465 ◽  
pp. 77-80
Author(s):  
Roman Gröger ◽  
Turab Lookman
Keyword(s):  
Free Energy ◽  
Local Density ◽  
Parent Phase ◽  
Mean Field ◽  
Continuum Theory ◽  
Energy Functional ◽  
Cubic Symmetry ◽  
Macroscopic Models ◽  
Multiphase Materials ◽  
Continuum Theory Of Dislocations

The continuum theory of dislocations, as developed predominantly by Kröner and Kosevich, views each dislocation as a source of incompatibility of strains. We show that this concept can be employed efficiently in the Landau free energy functional to develop a mean-field mesoscopic model of materials with dislocations. The order parameters that represent the distortion of the parent phase (often of cubic symmetry) are written in terms of elastic strains which are themselves coupled by incompatibility constraints. Since the “strength” of the incompatibility depends on the local density of dislocations, the presence of dislocations affects the evolution of the microstructure and vice versa. An advantage of this formulation is that long range anisotropic interactions between dislocations appear naturally in the formulation of the free energy. Owing to the distortion of the crystal structure around dislocations, their presence in multiphase materials causes heterogeneous nucleation of the product phase and thus also shifts of the transformation temperature. This novel field-theoretical approach is very convenient as it allows to bridge the gap in studying the behavior of materials at the length and time scales that are not attainable by atomistic or macroscopic models.

scholarly journals Transition pathways connecting crystals and quasicrystals

2021 ◽  
Vol 118 (49) ◽  
pp. e2106230118
Author(s):  
Jianyuan Yin ◽  
Kai Jiang ◽  
An-Chang Shi ◽  
Pingwen Zhang ◽  
Lei Zhang
Keyword(s):  
Phase Transition ◽  
Free Energy ◽  
Saddle Points ◽  
Energy Functional ◽  
Computational Method ◽  
Local Minima ◽  
Free Energy Functional ◽  
Transition Pathways ◽  
Critical Nuclei ◽  
Quasicrystalline State

Due to structural incommensurability, the emergence of a quasicrystal from a crystalline phase represents a challenge to computational physics. Here, the nucleation of quasicrystals is investigated by using an efficient computational method applied to a Landau free-energy functional. Specifically, transition pathways connecting different local minima of the Lifshitz–Petrich model are obtained by using the high-index saddle dynamics. Saddle points on these paths are identified as the critical nuclei of the 6-fold crystals and 12-fold quasicrystals. The results reveal that phase transitions between the crystalline and quasicrystalline phases could follow two possible pathways, corresponding to a one-stage phase transition and a two-stage phase transition involving a metastable lamellar quasicrystalline state, respectively.

Sign in / Sign up

Export Citation Format

Share Document