scholarly journals Efficient Learning in Boltzmann Machines Using Linear Response Theory

1998 ◽  
Vol 10 (5) ◽  
pp. 1137-1156 ◽  
Author(s):  
H. J. Kappen ◽  
F. B. Rodríguez
Keyword(s):  
Computational Complexity ◽  
Learning Process ◽  
Linear Response ◽  
Learning Algorithm ◽  
Mean Field Theory ◽  
Mean Field ◽  
Exact Algorithm ◽  
Linear Response Theory ◽  
Boltzmann Machines ◽  
Efficient Learning

The learning process in Boltzmann machines is computationally very expensive. The computational complexity of the exact algorithm is exponential in the number of neurons. We present a new approximate learning algorithm for Boltzmann machines, based on mean-field theory and the linear response theorem. The computational complexity of the algorithm is cubic in the number of neurons. In the absence of hidden units, we show how the weights can be directly computed from the fixed-point equation of the learning rules. Thus, in this case we do not need to use a gradient descent procedure for the learning process. We show that the solutions of this method are close to the optimal solutions and give a significant improvement when correlations play a significant role. Finally, we apply the method to a pattern completion task and show good performance for networks up to 100 neurons.

Learning in Boltzmann Trees

1994 ◽  
Vol 6 (6) ◽  
pp. 1174-1184 ◽  
Author(s):  
Lawrence Saul ◽  
Michael I. Jordan
Keyword(s):  
Supervised Learning ◽  
Gradient Descent ◽  
Learning Algorithm ◽  
Mean Field ◽  
Large Family ◽  
Weight Space ◽  
Hidden Symmetries ◽  
Boltzmann Machines ◽  
Field Annealing

We introduce a large family of Boltzmann machines that can be trained by standard gradient descent. The networks can have one or more layers of hidden units, with tree-like connectivity. We show how to implement the supervised learning algorithm for these Boltzmann machines exactly, without resort to simulated or mean-field annealing. The stochastic averages that yield the gradients in weight space are computed by the technique of decimation. We present results on the problems of N-bit parity and the detection of hidden symmetries.

scholarly journals Linear-Response Time-Dependent Embedded Mean-Field Theory

2017 ◽  
Vol 13 (9) ◽  
pp. 4216-4227 ◽  
Author(s):  
Feizhi Ding ◽  
Takashi Tsuchiya ◽  
Frederick R. Manby ◽  
Thomas F. Miller
Keyword(s):  
Field Theory ◽  
Response Time ◽  
Linear Response ◽  
Mean Field Theory ◽  
Mean Field ◽  
Time Dependent

Approximate Learning Algorithm in Boltzmann Machines

2009 ◽  
Vol 21 (11) ◽  
pp. 3130-3178 ◽  
Author(s):  
Muneki Yasuda ◽  
Kazuyuki Tanaka
Keyword(s):  
Markov Random Fields ◽  
Belief Propagation ◽  
Learning Algorithm ◽  
Learning Algorithms ◽  
Mean Field ◽  
Spin Model ◽  
Field Methods ◽  
Approximate Methods ◽  
Boltzmann Machines ◽  
Ising Spin

Boltzmann machines can be regarded as Markov random fields. For binary cases, they are equivalent to the Ising spin model in statistical mechanics. Learning systems in Boltzmann machines are one of the NP-hard problems. Thus, in general we have to use approximate methods to construct practical learning algorithms in this context. In this letter, we propose new and practical learning algorithms for Boltzmann machines by using the belief propagation algorithm and the linear response approximation, which are often referred as advanced mean field methods. Finally, we show the validity of our algorithm using numerical experiments.

scholarly journals FUNDAMENTAL CONSTRAINTS ON LINEAR RESPONSE THEORIES OF FERMI SUPERFLUIDS ABOVE AND BELOW Tc

2013 ◽  
Vol 27 (16) ◽  
pp. 1330010 ◽  
Author(s):  
HAO GUO ◽  
CHIH-CHUN CHIEN ◽  
YAN HE ◽  
K. LEVIN
Keyword(s):  
Gauge Invariance ◽  
Linear Response ◽  
Random Phase ◽  
Mean Field ◽  
Linear Response Theory ◽  
Bcs Theory ◽  
Einstein Condensation ◽  
Sum Rule ◽  
Many Body Theory ◽  
Theoretical Approaches

We present fundamental constraints required for a consistent linear response theory of fermionic superfluids and address temperatures both above and below the transition temperature Tc. We emphasize two independent constraints, one associated with gauge invariance (and the related Ward identity) and another associated with the compressibility sum rule, both of which are satisfied in strict BCS theory. However, we point out that it is the rare many body theory which satisfies both of these. Indeed, well studied quantum Hall systems and random-phase approximations to the electron gas are found to have difficulties with meeting these constraints. We summarize two distinct theoretical approaches which are, however, demonstrably compatible with gauge invariance and the compressibility sum rule. The first of these involves an extension of BCS theory to a mean field description of the BCS-Bose Einstein condensation crossover. The second is the simplest Nozieres Schmitt–Rink (NSR) treatment of pairing correlations in the normal state. As a point of comparison we focus on the compressibility κ of each and contrast the predictions above Tc. We note here that despite the compliance with sum rules, this NSR based scheme leads to an unphysical divergence in κ at the transition. Because of the delicacy of the various consistency requirements, the results of this paper suggest that avoiding this divergence may repair one problem while at the same time introducing others.

Explorations of the mean field theory learning algorithm

1989 ◽  
Vol 2 (6) ◽  
pp. 475-494 ◽  
Author(s):  
Carsten Peterson ◽  
Eric Hartman
Keyword(s):  
Field Theory ◽  
Learning Algorithm ◽  
Mean Field Theory ◽  
Mean Field ◽  
The Mean

Modification of linear response theory for mean-field approximations

1996 ◽  
Vol 54 (3) ◽  
pp. 2526-2530 ◽  
Author(s):  
Markus Hütter ◽  
Hans Christian Öttinger
Keyword(s):  
Linear Response ◽  
Mean Field ◽  
Linear Response Theory ◽  
Response Theory ◽  
Mean Field Approximations

Linear response theory for macroscopic observables in high-dimensional systems: when is it valid and when not?

2020 ◽  
Author(s):  
Georg Gottwald ◽  
Caroline Wormell
Keyword(s):  
Thermodynamic Limit ◽  
Linear Response ◽  
Chaotic Systems ◽  
Mean Field ◽  
Finite Size ◽  
Linear Response Theory ◽  
High Dimensional ◽  
Response Theory ◽  
Finite Dimensional ◽  
Comprehensive Picture

<p>The long-term average response of observables of chaotic systems to dynamical perturbations can often be predicted using linear response theory, but not all chaotic systems possess a linear response. Macroscopic observables of complex dissipative chaotic systems, however, are widely assumed to have a linear response even if the microscopic variables do not, but the mechanism for this is not well-understood.<br><br>We present a comprehensive picture for the linear response of macroscopic observables in high-dimensional coupled deterministic dynamical systems, where the coupling is via a mean field and the microscopic subsystems may or may not obey linear response theory. We derive stochastic reductions of the dynamics of these observables from statistics of the microscopic system, and provide conditions for linear response theory to hold in finite dimensional systems and in the thermodynamic limit. In particular, we show that for large systems of finite size, linear response is induced via self-generated noise.<br><br>We present examples in the thermodynamic limit where the macroscopic observable satisfies LRT, although the microscopic subsystems individually violate LRT, as well a converse example where the macroscopic observable does not satisfy LRT despite all microscopic subsystems satisfying LRT when uncoupled. This latter, maybe surprising, example is associated with emergent non-trivial dynamics of the macroscopic observable. We provide numerical evidence for our results on linear response as well as some analytical intuition.</p>

Sign in / Sign up

Export Citation Format

Share Document