scholarly journals Correctness of Belief Propagation in Gaussian Graphical Models of Arbitrary Topology

2001 ◽  
Vol 13 (10) ◽  
pp. 2173-2200 ◽  
Author(s):  
Yair Weiss ◽  
William T. Freeman
Keyword(s):  
Graphical Models ◽  
Turbo Codes ◽  
Markov Random Fields ◽  
Belief Propagation ◽  
Analytical Formula ◽  
Sufficient Conditions ◽  
Posterior Probabilities ◽  
Theoretical Understanding ◽  
Single Loop ◽  
Loopy Belief Propagation

Graphical models, such as Bayesian networks and Markov random fields, represent statistical dependencies of variables by a graph. Local “belief propagation” rules of the sort proposed by Pearl (1988) are guaranteed to converge to the correct posterior probabilities in singly connected graphs. Recently, good performance has been obtained by using these same rules on graphs with loops, a method we refer to as loopy belief propagation. Perhaps the most dramatic instance is the near Shannon-limit performance of “Turbo codes,” whose decoding algorithm is equivalent to loopy propagation. Except for the case of graphs with a single loop, there has been little theoretical understanding of loopy propagation. Here we analyze belief propagation in networks with arbitrary topologies when the nodes in the graph describe jointly gaussian random variables. We give an analytical formula relating the true posterior probabilities with those calculated using loopy propagation. We give sufficient conditions for convergence and show that when belief propagation converges, it gives the correct posterior means for all graph topologies, not just networks with a single loop. These results motivate using the powerful belief propagation algorithm in a broader class of networks and help clarify the empirical performance results.

Correctness of Local Probability Propagation in Graphical Models with Loops

2000 ◽  
Vol 12 (1) ◽  
pp. 1-41 ◽  
Author(s):  
Yair Weiss
Keyword(s):  
Graphical Models ◽  
Turbo Codes ◽  
Error Correcting Codes ◽  
Analytical Relationship ◽  
Theoretical Understanding ◽  
Single Loop ◽  
Joint Distributions ◽  
Probability Propagation ◽  
Local Propagation ◽  
Local Probability

Graphical models, such as Bayesian networks and Markov networks, represent joint distributions over a set of variables by means of a graph. When the graph is singly connected, local propagation rules of the sort proposed by Pearl (1988) are guaranteed to converge to the correct posterior probabilities. Recently a number of researchers have empirically demonstrated good performance of these same local propagation schemes on graphs with loops, but a theoretical understanding of this performance has yet to be achieved. For graphical models with a single loop, we derive an analytical relationship between the probabilities computed using local propagation and the correct marginals. Using this relationship we show a category of graphical models with loops for which local propagation gives rise to provably optimal maximum a posteriori assignments (although the computed marginals will be incorrect). We also show how nodes can use local information in the messages they receive in order to correct their computed marginals. We discuss how these results can be extended to graphical models with multiple loops and show simulation results suggesting that some properties of propagation on single-loop graphs may hold for a larger class of graphs. Specifically we discuss the implication of our results for understanding a class of recently proposed error-correcting codes known as turbo codes.

scholarly journals Marginal Inference in Continuous Markov Random Fields Using Mixtures

2019 ◽  
Vol 33 ◽  
pp. 7834-7841
Author(s):  
Yuanzhen Guo ◽  
Hao Xiong ◽  
Nicholas Ruozzi
Keyword(s):  
Graphical Models ◽  
Markov Random Fields ◽  
Belief Propagation ◽  
Current State ◽  
Special Cases ◽  
Bethe Free Energy ◽  
Practical Performance ◽  
Belief Propagation Algorithm ◽  
Loopy Belief Propagation ◽  
Marginal Inference

Exact marginal inference in continuous graphical models is computationally challenging outside of a few special cases. Existing work on approximate inference has focused on approximately computing the messages as part of the loopy belief propagation algorithm either via sampling methods or moment matching relaxations. In this work, we present an alternative family of approximations that, instead of approximating the messages, approximates the beliefs in the continuous Bethe free energy using mixture distributions. We show that these types of approximations can be combined with numerical quadrature to yield algorithms with both theoretical guarantees on the quality of the approximation and significantly better practical performance in a variety of applications that are challenging for current state-of-the-art methods.

scholarly journals A Spectral Approach to Analysing Belief Propagation for 3-Colouring

2009 ◽  
Vol 18 (6) ◽  
pp. 881-912 ◽  
Author(s):  
AMIN COJA-OGHLAN ◽  
ELCHANAN MOSSEL ◽  
DAN VILENCHIK
Keyword(s):  
Graphical Models ◽  
Message Passing ◽  
Graphical Model ◽  
Belief Propagation ◽  
Graph Colouring ◽  
Rigorous Result ◽  
Theoretical Understanding ◽  
Message Passing Algorithm ◽  
Survey Propagation ◽  
Graphical Structure

Belief propagation (BP) is a message-passing algorithm that computes the exact marginal distributions at every vertex of a graphical model without cycles. While BP is designed to work correctly on trees, it is routinely applied to general graphical models that may contain cycles, in which case neither convergence, nor correctness in the case of convergence is guaranteed. Nonetheless, BP has gained popularity as it seems to remain effective in many cases of interest, even when the underlying graph is ‘far’ from being a tree. However, the theoretical understanding of BP (and its new relative survey propagation) when applied to CSPs is poor.Contributing to the rigorous understanding of BP, in this paper we relate the convergence of BP to spectral properties of the graph. This encompasses a result for random graphs with a ‘planted’ solution; thus, we obtain the first rigorous result on BP for graph colouring in the case of a complex graphical structure (as opposed to trees). In particular, the analysis shows how belief propagation breaks the symmetry between the 3! possible permutations of the colour classes.

scholarly journals Block Belief Propagation for Parameter Learning in Markov Random Fields

2019 ◽  
Vol 33 ◽  
pp. 4448-4455
Author(s):  
You Lu ◽  
Zhiyuan Liu ◽  
Bert Huang
Keyword(s):  
Graphical Models ◽  
Random Fields ◽  
Markov Random Fields ◽  
Graphical Model ◽  
Belief Propagation ◽  
Traditional Learning ◽  
Training Methods ◽  
Iteration Complexity ◽  
Markov Random ◽  
Standard Training

Traditional learning methods for training Markov random fields require doing inference over all variables to compute the likelihood gradient. The iteration complexity for those methods therefore scales with the size of the graphical models. In this paper, we propose block belief propagation learning (BBPL), which uses block-coordinate updates of approximate marginals to compute approximate gradients, removing the need to compute inference on the entire graphical model. Thus, the iteration complexity of BBPL does not scale with the size of the graphs. We prove that the method converges to the same solution as that obtained by using full inference per iteration, despite these approximations, and we empirically demonstrate its scalability improvements over standard training methods.

scholarly journals On the Uniqueness of Loopy Belief Propagation Fixed Points

2004 ◽  
Vol 16 (11) ◽  
pp. 2379-2413 ◽  
Author(s):  
Tom Heskes
Keyword(s):  
Free Energy ◽  
Fixed Points ◽  
Belief Propagation ◽  
Sufficient Conditions ◽  
Free Energies ◽  
Strengthened Version ◽  
Bethe Free Energy ◽  
Loopy Belief Propagation

We derive sufficient conditions for the uniqueness of loopy belief propagation fixed points. These conditions depend on both the structure of the graph and the strength of the potentials and naturally extend those for convexity of the Bethe free energy. We compare them with (a strengthened version of) conditions derived elsewhere for pairwise potentials. We discuss possible implications for convergent algorithms, as well as for other approximate free energies.

Sign in / Sign up

Export Citation Format

Share Document