Correctness of Local Probability Propagation in Graphical Models with Loops

2000 ◽  
Vol 12 (1) ◽  
pp. 1-41 ◽  
Author(s):  
Yair Weiss
Keyword(s):  
Graphical Models ◽  
Turbo Codes ◽  
Error Correcting Codes ◽  
Analytical Relationship ◽  
Theoretical Understanding ◽  
Single Loop ◽  
Joint Distributions ◽  
Probability Propagation ◽  
Local Propagation ◽  
Local Probability

Graphical models, such as Bayesian networks and Markov networks, represent joint distributions over a set of variables by means of a graph. When the graph is singly connected, local propagation rules of the sort proposed by Pearl (1988) are guaranteed to converge to the correct posterior probabilities. Recently a number of researchers have empirically demonstrated good performance of these same local propagation schemes on graphs with loops, but a theoretical understanding of this performance has yet to be achieved. For graphical models with a single loop, we derive an analytical relationship between the probabilities computed using local propagation and the correct marginals. Using this relationship we show a category of graphical models with loops for which local propagation gives rise to provably optimal maximum a posteriori assignments (although the computed marginals will be incorrect). We also show how nodes can use local information in the messages they receive in order to correct their computed marginals. We discuss how these results can be extended to graphical models with multiple loops and show simulation results suggesting that some properties of propagation on single-loop graphs may hold for a larger class of graphs. Specifically we discuss the implication of our results for understanding a class of recently proposed error-correcting codes known as turbo codes.

scholarly journals Correctness of Belief Propagation in Gaussian Graphical Models of Arbitrary Topology

2001 ◽  
Vol 13 (10) ◽  
pp. 2173-2200 ◽  
Author(s):  
Yair Weiss ◽  
William T. Freeman
Keyword(s):  
Graphical Models ◽  
Turbo Codes ◽  
Markov Random Fields ◽  
Belief Propagation ◽  
Analytical Formula ◽  
Sufficient Conditions ◽  
Posterior Probabilities ◽  
Theoretical Understanding ◽  
Single Loop ◽  
Loopy Belief Propagation

Graphical models, such as Bayesian networks and Markov random fields, represent statistical dependencies of variables by a graph. Local “belief propagation” rules of the sort proposed by Pearl (1988) are guaranteed to converge to the correct posterior probabilities in singly connected graphs. Recently, good performance has been obtained by using these same rules on graphs with loops, a method we refer to as loopy belief propagation. Perhaps the most dramatic instance is the near Shannon-limit performance of “Turbo codes,” whose decoding algorithm is equivalent to loopy propagation. Except for the case of graphs with a single loop, there has been little theoretical understanding of loopy propagation. Here we analyze belief propagation in networks with arbitrary topologies when the nodes in the graph describe jointly gaussian random variables. We give an analytical formula relating the true posterior probabilities with those calculated using loopy propagation. We give sufficient conditions for convergence and show that when belief propagation converges, it gives the correct posterior means for all graph topologies, not just networks with a single loop. These results motivate using the powerful belief propagation algorithm in a broader class of networks and help clarify the empirical performance results.

scholarly journals A Robust and High Capacity Data Hiding Method for H.265/HEVC Compressed Videos with Block Roughness Measure and Error Correcting Techniques

Symmetry ◽  
2019 ◽  
Vol 11 (11) ◽  
pp. 1360
Author(s):  
Kusan Biswas
Keyword(s):  
Video Coding ◽  
Data Hiding ◽  
Turbo Codes ◽  
High Capacity ◽  
Error Correcting Codes ◽  
Human Vision ◽  
Embedding Technique ◽  
Payload Capacity ◽  
Capacity Data ◽  
Jensen Shannon Divergence

Recently, the H.265/HEVC video coding has been standardised by the ITU-T VCEG and the ISO/IEC MPEG. The improvements in H.265/HEVC video coding structure (CTU, motion compensation, inter- and intra-prediction, etc.) open up new possibilities to realise better data hiding algorithms in terms of capacity and robustness. In this paper, we propose a new data hiding method for HEVC videos. The proposed method embeds data in 4 × 4 and some selected larger transform units. As theory of Human Visual System suggests that human vision is less sensitive to change in uneven areas, relatively coarser blocks among the 8 × 8 and 16 × 16 blocks are selected as embedding destinations based on the proposed Jensen-Shannon Divergence and Second Moment (JSD-SM) block coarseness measure. In addition, the SME(1,3,7) embedding technique is able to embed three bits of message by modifying only one coefficient and therefore exhibits superior distortion performance. Furthermore, to achieve better robustness against re-compression attacks, BCH and Turbo error correcting codes have been used. Comparative studies of BCH and Turbo codes show the effectiveness of Turbo codes. Experimental results show that the proposed method achieves greater payload capacity and robustness than many existing state-of-the-art techniques without compromising on the visual quality.

scholarly journals A Spectral Approach to Analysing Belief Propagation for 3-Colouring

2009 ◽  
Vol 18 (6) ◽  
pp. 881-912 ◽  
Author(s):  
AMIN COJA-OGHLAN ◽  
ELCHANAN MOSSEL ◽  
DAN VILENCHIK
Keyword(s):  
Graphical Models ◽  
Message Passing ◽  
Graphical Model ◽  
Belief Propagation ◽  
Graph Colouring ◽  
Rigorous Result ◽  
Theoretical Understanding ◽  
Message Passing Algorithm ◽  
Survey Propagation ◽  
Graphical Structure

Belief propagation (BP) is a message-passing algorithm that computes the exact marginal distributions at every vertex of a graphical model without cycles. While BP is designed to work correctly on trees, it is routinely applied to general graphical models that may contain cycles, in which case neither convergence, nor correctness in the case of convergence is guaranteed. Nonetheless, BP has gained popularity as it seems to remain effective in many cases of interest, even when the underlying graph is ‘far’ from being a tree. However, the theoretical understanding of BP (and its new relative survey propagation) when applied to CSPs is poor.Contributing to the rigorous understanding of BP, in this paper we relate the convergence of BP to spectral properties of the graph. This encompasses a result for random graphs with a ‘planted’ solution; thus, we obtain the first rigorous result on BP for graph colouring in the case of a complex graphical structure (as opposed to trees). In particular, the analysis shows how belief propagation breaks the symmetry between the 3! possible permutations of the colour classes.

scholarly journals Linear Response Algorithms for Approximate Inference in Graphical Models

2004 ◽  
Vol 16 (1) ◽  
pp. 197-221 ◽  
Author(s):  
Max Welling ◽  
Yee Whye Teh
Keyword(s):  
Graphical Models ◽  
Random Fields ◽  
Linear Response ◽  
Belief Propagation ◽  
Probability Distributions ◽  
Gaussian Random Fields ◽  
Marginal Probability ◽  
Joint Distributions ◽  
Propagation Algorithm ◽  
New Algorithms

Belief propagation (BP) on cyclic graphs is an efficient algorithm for computing approximate marginal probability distributions over single nodes and neighboring nodes in the graph. However, it does not prescribe a way to compute joint distributions over pairs of distant nodes in the graph. In this article, we propose two new algorithms for approximating these pairwise probabilities, based on the linear response theorem. The first is a propagation algorithm that is shown to converge if BP converges to a stable fixed point. The second algorithm is based on matrix inversion. Applying these ideas to gaussian random fields, we derive a propagation algorithm for computing the inverse of a matrix.

scholarly journals Improving MIMO-OFDM decision-directed channel estimation by utilizing error-correcting codes

2009 ◽  
Vol 7 ◽  
pp. 83-88
Author(s):  
P. Beinschob ◽  
M. Lieberei ◽  
U. Zölzer
Keyword(s):  
Channel Estimation ◽  
Turbo Codes ◽  
Adaptive Filters ◽  
Multiple Input Multiple Output ◽  
Error Correcting Codes ◽  
Mimo Channel ◽  
Time Frequency ◽  
Coding Schemes ◽  
Channel Estimate ◽  
Input Multiple Output

Abstract. In this paper a decision-directed Multiple-Input Multiple-Output (MIMO) channel tracking algorithm is enhanced to raise the channel estimate accuracy. While DDCE is prone to error propagation the enhancement employs channel decoding in the tracking process. Therefore, a quantized block of symbols is checked on consistency via the channel decoder, possibly corrected and then used. This yields a more robust tracking of the channel in terms of bit error rate and improves the channel estimate under certain conditions. Equalization is performed to prove the feasibility of the obtained channel estimate. Therefore a combined signal consisting of data and pilot symbols is sent. Adaptive filters are applied to exploit correlations in time, frequency and spatial domain. By using good error-correcting coding schemes like Turbo Codes or Low Density Parity Check (LDPC) codes, adequate channel estimates can be acquired even at low signal to noise ratios (SNR). The proposed algorithm among two others is applied for channel estimation and equalization and results are compared.

scholarly journals Integrated World Modeling Theory (IWMT) Expanded: Implications for Theories of Consciousness and Artificial Intelligence

2021 ◽  
Author(s):  
Adam Safron
Keyword(s):  
Graphical Models ◽  
Turbo Codes ◽  
Intelligent Systems ◽  
Causal Structure ◽  
Predictive Processing ◽  
Active Inference ◽  
Energy Principle ◽  
World Modeling ◽  
Modeling Theory ◽  
Integrated Information

Integrated World Modeling Theory (IWMT) is a synthetic theory of consciousness that uses the Free Energy Principle and Active Inference (FEP-AI) framework to combine insights from Integrated Information Theory (IIT) and Global Neuronal Workspace Theory (GNWT). Here, I first review philosophical principles and neural systems contributing to IWMT’s integrative perspective. I then go on to describe predictive processing models of brains and their connections to machine learning architectures, with particular emphasis on autoencoders (perceptual and active inference), turbo-codes (establishment of shared latent spaces for multi-modal integration and inferential synergy), and graph neural networks (spatial and somatic modeling and control). Particular emphasis is placed on the hippocampal/entorhinal system, which may provide a source of high-level reasoning via predictive contrasting and generalized navigation, so affording multiple kinds of conscious access. Future directions for IIT and GNWT are considered by exploring ways in which modules and workspaces may be evaluated as both complexes of integrated information and arenas for iterated Bayesian model selection. Based on these considerations, I suggest novel ways in which integrated information might be estimated using concepts from probabilistic graphical models, flow networks, and game theory. Mechanistic and computational principles are also considered with respect to the ongoing debate between IIT and GNWT regarding the physical substrates of different kinds of conscious and unconscious phenomena. I further explore how these ideas might relate to the “Bayesian blur problem”, or how it is that a seemingly discrete experience can be generated from probabilistic modeling, with some consideration of analogies from quantum mechanics as potentially revealing different varieties of inferential dynamics. Finally, I go on to describe parallels between FEP-AI and theories of universal intelligence, including with respect to implications for the future of artificially intelligent systems. Particular emphasis is given to recurrent computation and its relationships with feedforward processing, including potential means of addressing critiques of causal structure theories based on network unfolding, and the seeming absurdity of conscious expander graphs (without cybernetic symbol grounding). While not quite solving the Hard problem, this article expands on IWMT as a unifying model of consciousness and the potential future evolution of minds.

scholarly journals On the Difference between the Information Bottleneck and the Deep Information Bottleneck

Entropy ◽  
2020 ◽  
Vol 22 (2) ◽  
pp. 131 ◽  
Author(s):  
Aleksander Wieczorek ◽  
Volker Roth
Keyword(s):  
Lower Bound ◽  
Mutual Information ◽  
Graphical Models ◽  
Neural Nets ◽  
Joint Distributions ◽  
Information Bottleneck ◽  
Special Cases ◽  
The Family ◽  
The Difference ◽  
Generative Modelling

Combining the information bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proven successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the deep variational information bottleneck and the assumptions needed for its derivation. The two assumed properties of the data, X and Y, and their latent representation T, take the form of two Markov chains T − X − Y and X − T − Y . Requiring both to hold during the optimisation process can be limiting for the set of potential joint distributions P ( X , Y , T ) . We, therefore, show how to circumvent this limitation by optimising a lower bound for the mutual information between T and Y: I ( T ; Y ) , for which only the latter Markov chain has to be satisfied. The mutual information I ( T ; Y ) can be split into two non-negative parts. The first part is the lower bound for I ( T ; Y ) , which is optimised in deep variational information bottleneck (DVIB) and cognate models in practice. The second part consists of two terms that measure how much the former requirement T − X − Y is violated. Finally, we propose interpreting the family of information bottleneck models as directed graphical models, and show that in this framework, the original and deep information bottlenecks are special cases of a fundamental IB model.

scholarly journals Performance analysis of highly improved hybrid turbo codes for 4G wireless networks

2019 ◽  
Vol 8 (4) ◽  
pp. 398
Author(s):  
M. Jose Raj ◽  
Dr. Sharmini Enoch
Keyword(s):  
Wireless Networks ◽  
Turbo Codes ◽  
Communication Systems ◽  
Turbo Code ◽  
Error Correcting Code ◽  
Low Complexity ◽  
Error Correcting Codes ◽  
4G Wireless ◽  
Low Complex ◽  
Digital Communication Systems

Efficient error correcting codes are essential in modern digital communication systems. Highly Improved Hybrid Turbo Code (HIHTC) is a low complex error and efficient error correcting code with excellentBit Error Rate (BER) which is comparable to Low Complexity Hybrid Turbo Codes (LCHTC), Improved Low Complexity Hybrid Turbo Codes (ILCHTC) and other Hybrid Turbo Codes. Rate 1/3 HIHTC shows a BER of 10-5 for E b/No of 1.7 dB which is closer to the E b/No of Improved Low Complexity Hybrid Turbo Codes. In this paper we analyze the performance of HIHTC in comparison with otherLow Complexity Hybrid Turbo Codes, for their performance in 4G and 5G wireless networks  

Sign in / Sign up

Export Citation Format

Share Document