scholarly journals Deep Learning for Channel Coding via Neural Mutual Information Estimation

2019 ◽  
Author(s):  
Rick Fritschek ◽  
Rafael F. Schaefer ◽  
Gerhard Wunder
Keyword(s):  
Deep Learning ◽  
Mutual Information ◽  
Channel Coding

scholarly journals Directed Data-Processing Inequalities for Systems with Feedback

Entropy ◽  
2021 ◽  
Vol 23 (5) ◽  
pp. 533
Author(s):  
Milan S. Derpich ◽  
Jan Østergaard
Keyword(s):  
Mutual Information ◽  
Data Processing ◽  
Channel Coding ◽  
Communication Channel ◽  
Closed Loop ◽  
Transmission Rate ◽  
Information Rate ◽  
Random Input ◽  
Directed Information ◽  
Stochastic Mapping

We present novel data-processing inequalities relating the mutual information and the directed information in systems with feedback. The internal deterministic blocks within such systems are restricted only to be causal mappings, but are allowed to be non-linear and time varying, and randomized by their own external random input, can yield any stochastic mapping. These randomized blocks can for example represent source encoders, decoders, or even communication channels. Moreover, the involved signals can be arbitrarily distributed. Our first main result relates mutual and directed information and can be interpreted as a law of conservation of information flow. Our second main result is a pair of data-processing inequalities (one the conditional version of the other) between nested pairs of random sequences entirely within the closed loop. Our third main result introduces and characterizes the notion of in-the-loop (ITL) transmission rate for channel coding scenarios in which the messages are internal to the loop. Interestingly, in this case the conventional notions of transmission rate associated with the entropy of the messages and of channel capacity based on maximizing the mutual information between the messages and the output turn out to be inadequate. Instead, as we show, the ITL transmission rate is the unique notion of rate for which a channel code attains zero error probability if and only if such an ITL rate does not exceed the corresponding directed information rate from messages to decoded messages. We apply our data-processing inequalities to show that the supremum of achievable (in the usual channel coding sense) ITL transmission rates is upper bounded by the supremum of the directed information rate across the communication channel. Moreover, we present an example in which this upper bound is attained. Finally, we further illustrate the applicability of our results by discussing how they make possible the generalization of two fundamental inequalities known in networked control literature.

SMILE: Mutual Information Learning for Integration of Single-cell Omics Data

2021 ◽  
Author(s):  
Yang Xu ◽  
Priyojit Das ◽  
Rachel Patton McCord
Keyword(s):  
Deep Learning ◽  
Mutual Information ◽  
Single Cell ◽  
Learning Algorithm ◽  
Cellular Systems ◽  
Supplementary Information ◽  
Omics Data ◽  
Learning Approaches ◽  
Rna Seq ◽  
Integrate Data

Abstract Motivation Deep learning approaches have empowered single-cell omics data analysis in many ways and generated new insights from complex cellular systems. As there is an increasing need for single cell omics data to be integrated across sources, types, and features of data, the challenges of integrating single-cell omics data are rising. Here, we present an unsupervised deep learning algorithm that learns discriminative representations for single-cell data via maximizing mutual information, SMILE (Single-cell Mutual Information Learning). Results Using a unique cell-pairing design, SMILE successfully integrates multi-source single-cell transcriptome data, removing batch effects and projecting similar cell types, even from different tissues, into the shared space. SMILE can also integrate data from two or more modalities, such as joint profiling technologies using single-cell ATAC-seq, RNA-seq, DNA methylation, Hi-C, and ChIP data. When paired cells are known, SMILE can integrate data with unmatched feature, such as genes for RNA-seq and genome wide peaks for ATAC-seq. Integrated representations learned from joint profiling technologies can then be used as a framework for comparing independent single source data. Supplementary information Supplementary data are available at Bioinformatics online. The source code of SMILE including analyses of key results in the study can be found at: https://github.com/rpmccordlab/SMILE.

Joint Channel Coding and Modulation via Deep Learning

2020 ◽  
Author(s):  
Yihan Jiang ◽  
Hyeji Kim ◽  
Himanshu Asnani ◽  
Sreeram Kannan ◽  
Sewoong Oh ◽  
...  
Keyword(s):  
Deep Learning ◽  
Channel Coding ◽  
Joint Channel

Mutual information for unsupervised deep learning image registration

2020 ◽  
Author(s):  
Bob D. de Vos ◽  
Bas van der Velden ◽  
Jörg Sander ◽  
Kenneth Gilhuijs ◽  
Marius Staring ◽  
...  
Keyword(s):  
Deep Learning ◽  
Image Registration ◽  
Mutual Information ◽  
Unsupervised Deep Learning

scholarly journals Dissecting Deep Learning Networks—Visualizing Mutual Information

Entropy ◽  
2018 ◽  
Vol 20 (11) ◽  
pp. 823 ◽  
Author(s):  
Hui Fang ◽  
Victoria Wang ◽  
Motonori Yamaguchi
Keyword(s):  
Deep Learning ◽  
Mutual Information ◽  
Network Performance ◽  
Classification Performance ◽  
Knowledge Gap ◽  
Accuracy Evaluation ◽  
Learning Networks ◽  
Theoretical Understanding ◽  
Effective Network ◽  
Visualization Techniques

Deep Learning (DL) networks are recent revolutionary developments in artificial intelligence research. Typical networks are stacked by groups of layers that are further composed of many convolutional kernels or neurons. In network design, many hyper-parameters need to be defined heuristically before training in order to achieve high cross-validation accuracies. However, accuracy evaluation from the output layer alone is not sufficient to specify the roles of the hidden units in associated networks. This results in a significant knowledge gap between DL’s wider applications and its limited theoretical understanding. To narrow the knowledge gap, our study explores visualization techniques to illustrate the mutual information (MI) in DL networks. The MI is a theoretical measurement, reflecting the relationship between two sets of random variables even if their relationship is highly non-linear and hidden in high-dimensional data. Our study aims to understand the roles of DL units in classification performance of the networks. Via a series of experiments using several popular DL networks, it shows that the visualization of MI and its change patterns between the input/output with the hidden layers and basic units can facilitate a better understanding of these DL units’ roles. Our investigation on network convergence suggests a more objective manner to potentially evaluate DL networks. Furthermore, the visualization provides a useful tool to gain insights into the network performance, and thus to potentially facilitate the design of better network architectures by identifying redundancy and less-effective network units.

Sign in / Sign up

Export Citation Format

Share Document