scholarly journals Nonlinear Canonical Correlation Analysis:A Compressed Representation Approach

Entropy ◽  
2020 ◽  
Vol 22 (2) ◽  
pp. 208
Author(s):  
Amichai Painsky ◽  
Meir Feder ◽  
Naftali Tishby
Keyword(s):  
Canonical Correlation ◽  
Joint Probability ◽  
Optimal Solution ◽  
Rate Distortion ◽  
Linear Representation ◽  
Representation Learning ◽  
Information Theoretic ◽  
Compact Representations ◽  
Simple Implementation ◽  
Ace Algorithm

Canonical Correlation Analysis (CCA) is a linear representation learning method that seeks maximally correlated variables in multi-view data. Nonlinear CCA extends this notion to a broader family of transformations, which are more powerful in many real-world applications. Given the joint probability, the Alternating Conditional Expectation (ACE) algorithm provides an optimal solution to the nonlinear CCA problem. However, it suffers from limited performance and an increasing computational burden when only a finite number of samples is available. In this work, we introduce an information-theoretic compressed representation framework for the nonlinear CCA problem (CRCCA), which extends the classical ACE approach. Our suggested framework seeks compact representations of the data that allow a maximal level of correlation. This way, we control the trade-off between the flexibility and the complexity of the model. CRCCA provides theoretical bounds and optimality conditions, as we establish fundamental connections to rate-distortion theory, the information bottleneck and remote source coding. In addition, it allows a soft dimensionality reduction, as the compression level is determined by the mutual information between the original noisy data and the extracted signals. Finally, we introduce a simple implementation of the CRCCA framework, based on lattice quantization.

scholarly journals An Information-Theoretic Account of Semantic Interference in Word Production

2021 ◽  
Vol 12 ◽  
Author(s):  
Richard Futrell
Keyword(s):  
Mutual Information ◽  
Semantic Similarity ◽  
Rate Distortion ◽  
Word Production ◽  
Interference Effects ◽  
Semantic Interference ◽  
Information Theoretic ◽  
Human Data ◽  
Level Model ◽  
Computational Level

I present a computational-level model of semantic interference effects in online word production within a rate–distortion framework. I consider a bounded-rational agent trying to produce words. The agent's action policy is determined by maximizing accuracy in production subject to computational constraints. These computational constraints are formalized using mutual information. I show that semantic similarity-based interference among words falls out naturally from this setup, and I present a series of simulations showing that the model captures some of the key empirical patterns observed in Stroop and Picture–Word Interference paradigms, including comparisons to human data from previous experiments.

scholarly journals Tests for quantum contextuality in terms of Q-entropies

2014 ◽  
Vol 14 (11&12) ◽  
pp. 996-1013
Author(s):  
Alexey E. Rastegin
Keyword(s):  
Probability Distribution ◽  
Probability Distributions ◽  
Joint Probability ◽  
Joint Probability Distribution ◽  
Theoretic Approach ◽  
Entropic Inequalities ◽  
Information Theoretic ◽  
The Family ◽  
Information Theoretic Approach ◽  
New Inequalities

The information-theoretic approach to Bell's theorem is developed with use of the conditional $q$-entropies. The $q$-entropic measures fulfill many similar properties to the standard Shannon entropy. In general, both the locality and noncontextuality notions are usually treated with use of the so-called marginal scenarios. These hypotheses lead to the existence of a joint probability distribution, which marginalizes to all particular ones. Assuming the existence of such a joint probability distribution, we derive the family of inequalities of Bell's type in terms of conditional $q$-entropies for all $q\geq1$. Quantum violations of the new inequalities are exemplified within the Clauser--Horne--Shimony--Holt (CHSH) and Klyachko--Can--Binicio\v{g}lu--Shumovsky (KCBS) scenarios. An extension to the case of $n$-cycle scenario is briefly mentioned. The new inequalities with conditional $q$-entropies allow to expand a class of probability distributions, for which the nonlocality or contextuality can be detected within entropic formulation. The $q$-entropic inequalities can also be useful in analyzing cases with detection inefficiencies. Using two models of such a kind, we consider some potential advantages of the $q$-entropic formulation.

An Integrated Probability-Based Propulsor-Hull Matching Methodology

2013 ◽  
Vol 135 (1) ◽  
Author(s):  
Matthew R. Kramer ◽  
Michael R. Motley ◽  
Yin L. Young
Keyword(s):  
Structural Integrity ◽  
Surface Effect ◽  
Weighting Function ◽  
Full Range ◽  
Joint Probability ◽  
Design Procedure ◽  
Optimal Solution ◽  
Operating Conditions ◽  
Operational Space ◽  
Surface Effect Ship

Traditionally, designers of marine propulsors select a discrete number of critical design points for which to optimize the propulsor geometry. The design procedure carefully weighs the needs to be fuel efficient, to minimize cavitation, to maintain structural integrity, and to provide enough thrust to reach the desired speed, including the need to overcome any resistance humps. The current work proposes a new, alternative propulsor-hull matching methodology that is able to systematically consider the full range of expected operating conditions. A joint probability density function is used to represent the probabilistic operational space as a function of ship speed and sea state, and is used as a weighting function to select the propulsor that will minimize the annual fuel consumption while satisfying a set of constraints. The new probabilistic design approach is able to automatically locate the globally optimal solution by considering the probability of occurrence along with system performance characteristics. Hence, it is able to avoid the inherent ambiguity of selecting the proper design points. The proposed methodology is general to the design of marine propulsors for any type of vessel and engine system. It is applied in the current study for the sizing of waterjets for a surface effect ship.

scholarly journals Population Risk Improvement with Model Compression: An Information-Theoretic Approach

Entropy ◽  
2021 ◽  
Vol 23 (10) ◽  
pp. 1255
Author(s):  
Yuheng Bu ◽  
Weihao Gao ◽  
Shaofeng Zou ◽  
Venugopal V. Veeravalli
Keyword(s):  
Rate Distortion ◽  
Theoretic Approach ◽  
Generalization Error ◽  
Regularization Technique ◽  
Information Theoretic ◽  
Rate Distortion Theory ◽  
Model Compression ◽  
Empirical Risk ◽  
Deep Model ◽  
Information Theoretic Approach

It has been reported in many recent works on deep model compression that the population risk of a compressed model can be even better than that of the original model. In this paper, an information-theoretic explanation for this population risk improvement phenomenon is provided by jointly studying the decrease in the generalization error and the increase in the empirical risk that results from model compression. It is first shown that model compression reduces an information-theoretic bound on the generalization error, which suggests that model compression can be interpreted as a regularization technique to avoid overfitting. The increase in empirical risk caused by model compression is then characterized using rate distortion theory. These results imply that the overall population risk could be improved by model compression if the decrease in generalization error exceeds the increase in empirical risk. A linear regression example is presented to demonstrate that such a decrease in population risk due to model compression is indeed possible. Our theoretical results further suggest a way to improve a widely used model compression algorithm, i.e., Hessian-weighted K-means clustering, by regularizing the distance between the clustering centers. Experiments with neural networks are provided to validate our theoretical assertions.

scholarly journals What do we gain when tolerating loss? The information bottleneck, lossy compression, and detecting horizontal gene transfer

2021 ◽  
Author(s):  
Apurva Narechania ◽  
Rob DeSalle ◽  
Barun Mathema ◽  
Barry N Kreiswirth ◽  
Paul J Planet
Keyword(s):  
Gene Transfer ◽  
Horizontal Gene Transfer ◽  
Language Processing ◽  
Large Scale ◽  
Genetic Material ◽  
Joint Probability ◽  
Rate Distortion ◽  
Joint Probability Distribution ◽  
Model Free ◽  
Information Bottleneck

Most microbes have the capacity to acquire genetic material from their environment. Recombination of foreign DNA yields genomes that are, at least in part, incongruent with the vertical history of their species. Dominant approaches for detecting such horizontal gene transfer (HGT) and recombination are phylogenetic, requiring a painstaking series of analyses including sequence-based clustering, alignment, and phylogenetic tree reconstruction. Given the breakneck pace of genome sequencing, these traditional pan-genomic methods do not scale. Here we propose an alignment-free and tree-free technique based on the sequential information bottleneck (SIB), an optimization procedure designed to extract some portion of relevant information from one random variable conditioned on another. In our case, this joint probability distribution tabulates occurrence counts of k-mers with respect to their genomes of origin (the relevance information) with the expectation that HGT and recombination will create a strong signal that distinguishes certain sets of co-occuring k-mers. The technique is conceptualized as a rate-distortion problem. We measure distortion in the relevance information as k-mers are compressed into clusters based on their co-occurrence in the source genomes. This approach is similar to topic mining in the Natural Language Processing (NLP) literature. The result is model-free, unsupervised compression of k-mers into genomic topics that trace tracts of shared genome sequence whether vertically or horizontally acquired. We examine the performance of SIB on simulated data and on the known large-scale recombination event that formed the Staphylococcus aureus ST239 clade. We use this technique to detect recombined regions and recover the vertically inherited core genome with a fraction of the computing power required of current phylogenetic methods.

Real-time recognition of team behaviors by multisensory graph-embedded robot learning

2021 ◽  
pp. 027836492110431
Author(s):  
Brian Reily ◽  
Peng Gao ◽  
Fei Han ◽  
Hua Wang ◽  
Hao Zhang
Keyword(s):  
Real Time ◽  
Latent Variables ◽  
Optimization Problem ◽  
Autonomous Robots ◽  
Critical Role ◽  
Optimal Solution ◽  
Representation Learning ◽  
Robot Learning ◽  
Intent Recognition ◽  
Regularized Optimization

Awareness of team behaviors (e.g., individual activities and team intents) plays a critical role in human–robot teaming. Autonomous robots need to be aware of the overall intent of the team they are collaborating with in order to effectively aid their human peers or augment the team’s capabilities. Team intents encode the goal of the team, which cannot be simply identified from a collection of individual activities. Instead, teammate relationships must also be encoded for team intent recognition. In this article, we introduce a novel representation learning approach to recognizing team intent awareness in real-time, based upon both individual human activities and the relationship between human peers in the team. Our approach formulates the task of robot learning for team intent recognition as a joint regularized optimization problem, which encodes individual activities as latent variables and represents teammate relationships through graph embedding. In addition, we design a new algorithm to efficiently solve the formulated regularized optimization problem, which possesses a theoretical guarantee to converge to the optimal solution. To evaluate our approach’s performance on team intent recognition, we test our approach on a public benchmark group activity dataset and a multisensory underground search and rescue team behavior dataset newly collected from robots in an underground environment, as well as perform a proof-of-concept case study on a physical robot. The experimental results have demonstrated both the superior accuracy of our proposed approach and its suitability for real-time applications on mobile robots.

Sign in / Sign up

Export Citation Format

Share Document