Collective dynamics of repeated inference in variational autoencoder rapidly find cluster structure

Abstract Deep neural networks are good at extracting low-dimensional subspaces (latent spaces) that represent the essential features inside a high-dimensional dataset. Deep generative models represented by variational autoencoders (VAEs) can generate and infer high-quality datasets, such as images. In particular, VAEs can eliminate the noise contained in an image by repeating the mapping between latent and data space. To clarify the mechanism of such denoising, we numerically analyzed how the activity pattern of trained networks changes in the latent space during inference. We considered the time development of the activity pattern for specific data as one trajectory in the latent space and investigated the collective behavior of these inference trajectories for many data. Our study revealed that when a cluster structure exists in the dataset, the trajectory rapidly approaches the center of the cluster. This behavior was qualitatively consistent with the concept retrieval reported in associative memory models. Additionally, the larger the noise contained in the data, the closer the trajectory was to a more global cluster. It was demonstrated that by increasing the number of the latent variables, the trend of the approach a cluster center can be enhanced, and the generalization ability of the VAE can be improved.

Download Full-text

A Flow-Based Deep Latent Variable Model for Speech Spectrogram Modeling and Enhancement

10.36227/techrxiv.12375284 ◽

2020 ◽

Author(s):

Aditya Arie Nugraha ◽

Kouhei Sekiguchi ◽

Kazuyoshi Yoshii

Keyword(s):

Speech Enhancement ◽

Latent Variables ◽

Latent Variable ◽

Generative Models ◽

Latent Variable Model ◽

Variable Model ◽

Variational Autoencoder ◽

Latent Representations ◽

Low Dimensional ◽

Better Than

This paper describes a deep latent variable model of speech power spectrograms and its application to semi-supervised speech enhancement with a deep speech prior. By integrating two major deep generative models, a variational autoencoder (VAE) and a normalizing flow (NF), in a mutually-beneficial manner, we formulate a flexible latent variable model called the NF-VAE that can extract low-dimensional latent representations from high-dimensional observations, akin to the VAE, and does not need to explicitly represent the distribution of the observations, akin to the NF. In this paper, we consider a variant of NF called the generative flow (GF a.k.a. Glow) and formulate a latent variable model called the GF-VAE. We experimentally show that the proposed GF-VAE is better than the standard VAE at capturing fine-structured harmonics of speech spectrograms, especially in the high-frequency range. A similar finding is also obtained when the GF-VAE and the VAE are used to generate speech spectrograms from latent variables randomly sampled from the standard Gaussian distribution. Lastly, when these models are used as speech priors for statistical multichannel speech enhancement, the GF-VAE outperforms the VAE and the GF.

Download Full-text

A Flow-Based Deep Latent Variable Model for Speech Spectrogram Modeling and Enhancement

10.36227/techrxiv.12375284.v1 ◽

2020 ◽

Author(s):

Aditya Arie Nugraha ◽

Kouhei Sekiguchi ◽

Kazuyoshi Yoshii

Keyword(s):

Speech Enhancement ◽

Latent Variables ◽

Latent Variable ◽

Generative Models ◽

Latent Variable Model ◽

Variable Model ◽

Variational Autoencoder ◽

Latent Representations ◽

Low Dimensional ◽

Better Than

Download Full-text

Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2021/408 ◽

2021 ◽

Author(s):

Dazhong Shen ◽

Chuan Qin ◽

Chao Wang ◽

Hengshu Zhu ◽

Enhong Chen ◽

...

Keyword(s):

Latent Variables ◽

State Of The Art ◽

Likelihood Estimation ◽

Generative Models ◽

Latent Space ◽

Variational Autoencoder ◽

Benchmark Datasets ◽

Classification Tasks ◽

Latent Representations ◽

Low Uncertainty

As one of the most popular generative models, Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. However, when the decoder network is sufficiently expressive, VAE may lead to posterior collapse; that is, uninformative latent representations may be learned. To this end, in this paper, we propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space, and thus the representation can be learned in a meaningful and compact manner. Specifically, we first theoretically demonstrate that it will result in better latent space with high diversity and low uncertainty awareness by controlling the distribution of posterior’s parameters across the whole data accordingly. Then, without the introduction of new loss terms or modifying training strategies, we propose to exploit Dropout on the variances and Batch-Normalization on the means simultaneously to regularize their distributions implicitly. Furthermore, to evaluate the generalization effect, we also exploit DU-VAE for inverse autoregressive flow based-VAE (VAE-IAF) empirically. Finally, extensive experiments on three benchmark datasets clearly show that our approach can outperform state-of-the-art baselines on both likelihood estimation and underlying classification tasks.

Download Full-text

Interpretable Variational Graph Autoencoder with Noninformative Prior

Future Internet ◽

10.3390/fi13020051 ◽

2021 ◽

Vol 13 (2) ◽

pp. 51

Author(s):

Lili Sun ◽

Xueyan Liu ◽

Min Zhao ◽

Bo Yang

Keyword(s):

Latent Variables ◽

Latent Variable ◽

Expert Knowledge ◽

Structural Information ◽

Standard Normal Distribution ◽

Noninformative Prior ◽

Latent Space ◽

Distribution Parameters ◽

Standard Normal ◽

Low Dimensional

Variational graph autoencoder, which can encode structural information and attribute information in the graph into low-dimensional representations, has become a powerful method for studying graph-structured data. However, most existing methods based on variational (graph) autoencoder assume that the prior of latent variables obeys the standard normal distribution which encourages all nodes to gather around 0. That leads to the inability to fully utilize the latent space. Therefore, it becomes a challenge on how to choose a suitable prior without incorporating additional expert knowledge. Given this, we propose a novel noninformative prior-based interpretable variational graph autoencoder (NPIVGAE). Specifically, we exploit the noninformative prior as the prior distribution of latent variables. This prior enables the posterior distribution parameters to be almost learned from the sample data. Furthermore, we regard each dimension of a latent variable as the probability that the node belongs to each block, thereby improving the interpretability of the model. The correlation within and between blocks is described by a block–block correlation matrix. We compare our model with state-of-the-art methods on three real datasets, verifying its effectiveness and superiority.

Download Full-text

Predictive learning as a network mechanism for extracting low-dimensional latent space representations

Nature Communications ◽

10.1038/s41467-021-21696-1 ◽

2021 ◽

Vol 12 (1) ◽

Author(s):

Stefano Recanatesi ◽

Matthew Farrell ◽

Guillaume Lajoie ◽

Sophie Deneve ◽

Mattia Rigotti ◽

...

Keyword(s):

Latent Variables ◽

Latent Structure ◽

Network Activity ◽

Semantic Organization ◽

Sequential Processing ◽

Predictive Learning ◽

Neural Representations ◽

Latent Space ◽

Low Dimensional ◽

Sensory Prediction

AbstractArtificial neural networks have recently achieved many successes in solving sequential processing and planning tasks. Their success is often ascribed to the emergence of the task’s low-dimensional latent structure in the network activity – i.e., in the learned neural representations. Here, we investigate the hypothesis that a means for generating representations with easily accessed low-dimensional latent structure, possibly reflecting an underlying semantic organization, is through learning to predict observations about the world. Specifically, we ask whether and when network mechanisms for sensory prediction coincide with those for extracting the underlying latent variables. Using a recurrent neural network model trained to predict a sequence of observations we show that network dynamics exhibit low-dimensional but nonlinearly transformed representations of sensory inputs that map the latent structure of the sensory environment. We quantify these results using nonlinear measures of intrinsic dimensionality and linear decodability of latent variables, and provide mathematical arguments for why such useful predictive representations emerge. We focus throughout on how our results can aid the analysis and interpretation of experimental data.

Download Full-text

Extracting Low-Dimensional Latent Structure from Time Series in the Presence of Delays

Neural Computation ◽

10.1162/neco_a_00759 ◽

2015 ◽

Vol 27 (9) ◽

pp. 1825-1856 ◽

Cited By ~ 19

Author(s):

Karthik C. Lakshmanan ◽

Patrick T. Sadtler ◽

Elizabeth C. Tyler-Kabara ◽

Aaron P. Batista ◽

Byron M. Yu

Keyword(s):

Time Series ◽

Time Delay ◽

Motor Cortex ◽

Dimensionality Reduction ◽

Gaussian Process ◽

Latent Variables ◽

Time Delays ◽

High Dimensional ◽

Latent Space ◽

Low Dimensional

Noisy, high-dimensional time series observations can often be described by a set of low-dimensional latent variables. Commonly used methods to extract these latent variables typically assume instantaneous relationships between the latent and observed variables. In many physical systems, changes in the latent variables manifest as changes in the observed variables after time delays. Techniques that do not account for these delays can recover a larger number of latent variables than are present in the system, thereby making the latent representation more difficult to interpret. In this work, we introduce a novel probabilistic technique, time-delay gaussian-process factor analysis (TD-GPFA), that performs dimensionality reduction in the presence of a different time delay between each pair of latent and observed variables. We demonstrate how using a gaussian process to model the evolution of each latent variable allows us to tractably learn these delays over a continuous domain. Additionally, we show how TD-GPFA combines temporal smoothing and dimensionality reduction into a common probabilistic framework. We present an expectation/conditional maximization either (ECME) algorithm to learn the model parameters. Our simulations demonstrate that when time delays are present, TD-GPFA is able to correctly identify these delays and recover the latent space. We then applied TD-GPFA to the activity of tens of neurons recorded simultaneously in the macaque motor cortex during a reaching task. TD-GPFA is able to better describe the neural activity using a more parsimonious latent space than GPFA, a method that has been used to interpret motor cortex data but does not account for time delays. More broadly, TD-GPFA can help to unravel the mechanisms underlying high-dimensional time series data by taking into account physical delays in the system.

Download Full-text

Weakly Supervised Disentanglement by Pairwise Similarities

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i04.5754 ◽

2020 ◽

Vol 34 (04) ◽

pp. 3495-3502 ◽

Cited By ~ 1

Author(s):

Junxiang Chen ◽

Kayhan Batmanghelich

Keyword(s):

Real World ◽

Latent Variables ◽

Generative Models ◽

Experimental Results ◽

New Method ◽

Weak Supervision ◽

Real World Problem ◽

Variational Autoencoder ◽

Weakly Supervised

Recently, researches related to unsupervised disentanglement learning with deep generative models have gained substantial popularity. However, without introducing supervision, there is no guarantee that the factors of interest can be successfully recovered (Locatello et al. 2018). Motivated by a real-world problem, we propose a setting where the user introduces weak supervision by providing similarities between instances based on a factor to be disentangled. The similarity is provided as either a binary (yes/no) or real-valued label describing whether a pair of instances are similar or not. We propose a new method for weakly supervised disentanglement of latent variables within the framework of Variational Autoencoder. Experimental results demonstrate that utilizing weak supervision improves the performance of the disentanglement method substantially.

Download Full-text

A generative modeling approach for interpreting population-level variability in brain structure

10.1101/2020.06.04.134635 ◽

2020 ◽

Author(s):

Ran Liu ◽

Cem Subakan ◽

Aishwarya H. Balwani ◽

Jennifer Whitesell ◽

Julie Harris ◽

...

Keyword(s):

Brain Structure ◽

Latent Variables ◽

Population Level ◽

Individual Variability ◽

Neural Structure ◽

Modeling Approach ◽

Structured Perturbations ◽

Generative Modeling ◽

Latent Space ◽

Low Dimensional

AbstractUnderstanding how neural structure varies across individuals is critical for characterizing the effects of disease, learning, and aging on the brain. However, disentangling the different factors that give rise to individual variability is still an outstanding challenge. In this paper, we introduce a deep generative modeling approach to find different modes of variation across many individuals. To do this, we start by training a variational autoencoder on a collection of auto-fluorescence images from a little over 1,700 mouse brains at 25 micron resolution. To then tap into the learned factors and validate the model’s expressiveness, we developed a novel bi-directional technique to interpret the latent space–by making structured perturbations to both, the high-dimensional inputs of the network, as well as the low-dimensional latent variables in its bottleneck. Our results demonstrate that through coupling generative modeling frameworks with structured perturbations, it is possible to probe the latent space to provide insights into the representations of brain structure formed in deep neural networks.

Download Full-text

Synthesizing Designs With Interpart Dependencies Using Hierarchical Generative Adversarial Networks

Journal of Mechanical Design ◽

10.1115/1.4044076 ◽

2019 ◽

Vol 141 (11) ◽

Cited By ~ 3

Author(s):

Wei Chen ◽

Mark Fuge

Keyword(s):

Constraint Satisfaction ◽

Real World ◽

Generative Models ◽

Dependency Graph ◽

Generative Adversarial Networks ◽

Engineering Systems ◽

Adversarial Networks ◽

Latent Space ◽

Branching Factor ◽

Low Dimensional

Abstract Real-world designs usually consist of parts with interpart dependencies, i.e., the geometry of one part is dependent on one or multiple other parts. We can represent such dependency in a part dependency graph. This paper presents a method for synthesizing these types of hierarchical designs using generative models learned from examples. It decomposes the problem of synthesizing the whole design into synthesizing each part separately but keeping the interpart dependencies satisfied. Specifically, this method constructs multiple generative models, the interaction of which is based on the part dependency graph. We then use the trained generative models to synthesize or explore each part design separately via a low-dimensional latent representation, conditioned on the corresponding parent part(s). We verify our model on multiple design examples with different interpart dependencies. We evaluate our model by analyzing the constraint satisfaction performance, the synthesis quality, the latent space quality, and the effects of part dependency depth and branching factor. This paper’s techniques for capturing dependencies among parts lay the foundation for learned generative models to extend to more realistic engineering systems where such relationships are widespread.

Download Full-text

Context-Aware Content Generation for Virtual Environments

Volume 1B: 36th Computers and Information in Engineering Conference ◽

10.1115/detc2016-59997 ◽

2016 ◽

Author(s):

Andrew Brock ◽

Theodore Lim ◽

J. M. Ritchie ◽

Nick Weston

Keyword(s):

Large Scale ◽

Dimensional Space ◽

Context Aware ◽

3 Dimensional ◽

Latent Space ◽

Variational Autoencoder ◽

Computationally Intensive ◽

Expert Input ◽

Low Dimensional ◽

Content Generation

Large scale scene generation is a computationally intensive operation, and added complexities arise when dynamic content generation is required. We propose a system capable of generating virtual content from non-expert input. The proposed system uses a 3-dimensional variational autoencoder to interactively generate new virtual objects by interpolating between extant objects in a learned low-dimensional space, as well as by randomly sampling in that space. We present an interface that allows a user to intuitively explore the latent manifold, taking advantage of the network’s ability to perform algebra in the latent space to help infer context and generalize to previously unseen inputs.

Download Full-text