scholarly journals Predictive learning as a network mechanism for extracting low-dimensional latent space representations

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Stefano Recanatesi ◽  
Matthew Farrell ◽  
Guillaume Lajoie ◽  
Sophie Deneve ◽  
Mattia Rigotti ◽  
...  
Keyword(s):  
Latent Variables ◽  
Latent Structure ◽  
Network Activity ◽  
Semantic Organization ◽  
Sequential Processing ◽  
Predictive Learning ◽  
Neural Representations ◽  
Latent Space ◽  
Low Dimensional ◽  
Sensory Prediction

AbstractArtificial neural networks have recently achieved many successes in solving sequential processing and planning tasks. Their success is often ascribed to the emergence of the task’s low-dimensional latent structure in the network activity – i.e., in the learned neural representations. Here, we investigate the hypothesis that a means for generating representations with easily accessed low-dimensional latent structure, possibly reflecting an underlying semantic organization, is through learning to predict observations about the world. Specifically, we ask whether and when network mechanisms for sensory prediction coincide with those for extracting the underlying latent variables. Using a recurrent neural network model trained to predict a sequence of observations we show that network dynamics exhibit low-dimensional but nonlinearly transformed representations of sensory inputs that map the latent structure of the sensory environment. We quantify these results using nonlinear measures of intrinsic dimensionality and linear decodability of latent variables, and provide mathematical arguments for why such useful predictive representations emerge. We focus throughout on how our results can aid the analysis and interpretation of experimental data.

scholarly journals Predictive learning as a network mechanism for extracting low-dimensional latent space representations

2018 ◽  
Author(s):  
Stefano Recanatesi ◽  
Matthew Farrell ◽  
Guillaume Lajoie ◽  
Sophie Deneve ◽  
Mattia Rigotti ◽  
...  
Keyword(s):  
Latent Variables ◽  
Latent Structure ◽  
Network Activity ◽  
Semantic Organization ◽  
Sequential Processing ◽  
Predictive Learning ◽  
Neural Representations ◽  
Latent Space ◽  
Low Dimensional ◽  
Sensory Prediction

Artificial neural networks have recently achieved many successes in solving sequential processing and planning tasks. Their success is often ascribed to the emergence of the task’s low-dimensional latent structure in the network activity – i.e., in the learned neural representations. Here, we investigate the hypothesis that a means for generating representations with easily accessed low-dimensional latent structure, possibly reflecting an underlying semantic organization, is through learning to predict observations about the world. Specifically, we ask whether and when network mechanisms for sensory prediction coincide with those for extracting the underlying latent variables. Using a recurrent neural network model trained to predict a sequence of observations we show that network dynamics exhibit low-dimensional but nonlinearly transformed representations of sensory inputs that map the latent structure of the sensory environment. We quantify these results using nonlinear measures of intrinsic dimensionality and linear decodability of latent variables, and provide mathematical arguments for why such useful predictive representations emerge. We focus throughout on how our results can aid the analysis and interpretation of experimental data.

scholarly journals Interpretable Variational Graph Autoencoder with Noninformative Prior

2021 ◽  
Vol 13 (2) ◽  
pp. 51
Author(s):  
Lili Sun ◽  
Xueyan Liu ◽  
Min Zhao ◽  
Bo Yang
Keyword(s):  
Latent Variables ◽  
Latent Variable ◽  
Expert Knowledge ◽  
Structural Information ◽  
Standard Normal Distribution ◽  
Noninformative Prior ◽  
Latent Space ◽  
Distribution Parameters ◽  
Standard Normal ◽  
Low Dimensional

Variational graph autoencoder, which can encode structural information and attribute information in the graph into low-dimensional representations, has become a powerful method for studying graph-structured data. However, most existing methods based on variational (graph) autoencoder assume that the prior of latent variables obeys the standard normal distribution which encourages all nodes to gather around 0. That leads to the inability to fully utilize the latent space. Therefore, it becomes a challenge on how to choose a suitable prior without incorporating additional expert knowledge. Given this, we propose a novel noninformative prior-based interpretable variational graph autoencoder (NPIVGAE). Specifically, we exploit the noninformative prior as the prior distribution of latent variables. This prior enables the posterior distribution parameters to be almost learned from the sample data. Furthermore, we regard each dimension of a latent variable as the probability that the node belongs to each block, thereby improving the interpretability of the model. The correlation within and between blocks is described by a block–block correlation matrix. We compare our model with state-of-the-art methods on three real datasets, verifying its effectiveness and superiority.

scholarly journals Collective dynamics of repeated inference in variational autoencoder rapidly find cluster structure

2020 ◽  
Vol 10 (1) ◽  
Author(s):  
Yoshihiro Nagano ◽  
Ryo Karakida ◽  
Masato Okada
Keyword(s):  
Activity Pattern ◽  
Latent Variables ◽  
Cluster Structure ◽  
Generative Models ◽  
Cluster Center ◽  
Specific Data ◽  
Global Cluster ◽  
Latent Space ◽  
Variational Autoencoder ◽  
Low Dimensional

Abstract Deep neural networks are good at extracting low-dimensional subspaces (latent spaces) that represent the essential features inside a high-dimensional dataset. Deep generative models represented by variational autoencoders (VAEs) can generate and infer high-quality datasets, such as images. In particular, VAEs can eliminate the noise contained in an image by repeating the mapping between latent and data space. To clarify the mechanism of such denoising, we numerically analyzed how the activity pattern of trained networks changes in the latent space during inference. We considered the time development of the activity pattern for specific data as one trajectory in the latent space and investigated the collective behavior of these inference trajectories for many data. Our study revealed that when a cluster structure exists in the dataset, the trajectory rapidly approaches the center of the cluster. This behavior was qualitatively consistent with the concept retrieval reported in associative memory models. Additionally, the larger the noise contained in the data, the closer the trajectory was to a more global cluster. It was demonstrated that by increasing the number of the latent variables, the trend of the approach a cluster center can be enhanced, and the generalization ability of the VAE can be improved.

scholarly journals Extracting Low-Dimensional Latent Structure from Time Series in the Presence of Delays

2015 ◽  
Vol 27 (9) ◽  
pp. 1825-1856 ◽  
Author(s):  
Karthik C. Lakshmanan ◽  
Patrick T. Sadtler ◽  
Elizabeth C. Tyler-Kabara ◽  
Aaron P. Batista ◽  
Byron M. Yu
Keyword(s):  
Time Series ◽  
Time Delay ◽  
Motor Cortex ◽  
Dimensionality Reduction ◽  
Gaussian Process ◽  
Latent Variables ◽  
Time Delays ◽  
High Dimensional ◽  
Latent Space ◽  
Low Dimensional

Noisy, high-dimensional time series observations can often be described by a set of low-dimensional latent variables. Commonly used methods to extract these latent variables typically assume instantaneous relationships between the latent and observed variables. In many physical systems, changes in the latent variables manifest as changes in the observed variables after time delays. Techniques that do not account for these delays can recover a larger number of latent variables than are present in the system, thereby making the latent representation more difficult to interpret. In this work, we introduce a novel probabilistic technique, time-delay gaussian-process factor analysis (TD-GPFA), that performs dimensionality reduction in the presence of a different time delay between each pair of latent and observed variables. We demonstrate how using a gaussian process to model the evolution of each latent variable allows us to tractably learn these delays over a continuous domain. Additionally, we show how TD-GPFA combines temporal smoothing and dimensionality reduction into a common probabilistic framework. We present an expectation/conditional maximization either (ECME) algorithm to learn the model parameters. Our simulations demonstrate that when time delays are present, TD-GPFA is able to correctly identify these delays and recover the latent space. We then applied TD-GPFA to the activity of tens of neurons recorded simultaneously in the macaque motor cortex during a reaching task. TD-GPFA is able to better describe the neural activity using a more parsimonious latent space than GPFA, a method that has been used to interpret motor cortex data but does not account for time delays. More broadly, TD-GPFA can help to unravel the mechanisms underlying high-dimensional time series data by taking into account physical delays in the system.

scholarly journals A generative modeling approach for interpreting population-level variability in brain structure

2020 ◽  
Author(s):  
Ran Liu ◽  
Cem Subakan ◽  
Aishwarya H. Balwani ◽  
Jennifer Whitesell ◽  
Julie Harris ◽  
...  
Keyword(s):  
Brain Structure ◽  
Latent Variables ◽  
Population Level ◽  
Individual Variability ◽  
Neural Structure ◽  
Modeling Approach ◽  
Structured Perturbations ◽  
Generative Modeling ◽  
Latent Space ◽  
Low Dimensional

AbstractUnderstanding how neural structure varies across individuals is critical for characterizing the effects of disease, learning, and aging on the brain. However, disentangling the different factors that give rise to individual variability is still an outstanding challenge. In this paper, we introduce a deep generative modeling approach to find different modes of variation across many individuals. To do this, we start by training a variational autoencoder on a collection of auto-fluorescence images from a little over 1,700 mouse brains at 25 micron resolution. To then tap into the learned factors and validate the model’s expressiveness, we developed a novel bi-directional technique to interpret the latent space–by making structured perturbations to both, the high-dimensional inputs of the network, as well as the low-dimensional latent variables in its bottleneck. Our results demonstrate that through coupling generative modeling frameworks with structured perturbations, it is possible to probe the latent space to provide insights into the representations of brain structure formed in deep neural networks.

scholarly journals VEGA is an interpretable generative model for inferring biological network activity in single-cell transcriptomics

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Lucas Seninge ◽  
Ioannis Anastopoulos ◽  
Hongxu Ding ◽  
Joshua Stuart
Keyword(s):  
Latent Variables ◽  
Regulatory Networks ◽  
Network Activity ◽  
Specific Response ◽  
Cell Type ◽  
State Identity ◽  
Latent Space ◽  
The Status ◽  
Transcriptomics Data ◽  
Gene Modules

AbstractDeep learning architectures such as variational autoencoders have revolutionized the analysis of transcriptomics data. However, the latent space of these variational autoencoders offers little to no interpretability. To provide further biological insights, we introduce a novel sparse Variational Autoencoder architecture, VEGA (VAE Enhanced by Gene Annotations), whose decoder wiring mirrors user-provided gene modules, providing direct interpretability to the latent variables. We demonstrate the performance of VEGA in diverse biological contexts using pathways, gene regulatory networks and cell type identities as the gene modules that define its latent space. VEGA successfully recapitulates the mechanism of cellular-specific response to treatments, the status of master regulators as well as jointly revealing the cell type and cellular state identity in developing cells. We envision the approach could serve as an explanatory biological model for development and drug treatment experiments.

scholarly journals Evolving Matrix-Factorization-Based Collaborative Filtering Using Genetic Programming

2020 ◽  
Vol 10 (2) ◽  
pp. 675 ◽  
Author(s):  
Raúl Lara-Cabrera ◽  
Ángel González-Prieto ◽  
Fernando Ortega ◽  
Jesús Bobadilla
Keyword(s):  
Genetic Programming ◽  
Collaborative Filtering ◽  
Recommender Systems ◽  
Matrix Factorization ◽  
Latent Structure ◽  
Matrix Factorizations ◽  
Evolutionary Approach ◽  
Similar Performance ◽  
Latent Space ◽  
Low Dimensional

Recommender systems aim to estimate the judgment or opinion that a user might offer to an item. Matrix-factorization-based collaborative filtering typifies both users and items as vectors of factors inferred from item rating patterns. This method finds latent structure in the data, assuming that observations lie close to a low-dimensional latent space. However, matrix factorizations have been traditionally designed by hand. Here, we present Evolutionary Matrix Factorization (EMF), an evolutionary approach that automatically generates matrix factorizations aimed at improving the performance of recommender systems. Initial experiments using this approach show that EMF generally outperforms baseline methods when applied to MovieLens and FilmTrust datasets, having a similar performance to those baselines on the worst cases. These results serve as an incentive to continue improving and studying the application of an evolutionary approach to collaborative filtering based on Matrix Factorization.

scholarly journals Representing Closed Transformation Paths in Encoded Network Latent Space

2020 ◽  
Vol 34 (04) ◽  
pp. 3666-3675
Author(s):  
Marissa Connor ◽  
Christopher Rozell
Keyword(s):  
Complex Systems ◽  
Latent Structure ◽  
High Dimensional ◽  
Dimensional Manifold ◽  
Data Transformations ◽  
Data Space ◽  
Latent Space ◽  
Starting Point ◽  
Low Dimensional ◽  
Low Dimensional Manifold

Deep generative networks have been widely used for learning mappings from a low-dimensional latent space to a high-dimensional data space. In many cases, data transformations are defined by linear paths in this latent space. However, the Euclidean structure of the latent space may be a poor match for the underlying latent structure in the data. In this work, we incorporate a generative manifold model into the latent space of an autoencoder in order to learn the low-dimensional manifold structure from the data and adapt the latent space to accommodate this structure. In particular, we focus on applications in which the data has closed transformation paths which extend from a starting point and return to nearly the same point. Through experiments on data with natural closed transformation paths, we show that this model introduces the ability to learn the latent dynamics of complex systems, generate transformation paths, and classify samples that belong on the same transformation path.

scholarly journals Multiscale low-dimensional motor cortical state dynamics predict naturalistic reach-and-grasp behavior

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Hamidreza Abbaspourazad ◽  
Mahdi Choudhury ◽  
Yan T. Wong ◽  
Bijan Pesaran ◽  
Maryam M. Shanechi
Keyword(s):  
Neural Control ◽  
Multiple Scales ◽  
Network Activity ◽  
Neural Dynamics ◽  
Population Activity ◽  
Reach And Grasp ◽  
Spatiotemporal Scales ◽  
Motor Cortical ◽  
Low Dimensional ◽  
Low Dimensional Dynamics

AbstractMotor function depends on neural dynamics spanning multiple spatiotemporal scales of population activity, from spiking of neurons to larger-scale local field potentials (LFP). How multiple scales of low-dimensional population dynamics are related in control of movements remains unknown. Multiscale neural dynamics are especially important to study in naturalistic reach-and-grasp movements, which are relatively under-explored. We learn novel multiscale dynamical models for spike-LFP network activity in monkeys performing naturalistic reach-and-grasps. We show low-dimensional dynamics of spiking and LFP activity exhibited several principal modes, each with a unique decay-frequency characteristic. One principal mode dominantly predicted movements. Despite distinct principal modes existing at the two scales, this predictive mode was multiscale and shared between scales, and was shared across sessions and monkeys, yet did not simply replicate behavioral modes. Further, this multiscale mode’s decay-frequency explained behavior. We propose that multiscale, low-dimensional motor cortical state dynamics reflect the neural control of naturalistic reach-and-grasp behaviors.

scholarly journals Generalization-Based Acquisition of Training Data for Motor Primitive Learning by Neural Networks

2021 ◽  
Vol 11 (3) ◽  
pp. 1013
Author(s):  
Zvezdan Lončarević ◽  
Rok Pahič ◽  
Aleš Ude ◽  
Andrej Gams
Keyword(s):  
Neural Networks ◽  
Dimensionality Reduction ◽  
Gaussian Process Regression ◽  
Search Space ◽  
Robot Learning ◽  
Training Data ◽  
Practical Applications ◽  
Latent Space ◽  
Real Robot ◽  
Low Dimensional

Autonomous robot learning in unstructured environments often faces the problem that the dimensionality of the search space is too large for practical applications. Dimensionality reduction techniques have been developed to address this problem and describe motor skills in low-dimensional latent spaces. Most of these techniques require the availability of a sufficiently large database of example task executions to compute the latent space. However, the generation of many example task executions on a real robot is tedious, and prone to errors and equipment failures. The main result of this paper is a new approach for efficient database gathering by performing a small number of task executions with a real robot and applying statistical generalization, e.g., Gaussian process regression, to generate more data. We have shown in our experiments that the data generated this way can be used for dimensionality reduction with autoencoder neural networks. The resulting latent spaces can be exploited to implement robot learning more efficiently. The proposed approach has been evaluated on the problem of robotic throwing at a target. Simulation and real-world results with a humanoid robot TALOS are provided. They confirm the effectiveness of generalization-based database acquisition and the efficiency of learning in a low-dimensional latent space.

Sign in / Sign up

Export Citation Format

Share Document