Extracting Low-Dimensional Latent Structure from Time Series in the Presence of Delays

Noisy, high-dimensional time series observations can often be described by a set of low-dimensional latent variables. Commonly used methods to extract these latent variables typically assume instantaneous relationships between the latent and observed variables. In many physical systems, changes in the latent variables manifest as changes in the observed variables after time delays. Techniques that do not account for these delays can recover a larger number of latent variables than are present in the system, thereby making the latent representation more difficult to interpret. In this work, we introduce a novel probabilistic technique, time-delay gaussian-process factor analysis (TD-GPFA), that performs dimensionality reduction in the presence of a different time delay between each pair of latent and observed variables. We demonstrate how using a gaussian process to model the evolution of each latent variable allows us to tractably learn these delays over a continuous domain. Additionally, we show how TD-GPFA combines temporal smoothing and dimensionality reduction into a common probabilistic framework. We present an expectation/conditional maximization either (ECME) algorithm to learn the model parameters. Our simulations demonstrate that when time delays are present, TD-GPFA is able to correctly identify these delays and recover the latent space. We then applied TD-GPFA to the activity of tens of neurons recorded simultaneously in the macaque motor cortex during a reaching task. TD-GPFA is able to better describe the neural activity using a more parsimonious latent space than GPFA, a method that has been used to interpret motor cortex data but does not account for time delays. More broadly, TD-GPFA can help to unravel the mechanisms underlying high-dimensional time series data by taking into account physical delays in the system.

Download Full-text

Interpretable Variational Graph Autoencoder with Noninformative Prior

Future Internet ◽

10.3390/fi13020051 ◽

2021 ◽

Vol 13 (2) ◽

pp. 51

Author(s):

Lili Sun ◽

Xueyan Liu ◽

Min Zhao ◽

Bo Yang

Keyword(s):

Latent Variables ◽

Latent Variable ◽

Expert Knowledge ◽

Structural Information ◽

Standard Normal Distribution ◽

Noninformative Prior ◽

Latent Space ◽

Distribution Parameters ◽

Standard Normal ◽

Low Dimensional

Variational graph autoencoder, which can encode structural information and attribute information in the graph into low-dimensional representations, has become a powerful method for studying graph-structured data. However, most existing methods based on variational (graph) autoencoder assume that the prior of latent variables obeys the standard normal distribution which encourages all nodes to gather around 0. That leads to the inability to fully utilize the latent space. Therefore, it becomes a challenge on how to choose a suitable prior without incorporating additional expert knowledge. Given this, we propose a novel noninformative prior-based interpretable variational graph autoencoder (NPIVGAE). Specifically, we exploit the noninformative prior as the prior distribution of latent variables. This prior enables the posterior distribution parameters to be almost learned from the sample data. Furthermore, we regard each dimension of a latent variable as the probability that the node belongs to each block, thereby improving the interpretability of the model. The correlation within and between blocks is described by a block–block correlation matrix. We compare our model with state-of-the-art methods on three real datasets, verifying its effectiveness and superiority.

Download Full-text

Predictive learning as a network mechanism for extracting low-dimensional latent space representations

Nature Communications ◽

10.1038/s41467-021-21696-1 ◽

2021 ◽

Vol 12 (1) ◽

Author(s):

Stefano Recanatesi ◽

Matthew Farrell ◽

Guillaume Lajoie ◽

Sophie Deneve ◽

Mattia Rigotti ◽

...

Keyword(s):

Latent Variables ◽

Latent Structure ◽

Network Activity ◽

Semantic Organization ◽

Sequential Processing ◽

Predictive Learning ◽

Neural Representations ◽

Latent Space ◽

Low Dimensional ◽

Sensory Prediction

AbstractArtificial neural networks have recently achieved many successes in solving sequential processing and planning tasks. Their success is often ascribed to the emergence of the task’s low-dimensional latent structure in the network activity – i.e., in the learned neural representations. Here, we investigate the hypothesis that a means for generating representations with easily accessed low-dimensional latent structure, possibly reflecting an underlying semantic organization, is through learning to predict observations about the world. Specifically, we ask whether and when network mechanisms for sensory prediction coincide with those for extracting the underlying latent variables. Using a recurrent neural network model trained to predict a sequence of observations we show that network dynamics exhibit low-dimensional but nonlinearly transformed representations of sensory inputs that map the latent structure of the sensory environment. We quantify these results using nonlinear measures of intrinsic dimensionality and linear decodability of latent variables, and provide mathematical arguments for why such useful predictive representations emerge. We focus throughout on how our results can aid the analysis and interpretation of experimental data.

Download Full-text

Generalization-Based Acquisition of Training Data for Motor Primitive Learning by Neural Networks

Applied Sciences ◽

10.3390/app11031013 ◽

2021 ◽

Vol 11 (3) ◽

pp. 1013

Author(s):

Zvezdan Lončarević ◽

Rok Pahič ◽

Aleš Ude ◽

Andrej Gams

Keyword(s):

Neural Networks ◽

Dimensionality Reduction ◽

Gaussian Process Regression ◽

Search Space ◽

Robot Learning ◽

Training Data ◽

Practical Applications ◽

Latent Space ◽

Real Robot ◽

Low Dimensional

Autonomous robot learning in unstructured environments often faces the problem that the dimensionality of the search space is too large for practical applications. Dimensionality reduction techniques have been developed to address this problem and describe motor skills in low-dimensional latent spaces. Most of these techniques require the availability of a sufficiently large database of example task executions to compute the latent space. However, the generation of many example task executions on a real robot is tedious, and prone to errors and equipment failures. The main result of this paper is a new approach for efficient database gathering by performing a small number of task executions with a real robot and applying statistical generalization, e.g., Gaussian process regression, to generate more data. We have shown in our experiments that the data generated this way can be used for dimensionality reduction with autoencoder neural networks. The resulting latent spaces can be exploited to implement robot learning more efficiently. The proposed approach has been evaluated on the problem of robotic throwing at a target. Simulation and real-world results with a humanoid robot TALOS are provided. They confirm the effectiveness of generalization-based database acquisition and the efficiency of learning in a low-dimensional latent space.

Download Full-text

LSTM-Guided Coaching Assistant for Table Tennis Practice

Sensors ◽

10.3390/s18124112 ◽

2018 ◽

Vol 18 (12) ◽

pp. 4112 ◽

Cited By ~ 6

Author(s):

Se-Min Lim ◽

Hyeong-Cheol Oh ◽

Jaein Kim ◽

Juwon Lee ◽

Jooyoung Park

Keyword(s):

Time Series ◽

State Space ◽

Time Series Data ◽

State Space Model ◽

Skill Assessment ◽

Series Data ◽

High Dimensional ◽

Table Tennis ◽

Space Model ◽

Low Dimensional

Recently, wearable devices have become a prominent health care application domain by incorporating a growing number of sensors and adopting smart machine learning technologies. One closely related topic is the strategy of combining the wearable device technology with skill assessment, which can be used in wearable device apps for coaching and/or personal training. Particularly pertinent to skill assessment based on high-dimensional time series data from wearable sensors is classifying whether a player is an expert or a beginner, which skills the player is exercising, and extracting some low-dimensional representations useful for coaching. In this paper, we present a deep learning-based coaching assistant method, which can provide useful information in supporting table tennis practice. Our method uses a combination of LSTM (Long short-term memory) with a deep state space model and probabilistic inference. More precisely, we use the expressive power of LSTM when handling high-dimensional time series data, and state space model and probabilistic inference to extract low-dimensional latent representations useful for coaching. Experimental results show that our method can yield promising results for characterizing high-dimensional time series patterns and for providing useful information when working with wearable IMU (Inertial measurement unit) sensors for table tennis coaching.

Download Full-text

Unsupervised Text Feature Learning via Deep Variational Auto-encoder

Information Technology And Control ◽

10.5755/j01.itc.49.3.25918 ◽

2020 ◽

Vol 49 (3) ◽

pp. 421-437

Author(s):

Genggeng Liu ◽

Lin Xie ◽

Chi-Hua Chen

Keyword(s):

Dimensionality Reduction ◽

High Dimensional Data ◽

Image Data ◽

Original Data ◽

Feature Representation ◽

High Dimensional ◽

Learning To Learn ◽

Text Feature ◽

Reduction Methods ◽

Low Dimensional

Dimensionality reduction plays an important role in the data processing of machine learning and data mining, which makes the processing of high-dimensional data more efficient. Dimensionality reduction can extract the low-dimensional feature representation of high-dimensional data, and an effective dimensionality reduction method can not only extract most of the useful information of the original data, but also realize the function of removing useless noise. The dimensionality reduction methods can be applied to all types of data, especially image data. Although the supervised learning method has achieved good results in the application of dimensionality reduction, its performance depends on the number of labeled training samples. With the growing of information from internet, marking the data requires more resources and is more difficult. Therefore, using unsupervised learning to learn the feature of data has extremely important research value. In this paper, an unsupervised multilayered variational auto-encoder model is studied in the text data, so that the high-dimensional feature to the low-dimensional feature becomes efficient and the low-dimensional feature can retain mainly information as much as possible. Low-dimensional feature obtained by different dimensionality reduction methods are used to compare with the dimensionality reduction results of variational auto-encoder (VAE), and the method can be significantly improved over other comparison methods.

Download Full-text

Synchronization Measure Based on a Geometric Approach to Attractor Embedding Using Finite Observation Windows

Complexity ◽

10.1155/2018/8259496 ◽

2018 ◽

Vol 2018 ◽

pp. 1-16 ◽

Cited By ~ 2

Author(s):

Inga Timofejeva ◽

Kristina Poskuviene ◽

Maosen Cao ◽

Minvydas Ragulskis

Keyword(s):

Time Series ◽

Time Delay ◽

Time Delays ◽

Geometric Approach ◽

Optimal Time ◽

Effective Algorithm ◽

Geometrical Properties ◽

Finite Observation

A simple and effective algorithm for the identification of optimal time delays based on the geometrical properties of the embedded attractor is presented in this paper. A time series synchronization measure based on optimal time delays is derived. The approach is based on the comparison of optimal time delay sequences that are computed for segments of the considered time series. The proposed technique is validated using coupled chaotic Rössler systems.

Download Full-text

Capturing discrete latent structures: choose LDs over PCs

Biostatistics ◽

10.1093/biostatistics/kxab030 ◽

2021 ◽

Author(s):

Theresa A Alexander ◽

Rafael A Irizarry ◽

Héctor Corrada Bravo

Keyword(s):

Dimensionality Reduction ◽

Biological Data ◽

Reduction Technique ◽

Latent Structure ◽

High Dimensional ◽

Underlying Structure ◽

Linear Transformations ◽

Latent Structures ◽

Low Dimensional ◽

Discriminatory Information

Summary High-dimensional biological data collection across heterogeneous groups of samples has become increasingly common, creating high demand for dimensionality reduction techniques that capture underlying structure of the data. Discovering low-dimensional embeddings that describe the separation of any underlying discrete latent structure in data is an important motivation for applying these techniques since these latent classes can represent important sources of unwanted variability, such as batch effects, or interesting sources of signal such as unknown cell types. The features that define this discrete latent structure are often hard to identify in high-dimensional data. Principal component analysis (PCA) is one of the most widely used methods as an unsupervised step for dimensionality reduction. This reduction technique finds linear transformations of the data which explain total variance. When the goal is detecting discrete structure, PCA is applied with the assumption that classes will be separated in directions of maximum variance. However, PCA will fail to accurately find discrete latent structure if this assumption does not hold. Visualization techniques, such as t-Distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP), attempt to mitigate these problems with PCA by creating a low-dimensional space where similar objects are modeled by nearby points in the low-dimensional embedding and dissimilar objects are modeled by distant points with high probability. However, since t-SNE and UMAP are computationally expensive, often a PCA reduction is done before applying them which makes it sensitive to PCAs downfalls. Also, tSNE is limited to only two or three dimensions as a visualization tool, which may not be adequate for retaining discriminatory information. The linear transformations of PCA are preferable to non-linear transformations provided by methods like t-SNE and UMAP for interpretable feature weights. Here, we propose iterative discriminant analysis (iDA), a dimensionality reduction technique designed to mitigate these limitations. iDA produces an embedding that carries discriminatory information which optimally separates latent clusters using linear transformations that permit post hoc analysis to determine features that define these latent structures.

Download Full-text

Projection Analysis Optimization for Human Transition Motion Estimation

International Journal of Digital Multimedia Broadcasting ◽

10.1155/2019/6816453 ◽

2019 ◽

Vol 2019 ◽

pp. 1-9

Author(s):

Wanyi Li ◽

Feifei Zhang ◽

Qiang Chen ◽

Qian Zhang

Keyword(s):

Video Game ◽

Human Motion ◽

High Dimensional ◽

Dynamical Models ◽

Excellent Performance ◽

3 Dimensional ◽

Latent Space ◽

Projection Analysis ◽

Low Dimensional ◽

Working Efficiency

It is a difficult task to estimate the human transition motion without the specialized software. The 3-dimensional (3D) human motion animation is widely used in video game, movie, and so on. When making the animation, human transition motion is necessary. If there is a method that can generate the transition motion, the making time will cost less and the working efficiency will be improved. Thus a new method called latent space optimization based on projection analysis (LSOPA) is proposed to estimate the human transition motion. LSOPA is carried out under the assistance of Gaussian process dynamical models (GPDM); it builds the object function to optimize the data in the low dimensional (LD) space, and the optimized data in LD space will be obtained to generate the human transition motion. The LSOPA can make the GPDM learn the high dimensional (HD) data to estimate the needed transition motion. The excellent performance of LSOPA will be tested by the experiments.

Download Full-text

Collective dynamics of repeated inference in variational autoencoder rapidly find cluster structure

Scientific Reports ◽

10.1038/s41598-020-72593-4 ◽

2020 ◽

Vol 10 (1) ◽

Author(s):

Yoshihiro Nagano ◽

Ryo Karakida ◽

Masato Okada

Keyword(s):

Activity Pattern ◽

Latent Variables ◽

Cluster Structure ◽

Generative Models ◽

Cluster Center ◽

Specific Data ◽

Global Cluster ◽

Latent Space ◽

Variational Autoencoder ◽

Low Dimensional

Abstract Deep neural networks are good at extracting low-dimensional subspaces (latent spaces) that represent the essential features inside a high-dimensional dataset. Deep generative models represented by variational autoencoders (VAEs) can generate and infer high-quality datasets, such as images. In particular, VAEs can eliminate the noise contained in an image by repeating the mapping between latent and data space. To clarify the mechanism of such denoising, we numerically analyzed how the activity pattern of trained networks changes in the latent space during inference. We considered the time development of the activity pattern for specific data as one trajectory in the latent space and investigated the collective behavior of these inference trajectories for many data. Our study revealed that when a cluster structure exists in the dataset, the trajectory rapidly approaches the center of the cluster. This behavior was qualitatively consistent with the concept retrieval reported in associative memory models. Additionally, the larger the noise contained in the data, the closer the trajectory was to a more global cluster. It was demonstrated that by increasing the number of the latent variables, the trend of the approach a cluster center can be enhanced, and the generalization ability of the VAE can be improved.

Download Full-text

Bayesian recurrent neural network as a tool for reconstructing dynamical systems from multidimensional data

10.5194/egusphere-egu2020-8787 ◽

2020 ◽

Author(s):

Alexander Feigin ◽

Aleksei Seleznev ◽

Dmitry Mukhin ◽

Andrey Gavrilov ◽

Evgeny Loskutov

Keyword(s):

Neural Network ◽

Time Series ◽

Cost Function ◽

Recurrent Neural Network ◽

Bayesian Optimization ◽

Data Driven ◽

High Dimensional ◽

Model Learning ◽

Low Dimensional ◽

The Cost

We suggest a new method for construction of data-driven dynamical models from observed multidimensional time series. The method is based on a recurrent neural network (RNN) with specific structure, which allows for the joint reconstruction of both a low-dimensional embedding for dynamical components in the data and an operator describing the low-dimensional evolution of the system. The key link of the method is a Bayesian optimization of both model structure and the hypothesis about the data generating law, which is needed for constructing the cost function for model learning. &#160;The form of the model we propose allows us to construct a stochastic dynamical system of moderate dimension that copies dynamical properties of the original high-dimensional system. An advantage of the proposed method is the data-adaptive properties of the RNN model: it is based on the adjustable nonlinear elements and has easily scalable structure. The combination of the RNN with the Bayesian optimization procedure efficiently provides the model with statistically significant nonlinearity and dimension. The method developed for the model optimization aims to detect the long-term connections between system&#8217;s states &#8211; the memory of the system: the cost-function used for model learning is constructed taking into account this factor. In particular, in the case of absence of interaction between the dynamical component and noise, the method provides unbiased reconstruction of the hidden deterministic system. In the opposite case when the noise has strong impact on the dynamics, the method yield a model in the form of a nonlinear stochastic map determining the Markovian process with memory. Bayesian approach used for selecting both the optimal model&#8217;s structure and the appropriate cost function allows to obtain the statistically significant inferences about the dynamical signal in data as well as its interaction with the noise components. Data driven model derived from the relatively short time series of the QG3 model &#8211; the high dimensional nonlinear system producing chaotic behavior &#8211; is shown be able to serve as a good simulator for the QG3 LFV components. The statistically significant recurrent states of the QG3 model, i.e. the well-known teleconnections in NH, are all reproduced by the model obtained. Moreover, statistics of the residence times of the model near these states is very close to the corresponding statistics of the original QG3 model. These results demonstrate that the method can be useful in modeling the variability of the real atmosphere.The work was supported by the Russian Science Foundation (Grant No. 19-42-04121).

Download Full-text