Robust Regression Estimation Based on Low-Dimensional Recurrent Neural Networks

2018 ◽  
Vol 29 (12) ◽  
pp. 5935-5946 ◽  
Author(s):  
Youshen Xia ◽  
Jun Wang
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Robust Regression ◽  
Regression Estimation ◽  
Low Dimensional

Reduced Order Model for Unsteady Fluid Flows via Recurrent Neural Networks

2019 ◽  
Author(s):  
Sandeep B. Reddy ◽  
Allan Ross Magee ◽  
Rajeev K. Jaiman ◽  
J. Liu ◽  
W. Xu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Reynolds Number ◽  
Flow Field ◽  
Unsteady Flow ◽  
Recurrent Neural Networks ◽  
Reduced Order Model ◽  
Data Driven ◽  
Order Model ◽  
Reduced Order ◽  
Low Dimensional

Abstract In this paper, we present a data-driven approach to construct a reduced-order model (ROM) for the unsteady flow field and fluid-structure interaction. This proposed approach relies on (i) a projection of the high-dimensional data from the Navier-Stokes equations to a low-dimensional subspace using the proper orthogonal decomposition (POD) and (ii) integration of the low-dimensional model with the recurrent neural networks. For the hybrid ROM formulation, we consider long short term memory networks with encoder-decoder architecture, which is a special variant of recurrent neural networks. The mathematical structure of recurrent neural networks embodies a non-linear state space form of the underlying dynamical behavior. This particular attribute of an RNN makes it suitable for non-linear unsteady flow problems. In the proposed hybrid RNN method, the spatial and temporal features of the unsteady flow system are captured separately. Time-invariant modes obtained by low-order projection embodies the spatial features of the flow field, while the temporal behavior of the corresponding modal coefficients is learned via recurrent neural networks. The effectiveness of the proposed method is first demonstrated on a canonical problem of flow past a cylinder at low Reynolds number. With regard to a practical marine/offshore engineering demonstration, we have applied and examined the reliability of the proposed data-driven framework for the predictions of vortex-induced vibrations of a flexible offshore riser at high Reynolds number.

Learning Brain Dynamics with Coupled Low-Dimensional Nonlinear Oscillators and Deep Recurrent Networks

2021 ◽  
pp. 1-40
Author(s):  
Germán Abrevaya ◽  
Guillaume Dumas ◽  
Aleksandr Y. Aravkin ◽  
Peng Zheng ◽  
Jean-Christophe Gagnon-Audet ◽  
...  
Keyword(s):  
Neural Networks ◽  
Dynamical Systems ◽  
Brain Imaging ◽  
Recurrent Neural Networks ◽  
Practical Importance ◽  
Parameters Estimation ◽  
Training Data ◽  
Autoregressive Models ◽  
Unseen Data ◽  
Low Dimensional

Abstract Many natural systems, especially biological ones, exhibit complex multivariate nonlinear dynamical behaviors that can be hard to capture by linear autoregressive models. On the other hand, generic nonlinear models such as deep recurrent neural networks often require large amounts of training data, not always available in domains such as brain imaging; also, they often lack interpretability. Domain knowledge about the types of dynamics typically observed in such systems, such as a certain type of dynamical systems models, could complement purely data-driven techniques by providing a good prior. In this work, we consider a class of ordinary differential equation (ODE) models known as van der Pol (VDP) oscil lators and evaluate their ability to capture a low-dimensional representation of neural activity measured by different brain imaging modalities, such as calcium imaging (CaI) and fMRI, in different living organisms: larval zebrafish, rat, and human. We develop a novel and efficient approach to the nontrivial problem of parameters estimation for a network of coupled dynamical systems from multivariate data and demonstrate that the resulting VDP models are both accurate and interpretable, as VDP's coupling matrix reveals anatomically meaningful excitatory and inhibitory interactions across different brain subsystems. VDP outperforms linear autoregressive models (VAR) in terms of both the data fit accuracy and the quality of insight provided by the coupling matrices and often tends to generalize better to unseen data when predicting future brain activity, being comparable to and sometimes better than the recurrent neural networks (LSTMs). Finally, we demonstrate that our (generative) VDP model can also serve as a data-augmentation tool leading to marked improvements in predictive accuracy of recurrent neural networks. Thus, our work contributes to both basic and applied dimensions of neuroimaging: gaining scientific insights and improving brain-based predictive models, an area of potentially high practical importance in clinical diagnosis and neurotechnology.

Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks

2013 ◽  
Vol 25 (3) ◽  
pp. 626-649 ◽  
Author(s):  
David Sussillo ◽  
Omri Barak
Keyword(s):  
Neural Networks ◽  
Fixed Points ◽  
Recurrent Neural Networks ◽  
Moving Average ◽  
Optimization Technique ◽  
Sine Wave ◽  
High Dimensional ◽  
Flip Flop ◽  
Low Dimensional ◽  
Sine Wave Generator

Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships between time-varying inputs and outputs with complex temporal dependencies. Recently developed algorithms have been successful at training RNNs to perform a wide variety of tasks, but the resulting networks have been treated as black boxes: their mechanism of operation remains unknown. Here we explore the hypothesis that fixed points, both stable and unstable, and the linearized dynamics around them, can reveal crucial aspects of how RNNs implement their computations. Further, we explore the utility of linearization in areas of phase space that are not true fixed points but merely points of very slow movement. We present a simple optimization technique that is applied to trained RNNs to find the fixed and slow points of their dynamics. Linearization around these slow regions can be used to explore, or reverse-engineer, the behavior of the RNN. We describe the technique, illustrate it using simple examples, and finally showcase it on three high-dimensional RNN examples: a 3-bit flip-flop device, an input-dependent sine wave generator, and a two-point moving average. In all cases, the mechanisms of trained networks could be inferred from the sets of fixed and slow points and the linearized dynamics around them.

scholarly journals Considerations in using recurrent neural networks to probe neural dynamics

2018 ◽  
Author(s):  
Jonathan C Kao
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Single Neuron ◽  
Motor Behavior ◽  
Neural Dynamics ◽  
Motor Tasks ◽  
Dynamical Mechanism ◽  
Model Complex ◽  
Low Dimensional ◽  
Robust To Noise

AbstractRecurrent neural networks (RNNs) are increasingly being used to model complex cognitive and motor tasks performed by behaving animals. Here, RNNs are trained to reproduce animal behavior while also recapitulating key statistics of empirically recorded neural activity. In this manner, the RNN can be viewed as an in silico circuit whose computational elements share similar motifs with the cortical area it is modeling. Further, as the RNN’s governing equations and parameters are fully known, they can be analyzed to propose hypotheses for how neural populations compute. In this context, we present important considerations when using RNNs to model motor behavior in a delayed reach task. First, by varying the network’s nonlinear activation and rate regularization, we show that RNNs reproducing single neuron firing rate motifs may not adequately capture important population motifs. Second, by visualizing the RNN’s dynamics in low-dimensional projections, we demonstrate that even when RNNs recapitulate key neurophysiological features on both the single neuron and population levels, it can do so through distinctly different dynamical mechanisms. To militate between these mechanisms, we show that an RNN consistent with a previously proposed dynamical mechanism is more robust to noise. Finally, we show that these dynamics are sufficient for the RNN to generalize to a target switch task it was not trained on. Together, these results emphasize important considerations when using RNN models to probe neural dynamics.

scholarly journals Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks

2021 ◽  
pp. 1-50
Author(s):  
Arnaud Fanthomme ◽  
Rémi Monasson
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Internal State ◽  
Activation Function ◽  
Dimensional Manifold ◽  
Learning Dynamics ◽  
Numerical Investigations ◽  
Low Dimensional

We study the learning dynamics and the representations emerging in recurrent neural networks (RNNs) trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which an RNN with [Formula: see text] neurons learns to integrate [Formula: see text] scalar signals of arbitrary duration. We show, for linear, ReLU, and sigmoidal neurons, that the internal state lives close to a [Formula: see text]-dimensional manifold, whose shape is related to the activation function. Each neuron therefore carries, to various degrees, information about the value of all integrals. We discuss the deep analogy between our results and the concept of mixed selectivity forged by computational neuroscientists to interpret cortical recordings.

scholarly journals Converting POMDPs into MDPs using History Representation

2021 ◽  
Author(s):  
Khanh Xuan Nguyen
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Recurrent Neural Networks ◽  
Utility Functions ◽  
Compact Representation ◽  
Common Solution ◽  
Trajectory Length ◽  
Practice Gap ◽  
Low Dimensional ◽  
Theory Practice Gap

Defining utility functions like the V and Q functions is essential for developing reinforcement learning (RL) solutions for POMDPs. Ideally, we want to define these functions over histories of observations and actions, which the agent can observe. However, the number of possible histories grows exponentially with the trajectory length, making it impractical to reliably estimate history-based utility functions. A common solution is to construct a compact representation of the history. Recently, with the resurgence of deep learning, researchers use recurrent neural networks to learn low-dimensional history representations. Conversion from POMDPs to MDPs using history representation appears to be so effective and seamless that it is common to see a theory-practice gap in deep RL papers, where an algorithm is theoretically formulated in an MDP setting but is empirically evaluated on POMDP tasks without any justifications on why it would work in the latter setting. Thisdocument provides a justification for the conversion from POMDPs to MDPs using history representation.

scholarly journals Differential geometry methods for constructing manifold-targeted recurrent neural networks.

2021 ◽  
Author(s):  
Federico Claudi ◽  
Tiago Branco
Keyword(s):  
Neural Networks ◽  
Differential Geometry ◽  
State Space ◽  
Recurrent Neural Networks ◽  
Network Connectivity ◽  
Neural Dynamics ◽  
Target Manifold ◽  
Neural Computations ◽  
Dimensional State Space ◽  
Low Dimensional

Neural computations can be framed as dynamical processes, whereby the structure of the dynamics within a neural network are a direct reflection of the computations that the network performs. A key step in generating mechanistic interpretations within this computation through dynamics framework is to establish the link between network connectivity, dynamics and computation. This link is only partly understood. Recent work has focused on producing algorithms for engineering artificial recurrent neural networks (RNN) with dynamics targeted to a specific goal manifold. Some of these algorithms only require a set of vectors tangent to the target manifold to be computed, and thus provide a general method that can be applied to a diverse set of problems. Nevertheless, computing such vectors for an arbitrary manifold in a high dimensional state space remains highly challenging, which in practice limits the applicability of this approach. Here we demonstrate how topology and differential geometry can be leveraged to simplify this task, by first computing tangent vectors on a low-dimensional topological manifold and then embedding these in state space. The simplicity of this procedure greatly facilitates the creation of manifold-targeted RNNs, as well as the process of designing task-solving on-manifold dynamics. This new method should enable the application of network engineering-based approaches to a wide set of problems in neuroscience and machine learning. Furthermore, our description of how fundamental concepts from differential geometry can be mapped onto different aspects of neural dynamics is a further demonstration of how the language of differential geometry can enrich the conceptual framework for describing neural dynamics and computation.

Improving circRNA-disease association prediction by sequence and ontology representations with convolutional and recurrent neural networks

2020 ◽  
Author(s):  
Chengqian Lu ◽  
Min Zeng ◽  
Fang-Xiang Wu ◽  
Min Li ◽  
Jianxin Wang
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
De Novo ◽  
Supplementary Information ◽  
Circular Rnas ◽  
Small Scale ◽  
Biological Functions ◽  
Disease Ontology ◽  
Disease Associations ◽  
Low Dimensional

Abstract Motivation Emerging studies indicate that circular RNAs (circRNAs) are widely involved in the progression of human diseases. Due to its special structure which is stable, circRNAs are promising diagnostic and prognostic biomarkers for diseases. However, the experimental verification of circRNA-disease associations is expensive and limited to small-scale. Effective computational methods for predicting potential circRNA-disease associations are regarded as a matter of urgency. Although several models have been proposed, over-reliance on known associations and the absence of characteristics of biological functions make precise predictions are still challenging. Results In this study, we propose a method for predicting CircRNA-Disease Associations based on Sequence and Ontology Representations, named CDASOR, with convolutional and recurrent neural networks. For sequences of circRNAs, we encode them with continuous k-mers, get low-dimensional vectors of k-mers, extract their local feature vectors with 1 D CNN and learn their long-term dependencies with bi-directional long short-term memory. For diseases, we serialize disease ontology into sentences containing the hierarchy of ontology, obtain low-dimensional vectors for disease ontology terms and get terms’ dependencies. Furthermore, we get association patterns of circRNAs and diseases from known circRNA-disease associations with neural networks. After the above steps, we get circRNAs’ and diseases’ high-level representations which are informative to improve the prediction. The experimental results show that CDASOR provides an accurate prediction. Importing the characteristics of biological functions, CDASOR achieves impressive predictions in the de novo test. In addition, 6 of the top-10 predicted results are verified by the published literature in the case studies. Availability The code of CDASOR is freely available at https://github.com/BioinformaticsCSU/CDASOR Supplementary information Supplementary data are available at Bioinformatics online.

Sign in / Sign up

Export Citation Format

Share Document