scholarly journals Clustered Echo State Networks for Signal Observation and Frequency Filtering

2020 ◽  
Author(s):  
Laércio Oliveira Junior ◽  
Florian Stelzer ◽  
Liang Zhao
Keyword(s):  
Dynamical System ◽  
Neural Networks ◽  
Input Signal ◽  
Recurrent Neural Networks ◽  
High Dimensional ◽  
Frequency Filtering ◽  
Echo State Networks ◽  
Chaotic Signals ◽  
Network Topologies ◽  
Dimensional Dynamical System

Echo State Networks (ESNs) are recurrent neural networks that map an input signal to a high-dimensional dynamical system, called reservoir, and possess adaptive output weights. The output weights are trained such that the ESN’s output signal fits the desired target signal. Classical reservoirs are sparse and randomly connected networks. In this article, we investigate the effect of different network topologies on the performance of ESNs. Specifically, we use two types of networks to construct clustered reservoirs of ESN: the clustered Erdös–Rényi and the clustered Barabási-Albert network model. Moreover, we compare the performance of these clustered ESNs (CESNs) and classical ESNs with the random reservoir by employing them to two different tasks: frequency filtering and the reconstruction of chaotic signals. By using a clustered topology, one can achieve a significant increase in the ESN’s performance.

scholarly journals Recurrent Kernel Machines: Computing with Infinite Echo State Networks

2012 ◽  
Vol 24 (1) ◽  
pp. 104-133 ◽  
Author(s):  
Michiel Hermans ◽  
Benjamin Schrauwen
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Input Data ◽  
Dimensional Space ◽  
Theoretical Research ◽  
High Dimensional ◽  
Support Vector ◽  
Practical Applications ◽  
Echo State Networks ◽  
Vector Machines

Echo state networks (ESNs) are large, random recurrent neural networks with a single trained linear readout layer. Despite the untrained nature of the recurrent weights, they are capable of performing universal computations on temporal input data, which makes them interesting for both theoretical research and practical applications. The key to their success lies in the fact that the network computes a broad set of nonlinear, spatiotemporal mappings of the input data, on which linear regression or classification can easily be performed. One could consider the reservoir as a spatiotemporal kernel, in which the mapping to a high-dimensional space is computed explicitly. In this letter, we build on this idea and extend the concept of ESNs to infinite-sized recurrent neural networks, which can be considered recursive kernels that subsequently can be used to create recursive support vector machines. We present the theoretical framework, provide several practical examples of recursive kernels, and apply them to typical temporal tasks.

Adaptive Capability of Recurrent Neural Networks with Fixed Weights for Series-Parallel System Identification

2009 ◽  
Vol 21 (11) ◽  
pp. 3214-3227
Author(s):  
James Ting-Ho Lo
Keyword(s):  
Neural Network ◽  
Dynamical System ◽  
Neural Networks ◽  
System Identification ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Additional Input ◽  
Adaptive Capability ◽  
Environmental Process ◽  
Optimal Series

By a fundamental neural filtering theorem, a recurrent neural network with fixed weights is known to be capable of adapting to an uncertain environment. This letter reports some mathematical results on the performance of such adaptation for series-parallel identification of a dynamical system as compared with the performance of the best series-parallel identifier possible under the assumption that the precise value of the uncertain environmental process is given. In short, if an uncertain environmental process is observable (not necessarily constant) from the output of a dynamical system or constant (not necessarily observable), then a recurrent neural network exists as a series-parallel identifier of the dynamical system whose output approaches the output of an optimal series-parallel identifier using the environmental process as an additional input.

scholarly journals Synthesis of recurrent neural networks for dynamical system simulation

2016 ◽  
Vol 80 ◽  
pp. 67-78 ◽  
Author(s):  
Adam P. Trischler ◽  
Gabriele M.T. D’Eleuterio
Keyword(s):  
Dynamical System ◽  
Neural Networks ◽  
Recurrent Neural Networks ◽  
System Simulation

scholarly journals A Novel Method for Training an Echo State Network with Feedback-Error Learning

2013 ◽  
Vol 2013 ◽  
pp. 1-9
Author(s):  
Rikke Amilde Løvlid
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Training Data ◽  
Echo State Network ◽  
Training Method ◽  
Computationally Efficient ◽  
Echo State Networks ◽  
Feedback Error Learning ◽  
Feedback Error ◽  
Error Learning

Echo state networks are a relatively new type of recurrent neural networks that have shown great potentials for solving non-linear, temporal problems. The basic idea is to transform the low dimensional temporal input into a higher dimensional state, and then train the output connection weights to make the system output the target information. Because only the output weights are altered, training is typically quick and computationally efficient compared to training of other recurrent neural networks. This paper investigates using an echo state network to learn the inverse kinematics model of a robot simulator with feedback-error-learning. In this scheme teacher forcing is not perfect, and joint constraints on the simulator makes the feedback error inaccurate. A novel training method which is less influenced by the noise in the training data is proposed and compared to the traditional ESN training method.

scholarly journals Asymptotic (statistical) periodicity in two-dimensional maps

2021 ◽  
Vol 0 (0) ◽  
pp. 0
Author(s):  
Fumihiko Nakamura ◽  
Michael C. Mackey
Keyword(s):  
Dynamical System ◽  
Bounded Variation ◽  
High Dimensional ◽  
Sufficient Condition ◽  
Two Dimensional ◽  
Function Of Bounded Variation ◽  
Dimensional Dynamical System ◽  
Asymptotic Periodicity ◽  
Parameter Values

<p style='text-indent:20px;'>In this paper we give a new sufficient condition for the existence of asymptotic periodicity of Frobenius–Perron operators corresponding to two–dimensional maps. Asymptotic periodicity for strictly expanding systems, that is, all eigenvalues of the system are greater than one, in a high-dimensional dynamical system was already known. Our new result enables one to deal with systems having an eigenvalue smaller than one. The key idea for the proof is to use a function of bounded variation defined by line integration. Finally, we introduce a new two-dimensional dynamical system numerically exhibiting asymptotic periodicity with different periods depending on parameter values, and discuss the application of our theorem to the example.</p>

RECURRENT NEURAL NETWORKS ARE UNIVERSAL APPROXIMATORS

2007 ◽  
Vol 17 (04) ◽  
pp. 253-263 ◽  
Author(s):  
ANTON MAXIMILIAN SCHÄFER ◽  
HANS-GEORG ZIMMERMANN
Keyword(s):  
Dynamical System ◽  
Neural Networks ◽  
Dynamical Systems ◽  
Error Correction ◽  
Recurrent Neural Networks ◽  
State Space Model ◽  
Universal Approximation ◽  
Space Model ◽  
Open Dynamical System ◽  
Universal Approximators

Recurrent Neural Networks (RNN) have been developed for a better understanding and analysis of open dynamical systems. Still the question often arises if RNN are able to map every open dynamical system, which would be desirable for a broad spectrum of applications. In this article we give a proof for the universal approximation ability of RNN in state space model form and even extend it to Error Correction and Normalized Recurrent Neural Networks.

ON THE MATHEMATICAL CLARIFICATION OF THE SNAP-BACK-REPELLER IN HIGH-DIMENSIONAL SYSTEMS AND CHAOS IN A DISCRETE NEURAL NETWORK MODEL

2002 ◽  
Vol 12 (05) ◽  
pp. 1129-1139 ◽  
Author(s):  
WEI LIN ◽  
JIONG RUAN ◽  
WEIRUI ZHAO
Keyword(s):  
Neural Network ◽  
Dynamical System ◽  
Neural Networks ◽  
Fixed Point ◽  
Numerical Simulations ◽  
Neural Network Model ◽  
Jacobian Matrix ◽  
Nonlinear Dynamical System ◽  
High Dimensional ◽  
Nonlinear Dynamical

We investigate the differences among several definitions of the snap-back-repeller, which is always regarded as an inducement to produce chaos in nonlinear dynamical system. By analyzing the norms in different senses and the illustrative examples, we clarify why a snap-back-repeller in the neighborhood of the fixed point, where all eigenvalues of the corresponding variable Jacobian Matrix are absolutely larger than 1 in norm, might not imply chaos. Furthermore, we theoretically prove the existence of chaos in a discrete neural networks model in the sense of Marotto with some parameters of the systems entering some regions. And the following numerical simulations and corresponding calculation, as concrete examples, reinforce our theoretical proof.

Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks

2013 ◽  
Vol 25 (3) ◽  
pp. 626-649 ◽  
Author(s):  
David Sussillo ◽  
Omri Barak
Keyword(s):  
Neural Networks ◽  
Fixed Points ◽  
Recurrent Neural Networks ◽  
Moving Average ◽  
Optimization Technique ◽  
Sine Wave ◽  
High Dimensional ◽  
Flip Flop ◽  
Low Dimensional ◽  
Sine Wave Generator

Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships between time-varying inputs and outputs with complex temporal dependencies. Recently developed algorithms have been successful at training RNNs to perform a wide variety of tasks, but the resulting networks have been treated as black boxes: their mechanism of operation remains unknown. Here we explore the hypothesis that fixed points, both stable and unstable, and the linearized dynamics around them, can reveal crucial aspects of how RNNs implement their computations. Further, we explore the utility of linearization in areas of phase space that are not true fixed points but merely points of very slow movement. We present a simple optimization technique that is applied to trained RNNs to find the fixed and slow points of their dynamics. Linearization around these slow regions can be used to explore, or reverse-engineer, the behavior of the RNN. We describe the technique, illustrate it using simple examples, and finally showcase it on three high-dimensional RNN examples: a 3-bit flip-flop device, an input-dependent sine wave generator, and a two-point moving average. In all cases, the mechanisms of trained networks could be inferred from the sets of fixed and slow points and the linearized dynamics around them.

Sign in / Sign up

Export Citation Format

Share Document