Recurrent Neural Networks and Efficiency in High-Dimensional EEG Classification

2021 ◽  
pp. 297-310
Author(s):  
Javier León ◽  
Juan José Escobar ◽  
Jesús González ◽  
Julio Ortega ◽  
Francisco Manuel Arrabal-Campos ◽  
...  
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
High Dimensional ◽  
Eeg Classification

scholarly journals Clustered Echo State Networks for Signal Observation and Frequency Filtering

2020 ◽  
Author(s):  
Laércio Oliveira Junior ◽  
Florian Stelzer ◽  
Liang Zhao
Keyword(s):  
Dynamical System ◽  
Neural Networks ◽  
Input Signal ◽  
Recurrent Neural Networks ◽  
High Dimensional ◽  
Frequency Filtering ◽  
Echo State Networks ◽  
Chaotic Signals ◽  
Network Topologies ◽  
Dimensional Dynamical System

Echo State Networks (ESNs) are recurrent neural networks that map an input signal to a high-dimensional dynamical system, called reservoir, and possess adaptive output weights. The output weights are trained such that the ESN’s output signal fits the desired target signal. Classical reservoirs are sparse and randomly connected networks. In this article, we investigate the effect of different network topologies on the performance of ESNs. Specifically, we use two types of networks to construct clustered reservoirs of ESN: the clustered Erdös–Rényi and the clustered Barabási-Albert network model. Moreover, we compare the performance of these clustered ESNs (CESNs) and classical ESNs with the random reservoir by employing them to two different tasks: frequency filtering and the reconstruction of chaotic signals. By using a clustered topology, one can achieve a significant increase in the ESN’s performance.

scholarly journals Recurrent Kernel Machines: Computing with Infinite Echo State Networks

2012 ◽  
Vol 24 (1) ◽  
pp. 104-133 ◽  
Author(s):  
Michiel Hermans ◽  
Benjamin Schrauwen
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Input Data ◽  
Dimensional Space ◽  
Theoretical Research ◽  
High Dimensional ◽  
Support Vector ◽  
Practical Applications ◽  
Echo State Networks ◽  
Vector Machines

Echo state networks (ESNs) are large, random recurrent neural networks with a single trained linear readout layer. Despite the untrained nature of the recurrent weights, they are capable of performing universal computations on temporal input data, which makes them interesting for both theoretical research and practical applications. The key to their success lies in the fact that the network computes a broad set of nonlinear, spatiotemporal mappings of the input data, on which linear regression or classification can easily be performed. One could consider the reservoir as a spatiotemporal kernel, in which the mapping to a high-dimensional space is computed explicitly. In this letter, we build on this idea and extend the concept of ESNs to infinite-sized recurrent neural networks, which can be considered recursive kernels that subsequently can be used to create recursive support vector machines. We present the theoretical framework, provide several practical examples of recursive kernels, and apply them to typical temporal tasks.

Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks

2013 ◽  
Vol 25 (3) ◽  
pp. 626-649 ◽  
Author(s):  
David Sussillo ◽  
Omri Barak
Keyword(s):  
Neural Networks ◽  
Fixed Points ◽  
Recurrent Neural Networks ◽  
Moving Average ◽  
Optimization Technique ◽  
Sine Wave ◽  
High Dimensional ◽  
Flip Flop ◽  
Low Dimensional ◽  
Sine Wave Generator

Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships between time-varying inputs and outputs with complex temporal dependencies. Recently developed algorithms have been successful at training RNNs to perform a wide variety of tasks, but the resulting networks have been treated as black boxes: their mechanism of operation remains unknown. Here we explore the hypothesis that fixed points, both stable and unstable, and the linearized dynamics around them, can reveal crucial aspects of how RNNs implement their computations. Further, we explore the utility of linearization in areas of phase space that are not true fixed points but merely points of very slow movement. We present a simple optimization technique that is applied to trained RNNs to find the fixed and slow points of their dynamics. Linearization around these slow regions can be used to explore, or reverse-engineer, the behavior of the RNN. We describe the technique, illustrate it using simple examples, and finally showcase it on three high-dimensional RNN examples: a 3-bit flip-flop device, an input-dependent sine wave generator, and a two-point moving average. In all cases, the mechanisms of trained networks could be inferred from the sets of fixed and slow points and the linearized dynamics around them.

scholarly journals Reservoir computing and the Sooner-is-Better bottleneck

2016 ◽  
Vol 39 ◽  
Author(s):  
Stefan L. Frank ◽  
Hartmut Fitz
Keyword(s):  
Neural Networks ◽  
State Space ◽  
Recurrent Neural Networks ◽  
Random Point ◽  
High Dimensional ◽  
Reservoir Computing ◽  
Current Input ◽  
Dimensional State Space ◽  
Language Input

AbstractPrior language input is not lost but integrated with the current input. This principle is demonstrated by “reservoir computing”: Untrained recurrent neural networks project input sequences onto a random point in high-dimensional state space. Earlier inputs can be retrieved from this projection, albeit less reliably so as more input is received. The bottleneck is therefore not “Now-or-Never” but “Sooner-is-Better.”

The Crystallizing Substochastic Sequential Machine Extractor: CrySSMEx

2006 ◽  
Vol 18 (9) ◽  
pp. 2211-2255 ◽  
Author(s):  
Henrik Jacobsson
Keyword(s):  
Neural Networks ◽  
Stochastic Model ◽  
Vector Quantization ◽  
Dynamic Systems ◽  
Recurrent Neural Networks ◽  
Finite State Machine ◽  
Chaotic Systems ◽  
High Dimensional ◽  
Finite State ◽  
Space Dynamics

This letter presents an algorithm, CrySSMEx, for extracting minimal finite state machine descriptions of dynamic systems such as recurrent neural networks. Unlike previous algorithms, CrySSMEx is parameter free and deterministic, and it efficiently generates a series of increasingly refined models. A novel finite stochastic model of dynamic systems and a novel vector quantization function have been developed to take into account the state-space dynamics of the system. The experiments show that (1) extraction from systems that can be described as regular grammars is trivial, (2) extraction from high-dimensional systems is feasible, and (3) extraction of approximative models from chaotic systems is possible. The results are promising, and an analysis of shortcomings suggests some possible further improvements. Some largely overlooked connections, of the field of rule extraction from recurrent neural networks, to other fields are also identified.

Sign in / Sign up

Export Citation Format

Share Document