scholarly journals Simple Framework for Constructing Functional Spiking Recurrent Neural Networks

2019 ◽  
Author(s):  
Robert Kim ◽  
Yinghao Li ◽  
Terrence J. Sejnowski
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Continuous Variable ◽  
Model Parameters ◽  
Additional Parameter ◽  
Wide Range ◽  
Close Relationship ◽  
Rate Coding ◽  
One To One ◽  
The One

AbstractCortical microcircuits exhibit complex recurrent architectures that possess dynamically rich properties. The neurons that make up these microcircuits communicate mainly via discrete spikes, and it is not clear how spikes give rise to dynamics that can be used to perform computationally challenging tasks. In contrast, continuous models of rate-coding neurons can be trained to perform complex tasks. Here, we present a simple framework to construct biologically realistic spiking recurrent neural networks (RNNs) capable of learning a wide range of tasks. Our framework involves training a continuous-variable rate RNN with important biophysical constraints and transferring the learned dynamics and constraints to a spiking RNN in a one-to-one manner. The proposed framework introduces only one additional parameter to establish the equivalence between rate and spiking RNN models. We also study other model parameters related to the rate and spiking networks to optimize the one-to-one mapping. By establishing a close relationship between rate and spiking models, we demonstrate that spiking RNNs could be constructed to achieve similar performance as their counterpart continuous rate networks.

scholarly journals Simple framework for constructing functional spiking recurrent neural networks

2019 ◽  
Vol 116 (45) ◽  
pp. 22811-22820 ◽  
Author(s):  
Robert Kim ◽  
Yinghao Li ◽  
Terrence J. Sejnowski
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Continuous Variable ◽  
Model Parameters ◽  
Additional Parameter ◽  
Wide Range ◽  
Close Relationship ◽  
Rate Coding ◽  
One To One ◽  
The One

Cortical microcircuits exhibit complex recurrent architectures that possess dynamically rich properties. The neurons that make up these microcircuits communicate mainly via discrete spikes, and it is not clear how spikes give rise to dynamics that can be used to perform computationally challenging tasks. In contrast, continuous models of rate-coding neurons can be trained to perform complex tasks. Here, we present a simple framework to construct biologically realistic spiking recurrent neural networks (RNNs) capable of learning a wide range of tasks. Our framework involves training a continuous-variable rate RNN with important biophysical constraints and transferring the learned dynamics and constraints to a spiking RNN in a one-to-one manner. The proposed framework introduces only 1 additional parameter to establish the equivalence between rate and spiking RNN models. We also study other model parameters related to the rate and spiking networks to optimize the one-to-one mapping. By establishing a close relationship between rate and spiking models, we demonstrate that spiking RNNs could be constructed to achieve similar performance as their counterpart continuous rate networks.

Locally Recurrent Neural Networks and Their Applications

2010 ◽  
pp. 195-222
Author(s):  
Todor D. Ganchev
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Computational Models ◽  
Machine Learning Techniques ◽  
Spatial Correlations ◽  
Temporal Correlations ◽  
Learning Techniques ◽  
Locally Recurrent ◽  
The One ◽  
And Control

In this chapter we review various computational models of locally recurrent neurons and deliberate the architecture of some archetypal locally recurrent neural networks (LRNNs) that are based on them. Generalizations of these structures are discussed as well. Furthermore, we point at a number of realworld applications of LRNNs that have been reported in past and recent publications. These applications involve classification or prediction of temporal sequences, discovering and modeling of spatial and temporal correlations, process identification and control, etc. Validation experiments reported in these developments provide evidence that locally recurrent architectures are capable of identifying and exploiting temporal and spatial correlations (i.e., the context in which events occur), which is the main reason for their advantageous performance when compared with the one of their non-recurrent counterparts or other reasonable machine learning techniques.

scholarly journals Automatic sleep staging using heart rate variability, body movements, and recurrent neural networks in a sleep disordered population

SLEEP ◽  
2020 ◽  
Vol 43 (9) ◽  
Author(s):  
Pedro Fonseca ◽  
Merel M van Gilst ◽  
Mustafa Radha ◽  
Marco Ross ◽  
Arnaud Moreau ◽  
...  
Keyword(s):  
Neural Networks ◽  
Heart Rate ◽  
Heart Rate Variability ◽  
Sleep Disorders ◽  
Recurrent Neural Networks ◽  
Clinical Diagnostics ◽  
Sleep Staging ◽  
Substantial Agreement ◽  
Body Movements ◽  
Wide Range

Abstract Study Objectives To validate a previously developed sleep staging algorithm using heart rate variability (HRV) and body movements in an independent broad cohort of unselected sleep disordered patients. Methods We applied a previously designed algorithm for automatic sleep staging using long short-term memory recurrent neural networks to model sleep architecture. The classifier uses 132 HRV features computed from electrocardiography and activity counts from accelerometry. We retrained our algorithm using two public datasets containing both healthy sleepers and sleep disordered patients. We then tested the performance of the algorithm on an independent hold-out validation set of sleep recordings from a wide range of sleep disorders collected in a tertiary sleep medicine center. Results The classifier achieved substantial agreement on four-class sleep staging (wake/N1–N2/N3/rapid eye movement [REM]), with an average κ of 0.60 and accuracy of 75.9%. The performance of the sleep staging algorithm was significantly higher in insomnia patients (κ = 0.62, accuracy = 77.3%). Only in REM parasomnias, the performance was significantly lower (κ = 0.47, accuracy = 70.5%). For two-class wake/sleep classification, the classifier achieved a κ of 0.65, with a sensitivity (to wake) of 72.9% and specificity of 94.0%. Conclusions This study shows that the combination of HRV, body movements, and a state-of-the-art deep neural network can reach substantial agreement in automatic sleep staging compared with polysomnography, even in patients suffering from a multitude of sleep disorders. The physiological signals required can be obtained in various ways, including non-obtrusive wrist-worn sensors, opening up new avenues for clinical diagnostics.

scholarly journals Exploring Efficient Neural Architectures for Linguistic–Acoustic Mapping in Text-To-Speech

2019 ◽  
Vol 9 (16) ◽  
pp. 3391 ◽  
Author(s):  
Santiago Pascual ◽  
Joan Serrà ◽  
Antonio Bonafonte
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Affine Transformations ◽  
Text To Speech ◽  
Recursive Structure ◽  
The One ◽  
Acoustic Mapping ◽  
Symbol Sequences

Conversion from text to speech relies on the accurate mapping from linguistic to acoustic symbol sequences, for which current practice employs recurrent statistical models such as recurrent neural networks. Despite the good performance of such models (in terms of low distortion in the generated speech), their recursive structure with intermediate affine transformations tends to make them slow to train and to sample from. In this work, we explore two different mechanisms that enhance the operational efficiency of recurrent neural networks, and study their performance–speed trade-off. The first mechanism is based on the quasi-recurrent neural network, where expensive affine transformations are removed from temporal connections and placed only on feed-forward computational directions. The second mechanism includes a module based on the transformer decoder network, designed without recurrent connections but emulating them with attention and positioning codes. Our results show that the proposed decoder networks are competitive in terms of distortion when compared to a recurrent baseline, whilst being significantly faster in terms of CPU and GPU inference time. The best performing model is the one based on the quasi-recurrent mechanism, reaching the same level of naturalness as the recurrent neural network based model with a speedup of 11.2 on CPU and 3.3 on GPU.

The significance of relevance trees in the identification of artificial neural networks input vectors

2013 ◽  
Vol 24 (1) ◽  
pp. 27-34
Author(s):  
G. Manuel ◽  
J.H.C. Pretorius
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Statistical Methods ◽  
Demand Forecasting ◽  
Forecast Model ◽  
Optimization Techniques ◽  
Model Parameters ◽  
Wide Range ◽  
Accurate Forecast ◽  
Artificial Neural

In the 1980s a renewed interest in artificial neural networks (ANN) has led to a wide range of applications which included demand forecasting. ANN demand forecasting algorithms were found to be preferable over parametric or also referred to as statistical based techniques. For an ANN demand forecasting algorithm, the demand may be stochastic or deterministic, linear or nonlinear. Comparative studies conducted on the two broad streams of demand forecasting methodologies, namely artificial intelligence methods and statistical methods has revealed that AI methods tend to hide the complexities of correlation analysis. In parametric methods, correlation is found by means of sometimes difficult and rigorous mathematics. Most statistical methods extract and correlate various demand elements which are usually broadly classed into weather and non-weather variables. Several models account for noise and random factors and suggest optimization techniques specific to certain model parameters. However, for an ANN algorithm, the identification of input and output vectors is critical. Predicting the future demand is conducted by observing previous demand values and how underlying factors influence the overall demand. Trend analyses are conducted on these influential variables and a medium and long term forecast model is derived. In order to perform an accurate forecast, the changes in the demand have to be defined in terms of how these input vectors correlate to the final demand. The elements of the input vectors have to be identifiable and quantifiable. This paper proposes a method known as relevance trees to identify critical elements of the input vector. The case study is of a rapid railway operator, namely the Gautrain.

NUMERICAL STUDIES ON THE VIBRATIONAL SPECTRUM OF FIBONACCI CHAIN: A MULTIFRACTAL ANALYSIS

1991 ◽  
Vol 05 (05) ◽  
pp. 825-841 ◽  
Author(s):  
WLODZIMIERZ SALEJDA
Keyword(s):  
Vibrational Spectrum ◽  
Multifractal Analysis ◽  
Scaling Laws ◽  
Model Parameters ◽  
Local Scaling ◽  
One Dimensional ◽  
Dynamic Matrix ◽  
Numerical Studies ◽  
Wide Range ◽  
The One

A harmonic Hamiltonian modelling the lattice dynamics of the one-dimensional Fibonacci-type quasicrystal is studied numerically. The multifractal analysis of vibrational spectrum is performed. It is found that the integrated normalized density of states [Formula: see text], where x denotes the square of the eigenenergy of the dynamic matrix, exhibits a finite range of scaling indices α (i.e. α min ≤α≤ α max ) describing the local scaling laws of [Formula: see text]. The α-f spectra and the Renyi dimensions [Formula: see text] are calculated in a wide range of model parameters taking into account the next-nearest-neighbour (NNN) interactions of atoms. In particular, we have observed that: (1) The α-f spectra are smooth in the interval [Formula: see text]; (2) If the so-called parameter of quasi-periodicity Q increases, then αmin and the fractal dimension of vibrational spectra [Formula: see text] decrease; (3) If the strength of NNN interactions h grows then α min decreases but D increases.

Elastic isotropic and anisotropic full wave-form inversions using automatic differentiation for gradient calculations in a framework of recurrent neural networks

Geophysics ◽  
2021 ◽  
pp. 1-56
Author(s):  
Wenlong Wang ◽  
George A. McMechan ◽  
Jianwei Ma
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Automatic Differentiation ◽  
Staggered Grid ◽  
Wave Form ◽  
Model Parameters ◽  
Elastic Model ◽  
Coherent Noise ◽  
Full Waveform ◽  
Isotropic Media

We implement multiparameter full waveform inversions in the framework of recurrent neural networks in elastic isotropic and transversely isotropic media. A staggered-grid velocity-stress scheme is used to solve the first-order elastodynamic equations for forward modeling. The gradients of loss with respect to model parameters are obtained by automatic differentiation. Multiple elastic model parameters are inverted simultaneously with a mini-batch optimizer. We prove the equivalency of full-batch automatic differentiation and the conventional adjoint-state method for inversions in elastic isotropic media. Synthetic tests on elastic isotropic models show that the mini-batch configuration has a better convergence rate and higher inversion accuracy than full-batch elastic full waveform inversions. Inversions with data that contain incoherent and coherent noise are tested, respectively. With automatic differentiation, we demonstrate the ease of extension to anisotropic media with two parameterizations, and the potential to implement it for more general media.

scholarly journals Gender Classification Based on The Non-Lexical Cues Of Emergency Calls With Recurrent Neural Networks (RNN)

Symmetry ◽  
2019 ◽  
Vol 11 (4) ◽  
pp. 525 ◽  
Author(s):  
SON ◽  
KWON ◽  
PARK
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Spectral Characteristics ◽  
Behavioral Approach ◽  
Support Vector ◽  
Gender Classification ◽  
Acoustic Feature ◽  
Emergency Calls ◽  
Wide Range ◽  
Speech Features

Automatic gender classification in speech is a challenging research field with a wide range of applications in HCI (humancomputer interaction). A couple of decades of research have shown promising results, but there is still a need for improvement. Until now, gender classification has been made using differences in the spectral characteristics of males and females. We assumed that a neutral margin exists between the male and female spectral range. This margin causes misclassification of gender. To address this limitation, we studied three non-lexical speech features (fillers, overlapping, and lengthening). From the statistical analysis, we found that overlapping and lengthening are effective in gender classification. Next, we performed gender classification using overlapping, lengthening, and the baseline acoustic feature, Mel Frequency Cepstral Coefficient (MFCC). We have tried to achieve the best results by using various combinations of features at the same time or sequentially. We used two types of machine-learning methods, support vector machine (SVM) and recurrent neural networks (RNN), to classify the gender. We achieved 89.61% with RNN using a feature set including MFCC, overlapping, and lengthening at the same time. Also, we have reclassified using non-lexical features with only data belonging to the neutral margin which was empirically selected based on the result of gender classification with only MFCC. As a result, we determined that the accuracy of classification with RNN using lengthening was 1.83% better than when MFCC alone was used. We concluded that new speech features could be effective in improving gender classification through a behavioral approach, notably including emergency calls.

scholarly journals Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing Its Gradient Estimator Bias

2021 ◽  
Vol 15 ◽  
Author(s):  
Axel Laborieux ◽  
Maxence Ernoult ◽  
Benjamin Scellier ◽  
Yoshua Bengio ◽  
Julie Grollier ◽  
...  
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Learning Rule ◽  
Deep Convolutional Neural Networks ◽  
Test Error ◽  
Backpropagation Through Time ◽  
Visual Tasks ◽  
Estimator Bias ◽  
Two Phases ◽  
The One

Equilibrium Propagation is a biologically-inspired algorithm that trains convergent recurrent neural networks with a local learning rule. This approach constitutes a major lead to allow learning-capable neuromophic systems and comes with strong theoretical guarantees. Equilibrium propagation operates in two phases, during which the network is let to evolve freely and then “nudged” toward a target; the weights of the network are then updated based solely on the states of the neurons that they connect. The weight updates of Equilibrium Propagation have been shown mathematically to approach those provided by Backpropagation Through Time (BPTT), the mainstream approach to train recurrent neural networks, when nudging is performed with infinitely small strength. In practice, however, the standard implementation of Equilibrium Propagation does not scale to visual tasks harder than MNIST. In this work, we show that a bias in the gradient estimate of equilibrium propagation, inherent in the use of finite nudging, is responsible for this phenomenon and that canceling it allows training deep convolutional neural networks. We show that this bias can be greatly reduced by using symmetric nudging (a positive nudging and a negative one). We also generalize Equilibrium Propagation to the case of cross-entropy loss (by opposition to squared error). As a result of these advances, we are able to achieve a test error of 11.7% on CIFAR-10, which approaches the one achieved by BPTT and provides a major improvement with respect to the standard Equilibrium Propagation that gives 86% test error. We also apply these techniques to train an architecture with unidirectional forward and backward connections, yielding a 13.2% test error. These results highlight equilibrium propagation as a compelling biologically-plausible approach to compute error gradients in deep neuromorphic systems.

scholarly journals Stochasticity invariance control in Pr1-xCaxMnO3 RRAM to enable large-scale stochastic recurrent neural networks

2021 ◽  
Author(s):  
Vivek Saraswat ◽  
Udayan Ganguly
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Time Distribution ◽  
Large Scale ◽  
Internal State ◽  
External Input ◽  
Problem Size ◽  
Boltzmann Machines ◽  
Wide Range ◽  
The Impact

Abstract Emerging non-volatile memories have been proposed for a wide range of applications, from easing the von-Neumann bottleneck to neuromorphic applications. Specifically, scalable RRAMs based on Pr1-xCaxMnO3 (PCMO) exhibit analog switching have been demonstrated as an integrating neuron, an analog synapse, and a voltage-controlled oscillator. More recently, the inherent stochasticity of memristors has been proposed for efficient hardware implementations of Boltzmann Machines. However, as the problem size scales, the number of neurons increases and controlling the stochastic distribution tightly over many iterations is necessary. This requires parametric control over stochasticity. Here, we characterize the stochastic Set in PCMO RRAMs. We identify that the Set time distribution depends on the internal state of the device (i.e., resistance) in addition to external input (i.e., voltage pulse). This requires the confluence of contradictory properties like stochastic switching as well as deterministic state control in the same device. Unlike ‘stochastic-everywhere’ filamentary memristors, in PCMO RRAMs, we leverage the (i) stochastic Set in negative polarity and (ii) deterministic analog Reset in positive polarity to demonstrate 100× reduced Set time distribution drift. The impact on Boltzmann Machines’ performance is analyzed and as opposed to the “fixed external input stochasticity”, the “state-monitored stochasticity” can solve problems 20× larger in size. State monitoring also tunes out the device-to-device variability effect on distributions providing 10× better performance. In addition to the physical insights, this study establishes the use of experimental stochasticity in PCMO RRAMs in stochastic recurrent neural networks reliably over many iterations.

Sign in / Sign up

Export Citation Format

Share Document