scholarly journals Event-based backpropagation can compute exact gradients for spiking neural networks

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Timo C. Wunderlich ◽  
Christian Pehle
Keyword(s):  
Neural Networks ◽  
Loss Function ◽  
Spiking Neural Networks ◽  
Backpropagation Algorithm ◽  
General Loss Function ◽  
Gradient Based ◽  
Rigorous Study ◽  
Spiking Networks ◽  
Event Based ◽  
First Time

AbstractSpiking neural networks combine analog computation with event-based communication using discrete spikes. While the impressive advances of deep learning are enabled by training non-spiking artificial neural networks using the backpropagation algorithm, applying this algorithm to spiking networks was previously hindered by the existence of discrete spike events and discontinuities. For the first time, this work derives the backpropagation algorithm for a continuous-time spiking neural network and a general loss function by applying the adjoint method together with the proper partial derivative jumps, allowing for backpropagation through discrete spike events without approximations. This algorithm, EventProp, backpropagates errors at spike times in order to compute the exact gradient in an event-based, temporally and spatially sparse fashion. We use gradients computed via EventProp to train networks on the Yin-Yang and MNIST datasets using either a spike time or voltage based loss function and report competitive performance. Our work supports the rigorous study of gradient-based learning algorithms in spiking neural networks and provides insights toward their implementation in novel brain-inspired hardware.

scholarly journals Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms

Sensors ◽  
2021 ◽  
Vol 21 (9) ◽  
pp. 3240
Author(s):  
Tehreem Syed ◽  
Vijay Kakani ◽  
Xuenan Cui ◽  
Hakil Kim
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Spiking Neural Networks ◽  
License Plate ◽  
Training Techniques ◽  
Neuromorphic Hardware ◽  
Private And Public ◽  
Embedded Platforms ◽  
Public Datasets ◽  
Event Based

In recent times, the usage of modern neuromorphic hardware for brain-inspired SNNs has grown exponentially. In the context of sparse input data, they are undertaking low power consumption for event-based neuromorphic hardware, specifically in the deeper layers. However, using deep ANNs for training spiking models is still considered as a tedious task. Until recently, various ANN to SNN conversion methods in the literature have been proposed to train deep SNN models. Nevertheless, these methods require hundreds to thousands of time-steps for training and still cannot attain good SNN performance. This work proposes a customized model (VGG, ResNet) architecture to train deep convolutional spiking neural networks. In this current study, the training is carried out using deep convolutional spiking neural networks with surrogate gradient descent backpropagation in a customized layer architecture similar to deep artificial neural networks. Moreover, this work also proposes fewer time-steps for training SNNs with surrogate gradient descent. During the training with surrogate gradient descent backpropagation, overfitting problems have been encountered. To overcome these problems, this work refines the SNN based dropout technique with surrogate gradient descent. The proposed customized SNN models achieve good classification results on both private and public datasets. In this work, several experiments have been carried out on an embedded platform (NVIDIA JETSON TX2 board), where the deployment of customized SNN models has been extensively conducted. Performance validations have been carried out in terms of processing time and inference accuracy between PC and embedded platforms, showing that the proposed customized models and training techniques are feasible for achieving a better performance on various datasets such as CIFAR-10, MNIST, SVHN, and private KITTI and Korean License plate dataset.

scholarly journals The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks

2020 ◽  
Author(s):  
Friedemann Zenke ◽  
Tim P. Vogels
Keyword(s):  
Neural Networks ◽  
Complex Function ◽  
Spiking Neural Networks ◽  
Learning Performance ◽  
Design Parameters ◽  
Classification Problems ◽  
Systematic Account ◽  
Practical Algorithms ◽  
Spiking Networks ◽  
Gradient Learning

AbstractBrains process information in spiking neural networks. Their intricate connections shape the diverse functions these networks perform. In comparison, the functional capabilities of models of spiking networks are still rudimentary. This shortcoming is mainly due to the lack of insight and practical algorithms to construct the necessary connectivity. Any such algorithm typically attempts to build networks by iteratively reducing the error compared to a desired output. But assigning credit to hidden units in multi-layered spiking networks has remained challenging due to the non-differentiable nonlinearity of spikes. To avoid this issue, one can employ surrogate gradients to discover the required connectivity in spiking network models. However, the choice of a surrogate is not unique, raising the question of how its implementation influences the effectiveness of the method. Here, we use numerical simulations to systematically study how essential design parameters of surrogate gradients impact learning performance on a range of classification problems. We show that surrogate gradient learning is robust to different shapes of underlying surrogate derivatives, but the choice of the derivative’s scale can substantially affect learning performance. When we combine surrogate gradients with a suitable activity regularization technique, robust information processing can be achieved in spiking networks even at the sparse activity limit. Our study provides a systematic account of the remarkable robustness of surrogate gradient learning and serves as a practical guide to model functional spiking neural networks.

A Bayesian Model of Polychronicity

2014 ◽  
Vol 26 (9) ◽  
pp. 2052-2073 ◽  
Author(s):  
Mira Guise ◽  
Alistair Knott ◽  
Lubica Benuskova
Keyword(s):  
Neural Networks ◽  
Spiking Neural Networks ◽  
Familiar Stimulus ◽  
Significant Feature ◽  
Neural Basis ◽  
Representational System ◽  
Large Numbers ◽  
Strongly Connected ◽  
Activation Response ◽  
First Time

A significant feature of spiking neural networks with varying connection delays, such as those in the brain, is the existence of strongly connected groups of neurons known as polychronous neural groups (PNGs). Polychronous groups are found in large numbers in these networks and are proposed by Izhikevich ( 2006a ) to provide a neural basis for representation and memory. When exposed to a familiar stimulus, spiking neural networks produce consistencies in the spiking output data that are the hallmarks of PNG activation. Previous methods for studying the PNG activation response to stimuli have been limited by the template-based methods used to identify PNG activation. In this letter, we outline a new method that overcomes these difficulties by establishing for the first time a probabilistic interpretation of PNG activation. We then demonstrate the use of this method by investigating the claim that PNGs might provide the foundation of a representational system.

Self-Supervised Optical Flow with Spiking Neural Networks and Event Based Cameras

2021 ◽  
Author(s):  
Kenneth Chaney ◽  
Artemis Panagopoulou ◽  
Chankyu Lee ◽  
Kaushik Roy ◽  
Kostas Daniilidis
Keyword(s):  
Neural Networks ◽  
Optical Flow ◽  
Spiking Neural Networks ◽  
Event Based

scholarly journals A Markovian event-based framework for stochastic spiking neural networks

2011 ◽  
Vol 31 (3) ◽  
pp. 485-507 ◽  
Author(s):  
Jonathan D. Touboul ◽  
Olivier D. Faugeras
Keyword(s):  
Neural Networks ◽  
Spiking Neural Networks ◽  
Event Based

scholarly journals An artificial spiking afferent nerve based on Mott memristors for neurorobotics

2020 ◽  
Vol 11 (1) ◽  
Author(s):  
Xumeng Zhang ◽  
Ye Zhuo ◽  
Qing Luo ◽  
Zuheng Wu ◽  
Rivu Midya ◽  
...  
Keyword(s):  
Neural Networks ◽  
Complementary Metal Oxide Semiconductor ◽  
Tactile Sensor ◽  
Metal Oxide Semiconductor ◽  
Afferent Nerve ◽  
Spiking Neural Networks ◽  
Oxide Semiconductor ◽  
Semiconductor Technology ◽  
Hardware Implementations ◽  
First Time

AbstractNeuromorphic computing based on spikes offers great potential in highly efficient computing paradigms. Recently, several hardware implementations of spiking neural networks based on traditional complementary metal-oxide semiconductor technology or memristors have been developed. However, an interface (called an afferent nerve in biology) with the environment, which converts the analog signal from sensors into spikes in spiking neural networks, is yet to be demonstrated. Here we propose and experimentally demonstrate an artificial spiking afferent nerve based on highly reliable NbOx Mott memristors for the first time. The spiking frequency of the afferent nerve is proportional to the stimuli intensity before encountering noxiously high stimuli, and then starts to reduce the spiking frequency at an inflection point. Using this afferent nerve, we further build a power-free spiking mechanoreceptor system with a passive piezoelectric device as the tactile sensor. The experimental results indicate that our afferent nerve is promising for constructing self-aware neurorobotics in the future.

scholarly journals eWB: Event-based weight binarization algorithm for spiking neural networks

IEEE Access ◽  
2021 ◽  
pp. 1-1
Author(s):  
Dohun Kim ◽  
Guhyun Kim ◽  
Cheol Seong Hwang ◽  
Doo Seok Jeong
Keyword(s):  
Neural Networks ◽  
Spiking Neural Networks ◽  
Binarization Algorithm ◽  
Event Based

scholarly journals The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks

2021 ◽  
pp. 1-27
Author(s):  
Friedemann Zenke ◽  
Tim P. Vogels
Keyword(s):  
Neural Networks ◽  
Information Processing ◽  
Complex Function ◽  
Spiking Neural Networks ◽  
Learning Performance ◽  
Design Parameters ◽  
Practical Algorithms ◽  
Neuromorphic Hardware ◽  
Spiking Networks ◽  
Gradient Learning

Brains process information in spiking neural networks. Their intricate connections shape the diverse functions these networks perform. Yet how network connectivity relates to function is poorly understood, and the functional capabilities of models of spiking networks are still rudimentary. The lack of both theoretical insight and practical algorithms to find the necessary connectivity poses a major impediment to both studying information processing in the brain and building efficient neuromorphic hardware systems. The training algorithms that solve this problem for artificial neural networks typically rely on gradient descent. But doing so in spiking networks has remained challenging due to the nondifferentiable nonlinearity of spikes. To avoid this issue, one can employ surrogate gradients to discover the required connectivity. However, the choice of a surrogate is not unique, raising the question of how its implementation influences the effectiveness of the method. Here, we use numerical simulations to systematically study how essential design parameters of surrogate gradients affect learning performance on a range of classification problems. We show that surrogate gradient learning is robust to different shapes of underlying surrogate derivatives, but the choice of the derivative's scale can substantially affect learning performance. When we combine surrogate gradients with suitable activity regularization techniques, spiking networks perform robust information processing at the sparse activity limit. Our study provides a systematic account of the remarkable robustness of surrogate gradient learning and serves as a practical guide to model functional spiking neural networks.

scholarly journals Evolving Spiking Networks with Variable Resistive Memories

2014 ◽  
Vol 22 (1) ◽  
pp. 79-103 ◽  
Author(s):  
Gerard Howard ◽  
Larry Bull ◽  
Ben de Lacy Costello ◽  
Ella Gale ◽  
Andrew Adamatzky
Keyword(s):  
Neural Networks ◽  
Adaptive Learning ◽  
Degrees Of Freedom ◽  
Spiking Neural Networks ◽  
Evolutionary Design ◽  
Evolutionary System ◽  
Learning Mechanisms ◽  
Resistive Networks ◽  
Self Adaptation ◽  
Spiking Networks

Neuromorphic computing is a brainlike information processing paradigm that requires adaptive learning mechanisms. A spiking neuro-evolutionary system is used for this purpose; plastic resistive memories are implemented as synapses in spiking neural networks. The evolutionary design process exploits parameter self-adaptation and allows the topology and synaptic weights to be evolved for each network in an autonomous manner. Variable resistive memories are the focus of this research; each synapse has its own conductance profile which modifies the plastic behaviour of the device and may be altered during evolution. These variable resistive networks are evaluated on a noisy robotic dynamic-reward scenario against two static resistive memories and a system containing standard connections only. The results indicate that the extra behavioural degrees of freedom available to the networks incorporating variable resistive memories enable them to outperform the comparative synapse types.

Sign in / Sign up

Export Citation Format

Share Document