scholarly journals Direct Training for Spiking Neural Networks: Faster, Larger, Better

2019 ◽  
Vol 33 ◽  
pp. 1311-1318 ◽  
Author(s):  
Yujie Wu ◽  
Lei Deng ◽  
Guoqi Li ◽  
Jun Zhu ◽  
Yuan Xie ◽  
...  
Keyword(s):  
Neural Networks ◽  
High Performance ◽  
Large Scale ◽  
Learning Algorithm ◽  
Efficient Implementation ◽  
Spiking Neural Networks ◽  
Direct Training ◽  
Neuromorphic Hardware ◽  
Rate Coding ◽  
Neural Selectivity

Spiking neural networks (SNNs) that enables energy efficient implementation on emerging neuromorphic hardware are gaining more attention. Yet now, SNNs have not shown competitive performance compared with artificial neural networks (ANNs), due to the lack of effective learning algorithms and efficient programming frameworks. We address this issue from two aspects: (1) We propose a neuron normalization technique to adjust the neural selectivity and develop a direct learning algorithm for deep SNNs. (2) Via narrowing the rate coding window and converting the leaky integrate-and-fire (LIF) model into an explicitly iterative version, we present a Pytorch-based implementation method towards the training of large-scale SNNs. In this way, we are able to train deep SNNs with tens of times speedup. As a result, we achieve significantly better accuracy than the reported works on neuromorphic datasets (N-MNIST and DVSCIFAR10), and comparable accuracy as existing ANNs and pre-trained SNNs on non-spiking datasets (CIFAR10). To our best knowledge, this is the first work that demonstrates direct training of deep SNNs with high performance on CIFAR10, and the efficient implementation provides a new way to explore the potential of SNNs.

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

2021 ◽  
pp. 1-1
Author(s):  
Xiaolei Zhu ◽  
Baixin Zhao ◽  
De Ma ◽  
Huajin Tang
Keyword(s):  
Neural Networks ◽  
Learning Algorithm ◽  
Spiking Neural Networks ◽  
Direct Training ◽  
Efficient Learning

A New Backpropagation Learning Algorithm for Layered Neural Networks with Nondifferentiable Units

2007 ◽  
Vol 19 (5) ◽  
pp. 1422-1435 ◽  
Author(s):  
Takahumi Oohori ◽  
Hidenori Naganuma ◽  
Kazuhisa Watanabe
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
High Performance ◽  
Large Scale ◽  
Learning Algorithm ◽  
Backpropagation Algorithm ◽  
Digital Version ◽  
Output Layer ◽  
Simulation Results ◽  
Large Scale Networks

We propose a digital version of the backpropagation algorithm (DBP) for three-layered neural networks with nondifferentiable binary units. This approach feeds teacher signals to both the middle and output layers, whereas with a simple perceptron, they are given only to the output layer. The additional teacher signals enable the DBP to update the coupling weights not only between the middle and output layers but also between the input and middle layers. A neural network based on DBP learning is fast and easy to implement in hardware. Simulation results for several linearly nonseparable problems such as XOR demonstrate that the DBP performs favorably when compared to the conventional approaches. Furthermore, in large-scale networks, simulation results indicate that the DBP provides high performance.

Large-Scale Spiking Neural Networks using Neuromorphic Hardware Compatible Models

2015 ◽  
Vol 11 (4) ◽  
pp. 1-18 ◽  
Author(s):  
Jeffrey L. Krichmar ◽  
Philippe Coussy ◽  
Nikil Dutt
Keyword(s):  
Neural Networks ◽  
Large Scale ◽  
Spiking Neural Networks ◽  
Neuromorphic Hardware

scholarly journals LISNN: Improving Spiking Neural Networks with Lateral Interactions for Robust Object Recognition

2020 ◽  
Author(s):  
Xiang Cheng ◽  
Yunzhe Hao ◽  
Jiaming Xu ◽  
Bo Xu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Object Recognition ◽  
Visual Information ◽  
High Performance ◽  
Spiking Neural Networks ◽  
Noise Robustness ◽  
Spiking Neural Network ◽  
Lateral Interactions ◽  
Neuromorphic Hardware

Spiking Neural Network (SNN) is considered more biologically plausible and energy-efficient on emerging neuromorphic hardware. Recently backpropagation algorithm has been utilized for training SNN, which allows SNN to go deeper and achieve higher performance. However, most existing SNN models for object recognition are mainly convolutional structures or fully-connected structures, which only have inter-layer connections, but no intra-layer connections. Inspired by Lateral Interactions in neuroscience, we propose a high-performance and noise-robust Spiking Neural Network (dubbed LISNN). Based on the convolutional SNN, we model the lateral interactions between spatially adjacent neurons and integrate it into the spiking neuron membrane potential formula, then build a multi-layer SNN on a popular deep learning framework, i.\,e., PyTorch. We utilize the pseudo-derivative method to solve the non-differentiable problem when applying backpropagation to train LISNN and test LISNN on multiple standard datasets. Experimental results demonstrate that the proposed model can achieve competitive or better performance compared to current state-of-the-art spiking neural networks on MNIST, Fashion-MNIST, and N-MNIST datasets. Besides, thanks to lateral interactions, our model processes stronger noise-robustness than other SNN. Our work brings a biologically plausible mechanism into SNN, hoping that it can help us understand the visual information processing in the brain.

A Developmental Method for Evolving Large-Scale Spiking Neural Networks

2012 ◽  
Vol 35 (12) ◽  
pp. 2633 ◽  
Author(s):  
Xiang-Hong LIN ◽  
Tian-Wen ZHANG ◽  
Gui-Cang ZHANG
Keyword(s):  
Neural Networks ◽  
Large Scale ◽  
Spiking Neural Networks

scholarly journals Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms

Sensors ◽  
2021 ◽  
Vol 21 (9) ◽  
pp. 3240
Author(s):  
Tehreem Syed ◽  
Vijay Kakani ◽  
Xuenan Cui ◽  
Hakil Kim
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Spiking Neural Networks ◽  
License Plate ◽  
Training Techniques ◽  
Neuromorphic Hardware ◽  
Private And Public ◽  
Embedded Platforms ◽  
Public Datasets ◽  
Event Based

In recent times, the usage of modern neuromorphic hardware for brain-inspired SNNs has grown exponentially. In the context of sparse input data, they are undertaking low power consumption for event-based neuromorphic hardware, specifically in the deeper layers. However, using deep ANNs for training spiking models is still considered as a tedious task. Until recently, various ANN to SNN conversion methods in the literature have been proposed to train deep SNN models. Nevertheless, these methods require hundreds to thousands of time-steps for training and still cannot attain good SNN performance. This work proposes a customized model (VGG, ResNet) architecture to train deep convolutional spiking neural networks. In this current study, the training is carried out using deep convolutional spiking neural networks with surrogate gradient descent backpropagation in a customized layer architecture similar to deep artificial neural networks. Moreover, this work also proposes fewer time-steps for training SNNs with surrogate gradient descent. During the training with surrogate gradient descent backpropagation, overfitting problems have been encountered. To overcome these problems, this work refines the SNN based dropout technique with surrogate gradient descent. The proposed customized SNN models achieve good classification results on both private and public datasets. In this work, several experiments have been carried out on an embedded platform (NVIDIA JETSON TX2 board), where the deployment of customized SNN models has been extensively conducted. Performance validations have been carried out in terms of processing time and inference accuracy between PC and embedded platforms, showing that the proposed customized models and training techniques are feasible for achieving a better performance on various datasets such as CIFAR-10, MNIST, SVHN, and private KITTI and Korean License plate dataset.

Effective Transfer Learning Algorithm in Spiking Neural Networks

2021 ◽  
pp. 1-13
Author(s):  
Qiugang Zhan ◽  
Guisong Liu ◽  
Xiurui Xie ◽  
Guolin Sun ◽  
Huajin Tang
Keyword(s):  
Neural Networks ◽  
Transfer Learning ◽  
Learning Algorithm ◽  
Spiking Neural Networks ◽  
Effective Transfer

A memory-efficient routing method for large-scale spiking neural networks

2013 ◽  
Author(s):  
Saber Moradi ◽  
Nabil Imam ◽  
Rajit Manohar ◽  
Giacomo Indiveri
Keyword(s):  
Neural Networks ◽  
Large Scale ◽  
Spiking Neural Networks ◽  
Routing Method ◽  
Memory Efficient

Chapter 9. Spike-Based Symbolic Computations on Bit Strings and Numbers

2021 ◽  
Author(s):  
Ceca Kraišniković ◽  
Wolfgang Maass ◽  
Robert Legenstein
Keyword(s):  
Neural Networks ◽  
Energy Efficient ◽  
Learning Rule ◽  
Spiking Neural Networks ◽  
New Paradigm ◽  
Symbolic Computations ◽  
Neuromorphic Hardware ◽  
State Of Research ◽  
Higher Cognitive Functions ◽  
The Brain

The brain uses recurrent spiking neural networks for higher cognitive functions such as symbolic computations, in particular, mathematical computations. We review the current state of research on spike-based symbolic computations of this type. In addition, we present new results which show that surprisingly small spiking neural networks can perform symbolic computations on bit sequences and numbers and even learn such computations using a biologically plausible learning rule. The resulting networks operate in a rather low firing rate regime, where they could not simply emulate artificial neural networks by encoding continuous values through firing rates. Thus, we propose here a new paradigm for symbolic computation in neural networks that provides concrete hypotheses about the organization of symbolic computations in the brain. The employed spike-based network models are the basis for drastically more energy-efficient computer hardware – neuromorphic hardware. Hence, our results can be seen as creating a bridge from symbolic artificial intelligence to energy-efficient implementation in spike-based neuromorphic hardware.

Sign in / Sign up

Export Citation Format

Share Document