scholarly journals Revisiting Batch Normalization for Training Low-Latency Deep Spiking Neural Networks From Scratch

2021 ◽  
Vol 15 ◽  
Author(s):  
Youngeun Kim ◽  
Priyadarshini Panda
Keyword(s):  
Neural Networks ◽  
Temporal Dynamics ◽  
Temporal Characteristic ◽  
Spiking Neural Networks ◽  
Time Step ◽  
Intelligent Computing ◽  
Early Exit ◽  
Batch Normalization ◽  
Binary Event ◽  
Neuromorphic Hardware

Spiking Neural Networks (SNNs) have recently emerged as an alternative to deep learning owing to sparse, asynchronous and binary event (or spike) driven processing, that can yield huge energy efficiency benefits on neuromorphic hardware. However, SNNs convey temporally-varying spike activation through time that is likely to induce a large variation of forward activation and backward gradients, resulting in unstable training. To address this training issue in SNNs, we revisit Batch Normalization (BN) and propose a temporal Batch Normalization Through Time (BNTT) technique. Different from previous BN techniques with SNNs, we find that varying the BN parameters at every time-step allows the model to learn the time-varying input distribution better. Specifically, our proposed BNTT decouples the parameters in a BNTT layer along the time axis to capture the temporal dynamics of spikes. We demonstrate BNTT on CIFAR-10, CIFAR-100, Tiny-ImageNet, event-driven DVS-CIFAR10 datasets, and Sequential MNIST and show near state-of-the-art performance. We conduct comprehensive analysis on the temporal characteristic of BNTT and showcase interesting benefits toward robustness against random and adversarial noise. Further, by monitoring the learnt parameters of BNTT, we find that we can do temporal early exit. That is, we can reduce the inference latency by ~5 − 20 time-steps from the original training latency. The code has been released at https://github.com/Intelligent-Computing-Lab-Yale/BNTT-Batch-Normalization-Through-Time.

scholarly journals Low‐Voltage Electrochemical Li x WO 3 Synapses with Temporal Dynamics for Spiking Neural Networks

2021 ◽  
pp. 2100021
Author(s):  
Qingzhou Wan ◽  
Marco Rasetto ◽  
Mohammad T. Sharbati ◽  
John R. Erickson ◽  
Sridhar Reddy Velagala ◽  
...  
Keyword(s):  
Neural Networks ◽  
Temporal Dynamics ◽  
Low Voltage ◽  
Spiking Neural Networks

scholarly journals Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms

Sensors ◽  
2021 ◽  
Vol 21 (9) ◽  
pp. 3240
Author(s):  
Tehreem Syed ◽  
Vijay Kakani ◽  
Xuenan Cui ◽  
Hakil Kim
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Spiking Neural Networks ◽  
License Plate ◽  
Training Techniques ◽  
Neuromorphic Hardware ◽  
Private And Public ◽  
Embedded Platforms ◽  
Public Datasets ◽  
Event Based

In recent times, the usage of modern neuromorphic hardware for brain-inspired SNNs has grown exponentially. In the context of sparse input data, they are undertaking low power consumption for event-based neuromorphic hardware, specifically in the deeper layers. However, using deep ANNs for training spiking models is still considered as a tedious task. Until recently, various ANN to SNN conversion methods in the literature have been proposed to train deep SNN models. Nevertheless, these methods require hundreds to thousands of time-steps for training and still cannot attain good SNN performance. This work proposes a customized model (VGG, ResNet) architecture to train deep convolutional spiking neural networks. In this current study, the training is carried out using deep convolutional spiking neural networks with surrogate gradient descent backpropagation in a customized layer architecture similar to deep artificial neural networks. Moreover, this work also proposes fewer time-steps for training SNNs with surrogate gradient descent. During the training with surrogate gradient descent backpropagation, overfitting problems have been encountered. To overcome these problems, this work refines the SNN based dropout technique with surrogate gradient descent. The proposed customized SNN models achieve good classification results on both private and public datasets. In this work, several experiments have been carried out on an embedded platform (NVIDIA JETSON TX2 board), where the deployment of customized SNN models has been extensively conducted. Performance validations have been carried out in terms of processing time and inference accuracy between PC and embedded platforms, showing that the proposed customized models and training techniques are feasible for achieving a better performance on various datasets such as CIFAR-10, MNIST, SVHN, and private KITTI and Korean License plate dataset.

Chapter 9. Spike-Based Symbolic Computations on Bit Strings and Numbers

2021 ◽  
Author(s):  
Ceca Kraišniković ◽  
Wolfgang Maass ◽  
Robert Legenstein
Keyword(s):  
Neural Networks ◽  
Energy Efficient ◽  
Learning Rule ◽  
Spiking Neural Networks ◽  
New Paradigm ◽  
Symbolic Computations ◽  
Neuromorphic Hardware ◽  
State Of Research ◽  
Higher Cognitive Functions ◽  
The Brain

The brain uses recurrent spiking neural networks for higher cognitive functions such as symbolic computations, in particular, mathematical computations. We review the current state of research on spike-based symbolic computations of this type. In addition, we present new results which show that surprisingly small spiking neural networks can perform symbolic computations on bit sequences and numbers and even learn such computations using a biologically plausible learning rule. The resulting networks operate in a rather low firing rate regime, where they could not simply emulate artificial neural networks by encoding continuous values through firing rates. Thus, we propose here a new paradigm for symbolic computation in neural networks that provides concrete hypotheses about the organization of symbolic computations in the brain. The employed spike-based network models are the basis for drastically more energy-efficient computer hardware – neuromorphic hardware. Hence, our results can be seen as creating a bridge from symbolic artificial intelligence to energy-efficient implementation in spike-based neuromorphic hardware.

Darwin: a neuromorphic hardware co-processor based on Spiking Neural Networks

2015 ◽  
Vol 59 (2) ◽  
pp. 1-5 ◽  
Author(s):  
Juncheng Shen ◽  
De Ma ◽  
Zonghua Gu ◽  
Ming Zhang ◽  
Xiaolei Zhu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Spiking Neural Networks ◽  
Neuromorphic Hardware

scholarly journals A Scatter-and-Gather Spiking Convolutional Neural Network on a Reconfigurable Neuromorphic Hardware

2021 ◽  
Vol 15 ◽  
Author(s):  
Chenglong Zou ◽  
Xiaoxin Cui ◽  
Yisong Kuang ◽  
Kefei Liu ◽  
Yuan Wang ◽  
...  
Keyword(s):  
Neural Networks ◽  
Large Scale ◽  
Average Power ◽  
Time Step ◽  
Mapping Algorithm ◽  
Learning Tasks ◽  
Main Challenge ◽  
Neuromorphic Hardware ◽  
Average Power Consumption ◽  
On Chip

Artificial neural networks (ANNs), like convolutional neural networks (CNNs), have achieved the state-of-the-art results for many machine learning tasks. However, inference with large-scale full-precision CNNs must cause substantial energy consumption and memory occupation, which seriously hinders their deployment on mobile and embedded systems. Highly inspired from biological brain, spiking neural networks (SNNs) are emerging as new solutions because of natural superiority in brain-like learning and great energy efficiency with event-driven communication and computation. Nevertheless, training a deep SNN remains a main challenge and there is usually a big accuracy gap between ANNs and SNNs. In this paper, we introduce a hardware-friendly conversion algorithm called “scatter-and-gather” to convert quantized ANNs to lossless SNNs, where neurons are connected with ternary {−1,0,1} synaptic weights. Each spiking neuron is stateless and more like original McCulloch and Pitts model, because it fires at most one spike and need be reset at each time step. Furthermore, we develop an incremental mapping framework to demonstrate efficient network deployments on a reconfigurable neuromorphic chip. Experimental results show our spiking LeNet on MNIST and VGG-Net on CIFAR-10 datasetobtain 99.37% and 91.91% classification accuracy, respectively. Besides, the presented mapping algorithm manages network deployment on our neuromorphic chip with maximum resource efficiency and excellent flexibility. Our four-spike LeNet and VGG-Net on chip can achieve respective real-time inference speed of 0.38 ms/image, 3.24 ms/image, and an average power consumption of 0.28 mJ/image and 2.3 mJ/image at 0.9 V, 252 MHz, which is nearly two orders of magnitude more efficient than traditional GPUs.

scholarly journals Pattern Recognition of Spiking Neural Networks Based on Visual Mechanism and Supervised Synaptic Learning

2020 ◽  
Vol 2020 ◽  
pp. 1-11
Author(s):  
Xiumin Li ◽  
Hao Yi ◽  
Shengyuan Luo
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Visual Cortex ◽  
Image Classification ◽  
Learning Rule ◽  
Spiking Neural Networks ◽  
Layer By Layer ◽  
Intelligent Computing ◽  
Training Samples ◽  
Feed Forward Network

Electrophysiological studies have shown that mammalian primary visual cortex are selective for the orientations of visual stimuli. Inspired by this mechanism, we propose a hierarchical spiking neural network (SNN) for image classification. Grayscale input images are fed through a feed-forward network consisting of orientation-selective neurons, which then projected to a layer of downstream classifier neurons through the spiking-based supervised tempotron learning rule. Based on the orientation-selective mechanism of the visual cortex and tempotron learning rule, the network can effectively classify images of the extensively studied MNIST database of handwritten digits, which achieves 96 % classification accuracy based on only 2000 training samples (traditional training set is 60000 ). Compared with other classification methods, our model not only guarantees the biological plausibility and the accuracy of image classification but also significantly reduces the needed training samples. Considering the fact that the most commonly used deep learning neural networks need big data samples and high power consumption in image recognition, this brain-inspired computational neural network model based on the layer-by-layer hierarchical image processing mechanism of the visual cortex may provide a basis for the wide application of spiking neural networks in the field of intelligent computing.

scholarly journals Run-time Mapping of Spiking Neural Networks to Neuromorphic Hardware

2020 ◽  
Vol 92 (11) ◽  
pp. 1293-1302
Author(s):  
Adarsha Balaji ◽  
Thibaut Marty ◽  
Anup Das ◽  
Francky Catthoor
Keyword(s):  
Neural Networks ◽  
Spiking Neural Networks ◽  
Neuromorphic Hardware ◽  
Run Time

scholarly journals Mapping Spiking Neural Networks to Neuromorphic Hardware

2020 ◽  
Vol 28 (1) ◽  
pp. 76-86 ◽  
Author(s):  
Adarsha Balaji ◽  
Francky Catthoor ◽  
Anup Das ◽  
Yuefeng Wu ◽  
Khanh Huynh ◽  
...  
Keyword(s):  
Neural Networks ◽  
Spiking Neural Networks ◽  
Neuromorphic Hardware
Sign in / Sign up

Export Citation Format

Share Document