A GPU Optimized Technique for Scalable Spiking Neural Network Simulation

2019 ◽  
Vol 8 (3) ◽  
pp. 4612-4616
Keyword(s):  
Neural Networks ◽  
High Performance ◽  
Parameter Tuning ◽  
Spiking Neural Networks ◽  
Internal Variables ◽  
Time Step ◽  
Simulation Speed ◽  
Active Synapse ◽  
And Performance ◽  
System Variables

Simulation studies, in general, heavily rely upon the internal variables of the system / entity in the studies. In case of simulation study of the Spiking Neural Networks (SNNs), the major internal system variables are membrane potentials of the neurons and their respective synaptic inputs which demand to be updated at a sub-millisecond resolution. It would be very apt here to note that this requires thousands of updates to simulate one second of an activity per neuron and this factor makes it imperative to have a highly scalable model to derive some inferences from the simulation. Conventionally, high performance CPUs with high degree of multi-threading were leveraged to conduct simulations and derive inferences. With the advances in the hardware, the degree of parallelism has also increased, especially the GPUs have opened a multitude of avenues to perform SNN simulations at scale. In our pervious works [1, 2, 3], we have demonstrated how GPUs can be leveraged to achieve scalability and performance by using hybrid CPU-GPU approach which have improved the performance as compared to multi-threading on high performance CPUs. In this work, we have focused on hyper parameter tuning of some of the key parameters such as delay insensitivity, time step grouping and the active synapse grouping to achieve greater simulation speed of scalable spiking neural networks

Spike: A GPU Optimised Spiking Neural Network Simulator

2018 ◽  
Author(s):  
Nasir Ahmad ◽  
James B. Isbister ◽  
Toby St. Clere Smithe ◽  
Simon M. Stringer
Keyword(s):  
Neural Network ◽  
High Performance ◽  
Internal Variables ◽  
Spiking Neural Network ◽  
Network Simulator ◽  
Synaptic Inputs ◽  
Active Synapse ◽  
Order Of Magnitude ◽  
Gpu Parallelism ◽  
Simulation Time

ABSTRACTSpiking Neural Network (SNN) simulations require internal variables – such as the membrane voltages of individual neurons and their synaptic inputs – to be updated on a sub-millisecond resolution. As a result, a single second of simulation time requires many thousands of update calculations per neuron. Furthermore, increases in the scale of SNN models have, accordingly, led to manyfold increases in the runtime of SNN simulations. Existing solutions to this problem of scale include high performance CPU based simulators capable of multithreaded execution (“CPU parallelism”). More recent GPU based simulators have emerged, which aim to utilise GPU parallelism for SNN execution. We have identified several key speedups, which give GPU based simulators up to an order of magnitude performance increase over CPU based simulators on several benchmarks. We present the Spike simulator with three key optimisations: timestep grouping, active synapse grouping, and delay insensitivity. Combined, these optimisations massively increase the speed of executing a SNN simulation and produce a simulator which is, on a single machine, faster than currently available simulators.

scholarly journals Revisiting Batch Normalization for Training Low-Latency Deep Spiking Neural Networks From Scratch

2021 ◽  
Vol 15 ◽  
Author(s):  
Youngeun Kim ◽  
Priyadarshini Panda
Keyword(s):  
Neural Networks ◽  
Temporal Dynamics ◽  
Temporal Characteristic ◽  
Spiking Neural Networks ◽  
Time Step ◽  
Intelligent Computing ◽  
Early Exit ◽  
Batch Normalization ◽  
Binary Event ◽  
Neuromorphic Hardware

Spiking Neural Networks (SNNs) have recently emerged as an alternative to deep learning owing to sparse, asynchronous and binary event (or spike) driven processing, that can yield huge energy efficiency benefits on neuromorphic hardware. However, SNNs convey temporally-varying spike activation through time that is likely to induce a large variation of forward activation and backward gradients, resulting in unstable training. To address this training issue in SNNs, we revisit Batch Normalization (BN) and propose a temporal Batch Normalization Through Time (BNTT) technique. Different from previous BN techniques with SNNs, we find that varying the BN parameters at every time-step allows the model to learn the time-varying input distribution better. Specifically, our proposed BNTT decouples the parameters in a BNTT layer along the time axis to capture the temporal dynamics of spikes. We demonstrate BNTT on CIFAR-10, CIFAR-100, Tiny-ImageNet, event-driven DVS-CIFAR10 datasets, and Sequential MNIST and show near state-of-the-art performance. We conduct comprehensive analysis on the temporal characteristic of BNTT and showcase interesting benefits toward robustness against random and adversarial noise. Further, by monitoring the learnt parameters of BNTT, we find that we can do temporal early exit. That is, we can reduce the inference latency by ~5 − 20 time-steps from the original training latency. The code has been released at https://github.com/Intelligent-Computing-Lab-Yale/BNTT-Batch-Normalization-Through-Time.

scholarly journals Visual explanations from spiking neural networks using inter-spike intervals

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Youngeun Kim ◽  
Priyadarshini Panda
Keyword(s):  
Neural Networks ◽  
Region Of Interest ◽  
Ground Truth ◽  
Research Area ◽  
Spiking Neural Networks ◽  
End User ◽  
Time Step ◽  
Efficient Alternative ◽  
New Research ◽  
Temporal Localization

AbstractBy emulating biological features in brain, Spiking Neural Networks (SNNs) offer an energy-efficient alternative to conventional deep learning. To make SNNs ubiquitous, a ‘visual explanation’ technique for analysing and explaining the internal spike behavior of such temporal deep SNNs is crucial. Explaining SNNs visually will make the network more transparent giving the end-user a tool to understand how SNNs make temporal predictions and why they make a certain decision. In this paper, we propose a bio-plausible visual explanation tool for SNNs, called Spike Activation Map (SAM). SAM yields a heatmap (i.e., localization map) corresponding to each time-step of input data by highlighting neurons with short inter-spike interval activity. Interestingly, without the use of gradients and ground truth, SAM produces a temporal localization map highlighting the region of interest in an image attributed to an SNN’s prediction at each time-step. Overall, SAM outsets the beginning of a new research area ‘explainable neuromorphic computing’ that will ultimately allow end-users to establish appropriate trust in predictions from SNNs.

Performance of a Method for Formulating Geometrically Exact Complementarity Constraints in Multibody Dynamic Simulation

2014 ◽  
Vol 10 (1) ◽  
Author(s):  
Daniel Montrallo Flickinger ◽  
Jedediyah Williams ◽  
Jeffrey C. Trinkle
Keyword(s):  
Dynamic Simulation ◽  
Collision Detection ◽  
High Performance ◽  
Parameter Tuning ◽  
Problem Formulation ◽  
Rigid Bodies ◽  
Detection Scheme ◽  
Practical Applications ◽  
System States ◽  
And Performance

Contemporary problem formulation methods used in the dynamic simulation of rigid bodies suffer from problems in accuracy, performance, and robustness. Significant allowances for parameter tuning, coupled with careful implementation of a broad-phase collision detection scheme are required to make dynamic simulation useful for practical applications. A constraint formulation method is presented herein that is more robust, and not dependent on broad-phase collision detection or system tuning for its behavior. Several uncomplicated benchmark examples are presented to give an analysis and make a comparison of the new polyhedral exact geometry (PEG) method with the well-known Stewart–Trinkle method. The behavior and performance for the two methods are discussed. This includes specific cases where contemporary methods fail to match theorized and observed system states in simulation, and how they are ameliorated by the new method presented here. The goal of this work is to complete the groundwork for further research into high performance simulation.

scholarly journals An efficient automated parameter tuning framework for spiking neural networks

2014 ◽  
Vol 8 ◽  
Author(s):  
Kristofor D. Carlson ◽  
Jayram Moorkanikara Nageswaran ◽  
Nikil Dutt ◽  
Jeffrey L. Krichmar
Keyword(s):  
Neural Networks ◽  
Parameter Tuning ◽  
Spiking Neural Networks

scholarly journals Brian2GeNN: a system for accelerating a large variety of spiking neural networks with graphics hardware

2018 ◽  
Author(s):  
Marcel Stimberg ◽  
Dan F. M. Goodman ◽  
Thomas Nowotny
Keyword(s):  
Neural Networks ◽  
Code Generation ◽  
Graphics Processing Units ◽  
High Performance ◽  
Spiking Neural Networks ◽  
Graphics Hardware ◽  
Performance Grade ◽  
Network Simulations ◽  
Nvidia Gpu ◽  
Graphics Processing

“Brian” is a popular Python-based simulator for spiking neural networks, commonly used in computational neuroscience. GeNN is a C++-based meta-compiler for accelerating spiking neural network simulations using consumer or high performance grade graphics processing units (GPUs). Here we introduce a new software package, Brian2GeNN, that connects the two systems so that users can make use of GeNN GPU acceleration when developing their models in Brian, without requiring any technical knowledge about GPUs, C++ or GeNN. The new Brian2GeNN software uses a pipeline of code generation to translate Brian scripts into C++ code that can be used as input to GeNN, and subsequently can be run on suitable NVIDIA GPU accelerators. From the user’s perspective, the entire pipeline is invoked by adding two simple lines to their Brian scripts. We have shown that using Brian2GeNN, typical models can run tens to hundreds of times faster than on CPU.

scholarly journals Direct Training for Spiking Neural Networks: Faster, Larger, Better

2019 ◽  
Vol 33 ◽  
pp. 1311-1318 ◽  
Author(s):  
Yujie Wu ◽  
Lei Deng ◽  
Guoqi Li ◽  
Jun Zhu ◽  
Yuan Xie ◽  
...  
Keyword(s):  
Neural Networks ◽  
High Performance ◽  
Large Scale ◽  
Learning Algorithm ◽  
Efficient Implementation ◽  
Spiking Neural Networks ◽  
Direct Training ◽  
Neuromorphic Hardware ◽  
Rate Coding ◽  
Neural Selectivity

Spiking neural networks (SNNs) that enables energy efficient implementation on emerging neuromorphic hardware are gaining more attention. Yet now, SNNs have not shown competitive performance compared with artificial neural networks (ANNs), due to the lack of effective learning algorithms and efficient programming frameworks. We address this issue from two aspects: (1) We propose a neuron normalization technique to adjust the neural selectivity and develop a direct learning algorithm for deep SNNs. (2) Via narrowing the rate coding window and converting the leaky integrate-and-fire (LIF) model into an explicitly iterative version, we present a Pytorch-based implementation method towards the training of large-scale SNNs. In this way, we are able to train deep SNNs with tens of times speedup. As a result, we achieve significantly better accuracy than the reported works on neuromorphic datasets (N-MNIST and DVSCIFAR10), and comparable accuracy as existing ANNs and pre-trained SNNs on non-spiking datasets (CIFAR10). To our best knowledge, this is the first work that demonstrates direct training of deep SNNs with high performance on CIFAR10, and the efficient implementation provides a new way to explore the potential of SNNs.

Sign in / Sign up

Export Citation Format

Share Document