Robustness from structure: Inference with hierarchical spiking networks on analog neuromorphic hardware

The problem with training spiking neural networks (SNNs) is relevant due to the ultra-low power consumption these networks could exhibit when implemented in neuromorphic hardware. The ongoing progress in the fabrication of memristors, a prospective basis for analogue synapses, gives relevance to studying the possibility of SNN learning on the base of synaptic plasticity models, obtained by fitting the experimental measurements of the memristor conductance change. The dynamics of memristor conductances is (necessarily) nonlinear, because conductance changes depend on the spike timings, which neurons emit in an all-or-none fashion. The ability to solve classification tasks was previously shown for spiking network models based on the bio-inspired local learning mechanism of spike-timing-dependent plasticity (STDP), as well as with the plasticity that models the conductance change of nanocomposite (NC) memristors. Input data were presented to the network encoded into the intensities of Poisson input spike sequences. This work considers another approach for encoding input data into input spike sequences presented to the network: temporal encoding, in which an input vector is transformed into relative timing of individual input spikes. Since temporal encoding uses fewer input spikes, the processing of each input vector by the network can be faster and more energy-efficient. The aim of the current work is to show the applicability of temporal encoding to training spiking networks with three synaptic plasticity models: STDP, NC memristor approximation, and PPX memristor approximation. We assess the accuracy of the proposed approach on several benchmark classification tasks: Fisher’s Iris, Wisconsin breast cancer, and the pole balancing task (CartPole). The accuracies achieved by SNN with memristor plasticity and conventional STDP are comparable and are on par with classic machine learning approaches.

Download Full-text

The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks

Neural Computation ◽

10.1162/neco_a_01367 ◽

2021 ◽

pp. 1-27

Author(s):

Friedemann Zenke ◽

Tim P. Vogels

Keyword(s):

Neural Networks ◽

Information Processing ◽

Complex Function ◽

Spiking Neural Networks ◽

Learning Performance ◽

Design Parameters ◽

Practical Algorithms ◽

Neuromorphic Hardware ◽

Spiking Networks ◽

Gradient Learning

Brains process information in spiking neural networks. Their intricate connections shape the diverse functions these networks perform. Yet how network connectivity relates to function is poorly understood, and the functional capabilities of models of spiking networks are still rudimentary. The lack of both theoretical insight and practical algorithms to find the necessary connectivity poses a major impediment to both studying information processing in the brain and building efficient neuromorphic hardware systems. The training algorithms that solve this problem for artificial neural networks typically rely on gradient descent. But doing so in spiking networks has remained challenging due to the nondifferentiable nonlinearity of spikes. To avoid this issue, one can employ surrogate gradients to discover the required connectivity. However, the choice of a surrogate is not unique, raising the question of how its implementation influences the effectiveness of the method. Here, we use numerical simulations to systematically study how essential design parameters of surrogate gradients affect learning performance on a range of classification problems. We show that surrogate gradient learning is robust to different shapes of underlying surrogate derivatives, but the choice of the derivative's scale can substantially affect learning performance. When we combine surrogate gradients with suitable activity regularization techniques, spiking networks perform robust information processing at the sparse activity limit. Our study provides a systematic account of the remarkable robustness of surrogate gradient learning and serves as a practical guide to model functional spiking neural networks.

Download Full-text

Solving a steady-state PDE using spiking networks and neuromorphic hardware

International Conference on Neuromorphic Systems 2020 ◽

10.1145/3407197.3407202 ◽

2020 ◽

Author(s):

J. Darby Smith ◽

William Severa ◽

Aaron J. Hill ◽

Leah Reeder ◽

Brian Franke ◽

...

Keyword(s):

Steady State ◽

Neuromorphic Hardware ◽

Spiking Networks

Download Full-text

PageRank Implemented with the MPI Paradigm Running on a Many-Core Neuromorphic Platform

Journal of Low Power Electronics and Applications ◽

10.3390/jlpea11020025 ◽

2021 ◽

Vol 11 (2) ◽

pp. 25

Author(s):

Evelina Forno ◽

Alessandro Salvato ◽

Enrico Macii ◽

Gianvito Urgese

Keyword(s):

Parallel Execution ◽

Spiking Neural Networks ◽

Massively Parallel ◽

Current Version ◽

Hardware Platform ◽

Pagerank Algorithm ◽

Programming Paradigm ◽

Neuromorphic Hardware ◽

Efficient Communication ◽

Many Core

SpiNNaker is a neuromorphic hardware platform, especially designed for the simulation of Spiking Neural Networks (SNNs). To this end, the platform features massively parallel computation and an efficient communication infrastructure based on the transmission of small packets. The effectiveness of SpiNNaker in the parallel execution of the PageRank (PR) algorithm has been tested by the realization of a custom SNN implementation. In this work, we propose a PageRank implementation fully realized with the MPI programming paradigm ported to the SpiNNaker platform. We compare the scalability of the proposed program with the equivalent SNN implementation, and we leverage the characteristics of the PageRank algorithm to benchmark our implementation of MPI on SpiNNaker when faced with massive communication requirements. Experimental results show that the algorithm exhibits favorable scaling for a mid-sized execution context, while highlighting that the performance of MPI-PageRank on SpiNNaker is bounded by memory size and speed limitations on the current version of the hardware.

Download Full-text

Feed-forward and noise-tolerant detection of feature homogeneity in spiking networks with a latency code

Biological Cybernetics ◽

10.1007/s00422-021-00866-w ◽

2021 ◽

Author(s):

Michael Schmuker ◽

Rüdiger Kupper ◽

Ad Aertsen ◽

Thomas Wachtler ◽

Marc-Oliver Gewaltig

Keyword(s):

Feed Forward ◽

Spiking Networks ◽

Noise Tolerant

Download Full-text

Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms

Sensors ◽

10.3390/s21093240 ◽

2021 ◽

Vol 21 (9) ◽

pp. 3240

Author(s):

Tehreem Syed ◽

Vijay Kakani ◽

Xuenan Cui ◽

Hakil Kim

Keyword(s):

Neural Networks ◽

Gradient Descent ◽

Spiking Neural Networks ◽

License Plate ◽

Training Techniques ◽

Neuromorphic Hardware ◽

Private And Public ◽

Embedded Platforms ◽

Public Datasets ◽

Event Based

In recent times, the usage of modern neuromorphic hardware for brain-inspired SNNs has grown exponentially. In the context of sparse input data, they are undertaking low power consumption for event-based neuromorphic hardware, specifically in the deeper layers. However, using deep ANNs for training spiking models is still considered as a tedious task. Until recently, various ANN to SNN conversion methods in the literature have been proposed to train deep SNN models. Nevertheless, these methods require hundreds to thousands of time-steps for training and still cannot attain good SNN performance. This work proposes a customized model (VGG, ResNet) architecture to train deep convolutional spiking neural networks. In this current study, the training is carried out using deep convolutional spiking neural networks with surrogate gradient descent backpropagation in a customized layer architecture similar to deep artificial neural networks. Moreover, this work also proposes fewer time-steps for training SNNs with surrogate gradient descent. During the training with surrogate gradient descent backpropagation, overfitting problems have been encountered. To overcome these problems, this work refines the SNN based dropout technique with surrogate gradient descent. The proposed customized SNN models achieve good classification results on both private and public datasets. In this work, several experiments have been carried out on an embedded platform (NVIDIA JETSON TX2 board), where the deployment of customized SNN models has been extensively conducted. Performance validations have been carried out in terms of processing time and inference accuracy between PC and embedded platforms, showing that the proposed customized models and training techniques are feasible for achieving a better performance on various datasets such as CIFAR-10, MNIST, SVHN, and private KITTI and Korean License plate dataset.

Download Full-text

A Review of Algorithms and Hardware Implementations for Spiking Neural Networks

Journal of Low Power Electronics and Applications ◽

10.3390/jlpea11020023 ◽

2021 ◽

Vol 11 (2) ◽

pp. 23

Author(s):

Duy-Anh Nguyen ◽

Xuan-Tu Tran ◽

Francesca Iacopi

Keyword(s):

Neural Networks ◽

Computational Cost ◽

Superior Performance ◽

Training Algorithms ◽

Current State ◽

Hardware Implementations ◽

Neuromorphic Hardware ◽

High Level ◽

Event Based ◽

Hardware Platforms

Deep Learning (DL) has contributed to the success of many applications in recent years. The applications range from simple ones such as recognizing tiny images or simple speech patterns to ones with a high level of complexity such as playing the game of Go. However, this superior performance comes at a high computational cost, which made porting DL applications to conventional hardware platforms a challenging task. Many approaches have been investigated, and Spiking Neural Network (SNN) is one of the promising candidates. SNN is the third generation of Artificial Neural Networks (ANNs), where each neuron in the network uses discrete spikes to communicate in an event-based manner. SNNs have the potential advantage of achieving better energy efficiency than their ANN counterparts. While generally there will be a loss of accuracy on SNN models, new algorithms have helped to close the accuracy gap. For hardware implementations, SNNs have attracted much attention in the neuromorphic hardware research community. In this work, we review the basic background of SNNs, the current state and challenges of the training algorithms for SNNs and the current implementations of SNNs on various hardware platforms.

Download Full-text