scholarly journals A biomimetic neural encoder for spiking neural network

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Shiva Subbulakshmi Radhakrishnan ◽  
Amritanand Sebastian ◽  
Aaryan Oberoi ◽  
Sarbashis Das ◽  
Saptarshi Das
Keyword(s):  
Neural Networks ◽  
Dynamic Range ◽  
Spike Trains ◽  
Spike Timing ◽  
Neural Encoding ◽  
Data Set ◽  
Analog Signals ◽  
Neuromorphic Hardware ◽  
Effect Transistor ◽  
Internal Stimulus

AbstractSpiking neural networks (SNNs) promise to bridge the gap between artificial neural networks (ANNs) and biological neural networks (BNNs) by exploiting biologically plausible neurons that offer faster inference, lower energy expenditure, and event-driven information processing capabilities. However, implementation of SNNs in future neuromorphic hardware requires hardware encoders analogous to the sensory neurons, which convert external/internal stimulus into spike trains based on specific neural algorithm along with inherent stochasticity. Unfortunately, conventional solid-state transducers are inadequate for this purpose necessitating the development of neural encoders to serve the growing need of neuromorphic computing. Here, we demonstrate a biomimetic device based on a dual gated MoS2 field effect transistor (FET) capable of encoding analog signals into stochastic spike trains following various neural encoding algorithms such as rate-based encoding, spike timing-based encoding, and spike count-based encoding. Two important aspects of neural encoding, namely, dynamic range and encoding precision are also captured in our demonstration. Furthermore, the encoding energy was found to be as frugal as ≈1–5 pJ/spike. Finally, we show fast (≈200 timesteps) encoding of the MNIST data set using our biomimetic device followed by more than 91% accurate inference using a trained SNN.

Spike-Timing Dependent Plasticity for Evolved Robots

2002 ◽  
Vol 10 (3-4) ◽  
pp. 243-263 ◽  
Author(s):  
Ezequiel Di Paolo
Keyword(s):  
Neural Networks ◽  
Synaptic Plasticity ◽  
Spike Trains ◽  
Spike Timing ◽  
Behavioral Strategies ◽  
Spike Timing Dependent Plasticity ◽  
Dependent Plasticity ◽  
Neural Noise ◽  
Functional Value ◽  
Millisecond Range

Plastic spiking neural networks are synthesized for phototactic robots using evolutionary techniques. Synaptic plasticity asymmetrically depends on the precise relative timing between presynaptic and postsynaptic spikes at the millisecond range and on longer-term activity-dependent regulatory scaling. Comparative studies have been carried out for different kinds of plastic neural networks with low and high levels of neural noise. In all cases, the evolved controllers are highly robust against internal synaptic decay and other perturbations. The importance of the precise timing of spikes is demonstrated by randomizing the spike trains. In the low neural noise scenario, weight values undergo rhythmic changes at the mesoscale due to bursting, but during periods of high activity they are finely regulated at the microscale by synchronous or entrained firing. Spike train randomization results in loss of performance in this case. In contrast, in the high neural noise scenario, robots are robust to loss of information in the timing of the spike trains, demonstrating the counterintuitive results that plasticity, which is dependent on precise spike timing, can work even in its absence, provided the behavioral strategies make use of robust longer-term invariants of sensorimotor interaction. A comparison with a rate-based model of synaptic plasticity shows that under similarly noisy conditions, asymmetric spike-timing dependent plasticity achieves better performance by means of efficient reduction in weight variance over time. Performance also presents negative sensitivity to reduced levels of noise, showing that random firing has a functional value.

scholarly journals SSTDP: Supervised Spike Timing Dependent Plasticity for Efficient Spiking Neural Network Training

2021 ◽  
Vol 15 ◽  
Author(s):  
Fangxin Liu ◽  
Wenbo Zhao ◽  
Yongbiao Chen ◽  
Zongwu Wang ◽  
Tao Yang ◽  
...  
Keyword(s):  
Neural Networks ◽  
Learning Algorithm ◽  
Activation Function ◽  
Spike Timing ◽  
Extraction Property ◽  
Dependent Plasticity ◽  
Neuromorphic Hardware ◽  
Gradient Based ◽  
Spatio Temporal ◽  
Spike Time Dependent Plasticity

Spiking Neural Networks (SNNs) are a pathway that could potentially empower low-power event-driven neuromorphic hardware due to their spatio-temporal information processing capability and high biological plausibility. Although SNNs are currently more efficient than artificial neural networks (ANNs), they are not as accurate as ANNs. Error backpropagation is the most common method for directly training neural networks, promoting the prosperity of ANNs in various deep learning fields. However, since the signals transmitted in the SNN are non-differentiable discrete binary spike events, the activation function in the form of spikes presents difficulties for the gradient-based optimization algorithms to be directly applied in SNNs, leading to a performance gap (i.e., accuracy and latency) between SNNs and ANNs. This paper introduces a new learning algorithm, called SSTDP, which bridges the gap between backpropagation (BP)-based learning and spike-time-dependent plasticity (STDP)-based learning to train SNNs efficiently. The scheme incorporates the global optimization process from BP and the efficient weight update derived from STDP. It not only avoids the non-differentiable derivation in the BP process but also utilizes the local feature extraction property of STDP. Consequently, our method can lower the possibility of vanishing spikes in BP training and reduce the number of time steps to reduce network latency. In SSTDP, we employ temporal-based coding and use Integrate-and-Fire (IF) neuron as the neuron model to provide considerable computational benefits. Our experiments show the effectiveness of the proposed SSTDP learning algorithm on the SNN by achieving the best classification accuracy 99.3% on the Caltech 101 dataset, 98.1% on the MNIST dataset, and 91.3% on the CIFAR-10 dataset compared to other SNNs trained with other learning methods. It also surpasses the best inference accuracy of the directly trained SNN with 25~32× less inference latency. Moreover, we analyze event-based computations to demonstrate the efficacy of the SNN for inference operation in the spiking domain, and SSTDP methods can achieve 1.3~37.7× fewer addition operations per inference. The code is available at: https://github.com/MXHX7199/SNN-SSTDP.

scholarly journals Jointly Learning Network Connections and Link Weights in Spiking Neural Networks

2018 ◽  
Author(s):  
Yu Qi ◽  
Jiangrong Shen ◽  
Yueming Wang ◽  
Huajin Tang ◽  
Hang Yu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Spike Timing ◽  
Spiking Neural Networks ◽  
Power Efficient ◽  
Network Connections ◽  
Neuromorphic Hardware ◽  
Stdp Rule ◽  
Benchmark Datasets ◽  
Network Topologies ◽  
The Brain

Spiking neural networks (SNNs) are considered to be biologically plausible and power-efficient on neuromorphic hardware. However, unlike the brain mechanisms, most existing SNN algorithms have fixed network topologies and connection relationships. This paper proposes a method to jointly learn network connections and link weights simultaneously. The connection structures are optimized by the spike-timing-dependent plasticity (STDP) rule with timing information, and the link weights are optimized by a supervised algorithm. The connection structures and the weights are learned alternately until a termination condition is satisfied. Experiments are carried out using four benchmark datasets. Our approach outperforms classical learning methods such as STDP, Tempotron, SpikeProp, and a state-of-the-art supervised algorithm. In addition, the learned structures effectively reduce the number of connections by about 24%, thus facilitate the computational efficiency of the network.

scholarly journals A Radial Basis Function Spike Model for Indirect Learning via Integrate-and-Fire Sampling and Reconstruction Techniques

2012 ◽  
Vol 2012 ◽  
pp. 1-16 ◽  
Author(s):  
X. Zhang ◽  
G. Foderaro ◽  
C. Henriquez ◽  
A. M. J. VanDongen ◽  
S. Ferrari
Keyword(s):  
Neural Networks ◽  
Synaptic Plasticity ◽  
Spike Trains ◽  
Spike Timing ◽  
Spiking Neural Networks ◽  
Learning Mechanisms ◽  
Integrate And Fire ◽  
Spike Model ◽  
Radial Basis ◽  
Input Spike

This paper presents a deterministic and adaptive spike model derived from radial basis functions and a leaky integrate-and-fire sampler developed for training spiking neural networks without direct weight manipulation. Several algorithms have been proposed for training spiking neural networks through biologically-plausible learning mechanisms, such as spike-timing-dependent synaptic plasticity and Hebbian plasticity. These algorithms typically rely on the ability to update the synaptic strengths, or weights, directly, through a weight update rule in which the weight increment can be decided and implemented based on the training equations. However, in several potential applications of adaptive spiking neural networks, including neuroprosthetic devices and CMOS/memristor nanoscale neuromorphic chips, the weights cannot be manipulated directly and, instead, tend to change over time by virtue of the pre- and postsynaptic neural activity. This paper presents an indirect learning method that induces changes in the synaptic weights by modulating spike-timing-dependent plasticity by means of controlled input spike trains. In place of the weights, the algorithm manipulates the input spike trains used to stimulate the input neurons by determining a sequence of spike timings that minimize a desired objective function and, indirectly, induce the desired synaptic plasticity in the network.

scholarly journals An All-in-One Biomimetic 2D Spiking Neural Network

2021 ◽  
Author(s):  
Shiva Subbulakshmi Radhakrishnan ◽  
Akhil Dodda ◽  
Saptarshi Das
Keyword(s):  
Neural Networks ◽  
Unsupervised Learning ◽  
Bright Light ◽  
Spike Trains ◽  
Spike Timing ◽  
Low Light ◽  
Computing Systems ◽  
Von Neumann ◽  
2D Layered Materials ◽  
Learning Challenges

Abstract In spite of recent advancements in bio-realistic artificial neural networks such as spiking neural networks (SNNs), the energy efficiency, multifunctionality, adaptability, and integrated nature of biological neural networks (BNNs) largely remain unimitated in hardware neuromorphic computing systems. Here we exploit optoelectronic and programmable memory devices based on emerging two-dimensional (2D) layered materials such as MoS2 to demonstrate an “all-in-one” hardware SNN system which is capable of sensing, encoding, unsupervised learning, and inference at miniscule energy expenditure. In short, we have utilized photogating effect in MoS2 based neuromorphic phototransistor for sensing and direct encoding of analog optical information into graded spike trains, we have designed MoS2 based neuromorphic encoding module for conversion of spike trains into spike-count and spike-timing based programming voltages, and finally we have used arrays of programmable MoS2 non-volatile synapses for spike-based unsupervised learning and inference. We also demonstrate adaptability of our SNN for learning under scotopic (low-light) and photopic (bright-light) conditions mimicking neuroplasticity of BNNs. Furthermore, we use our hardware SNN platform to show learning challenges under specific synaptic conditions, which can aid in understanding learning disabilities in BNNs. Our findings highlight the potential of in-memory computing and sensing based on emerging 2D materials, devices, and circuits not only to overcome the bottleneck of von Neumann computing in conventional CMOS designs but also aid in eliminating peripheral components necessary for competing technologies such as memristors, RRAM, PCM, etc. as well as bridge the understanding between neuroscience of learning and machine learning.

scholarly journals Localization of Scattering Objects Using Neural Networks

Sensors ◽  
2020 ◽  
Vol 21 (1) ◽  
pp. 11
Author(s):  
Domonkos Haffner ◽  
Ferenc Izsák
Keyword(s):  
Neural Networks ◽  
Plane Waves ◽  
Measurement Data ◽  
Scattered Wave ◽  
Training Data ◽  
Data Set ◽  
Main Compound ◽  
Incident Plane ◽  
The Neural Networks ◽  
Simulation Package

The localization of multiple scattering objects is performed while using scattered waves. An up-to-date approach: neural networks are used to estimate the corresponding locations. In the scattering phenomenon under investigation, we assume known incident plane waves, fully reflecting balls with known diameters and measurement data of the scattered wave on one fixed segment. The training data are constructed while using the simulation package μ-diff in Matlab. The structure of the neural networks, which are widely used for similar purposes, is further developed. A complex locally connected layer is the main compound of the proposed setup. With this and an appropriate preprocessing of the training data set, the number of parameters can be kept at a relatively low level. As a result, using a relatively large training data set, the unknown locations of the objects can be estimated effectively.

scholarly journals Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms

Sensors ◽  
2021 ◽  
Vol 21 (9) ◽  
pp. 3240
Author(s):  
Tehreem Syed ◽  
Vijay Kakani ◽  
Xuenan Cui ◽  
Hakil Kim
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Spiking Neural Networks ◽  
License Plate ◽  
Training Techniques ◽  
Neuromorphic Hardware ◽  
Private And Public ◽  
Embedded Platforms ◽  
Public Datasets ◽  
Event Based

In recent times, the usage of modern neuromorphic hardware for brain-inspired SNNs has grown exponentially. In the context of sparse input data, they are undertaking low power consumption for event-based neuromorphic hardware, specifically in the deeper layers. However, using deep ANNs for training spiking models is still considered as a tedious task. Until recently, various ANN to SNN conversion methods in the literature have been proposed to train deep SNN models. Nevertheless, these methods require hundreds to thousands of time-steps for training and still cannot attain good SNN performance. This work proposes a customized model (VGG, ResNet) architecture to train deep convolutional spiking neural networks. In this current study, the training is carried out using deep convolutional spiking neural networks with surrogate gradient descent backpropagation in a customized layer architecture similar to deep artificial neural networks. Moreover, this work also proposes fewer time-steps for training SNNs with surrogate gradient descent. During the training with surrogate gradient descent backpropagation, overfitting problems have been encountered. To overcome these problems, this work refines the SNN based dropout technique with surrogate gradient descent. The proposed customized SNN models achieve good classification results on both private and public datasets. In this work, several experiments have been carried out on an embedded platform (NVIDIA JETSON TX2 board), where the deployment of customized SNN models has been extensively conducted. Performance validations have been carried out in terms of processing time and inference accuracy between PC and embedded platforms, showing that the proposed customized models and training techniques are feasible for achieving a better performance on various datasets such as CIFAR-10, MNIST, SVHN, and private KITTI and Korean License plate dataset.

scholarly journals A Review of Algorithms and Hardware Implementations for Spiking Neural Networks

2021 ◽  
Vol 11 (2) ◽  
pp. 23
Author(s):  
Duy-Anh Nguyen ◽  
Xuan-Tu Tran ◽  
Francesca Iacopi
Keyword(s):  
Neural Networks ◽  
Computational Cost ◽  
Superior Performance ◽  
Training Algorithms ◽  
Current State ◽  
Hardware Implementations ◽  
Neuromorphic Hardware ◽  
High Level ◽  
Event Based ◽  
Hardware Platforms

Deep Learning (DL) has contributed to the success of many applications in recent years. The applications range from simple ones such as recognizing tiny images or simple speech patterns to ones with a high level of complexity such as playing the game of Go. However, this superior performance comes at a high computational cost, which made porting DL applications to conventional hardware platforms a challenging task. Many approaches have been investigated, and Spiking Neural Network (SNN) is one of the promising candidates. SNN is the third generation of Artificial Neural Networks (ANNs), where each neuron in the network uses discrete spikes to communicate in an event-based manner. SNNs have the potential advantage of achieving better energy efficiency than their ANN counterparts. While generally there will be a loss of accuracy on SNN models, new algorithms have helped to close the accuracy gap. For hardware implementations, SNNs have attracted much attention in the neuromorphic hardware research community. In this work, we review the basic background of SNNs, the current state and challenges of the training algorithms for SNNs and the current implementations of SNNs on various hardware platforms.

scholarly journals Generation of geometric interpolations of building types with deep variational autoencoders

2020 ◽  
Vol 6 ◽  
Author(s):  
Jaime de Miguel Rodríguez ◽  
Maria Eugenia Villafañe ◽  
Luka Piškorec ◽  
Fernando Sancho Caparrini
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Large Data ◽  
Learning Model ◽  
Large Data Sets ◽  
Data Sets ◽  
Connectivity Map ◽  
Data Set ◽  
3D Objects ◽  
Machine Learning Model

Abstract This work presents a methodology for the generation of novel 3D objects resembling wireframes of building types. These result from the reconstruction of interpolated locations within the learnt distribution of variational autoencoders (VAEs), a deep generative machine learning model based on neural networks. The data set used features a scheme for geometry representation based on a ‘connectivity map’ that is especially suited to express the wireframe objects that compose it. Additionally, the input samples are generated through ‘parametric augmentation’, a strategy proposed in this study that creates coherent variations among data by enabling a set of parameters to alter representative features on a given building type. In the experiments that are described in this paper, more than 150 k input samples belonging to two building types have been processed during the training of a VAE model. The main contribution of this paper has been to explore parametric augmentation for the generation of large data sets of 3D geometries, showcasing its problems and limitations in the context of neural networks and VAEs. Results show that the generation of interpolated hybrid geometries is a challenging task. Despite the difficulty of the endeavour, promising advances are presented.

Sign in / Sign up

Export Citation Format

Share Document