Gradient Descent Learning Algorithm Based on Spike Selection Mechanism for Multilayer Spiking Neural Networks

2021 ◽  
pp. 40-51
Author(s):  
Xianghong Lin ◽  
Tiandou Hu ◽  
Xiangwen Wang ◽  
Han Lu
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Learning Algorithm ◽  
Spiking Neural Networks ◽  
Selection Mechanism

A supervised multi-spike learning algorithm based on gradient descent for spiking neural networks

2013 ◽  
Vol 43 ◽  
pp. 99-113 ◽  
Author(s):  
Yan Xu ◽  
Xiaoqin Zeng ◽  
Lixin Han ◽  
Jing Yang
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Learning Algorithm ◽  
Spiking Neural Networks

scholarly journals Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms

Sensors ◽  
2021 ◽  
Vol 21 (9) ◽  
pp. 3240
Author(s):  
Tehreem Syed ◽  
Vijay Kakani ◽  
Xuenan Cui ◽  
Hakil Kim
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Spiking Neural Networks ◽  
License Plate ◽  
Training Techniques ◽  
Neuromorphic Hardware ◽  
Private And Public ◽  
Embedded Platforms ◽  
Public Datasets ◽  
Event Based

In recent times, the usage of modern neuromorphic hardware for brain-inspired SNNs has grown exponentially. In the context of sparse input data, they are undertaking low power consumption for event-based neuromorphic hardware, specifically in the deeper layers. However, using deep ANNs for training spiking models is still considered as a tedious task. Until recently, various ANN to SNN conversion methods in the literature have been proposed to train deep SNN models. Nevertheless, these methods require hundreds to thousands of time-steps for training and still cannot attain good SNN performance. This work proposes a customized model (VGG, ResNet) architecture to train deep convolutional spiking neural networks. In this current study, the training is carried out using deep convolutional spiking neural networks with surrogate gradient descent backpropagation in a customized layer architecture similar to deep artificial neural networks. Moreover, this work also proposes fewer time-steps for training SNNs with surrogate gradient descent. During the training with surrogate gradient descent backpropagation, overfitting problems have been encountered. To overcome these problems, this work refines the SNN based dropout technique with surrogate gradient descent. The proposed customized SNN models achieve good classification results on both private and public datasets. In this work, several experiments have been carried out on an embedded platform (NVIDIA JETSON TX2 board), where the deployment of customized SNN models has been extensively conducted. Performance validations have been carried out in terms of processing time and inference accuracy between PC and embedded platforms, showing that the proposed customized models and training techniques are feasible for achieving a better performance on various datasets such as CIFAR-10, MNIST, SVHN, and private KITTI and Korean License plate dataset.

Effective Transfer Learning Algorithm in Spiking Neural Networks

2021 ◽  
pp. 1-13
Author(s):  
Qiugang Zhan ◽  
Guisong Liu ◽  
Xiurui Xie ◽  
Guolin Sun ◽  
Huajin Tang
Keyword(s):  
Neural Networks ◽  
Transfer Learning ◽  
Learning Algorithm ◽  
Spiking Neural Networks ◽  
Effective Transfer

ACCELERATED LEARNING BY ACTIVE EXAMPLE SELECTION

1994 ◽  
Vol 05 (01) ◽  
pp. 67-75 ◽  
Author(s):  
BYOUNG-TAK ZHANG
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Learning Algorithm ◽  
Accelerated Learning ◽  
Training Set ◽  
Alternative Approach ◽  
Speed Up ◽  
Multilayer Neural Networks ◽  
Training Examples ◽  
The Given

Much previous work on training multilayer neural networks has attempted to speed up the backpropagation algorithm using more sophisticated weight modification rules, whereby all the given training examples are used in a random or predetermined sequence. In this paper we investigate an alternative approach in which the learning proceeds on an increasing number of selected training examples, starting with a small training set. We derive a measure of criticality of examples and present an incremental learning algorithm that uses this measure to select a critical subset of given examples for solving the particular task. Our experimental results suggest that the method can significantly improve training speed and generalization performance in many real applications of neural networks. This method can be used in conjunction with other variations of gradient descent algorithms.

scholarly journals Alopex: A Correlation-Based Learning Algorithm for Feedforward and Recurrent Neural Networks

1994 ◽  
Vol 6 (3) ◽  
pp. 469-490 ◽  
Author(s):  
K. P. Unnikrishnan ◽  
K. P. Venugopal
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Transfer Functions ◽  
Learning Algorithm ◽  
Recurrent Networks ◽  
Error Measure ◽  
Scaling Properties ◽  
Error Measures ◽  
Local Correlations ◽  
Temperature Parameter

We present a learning algorithm for neural networks, called Alopex. Instead of error gradient, Alopex uses local correlations between changes in individual weights and changes in the global error measure. The algorithm does not make any assumptions about transfer functions of individual neurons, and does not explicitly depend on the functional form of the error measure. Hence, it can be used in networks with arbitrary transfer functions and for minimizing a large class of error measures. The learning algorithm is the same for feedforward and recurrent networks. All the weights in a network are updated simultaneously, using only local computations. This allows complete parallelization of the algorithm. The algorithm is stochastic and it uses a “temperature” parameter in a manner similar to that in simulated annealing. A heuristic “annealing schedule” is presented that is effective in finding global minima of error surfaces. In this paper, we report extensive simulation studies illustrating these advantages and show that learning times are comparable to those for standard gradient descent methods. Feedforward networks trained with Alopex are used to solve the MONK's problems and symmetry problems. Recurrent networks trained with the same algorithm are used for solving temporal XOR problems. Scaling properties of the algorithm are demonstrated using encoder problems of different sizes and advantages of appropriate error measures are illustrated using a variety of problems.

scholarly journals Supervised Learning Algorithm for Multilayer Spiking Neural Networks with Long-Term Memory Spike Response Model

2021 ◽  
Vol 2021 ◽  
pp. 1-16
Author(s):  
Xianghong Lin ◽  
Mengwei Zhang ◽  
Xiangwen Wang
Keyword(s):  
Neural Networks ◽  
Supervised Learning ◽  
Learning Algorithm ◽  
Spike Trains ◽  
Spiking Neural Networks ◽  
Long Term Memory ◽  
Response Model ◽  
Spike Response ◽  
Spike Response Model

As a new brain-inspired computational model of artificial neural networks, spiking neural networks transmit and process information via precisely timed spike trains. Constructing efficient learning methods is a significant research field in spiking neural networks. In this paper, we present a supervised learning algorithm for multilayer feedforward spiking neural networks; all neurons can fire multiple spikes in all layers. The feedforward network consists of spiking neurons governed by biologically plausible long-term memory spike response model, in which the effect of earlier spikes on the refractoriness is not neglected to incorporate adaptation effects. The gradient descent method is employed to derive synaptic weight updating rule for learning spike trains. The proposed algorithm is tested and verified on spatiotemporal pattern learning problems, including a set of spike train learning tasks and nonlinear pattern classification problems on four UCI datasets. Simulation results indicate that the proposed algorithm can improve learning accuracy in comparison with other supervised learning algorithms.

scholarly journals Deep Convolutional Spiking Neural Networks for Image Classification

2021 ◽  
Author(s):  
Ruthvik Vaila
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Gradient Descent ◽  
Stochastic Gradient ◽  
Spiking Neural Networks ◽  
Stochastic Gradient Descent ◽  
Data Set ◽  
Learning Capabilities ◽  
Artificial Neural

Spiking neural networks are biologically plausible counterparts of artificial neural networks. Artificial neural networks are usually trained with stochastic gradient descent (SGD) and spiking neural networks are trained with bioinspired spike timing dependent plasticity (STDP). Spiking networks could potentially help in reducing power usage owing to their binary activations. In this work, we use unsupervised STDP in the feature extraction layers of a neural network with instantaneous neurons to extract meaningful features. The extracted binary feature vectors are then classified using classification layers containing neurons with binary activations. Gradient descent (backpropagation) is used only on the output layer to perform training for classification. Surrogate gradients are proposed to perform backpropagation with binary gradients. The accuracies obtained for MNIST and the balanced EMNIST data set compare favorably with other approaches. The effect of the stochastic gradient descent (SGD) approximations on learning capabilities of our network are also explored. We also studied catastrophic forgetting and its effect on spiking neural networks (SNNs). For the experiments regarding catastrophic forgetting, in the classification sections of the network we use a modified synaptic intelligence that we refer to as cost per synapse metric as a regularizer to immunize the network against catastrophic forgetting in a Single-Incremental-Task scenario (SIT). In catastrophic forgetting experiments, we use MNIST and EMNIST handwritten digits datasets that were divided into five and ten incremental subtasks respectively. We also examine behavior of the spiking neural network and empirically study the effect of various hyperparameters on its learning capabilities using the software tool SPYKEFLOW that we developed. We employ MNIST, EMNIST and NMNIST data sets to produce our results.

Sign in / Sign up

Export Citation Format

Share Document