scholarly journals Gradient Learning in Spiking Neural Networks by Dynamic Perturbation of Conductances

2006 ◽  
Vol 97 (4) ◽  
Author(s):  
Ila R. Fiete ◽  
H. Sebastian Seung
Keyword(s):  
Neural Networks ◽  
Spiking Neural Networks ◽  
Gradient Learning ◽  
Dynamic Perturbation

scholarly journals The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks

2020 ◽  
Author(s):  
Friedemann Zenke ◽  
Tim P. Vogels
Keyword(s):  
Neural Networks ◽  
Complex Function ◽  
Spiking Neural Networks ◽  
Learning Performance ◽  
Design Parameters ◽  
Classification Problems ◽  
Systematic Account ◽  
Practical Algorithms ◽  
Spiking Networks ◽  
Gradient Learning

AbstractBrains process information in spiking neural networks. Their intricate connections shape the diverse functions these networks perform. In comparison, the functional capabilities of models of spiking networks are still rudimentary. This shortcoming is mainly due to the lack of insight and practical algorithms to construct the necessary connectivity. Any such algorithm typically attempts to build networks by iteratively reducing the error compared to a desired output. But assigning credit to hidden units in multi-layered spiking networks has remained challenging due to the non-differentiable nonlinearity of spikes. To avoid this issue, one can employ surrogate gradients to discover the required connectivity in spiking network models. However, the choice of a surrogate is not unique, raising the question of how its implementation influences the effectiveness of the method. Here, we use numerical simulations to systematically study how essential design parameters of surrogate gradients impact learning performance on a range of classification problems. We show that surrogate gradient learning is robust to different shapes of underlying surrogate derivatives, but the choice of the derivative’s scale can substantially affect learning performance. When we combine surrogate gradients with a suitable activity regularization technique, robust information processing can be achieved in spiking networks even at the sparse activity limit. Our study provides a systematic account of the remarkable robustness of surrogate gradient learning and serves as a practical guide to model functional spiking neural networks.

scholarly journals The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks

2021 ◽  
pp. 1-27
Author(s):  
Friedemann Zenke ◽  
Tim P. Vogels
Keyword(s):  
Neural Networks ◽  
Information Processing ◽  
Complex Function ◽  
Spiking Neural Networks ◽  
Learning Performance ◽  
Design Parameters ◽  
Practical Algorithms ◽  
Neuromorphic Hardware ◽  
Spiking Networks ◽  
Gradient Learning

Brains process information in spiking neural networks. Their intricate connections shape the diverse functions these networks perform. Yet how network connectivity relates to function is poorly understood, and the functional capabilities of models of spiking networks are still rudimentary. The lack of both theoretical insight and practical algorithms to find the necessary connectivity poses a major impediment to both studying information processing in the brain and building efficient neuromorphic hardware systems. The training algorithms that solve this problem for artificial neural networks typically rely on gradient descent. But doing so in spiking networks has remained challenging due to the nondifferentiable nonlinearity of spikes. To avoid this issue, one can employ surrogate gradients to discover the required connectivity. However, the choice of a surrogate is not unique, raising the question of how its implementation influences the effectiveness of the method. Here, we use numerical simulations to systematically study how essential design parameters of surrogate gradients affect learning performance on a range of classification problems. We show that surrogate gradient learning is robust to different shapes of underlying surrogate derivatives, but the choice of the derivative's scale can substantially affect learning performance. When we combine surrogate gradients with suitable activity regularization techniques, spiking networks perform robust information processing at the sparse activity limit. Our study provides a systematic account of the remarkable robustness of surrogate gradient learning and serves as a practical guide to model functional spiking neural networks.

A Developmental Method for Evolving Large-Scale Spiking Neural Networks

2012 ◽  
Vol 35 (12) ◽  
pp. 2633 ◽  
Author(s):  
Xiang-Hong LIN ◽  
Tian-Wen ZHANG ◽  
Gui-Cang ZHANG
Keyword(s):  
Neural Networks ◽  
Large Scale ◽  
Spiking Neural Networks

scholarly journals Spiking Neural Networks and online learning: An overview and perspectives

2020 ◽  
Vol 121 ◽  
pp. 88-100 ◽  
Author(s):  
Jesus L. Lobo ◽  
Javier Del Ser ◽  
Albert Bifet ◽  
Nikola Kasabov
Keyword(s):  
Neural Networks ◽  
Online Learning ◽  
Spiking Neural Networks

scholarly journals Symmetry perception with spiking neural networks

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Jonathan K. George ◽  
Cesare Soci ◽  
Mario Miscuglio ◽  
Volker J. Sorger
Keyword(s):  
Neural Networks ◽  
Mirror Symmetry ◽  
Satellite Image ◽  
Image Data ◽  
Coincidence Detection ◽  
Spiking Neural Networks ◽  
Feed Forward Neural Network ◽  
Neuromorphic Computing ◽  
Symmetry Perception ◽  
High Level

AbstractMirror symmetry is an abundant feature in both nature and technology. Its successful detection is critical for perception procedures based on visual stimuli and requires organizational processes. Neuromorphic computing, utilizing brain-mimicked networks, could be a technology-solution providing such perceptual organization functionality, and furthermore has made tremendous advances in computing efficiency by applying a spiking model of information. Spiking models inherently maximize efficiency in noisy environments by placing the energy of the signal in a minimal time. However, many neuromorphic computing models ignore time delay between nodes, choosing instead to approximate connections between neurons as instantaneous weighting. With this assumption, many complex time interactions of spiking neurons are lost. Here, we show that the coincidence detection property of a spiking-based feed-forward neural network enables mirror symmetry. Testing this algorithm exemplary on geospatial satellite image data sets reveals how symmetry density enables automated recognition of man-made structures over vegetation. We further demonstrate that the addition of noise improves feature detectability of an image through coincidence point generation. The ability to obtain mirror symmetry from spiking neural networks can be a powerful tool for applications in image-based rendering, computer graphics, robotics, photo interpretation, image retrieval, video analysis and annotation, multi-media and may help accelerating the brain-machine interconnection. More importantly it enables a technology pathway in bridging the gap between the low-level incoming sensor stimuli and high-level interpretation of these inputs as recognized objects and scenes in the world.

Sign in / Sign up

Export Citation Format

Share Document