A simple selectionist learning rule for neural networks

1986 ◽  
Author(s):  
L. Personnaz ◽  
I. Guyon ◽  
A. Johannet ◽  
G. Dreyfus ◽  
G. Toulouse
Keyword(s):  
Neural Networks ◽  
Learning Rule

DeepTempo: A Hardware-Friendly Direct Feedback Alignment Multi-Layer Tempotron Learning Rule for Deep Spiking Neural Networks

2021 ◽  
pp. 1-1
Author(s):  
Cong Shi ◽  
Tengxiao Wang ◽  
Junxian He ◽  
Jianghao Zhang ◽  
Liyuan Liu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Learning Rule ◽  
Spiking Neural Networks ◽  
Direct Feedback

Chapter 9. Spike-Based Symbolic Computations on Bit Strings and Numbers

2021 ◽  
Author(s):  
Ceca Kraišniković ◽  
Wolfgang Maass ◽  
Robert Legenstein
Keyword(s):  
Neural Networks ◽  
Energy Efficient ◽  
Learning Rule ◽  
Spiking Neural Networks ◽  
New Paradigm ◽  
Symbolic Computations ◽  
Neuromorphic Hardware ◽  
State Of Research ◽  
Higher Cognitive Functions ◽  
The Brain

The brain uses recurrent spiking neural networks for higher cognitive functions such as symbolic computations, in particular, mathematical computations. We review the current state of research on spike-based symbolic computations of this type. In addition, we present new results which show that surprisingly small spiking neural networks can perform symbolic computations on bit sequences and numbers and even learn such computations using a biologically plausible learning rule. The resulting networks operate in a rather low firing rate regime, where they could not simply emulate artificial neural networks by encoding continuous values through firing rates. Thus, we propose here a new paradigm for symbolic computation in neural networks that provides concrete hypotheses about the organization of symbolic computations in the brain. The employed spike-based network models are the basis for drastically more energy-efficient computer hardware – neuromorphic hardware. Hence, our results can be seen as creating a bridge from symbolic artificial intelligence to energy-efficient implementation in spike-based neuromorphic hardware.

scholarly journals An approximate backpropagation learning rule for memristor based neural networks using synaptic plasticity

2017 ◽  
Vol 237 ◽  
pp. 193-199 ◽  
Author(s):  
D. Negrov ◽  
I. Karandashev ◽  
V. Shakirov ◽  
Yu. Matveyev ◽  
W. Dunin-Barkowski ◽  
...  
Keyword(s):  
Neural Networks ◽  
Synaptic Plasticity ◽  
Learning Rule ◽  
Backpropagation Learning

scholarly journals Biologically Plausible Sequence Learning with Spiking Neural Networks

2020 ◽  
Vol 34 (02) ◽  
pp. 1316-1323
Author(s):  
Zuozhu Liu ◽  
Thiparat Chotibut ◽  
Christopher Hillar ◽  
Shaowei Lin
Keyword(s):  
Neural Networks ◽  
Sequence Learning ◽  
Nervous Activity ◽  
Learning Rule ◽  
Generative Models ◽  
Spike Timing ◽  
Hopfield Network ◽  
Spiking Neural Networks ◽  
Theoretical Ground ◽  
Time Model

Motivated by the celebrated discrete-time model of nervous activity outlined by McCulloch and Pitts in 1943, we propose a novel continuous-time model, the McCulloch-Pitts network (MPN), for sequence learning in spiking neural networks. Our model has a local learning rule, such that the synaptic weight updates depend only on the information directly accessible by the synapse. By exploiting asymmetry in the connections between binary neurons, we show that MPN can be trained to robustly memorize multiple spatiotemporal patterns of binary vectors, generalizing the ability of the symmetric Hopfield network to memorize static spatial patterns. In addition, we demonstrate that the model can efficiently learn sequences of binary pictures as well as generative models for experimental neural spike-train data. Our learning rule is consistent with spike-timing-dependent plasticity (STDP), thus providing a theoretical ground for the systematic design of biologically inspired networks with large and robust long-range sequence storage capacity.

On Functional Approximation with Normalized Gaussian Units

1994 ◽  
Vol 6 (2) ◽  
pp. 319-333 ◽  
Author(s):  
Michel Benaim
Keyword(s):  
Neural Networks ◽  
Asymptotic Stability ◽  
Asymptotic Properties ◽  
Learning Rule ◽  
Adaptive Algorithms ◽  
Feedforward Neural Networks ◽  
Hybrid Learning ◽  
Universal Approximation ◽  
Ode Method ◽  
Hidden Layer

Feedforward neural networks with a single hidden layer using normalized gaussian units are studied. It is proved that such neural networks are capable of universal approximation in a satisfactory sense. Then, a hybrid learning rule as per Moody and Darken that combines unsupervised learning of hidden units and supervised learning of output units is considered. By using the method of ordinary differential equations for adaptive algorithms (ODE method) it is shown that the asymptotic properties of the learning rule may be studied in terms of an autonomous cascade of dynamical systems. Some recent results from Hirsch about cascades are used to show the asymptotic stability of the learning rule.

HOW TO MAKE GOOD USE OF MULTILAYER NEURAL NETWORKS

1995 ◽  
Vol 03 (04) ◽  
pp. 1177-1191 ◽  
Author(s):  
HÉLÈNE PAUGAM-MOISY
Keyword(s):  
Neural Networks ◽  
Open Problem ◽  
Back Propagation ◽  
Learning Rule ◽  
Multilayer Network ◽  
Odor Recognition ◽  
Short Summary ◽  
Multilayer Neural Networks ◽  
Hidden Layer ◽  
Universal Approximators

This article is a survey of recent advances on multilayer neural networks. The first section is a short summary on multilayer neural networks, their history, their architecture and their learning rule, the well-known back-propagation. In the following section, several theorems are cited, which present one-hidden-layer neural networks as universal approximators. The next section points out that two hidden layers are often required for exactly realizing d-dimensional dichotomies. Defining the frontier between one-hidden-layer and two-hidden-layer networks is still an open problem. Several bounds on the size of a multilayer network which learns from examples are presented and we enhance the fact that, even if all can be done with only one hidden layer, more often, things can be done better with two or more hidden layers. Finally, this assertion 'is supported by the behaviour of multilayer neural networks in two applications: prediction of pollution and odor recognition modelling.

scholarly journals One Step Back, Two Steps Forward: Interference and Learning in Recurrent Neural Networks

2019 ◽  
Vol 31 (10) ◽  
pp. 1985-2003 ◽  
Author(s):  
Chen Beer ◽  
Omri Barak
Keyword(s):  
Neural Networks ◽  
Neural Activity ◽  
Learning Rule ◽  
Pattern Generation ◽  
Learning Trajectory ◽  
Dependent Manner ◽  
Neural Recordings ◽  
Multiple Trials ◽  
Experimental Protocols ◽  
One Step

Artificial neural networks, trained to perform cognitive tasks, have recently been used as models for neural recordings from animals performing these tasks. While some progress has been made in performing such comparisons, the evolution of network dynamics throughout learning remains unexplored. This is paralleled by an experimental focus on recording from trained animals, with few studies following neural activity throughout training. In this work, we address this gap in the realm of artificial networks by analyzing networks that are trained to perform memory and pattern generation tasks. The functional aspect of these tasks corresponds to dynamical objects in the fully trained network—a line attractor or a set of limit cycles for the two respective tasks. We use these dynamical objects as anchors to study the effect of learning on their emergence. We find that the sequential nature of learning—one trial at a time—has major consequences for the learning trajectory and its final outcome. Specifically, we show that least mean squares (LMS), a simple gradient descent suggested as a biologically plausible version of the FORCE algorithm, is constantly obstructed by forgetting, which is manifested as the destruction of dynamical objects from previous trials. The degree of interference is determined by the correlation between different trials. We show which specific ingredients of FORCE avoid this phenomenon. Overall, this difference results in convergence that is orders of magnitude slower for LMS. Learning implies accumulating information across multiple trials to form the overall concept of the task. Our results show that interference between trials can greatly affect learning in a learning-rule-dependent manner. These insights can help design experimental protocols that minimize such interference, and possibly infer underlying learning rules by observing behavior and neural activity throughout learning.

Sign in / Sign up

Export Citation Format

Share Document