Weight discretization in backward error propagation neural networks

1988 ◽  
Vol 1 ◽  
pp. 380 ◽  
Author(s):  
E. Fiesler ◽  
A. Choudry ◽  
H.J. Caulfield
Keyword(s):  
Neural Networks ◽  
Error Propagation ◽  
Backward Error

Sufficient Conditions for Error Backflow Convergence in Dynamical Recurrent Neural Networks

2002 ◽  
Vol 14 (8) ◽  
pp. 1907-1927 ◽  
Author(s):  
Alex Aussem
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Error Propagation ◽  
Finite Impulse Response ◽  
Learning Algorithm ◽  
Previous Analysis ◽  
Sufficient Conditions ◽  
Recurrent Networks ◽  
Computational Overhead ◽  
Propagation Network

This article extends previous analysis of the gradient decay to a class of discrete-time fully recurrent networks, called dynamical recurrent neural networks, obtained by modeling synapses as finite impulse response (FIR) filters instead of multiplicative scalars. Using elementary matrix manipulations, we provide an upper bound on the norm of the weight matrix, ensuring that the gradient vector, when propagated in a reverse manner in time through the error-propagation network, decays exponentially to zero. This bound applies to all recurrent FIR architecture proposals, as well as fixed-point recurrent networks, regardless of delay and connectivity. In addition, we show that the computational overhead of the learning algorithm can be reduced drastically by taking advantage of the exponential decay of the gradient.

scholarly journals CREATING RLCP-COMPATIBLE VIRTUAL LABORATORIES FOR TRAINING BASIC ALGORITHMS ON NEURAL NETWORKS

2021 ◽  
Vol 2021 (3) ◽  
pp. 134-142
Author(s):  
Liubov Lisitsyna ◽  
Marina Senchilo ◽  
Sergei Teleshev
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Error Propagation ◽  
High Efficiency ◽  
Single Layer ◽  
Visual Display ◽  
Signal Propagation ◽  
Automatic Generation ◽  
Virtual Laboratories ◽  
Individual Task

The article describes the principles of developing RLCP-compatible virtual laboratories. There are build two virtual laboratories based on these principles for mastering the basic algo-rithms on neural networks: Algorithm for Sequential Signal Propagation in Perceptron and Algorithm for Training Perceptron Using Method of Backward Error Propagation. Virtual laboratories consist of two independent modules – a virtual stand and an RLCP server. The virtual stand implements a visual display of the task's data and provides the listener with tools for forming and editing intermediate solutions and responses. Since the virtual laboratories were assumed for the first acquaintance with neural networks, the simplest neural network architectures in the form of single-layer perceptrons were used as the initial data. And the algorithm of sequential propagation of signals in a neural network (VL1) and the algorithm of training a neural network with a teacher based on the method of inverse error propagation (VL2) are used as the basic algorithms. For automatic generation of equally complex and valid tasks there have been proposed algorithms with high efficiency (the average time for generating an individual task on the VL2 stand for a student was no longer than 3 seconds). It was found out experimentally that such virtual laboratories should be created in two modes: the mode of training and mode of certification. The training shop works for solving problems using the studied algorithms on the stands of virtual laboratories in the training mode with the diagnosis of admitted errors significantly increase the effectiveness of students' results

A Subgrouping Strategy that Reduces Complexity and Speeds Up Learning in Recurrent Networks

1989 ◽  
Vol 1 (4) ◽  
pp. 552-558 ◽  
Author(s):  
David Zipser
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Error Propagation ◽  
Computation Time ◽  
Great Power ◽  
Recurrent Networks ◽  
Original Network ◽  
Original Algorithm ◽  
Finite State ◽  
Unit Network

An algorithm, called RTRL, for training fully recurrent neural networks has recently been studied by Williams and Zipser (1989a, b). Whereas RTRL has been shown to have great power and generality, it has the disadvantage of requiring a great deal of computation time. A technique is described here for reducing the amount of computation required by RTRL without changing the connectivity of the networks. This is accomplished by dividing the original network into subnets for the purpose of error propagation while leaving them undivided for activity propagation. An example is given of a 12-unit network that learns to be the finite-state part of a Turing machine and runs 10 times faster using the subgrouping strategy than the original algorithm.

scholarly journals Predicting complex genetic phenotypes using error propagation in weighted networks

2018 ◽  
Author(s):  
El Mahdi El Mhamdi ◽  
Andrei Kucharavy ◽  
Rachid Guerraoui ◽  
Rong Li
Keyword(s):  
Neural Networks ◽  
Distributed System ◽  
Biological Networks ◽  
Error Propagation ◽  
Biological Systems ◽  
Genomic Data ◽  
Activation Functions ◽  
Mathematical Formalization ◽  
High Level ◽  
Universal Approximators

AbstractNetwork-biology view of biological systems is a ubiquitous abstraction that emerged in the last two decades to allow a high-level understanding of principles governing them. However, the principles according to which biological systems are organized are still unclear. Here, we investigate if biological networks could be approximated as overlapping, feed-forward networks where the nodes have non-linear activation functions. Such networks have been shown to be universal approximators and their stability has been explored in the context of artificial neural networks. Mathematical formalization of this model followed by numerical simulations based on genomic data allowed us to accurately predict the statistics of gene essentiality in yeast and hence indicate that biological networks might be better understood as a distributed system, comprising potentially unreliable components.

Sign in / Sign up

Export Citation Format

Share Document