Complexity of implementation has been a major difficulty in the development of gradient descent learning algorithms for dynamical neural networks with feedback and recurrent connections. Some insights from the stability properties of the equilibrium points of the network, which suggest an appropriate tailoring of the sigmoidal nonlinear functions, can however be utilized in obtaining simplified learning rules, as demonstrated in this paper. An analytical proof of convergence of the learning scheme under specific conditions is given and some upper bounds on the adaptation parameters for an efficient implementation of the training procedure are developed. The performance features of the learning algorithm are illustrated by applying it to two problems of importance, viz., design of associative memories and nonlinear input-output mapping. For the first application, a systematic procedure is given for training a network to store multiple memory vectors as its stable equilibrium points, whereas for the second application, specific training rules are developed for a three-layer network architecture comprising a dynamical hidden layer for the identification of nonlinear input-output maps. A comparison with the performance of a standard backpropagation network provides an illustration of the capabilities of the present network architecture and the learning algorithm.
Wavelet based non-parametric NARX models for nonlinear input–output system identification
