Convergence of Gradient Descent Algorithm with Penalty Term For Recurrent Neural Networks

2014 ◽  
Vol 9 (9) ◽  
pp. 151-158
Author(s):  
Xiaoshuai Ding ◽  
Kuaini Wang
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Gradient Descent ◽  
Penalty Term ◽  
Descent Algorithm ◽  
Gradient Descent Algorithm

scholarly journals Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory

2020 ◽  
Vol 34 (04) ◽  
pp. 4115-4122
Author(s):  
Kyle Helfrich ◽  
Qiang Ye
Keyword(s):  
Neural Networks ◽  
Unit Circle ◽  
Recurrent Neural Networks ◽  
Unit Disc ◽  
Gradient Descent ◽  
Short Term Memory ◽  
Short Term ◽  
Term Memory ◽  
Gradient Descent Algorithm

Several variants of recurrent neural networks (RNNs) with orthogonal or unitary recurrent matrices have recently been developed to mitigate the vanishing/exploding gradient problem and to model long-term dependencies of sequences. However, with the eigenvalues of the recurrent matrix on the unit circle, the recurrent state retains all input information which may unnecessarily consume model capacity. In this paper, we address this issue by proposing an architecture that expands upon an orthogonal/unitary RNN with a state that is generated by a recurrent matrix with eigenvalues in the unit disc. Any input to this state dissipates in time and is replaced with new inputs, simulating short-term memory. A gradient descent algorithm is derived for learning such a recurrent matrix. The resulting method, called the Eigenvalue Normalized RNN (ENRNN), is shown to be highly competitive in several experiments.

scholarly journals A Novel Technique for Solving Fully Fuzzy Nonlinear Systems Based on Neural Networks

2020 ◽  
Vol 07 (01) ◽  
pp. 93-107 ◽  
Author(s):  
Raheleh Jafari ◽  
Sina Razvarz ◽  
Alexander Gegov
Keyword(s):  
Neural Networks ◽  
Nonlinear System ◽  
Nonlinear Systems ◽  
Complex Systems ◽  
Gradient Descent ◽  
Descent Algorithm ◽  
Novel Technique ◽  
New Approaches ◽  
Gradient Descent Algorithm

Predicting the solutions of complex systems is a crucial challenge. Complexity exists because of the uncertainty as well as nonlinearity. The nonlinearity in complex systems makes uncertainty irreducible in several cases. In this paper, two new approaches based on neural networks are proposed in order to find the estimated solutions of the fully fuzzy nonlinear system (FFNS). For obtaining the estimated solutions, a gradient descent algorithm is proposed in order to train the proposed networks. An example is proposed in order to show the efficiency of the considered approaches.

Control of Multivariable Coupling System Based on Improved PID Neural Networks

2012 ◽  
Vol 157-158 ◽  
pp. 386-389
Author(s):  
Jun Kang ◽  
Wen Jun Meng
Keyword(s):  
Neural Networks ◽  
High Precision ◽  
Gradient Descent ◽  
Pso Algorithm ◽  
Initial Structure ◽  
Descent Algorithm ◽  
Coupling System ◽  
Gradient Descent Algorithm ◽  
Short Time

The paper analyzes the characteristics and the situation of the control of multivariable coupling system, combining improved PID neural networks with PSO algorithm, and finally designs a suitable controller model. To achieve a controller model, an initial structure of PID neural networks is established in the first place. Then weights in the network is initialized by PSO algorithm and optimized by improved gradient descent algorithm. A simulation, MIMO coupling system, prove this controller has such characteristics with short time and high precision. The research of the paper provides a new idea and approach for control of complex coupling system.

Sign in / Sign up

Export Citation Format

Share Document