Improvement of Learning in Recurrent Networks by Substituting the Sigmoid Activation Function

ICANN ’94 ◽  
1994 ◽  
pp. 417-420 ◽  
Author(s):  
J. M. Sopena ◽  
R. Alquezar
2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.


2003 ◽  
Vol 15 (8) ◽  
pp. 1897-1929 ◽  
Author(s):  
Barbara Hammer ◽  
Peter Tiňo

Recent experimental studies indicate that recurrent neural networks initialized with “small” weights are inherently biased toward definite memory machines (Tiňno, Čerňanský, & Beňušková, 2002a, 2002b). This article establishes a theoretical counterpart: transition function of recurrent network with small weights and squashing activation function is a contraction. We prove that recurrent networks with contractive transition function can be approximated arbitrarily well on input sequences of unbounded length by a definite memory machine. Conversely, every definite memory machine can be simulated by a recurrent network with contractive transition function. Hence, initialization with small weights induces an architectural bias into learning with recurrent neural networks. This bias might have benefits from the point of view of statistical learning theory: it emphasizes one possible region of the weight space where generalization ability can be formally proved. It is well known that standard recurrent neural networks are not distribution independent learnable in the probably approximately correct (PAC) sense if arbitrary precision and inputs are considered. We prove that recurrent networks with contractive transition function with a fixed contraction parameter fulfill the so-called distribution independent uniform convergence of empirical distances property and hence, unlike general recurrent networks, are distribution independent PAC learnable.


2019 ◽  
Vol 8 (2S11) ◽  
pp. 3523-3526

This paper describes an efficient algorithm for classification in large data set. While many algorithms exist for classification, they are not suitable for larger contents and different data sets. For working with large data sets various ELM algorithms are available in literature. However the existing algorithms using fixed activation function and it may lead deficiency in working with large data. In this paper, we proposed novel ELM comply with sigmoid activation function. The experimental evaluations demonstrate the our ELM-S algorithm is performing better than ELM,SVM and other state of art algorithms on large data sets.


Author(s):  
George Mourgias-Alexandris ◽  
George Dabos ◽  
Nikolaos Passalis ◽  
Anastasios Tefas ◽  
Angelina Totovic ◽  
...  

NeuroImage ◽  
2008 ◽  
Vol 42 (1) ◽  
pp. 147-157 ◽  
Author(s):  
André C. Marreiros ◽  
Jean Daunizeau ◽  
Stefan J. Kiebel ◽  
Karl J. Friston

2019 ◽  
Vol 27 (7) ◽  
pp. 9620 ◽  
Author(s):  
G. Mourgias-Alexandris ◽  
A. Tsakyridis ◽  
N. Passalis ◽  
A. Tefas ◽  
K. Vyrsokinos ◽  
...  

1999 ◽  
Author(s):  
Arturo Pacheco-Vega ◽  
Mihir Sen ◽  
K. T. Yang ◽  
Rodney L. McClain

Abstract In the present study we apply an artificial neural network to predict the operation of a humid air-water fin-tube compact heat exchanger. The network configuration is of the feedforward type with a sigmoid activation function and a backpropagation algorithm. Published experimental data, corresponding to humid air flowing over the heat exchanger tubes and water flowing inside them, are used to train the neural network. After training with known experimental values of the humid-air flow rates, dry-bulb and wet-bulb inlet temperatures for various geometrical configurations, the j-factor and heat transfer rate predictions of the network were tested against the experimental values. Comparisons were made with published predictions of power-law correlations which were obtained from the same data. The results demonstrate that the neural network is able to predict the performance of this heat exchanger much better than the correlations.


Sign in / Sign up

Export Citation Format

Share Document