Convergence of Batch Gradient Method Based on the Entropy Error Function for Feedforward Neural Networks

The article deals with the problem of recognition of handwritten digits using feedforward neural networks (perceptrons) using a correlation indicator. The proposed method is based on the mathematical model of the neural network as an oscillatory system similar to the information transmission system. The article uses theoretical developments of the authors to search for the global extremum of the error function in artificial neural networks. The handwritten digit image is considered as a one-dimensional input discrete signal representing a combination of "perfect digit writing" and noise, which describes the deviation of the input implementation from "perfect writing". The ideal observer criterion (Kotelnikov criterion), which is widely used in information transmission systems and describes the probability of correct recognition of the input signal, is used to form the loss function. In the article is carried out a comparative analysis of the convergence of learning and experimentally obtained sequences on the basis of the correlation indicator and widely used in the tasks of classification of the function CrossEntropyLoss with the use of the optimizer and without it. Based on the experiments carried out, it is concluded that the proposed correlation indicator has an advantage of 2-3 times.

Download Full-text

Classification and Prediction with Neural Networks

Data Mining and Medical Knowledge Management ◽

10.4018/978-1-60566-218-3.ch004 ◽

2011 ◽

pp. 76-107

Author(s):

Arnošt Veselý

Keyword(s):

Neural Networks ◽

Error Function ◽

Descent Method ◽

Cross Entropy ◽

Gradient Descent Method ◽

Classification Problems ◽

Network Training ◽

The Neural Network ◽

Gradient Calculation ◽

Entropy Error

This chapter deals with applications of artificial neural networks in classification and regression problems. Based on theoretical analysis it demonstrates that in classification problems one should use cross-entropy error function rather than the usual sum-of-square error function. Using gradient descent method for finding the minimum of the cross entropy error function, leads to the well-known backpropagation of error scheme of gradient calculation if at the output layer of the neural network the neurons with logistic or softmax output functions are used. The author believes that understanding the underlying theory presented in this chapter will help researchers in medical informatics to choose more suitable network architectures for medical applications and that it helps them to carry out the network training more effectively.

Download Full-text

Training Pi-Sigma Network by Online Gradient Algorithm with Penalty for Small Weight Update

Neural Computation ◽

10.1162/neco.2007.19.12.3356 ◽

2007 ◽

Vol 19 (12) ◽

pp. 3356-3368 ◽

Cited By ~ 10

Author(s):

Yan Xiong ◽

Wei Wu ◽

Xidai Kang ◽

Chao Zhang

Keyword(s):

Neural Networks ◽

Numerical Experiments ◽

Error Function ◽

Feedforward Neural Networks ◽

Gradient Algorithm ◽

Training Method ◽

Penalty Term ◽

Slow Convergence ◽

Output Layer ◽

Adaptive Penalty

A pi-sigma network is a class of feedforward neural networks with product units in the output layer. An online gradient algorithm is the simplest and most often used training method for feedforward neural networks. But there arises a problem when the online gradient algorithm is used for pi-sigma networks in that the update increment of the weights may become very small, especially early in training, resulting in a very slow convergence. To overcome this difficulty, we introduce an adaptive penalty term into the error function, so as to increase the magnitude of the update increment of the weights when it is too small. This strategy brings about faster convergence as shown by the numerical experiments carried out in this letter.

Download Full-text

Convergence of an Online Split-Complex Gradient Algorithm for Complex-Valued Neural Networks

Discrete Dynamics in Nature and Society ◽

10.1155/2010/829692 ◽

2010 ◽

Vol 2010 ◽

pp. 1-27

Author(s):

Huisheng Zhang ◽

Dongpo Xu ◽

Zhiping Wang

Keyword(s):

Neural Networks ◽

Fixed Point ◽

Adaptive Learning ◽

Gradient Method ◽

Error Function ◽

Gradient Algorithm ◽

Training Procedure ◽

Weight Sequence ◽

Complex Valued ◽

Complex Gradient

The online gradient method has been widely used in training neural networks. We consider in this paper an online split-complex gradient algorithm for complex-valued neural networks. We choose an adaptive learning rate during the training procedure. Under certain conditions, by firstly showing the monotonicity of the error function, it is proved that the gradient of the error function tends to zero and the weight sequence tends to a fixed point. A numerical example is given to support the theoretical findings.

Download Full-text