A novel approach to error function minimization for feedforward neural networks

The article deals with the problem of recognition of handwritten digits using feedforward neural networks (perceptrons) using a correlation indicator. The proposed method is based on the mathematical model of the neural network as an oscillatory system similar to the information transmission system. The article uses theoretical developments of the authors to search for the global extremum of the error function in artificial neural networks. The handwritten digit image is considered as a one-dimensional input discrete signal representing a combination of "perfect digit writing" and noise, which describes the deviation of the input implementation from "perfect writing". The ideal observer criterion (Kotelnikov criterion), which is widely used in information transmission systems and describes the probability of correct recognition of the input signal, is used to form the loss function. In the article is carried out a comparative analysis of the convergence of learning and experimentally obtained sequences on the basis of the correlation indicator and widely used in the tasks of classification of the function CrossEntropyLoss with the use of the optimizer and without it. Based on the experiments carried out, it is concluded that the proposed correlation indicator has an advantage of 2-3 times.

Download Full-text

Training Pi-Sigma Network by Online Gradient Algorithm with Penalty for Small Weight Update

Neural Computation ◽

10.1162/neco.2007.19.12.3356 ◽

2007 ◽

Vol 19 (12) ◽

pp. 3356-3368 ◽

Cited By ~ 10

Author(s):

Yan Xiong ◽

Wei Wu ◽

Xidai Kang ◽

Chao Zhang

Keyword(s):

Neural Networks ◽

Numerical Experiments ◽

Error Function ◽

Feedforward Neural Networks ◽

Gradient Algorithm ◽

Training Method ◽

Penalty Term ◽

Slow Convergence ◽

Output Layer ◽

Adaptive Penalty

A pi-sigma network is a class of feedforward neural networks with product units in the output layer. An online gradient algorithm is the simplest and most often used training method for feedforward neural networks. But there arises a problem when the online gradient algorithm is used for pi-sigma networks in that the update increment of the weights may become very small, especially early in training, resulting in a very slow convergence. To overcome this difficulty, we introduce an adaptive penalty term into the error function, so as to increase the magnitude of the update increment of the weights when it is too small. This strategy brings about faster convergence as shown by the numerical experiments carried out in this letter.

Download Full-text

Modified Neural Network Algorithms for Predicting Trading Signals of Stock Market Indices

Journal of Applied Mathematics and Decision Sciences ◽

10.1155/2009/125308 ◽

2009 ◽

Vol 2009 ◽

pp. 1-22 ◽

Cited By ~ 10

Author(s):

C. D. Tilakaratne ◽

M. A. Mammadov ◽

S. A. Morris

Keyword(s):

Neural Network ◽

Neural Networks ◽

Global Optimization ◽

Stock Market ◽

Historical Data ◽

Error Function ◽

Feedforward Neural Networks ◽

Ordinary Least Squares ◽

Global Optimization Algorithm ◽

Network Algorithms

The aim of this paper is to present modified neural network algorithms to predict whether it is best to buy, hold, or sell shares (trading signals) of stock market indices. Most commonly used classification techniques are not successful in predicting trading signals when the distribution of the actual trading signals, among these three classes, is imbalanced. The modified network algorithms are based on the structure of feedforward neural networks and a modified Ordinary Least Squares (OLSs) error function. An adjustment relating to the contribution from the historical data used for training the networks and penalisation of incorrectly classified trading signals were accounted for, when modifying the OLS function. A global optimization algorithm was employed to train these networks. These algorithms were employed to predict the trading signals of the Australian All Ordinary Index. The algorithms with the modified error functions introduced by this study produced better predictions.

Download Full-text

Neural networks for prediction of robot failures

Proceedings of the Institution of Mechanical Engineers Part C Journal of Mechanical Engineering Science ◽

10.1177/0954406213507704 ◽

2013 ◽

Vol 228 (8) ◽

pp. 1444-1458 ◽

Cited By ~ 6

Author(s):

Ali Diryag ◽

Marko Mitić ◽

Zoran Miljković

Keyword(s):

Neural Network ◽

Neural Networks ◽

Optimal Solution ◽

Feedforward Neural Networks ◽

Real Data ◽

Obstacle Detection ◽

Network Training ◽

Novel Approach ◽

The Neural Network ◽

Prediction Approach

It is known that the supervision and learning of robotic executions is not a trivial problem. Nowadays, robots must be able to tolerate and predict internal failures in order to successfully continue performing their tasks. This study presents a novel approach for prediction of robot execution failures based on neural networks. Real data consisting of robot forces and torques recorded immediately after the system failure are used for the neural network training. The multilayer feedforward neural networks are employed in order to find optimal solution for the failure prediction problem. In total, 7 learning algorithms and 24 neural architectures are implemented in two environments – Matlab and specially designed software titled BPnet. The results show that the neural networks can successfully be applied for the problem in hand with prediction rate of 95.4545%, despite having the erroneous or otherwise incomplete sensor measurements invoked in the dataset. Additionally, the real-world experiments are conducted on a mobile robot for obstacle detection and trajectory tracking problems in order to prove the robustness of the proposed prediction approach. In over 96% for the detection problem and 99% for the tracking experiments, neural network successfully predicted the failed information, which evidences the usefulness and the applicability of the developed intelligent method.

Download Full-text

FEEDFORWARD NEURAL NETWORK MODELS FOR HANDLING CLASS OVERLAP AND CLASS IMBALANCE

International Journal of Neural Systems ◽

10.1142/s012906570500030x ◽

2005 ◽

Vol 15 (05) ◽

pp. 323-338 ◽

Cited By ~ 3

Author(s):

RALF KRETZSCHMAR ◽

NICOLAOS B. KARAYIANNIS ◽

FRITZ EGGIMANN

Keyword(s):

Neural Network ◽

Neural Networks ◽

Error Function ◽

Class Imbalance ◽

Network Models ◽

Feedforward Neural Networks ◽

Feedforward Neural Network ◽

Neural Network Models ◽

Real World Data ◽

Special Case

This paper proposes a framework for training feedforward neural network models capable of handling class overlap and imbalance by minimizing an error function that compensates for such imperfections of the training set. A special case of the proposed error function can be used for training variance-controlled neural networks (VCNNs), which are developed to handle class overlap by minimizing an error function involving the class-specific variance (CSV) computed at their outputs. Another special case of the proposed error function can be used for training class-balancing neural networks (CBNNs), which are developed to handle class imbalance by relying on class-specific correction (CSC). VCNNs and CBNNs are compared with conventional feedforward neural networks (FFNNs), quantum neural networks (QNNs), and resampling techniques. The properties of VCNNs and CBNNs are illustrated by experiments on artificial data. Various experiments involving real-world data reveal the advantages offered by VCNNs and CBNNs in the presence of class overlap and class imbalance.

Download Full-text

Feedforward Neural Networks with a Hidden Layer Regularization Method

Symmetry ◽

10.3390/sym10100525 ◽

2018 ◽

Vol 10 (10) ◽

pp. 525 ◽

Cited By ~ 3

Author(s):

Habtamu Alemu ◽

Wei Wu ◽

Junhong Zhao

Keyword(s):

Neural Network ◽

Neural Networks ◽

Network Structure ◽

Regularization Method ◽

Error Function ◽

Feedforward Neural Networks ◽

Training Process ◽

Regularization Term ◽

Neural Network Structure ◽

Hidden Layer

In this paper, we propose a group Lasso regularization term as a hidden layer regularization method for feedforward neural networks. Adding a group Lasso regularization term into the standard error function as a hidden layer regularization term is a fruitful approach to eliminate the redundant or unnecessary hidden layer neurons from the feedforward neural network structure. As a comparison, a popular Lasso regularization method is introduced into standard error function of the network. Our novel hidden layer regularization method can force a group of outgoing weights to become smaller during the training process and can eventually be removed after the training process. This means it can simplify the neural network structure and it minimizes the computational cost. Numerical simulations are provided by using K-fold cross-validation method with K = 5 to avoid overtraining and to select the best learning parameters. The numerical results show that our proposed hidden layer regularization method prunes more redundant hidden layer neurons consistently for each benchmark dataset without loss of accuracy. In contrast, the existing Lasso regularization method prunes only the redundant weights of the network, but it cannot prune any redundant hidden layer neurons.

Download Full-text