Feedforward neural networks for Bayes-optimal classification: investigations into the influence of the composition of the training set on the cost function

Author(s):  
A. Doering ◽  
H. Witte
Symmetry ◽  
2020 ◽  
Vol 12 (4) ◽  
pp. 660 ◽  
Author(s):  
Jieun Park ◽  
Dokkyun Yi ◽  
Sangmin Ji

The process of machine learning is to find parameters that minimize the cost function constructed by learning the data. This is called optimization and the parameters at that time are called the optimal parameters in neural networks. In the process of finding the optimization, there were attempts to solve the symmetric optimization or initialize the parameters symmetrically. Furthermore, in order to obtain the optimal parameters, the existing methods have used methods in which the learning rate is decreased over the iteration time or is changed according to a certain ratio. These methods are a monotonically decreasing method at a constant rate according to the iteration time. Our idea is to make the learning rate changeable unlike the monotonically decreasing method. We introduce a method to find the optimal parameters which adaptively changes the learning rate according to the value of the cost function. Therefore, when the cost function is optimized, the learning is complete and the optimal parameters are obtained. This paper proves that the method ensures convergence to the optimal parameters. This means that our method achieves a minimum of the cost function (or effective learning). Numerical experiments demonstrate that learning is good effective when using the proposed learning rate schedule in various situations.


Author(s):  
TAO WANG ◽  
XIAOLIANG XING ◽  
XINHUA ZHUANG

In this paper, we describe an optimal learning algorithm for designing one-layer neural networks by means of global minimization. Taking the properties of a well-defined neural network into account, we derive a cost function to measure the goodness of the network quantitatively. The connection weights are determined by the gradient descent rule to minimize the cost function. The optimal learning algorithm is formed as either the unconstraint-based or the constraint-based minimization problem. It ensures the realization of each desired associative mapping with the best noise reduction ability in the sense of optimization. We also investigate the storage capacity of the neural network, the degree of noise reduction for a desired associative mapping, and the convergence of the learning algorithm in an analytic way. Finally, a large number of computer experimental results are presented.


1990 ◽  
Vol 01 (03) ◽  
pp. 237-245 ◽  
Author(s):  
Edgardo A. Ferrán ◽  
Roberto P. J. Perazzo

A model is proposed in which the synaptic efficacies of a feedforward neural network are adapted with a cost function that vanishes if the boolean function that is represented has the same symmetry properties as the target one. The function chosen according to this procedure is thus taken as an archetype of the whole symmetry class. Several examples are presented showing how this type of partial learning can produce a behaviour of the net that is reminiscent of that of dyslexic persons.


MAUSAM ◽  
2022 ◽  
Vol 53 (2) ◽  
pp. 225-232
Author(s):  
PANKAJ JAIN ◽  
ASHOK KUMAR ◽  
PARVINDER MAINI ◽  
S. V. SINGH

Feedforward Neural Networks are used for daily precipitation forecast using several test stations all over India. The six year European Centre of Medium Range Weather Forecasting (ECMWF) data is used with the training set consisting of the four year data from 1985-1988 and validation set consisting of the data from 1989-1990. Neural networks are used to develop a concurrent relationship between precipitation and other atmospheric variables. No attempt is made to select optimal variables for this study and the inputs are chosen to be same as the ones obtained earlier at National Center for Medium Range Weather Forecasting (NCMRWF) in developing a linear regression model. Neural networks are found to yield results which are atleast as good as linear regression and in several cases yield 10 - 20 % improvement. This is encouraging since the variable selection has so far been optimized for linear regression.


1995 ◽  
Vol 06 (01) ◽  
pp. 61-78 ◽  
Author(s):  
FOO SHOU KING ◽  
P. SARATCHANDRAN ◽  
N. SUNDARARAJAN

Training set parallelism and network based parallelism are two popular paradigms for parallelizing a feedforward (artificial) neural network. Training set parallelism is particularly suited to feedforward neural networks with backpropagation learning where the size of the training set is large in relation to the size of the network. This paper analyzes training set parallelism for feedforward neural networks when implemented on a transputer array configured in a pipelined ring topology. Theoretical expressions for the time per epoch (iteration) and optimal size of a processor network are derived when the training set is equally distributed among the processing nodes. These show that the speed up is a function of the number of patterns per processor, communication overhead per epoch and the total number of processors in the topology. Further analysis of how to optimally distribute the training set on a given processor network when the number of patterns in the training set is not an integer multiple of the number of processors, is also carried out. It is shown that optimal allocation of patterns in such cases is a mixed integer programming problem. Using this analysis it is found that equal distribution of training patterns among the processors is not the optimal way to allocate the patterns even when the training set is an integer multiple of the number of processors. Extension of the analysis to processor networks comprising processors of different speeds is also carried out. Experimental results from a T805 transputer array are presented to verify all the theoretical results.


atp magazin ◽  
2018 ◽  
Vol 60 (08) ◽  
pp. 46
Author(s):  
Stefan Löw ◽  
Dragan Obradovic

Nonlinear Model Predictive Control (NMPC) is an aspiring control method for the implementation of advanced controller behavior. The present work shows the symbolic math implementation of a mechatronic system model containing aerodynamic nonlinearities modeled by Feedforward Neural Networks. Gradients for the optimization are obtained efficiently by exploiting the feedforward property of the Neural Networks and symbolic computation. Current research on the implementation of damage metrics into the cost function is stated briefly. In order to achieve real-time capability, the method Real-time Iteration is used.


Author(s):  
Tuan Hoang ◽  
Thanh-Toan Do ◽  
Tam V. Nguyen ◽  
Ngai-Man Cheung

This paper proposes two novel techniques to train deep convolutional neural networks with low bit-width weights and activations. First, to obtain low bit-width weights, most existing methods obtain the quantized weights by performing quantization on the full-precision network weights. However, this approach would result in some mismatch: the gradient descent updates full-precision weights, but it does not update the quantized weights. To address this issue, we propose a novel method that enables direct updating of quantized weights with learnable quantization levels to minimize the cost function using gradient descent. Second, to obtain low bit-width activations, existing works consider all channels equally. However, the activation quantizers could be biased toward a few channels with high-variance. To address this issue, we propose a method to take into account the quantization errors of individual channels. With this approach, we can learn activation quantizers that minimize the quantization errors in the majority of channels. Experimental results demonstrate that our proposed method achieves state-of-the-art performance on the image classification task, using AlexNet, ResNet and MobileNetV2 architectures on CIFAR-100 and ImageNet datasets.


2021 ◽  
Author(s):  
Amir Valizadeh

Abstract In this paper, an alternative way to backpropagation is introduced and tested, which results in faster convergence of the cost function.


1990 ◽  
Vol 2 (2) ◽  
pp. 198-209 ◽  
Author(s):  
Marcus Frean

A general method for building and training multilayer perceptrons composed of linear threshold units is proposed. A simple recursive rule is used to build the structure of the network by adding units as they are needed, while a modified perceptron algorithm is used to learn the connection strengths. Convergence to zero errors is guaranteed for any boolean classification on patterns of binary variables. Simulations suggest that this method is efficient in terms of the numbers of units constructed, and the networks it builds can generalize over patterns not in the training set.


Sign in / Sign up

Export Citation Format

Share Document