An Overview of Predictive Learning and Function Approximation

1994 ◽  
pp. 1-61 ◽  
Author(s):  
Jerome H. Friedman
Keyword(s):  
Function Approximation ◽  
Predictive Learning ◽  
And Function

PRUNING AND MODEL-SELECTING ALGORITHMS IN THE RBF FRAMEWORKS CONSTRUCTED BY SUPPORT VECTOR LEARNING

2006 ◽  
Vol 16 (04) ◽  
pp. 283-293 ◽  
Author(s):  
PEI-YI HAO ◽  
JUNG-HSIEN CHIANG
Keyword(s):  
Function Approximation ◽  
Synthetic Data ◽  
Support Vector ◽  
Detection Problem ◽  
Data Simulation ◽  
Penalty Term ◽  
Rbf Network ◽  
Sample Classification ◽  
Pruning Algorithms ◽  
And Function

This paper presents the pruning and model-selecting algorithms to the support vector learning for sample classification and function regression. When constructing RBF network by support vector learning we occasionally obtain redundant support vectors which do not significantly affect the final classification and function approximation results. The pruning algorithms primarily based on the sensitivity measure and the penalty term. The kernel function parameters and the position of each support vector are updated in order to have minimal increase in error, and this makes the structure of SVM network more flexible. We illustrate this approach with synthetic data simulation and face detection problem in order to demonstrate the pruning effectiveness.

Decorrelated Hebbian Learning for Clustering and Function Approximation

1995 ◽  
Vol 7 (2) ◽  
pp. 338-348 ◽  
Author(s):  
G. Deco ◽  
D. Obradovic
Keyword(s):  
Cost Function ◽  
Function Approximation ◽  
Hebbian Learning ◽  
Clustering Problem ◽  
New Learning ◽  
Hidden Layer ◽  
And Function ◽  
The Cost ◽  
Winner Take All ◽  
Gaussian Network

This paper presents a new learning paradigm that consists of a Hebbian and anti-Hebbian learning. A layer of radial basis functions is adapted in an unsupervised fashion by minimizing a two-element cost function. The first element maximizes the output of each gaussian neuron and it can be seen as an implementation of the traditional Hebbian learning law. The second element of the cost function reinforces the competitive learning by penalizing the correlation between the nodes. Consequently, the second term has an “anti-Hebbian” effect that is learned by the gaussian neurons without the implementation of lateral inhibition synapses. Therefore, the decorrelated Hebbian learning (DHL) performs clustering in the input space avoiding the “nonbiological” winner-take-all rule. In addition to the standard clustering problem, this paper also presents an application of the DHL in function approximation. A scaled piece-wise linear approximation of a function is obtained in the supervised fashion within the local regions of its domain determined by the DHL. For comparison, a standard single hidden-layer gaussian network is optimized with the initial centers corresponding to the DHL. The efficiency of the algorithm is demonstrated on the chaotic Mackey-Glass time series.

On the Equivalence between Ordinary Neural Networks and Higher Order Neural Networks

2010 ◽  
pp. 138-158 ◽  
Author(s):  
Mohammed Sadiq Al-Rawi ◽  
Kamal R. Al-Rawi
Keyword(s):  
Neural Networks ◽  
Function Approximation ◽  
Input Data ◽  
Expressive Power ◽  
Higher Order ◽  
Training Algorithms ◽  
Neuron Activation ◽  
Higher Order Neural Networks ◽  
And Function ◽  
Complex Valued

In this chapter, we study the equivalence between multilayer feedforward neural networks referred as Ordinary Neural Networks (ONNs) that contain only summation (Sigma) as activation units, and multilayer feedforward Higher order Neural Networks (HONNs) that contains Sigma and product (PI) activation units. Since the time they were introduced by Giles and Maxwell (1987), HONNs have been used in many supervised classification and function approximation. Up to the date of writing this chapter, the most cited HONN article by ISI Thomson Web of Knowledge is the work of Kosmatopoulos et al., (1995) by which they introduced a recurrent HONN modeling. A simple comparison with ONNs is usually performed in order to demonstrate the performance of some newly introduced HONN architecture. Is it true that HONNs outperform ONNs, how much do they differ? And how much do they commute? Does equivalence exists between a HONN and an ONN? Is it possible to convert a HONN to an equivalent ONN? And how neural network equivalence is defined? This chapter tries to answer most of these questions. Due to the existence of huge neural networks architectures in the literature, the authors of this work are concerned and think that equivalence studies are necessary to give abstract definitions and unified approaches which might help in better understanding of HONNs performance and their respective design. On contrary to most of the previous works were HONN weights are non-negative integers, HONNs are given in this chapter in a form such that weights are adjustable real-valued numbers. In doing that, HONNs might have more expressive power and there is an increase probability of having complex valued neuron outputs. To enable the use of the real-valued weights that may result in a complex valued neuron output we introduce normalization to the input data as well as a modification to neuron activation functions. Using simple mathematics and the proposed normalization to input data, we showed that HONNs are equivalent to ONNs. The converted equivalent ONN posses the features of HONN and they have exactly the same functionality and output. The proposed conversion of HONN to ONN would permit using the huge amount of optimization algorithms to speed up the convergence of HONN and/or finding better topology. Recurrent HONNs, cascaded correlation HONNs, or any other complicated HONN can be simply defined via their equivalent ONNs and then trained with backpropagation, scaled conjugate gradient, Lavenberg-Marqudat algorithm, brain damage algorithms (Duda et al., 2000), etc. Using the developed equivalency model, this chapter also gives an easy bottom-up approach to convert a HONN to its equivalent ONN. Results on XOR and function approximation problems showed that ONNs obtained from their corresponding HONNs converged well to a solution. Different optimization training algorithms have been tested equivalent ONNs having feedforward structure and/or cascade correlation where the later have shown outstanding function approximation results.

Sign in / Sign up

Export Citation Format

Share Document