A novel learning objective function using localized generalization error bound for RBF network

2009 ◽  
Author(s):  
Daniel S. Yueng ◽  
Patrick P.K. Chan
Keyword(s):  
Objective Function ◽  
Error Bound ◽  
Learning Objective ◽  
Generalization Error ◽  
Rbf Network ◽  
Generalization Error Bound ◽  
Localized Generalization Error

THE LOCALIZED GENERALIZATION ERROR MODEL FOR SINGLE LAYER PERCEPTRON NEURAL NETWORK AND SIGMOID SUPPORT VECTOR MACHINE

2008 ◽  
Vol 22 (01) ◽  
pp. 121-135 ◽  
Author(s):  
WING W. Y. NG ◽  
DANIEL S. YEUNG ◽  
ERIC C. C. TSANG
Keyword(s):  
Neural Network ◽  
Support Vector Machine ◽  
Error Bound ◽  
Single Layer ◽  
Error Model ◽  
Support Vector ◽  
Generalization Error ◽  
Sensitivity Measure ◽  
Stochastic Sensitivity ◽  
Localized Generalization Error

We had developed the localized generalization error model for supervised learning with minimization of Mean Square Error. In this work, we extend the error model to Single Layer Perceptron Neural Network (SLPNN) and Support Vector Machine (SVM) with sigmoid kernel function. For a trained SLPNN or SVM and a given training dataset, the proposed error model bounds above the error for unseen samples which are similar to the training samples. As the major component of the localized generalization error model, the stochastic sensitivity measure formula for perceptron neural network derived in this work has relaxed the assumptions of same distribution for all inputs and each sample perturbed only once in previous works. These make the sensitivity measure applicable to pattern classification problems. The stochastic sensitivity measure of SVM with Sigmoid kernel is also derived in this work as a component of the localized generalization error model. At the end of this paper, we discuss the advantages of the proposed error bound over existing error bound.

scholarly journals A Novel Radial Basis Function Neural Network with High Generalization Performance for Nonlinear Process Modelling

Processes ◽  
2022 ◽  
Vol 10 (1) ◽  
pp. 140
Author(s):  
Yanxia Yang ◽  
Pu Wang ◽  
Xuejin Gao
Keyword(s):  
Neural Network ◽  
Radial Basis Function ◽  
Error Bound ◽  
Basis Function ◽  
Nonlinear Process ◽  
Generalization Error ◽  
Sample Mean ◽  
Generalization Ability ◽  
Radial Basis ◽  
Generalization Error Bound

A radial basis function neural network (RBFNN), with a strong function approximation ability, was proven to be an effective tool for nonlinear process modeling. However, in many instances, the sample set is limited and the model evaluation error is fixed, which makes it very difficult to construct an optimal network structure to ensure the generalization ability of the established nonlinear process model. To solve this problem, a novel RBFNN with a high generation performance (RBFNN-GP), is proposed in this paper. The proposed RBFNN-GP consists of three contributions. First, a local generalization error bound, introducing the sample mean and variance, is developed to acquire a small error bound to reduce the range of error. Second, the self-organizing structure method, based on a generalization error bound and network sensitivity, is established to obtain a suitable number of neurons to improve the generalization ability. Third, the convergence of this proposed RBFNN-GP is proved theoretically in the case of structure fixation and structure adjustment. Finally, the performance of the proposed RBFNN-GP is compared with some popular algorithms, using two numerical simulations and a practical application. The comparison results verified the effectiveness of RBFNN-GP.

scholarly journals The Generalization Error Bound for the Multiclass Analytical Center Classifier

2013 ◽  
Vol 2013 ◽  
pp. 1-5
Author(s):  
Zeng Fanzi ◽  
Ma Xiaolong
Keyword(s):  
Error Bound ◽  
Upper Bound ◽  
Linear Optimization ◽  
Generalization Error ◽  
Single Class ◽  
Benchmark Datasets ◽  
Generalization Error Bound ◽  
Multiclass Classifier ◽  
Feasible Space ◽  
Analytical Center

This paper presents the multiclass classifier based on analytical center of feasible space (MACM). This multiclass classifier is formulated as quadratic constrained linear optimization and does not need repeatedly constructing classifiers to separate a single class from all the others. Its generalization error upper bound is proved theoretically. The experiments on benchmark datasets validate the generalization performance of MACM.

scholarly journals Improving the Robustness of Wasserstein Embedding by Adversarial PAC-Bayesian Learning

2020 ◽  
Vol 34 (04) ◽  
pp. 3791-3800
Author(s):  
Daizong Ding ◽  
Mi Zhang ◽  
Xudong Pan ◽  
Min Yang ◽  
Xiangnan He
Keyword(s):  
Error Bound ◽  
Upper Bound ◽  
Bayesian Learning ◽  
State Of The Art ◽  
Comprehensive Analysis ◽  
Graph Analysis ◽  
Generalization Error ◽  
New Challenges ◽  
Generalization Error Bound ◽  
First Time

Node embedding is a crucial task in graph analysis. Recently, several methods are proposed to embed a node as a distribution rather than a vector to capture more information. Although these methods achieved noticeable improvements, their extra complexity brings new challenges. For example, the learned representations of nodes could be sensitive to external noises on the graph and vulnerable to adversarial behaviors. In this paper, we first derive an upper bound on generalization error for Wasserstein embedding via the PAC-Bayesian theory. Based on this, we propose an algorithm called Adversarial PAC-Bayesian Learning (APBL) in order to minimize the generalization error bound. Furthermore, we provide a model called Regularized Adversarial Wasserstein Embedding Network (RAWEN) as an implementation of APBL. Besides our comprehensive analysis of the robustness of RAWEN, our work for the first time explores more kinds of embedded distributions. For evaluations, we conduct extensive experiments to demonstrate the effectiveness and robustness of our proposed embedding model compared with the state-of-the-art methods.

scholarly journals Bridging the Gap between Few-Shot and Many-Shot Learning via Distribution Calibration

2021 ◽  
Author(s):  
Shuo Yang ◽  
Songhua Wu ◽  
Tongliang Liu ◽  
Min Xu
Keyword(s):  
Error Bound ◽  
Estimation Error ◽  
Data Distribution ◽  
Approximation Error ◽  
Ground Truth ◽  
Generalization Error ◽  
Learning To Learn ◽  
Ground Truth Data ◽  
Generalization Error Bound ◽  
Training Examples

A major gap between few-shot and many-shot learning is the data distribution empirically observed by the model during training. In few-shot learning, the learned model can easily become over-fitted based on the biased distribution formed by only a few training examples, while the ground-truth data distribution is more accurately uncovered in many-shot learning to learn a well-generalized model. In this paper, we propose to calibrate the distribution of these few-sample classes to be more unbiased to alleviate such an over-fitting problem. The distribution calibration is achieved by transferring statistics from the classes with sufficient examples to those few-sample classes. After calibration, an adequate number of examples can be sampled from the calibrated distribution to expand the inputs to the classifier. Extensive experiments on three datasets, miniImageNet, tieredImageNet, and CUB, show that a simple linear classifier trained using the features sampled from our calibrated distribution can outperform the state-of-the-art accuracy by a large margin. We also establish a generalization error bound for the proposed distribution-calibration-based few-shot learning, which consists of the distribution assumption error, the distribution approximation error, and the estimation error. This generalization error bound theoretically justifies the effectiveness of the proposed method.

Sign in / Sign up

Export Citation Format

Share Document