Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation

1999 ◽  
Vol 11 (6) ◽  
pp. 1427-1453 ◽  
Author(s):  
Michael Kearns ◽  
Dana Ron
Keyword(s):  
Error Estimate ◽  
Cross Validation ◽  
Nearest Neighbor ◽  
Error Minimization ◽  
Worst Case ◽  
Local Algorithms ◽  
Algorithmic Stability ◽  
Minimization Algorithms ◽  
Training Error ◽  
Leave One Out

In this article we prove sanity-check bounds for the error of the leave-oneout cross-validation estimate of the generalization error: that is, bounds showing that the worst-case error of this estimate is not much worse than that of the training error estimate. The name sanity check refers to the fact that although we often expect the leave-one-out estimate to perform considerably better than the training error estimate, we are here only seeking assurance that its performance will not be considerably worse. Perhaps surprisingly, such assurance has been given only for limited cases in the prior literature on cross-validation. Any nontrivial bound on the error of leave-one-out must rely on some notion of algorithmic stability. Previous bounds relied on the rather strong notion of hypothesis stability, whose application was primarily limited to nearest-neighbor and other local algorithms. Here we introduce the new and weaker notion of error stability and apply it to obtain sanity-check bounds for leave-one-out for other classes of learning algorithms, including training error minimization procedures and Bayesian algorithms. We also provide lower bounds demonstrating the necessity of some form of error stability for proving bounds on the error of the leave-one-out estimate, and the fact that for training error minimization algorithms, in the worst case such bounds must still depend on the Vapnik-Chervonenkis dimension of the hypothesis class.

HYPER-PARAMETER SELECTION FOR SPARSE LS-SVM VIA MINIMIZATION OF ITS LOCALIZED GENERALIZATION ERROR

2013 ◽  
Vol 11 (03) ◽  
pp. 1350030 ◽  
Author(s):  
BINBIN SUN ◽  
WING W. Y. NG ◽  
DANIEL S. YEUNG ◽  
PATRICK P. K. CHAN
Keyword(s):  
Cross Validation ◽  
Parameter Selection ◽  
Data Sets ◽  
Generalization Error ◽  
Generalization Capability ◽  
Sensitivity Measure ◽  
Minimum Sensitivity ◽  
Training Error ◽  
Leave One Out ◽  
Localized Generalization Error

Sparse LS-SVM yields better generalization capability and reduces prediction time in comparison to full dense LS-SVM. However, both methods require careful selection of hyper-parameters (HPS) to achieve high generalization capability. Leave-One-Out Cross Validation (LOO-CV) and k-fold Cross Validation (k-CV) are the two most widely used hyper-parameter selection methods for LS-SVMs. However, both fail to select good hyper-parameters for sparse LS-SVM. In this paper we propose a new hyper-parameter selection method, LGEM-HPS, for LS-SVM via minimization of the Localized Generalization Error (L-GEM). The L-GEM consists of two major components: empirical mean square error and sensitivity measure. A new sensitivity measure is derived for LS-SVM to enable the LGEM-HPS select hyper-parameters yielding LS-SVM with smaller training error and minimum sensitivity to minor changes in inputs. Experiments on eleven UCI data sets show the effectiveness of the proposed method for selecting hyper-parameters for sparse LS-SVM.

Mapping Arctic clam abundance using multiple datasets, models, and a spatially explicit accuracy assessment

2019 ◽  
Vol 76 (7) ◽  
pp. 2349-2361
Author(s):  
Benjamin Misiuk ◽  
Trevor Bell ◽  
Alec Aitken ◽  
Craig J Brown ◽  
Evan N Edinger
Keyword(s):  
Spatial Autocorrelation ◽  
Cross Validation ◽  
Predictive Accuracy ◽  
Accuracy Assessment ◽  
Small Scale ◽  
Suitable Habitat ◽  
Distribution Models ◽  
Management Tools ◽  
Seabed Topography ◽  
Leave One Out

Abstract Species distribution models are commonly used in the marine environment as management tools. The high cost of collecting marine data for modelling makes them finite, especially in remote locations. Underwater image datasets from multiple surveys were leveraged to model the presence–absence and abundance of Arctic soft-shell clam (Mya spp.) to support the management of a local small-scale fishery in Qikiqtarjuaq, Nunavut, Canada. These models were combined to predict Mya abundance, conditional on presence throughout the study area. Results suggested that water depth was the primary environmental factor limiting Mya habitat suitability, yet seabed topography and substrate characteristics influence their abundance within suitable habitat. Ten-fold cross-validation and spatial leave-one-out cross-validation (LOO CV) were used to assess the accuracy of combined predictions and to test whether this was inflated by the spatial autocorrelation of transect sample data. Results demonstrated that four different measures of predictive accuracy were substantially inflated due to spatial autocorrelation, and the spatial LOO CV results were therefore adopted as the best estimates of performance.

scholarly journals QSPR study on the gas/particle partition coefficient of PCBs by using molecular distance-edge vector index

2014 ◽  
Vol 79 (8) ◽  
pp. 965-975 ◽  
Author(s):  
Long Jiao ◽  
Xiaofei Wang ◽  
LI. Hua ◽  
Yunxia Wang
Keyword(s):  
Partition Coefficient ◽  
Cross Validation ◽  
External Validation ◽  
Quantitative Relationship ◽  
Structure Property ◽  
Prediction Ability ◽  
Vector Index ◽  
Artificial Neural Network Ann ◽  
Leave One Out ◽  
Edge Vector

The quantitative structure property relationship (QSPR) for gas/particle partition coefficient, Kp, of polychlorinated biphenyls (PCBs) was investigated. Molecular distance-edge vector (MDEV) index was used as the structural descriptor of PCBs. The quantitative relationship between the MDEV index and log Kp was modeled by multivariate linear regression (MLR) and artificial neural network (ANN) respectively. Leave one out cross validation and external validation were carried out to assess the prediction ability of the developed models. When the MLR method is used, the root mean square relative error (RMSRE) of prediction for leave one out cross validation and external validation is 4.72 and 8.62 respectively. When the ANN method is employed, the prediction RMSRE of leave one out cross validation and external validation is 3.87 and 7.47 respectively. It is demonstrated that the developed models are practicable for predicting the Kp of PCBs. The MDEV index is shown to be quantitatively related to the Kp of PCBs.

scholarly journals Study onYang-XuUsing Body Constitution Questionnaire and Blood Variables in Healthy Volunteers

2016 ◽  
Vol 2016 ◽  
pp. 1-7 ◽  
Author(s):  
Hong-Jhang Chen ◽  
Yii-Jeng Lin ◽  
Pei-Chen Wu ◽  
Wei-Hsiang Hsu ◽  
Wan-Chung Hu ◽  
...  
Keyword(s):  
Healthy Subjects ◽  
Logistic Regression Model ◽  
Cross Validation ◽  
Blood Biomarkers ◽  
Metabolic Characteristics ◽  
Body Constitution ◽  
Leave One Out ◽  
The Relationship ◽  
Fold Cross Validation ◽  
Blood Variables

Traditional Chinese medicine (TCM) formulates treatment according to body constitution (BC) differentiation. Different constitutions have specific metabolic characteristics and different susceptibility to certain diseases. This study aimed to assess theYang-Xuconstitution using a body constitution questionnaire (BCQ) and clinical blood variables. A BCQ was employed to assess the clinical manifestation ofYang-Xu. The logistic regression model was conducted to explore the relationship between BC scores and biomarkers. Leave-one-out cross-validation (LOOCV) and K-fold cross-validation were performed to evaluate the accuracy of a predictive model in practice. Decision trees (DTs) were conducted to determine the possible relationships between blood biomarkers and BC scores. According to the BCQ analysis, 49% participants without any BC were classified as healthy subjects. Among them, 130 samples were selected for further analysis and divided into two groups. One group comprised healthy subjects without any BC (68%), while subjects of the other group, named as the sub-healthy group, had three BCs (32%). Six biomarkers, CRE, TSH, HB, MONO, RBC, and LH, were found to have the greatest impact on BCQ outcomes inYang-Xusubjects. This study indicated significant biochemical differences inYang-Xusubjects, which may provide a connection between blood variables and theYang-XuBC.

Evaluation of Safety Performance Functions Based on Experimental Design Using a Cross-Validation Method

2017 ◽  
Vol 2636 (1) ◽  
pp. 62-72 ◽  
Author(s):  
Jung-Han Wang ◽  
Mohamed A. Abdel-Aty ◽  
Jaeyoung Lee
Keyword(s):  
Goodness Of Fit ◽  
Cross Validation ◽  
Safety Performance ◽  
Major Road ◽  
Highway Safety Manual ◽  
Annual Average Daily Traffic ◽  
Minor Road ◽  
Safety Performance Functions ◽  
The One ◽  
Leave One Out

The Highway Safety Manual (HSM) Part C provides a series of safety performance functions (SPFs) for different roadway conditions. The SPFs suggested in the HSM are formulated on the basis of exposure variables: the logarithms of the annual average daily traffic (AADT) on the major road and on the minor road under the base condition. In this research, data from 7,802 intersections in Florida were collected and processed. These intersections were categorized into seven types based on area type (rural or urban), number of legs (three or four), and number of approaches controlled by stop signs. Twenty-two SPF formulations, including the one suggested by the HSM, were developed for each intersection type for examination of the goodness-of-fit measures of the SPFs. In addition, the goodness of fit of each model of the 22 SPFs in each category was examined with 10-fold leave-one-out cross-validation (LOOCV). With a comparison of the delta values generated with the LOOCV method, it is suggested that the SPF with the logarithm of the total entering vehicle volume and the ratio of the AADT on the minor road and the AADT on the major road are important. In addition, the SPFs with the AADT on the major road and the AADT on the minor road and their logarithmic transformations are also important. Therefore, it is suggested that the future HSM compare these two SPF formulations—as suggested in the current research, along with the original SPF formulation in the manual—and select the one with the best model fit on the basis of the delta value using LOOCV.

scholarly journals IILLS: predicting virus-receptor interactions based on similarity and semi-supervised learning

2019 ◽  
Vol 20 (S23) ◽  
Author(s):  
Cheng Yan ◽  
Guihua Duan ◽  
Fang-Xiang Wu ◽  
Jianxin Wang
Keyword(s):  
Infectious Diseases ◽  
Cross Validation ◽  
Sequence Similarity ◽  
Least Square ◽  
Computational Method ◽  
Receptor Interaction ◽  
Virus Receptor ◽  
Receptor Interactions ◽  
Leave One Out ◽  
Fold Cross Validation

Abstract Background Viral infectious diseases are the serious threat for human health. The receptor-binding is the first step for the viral infection of hosts. To more effectively treat human viral infectious diseases, the hidden virus-receptor interactions must be discovered. However, current computational methods for predicting virus-receptor interactions are limited. Result In this study, we propose a new computational method (IILLS) to predict virus-receptor interactions based on Initial Interaction scores method via the neighbors and the Laplacian regularized Least Square algorithm. IILLS integrates the known virus-receptor interactions and amino acid sequences of receptors. The similarity of viruses is calculated by the Gaussian Interaction Profile (GIP) kernel. On the other hand, we also compute the receptor GIP similarity and the receptor sequence similarity. Then the sequence similarity is used as the final similarity of receptors according to the prediction results. The 10-fold cross validation (10CV) and leave one out cross validation (LOOCV) are used to assess the prediction performance of our method. We also compare our method with other three competing methods (BRWH, LapRLS, CMF). Conlusion The experiment results show that IILLS achieves the AUC values of 0.8675 and 0.9061 with the 10-fold cross validation and leave-one-out cross validation (LOOCV), respectively, which illustrates that IILLS is superior to the competing methods. In addition, the case studies also further indicate that the IILLS method is effective for the virus-receptor interaction prediction.

scholarly journals Analisis Metode Pengenalan Wajah Two Dimensial Principal Component Analysis (2DPCA) dan Kernel Fisher Discriminant Analysis Menggunakan Klasifikasi KNN (K- Nearest Neighbor)

2020 ◽  
Vol 2 (2) ◽  
pp. 29-38
Author(s):  
Abdur Rohman Harits Martawireja ◽  
Hilman Mujahid Purnama ◽  
Atika Nur Rahmawati
Keyword(s):  
Principal Component Analysis ◽  
Discriminant Analysis ◽  
Cross Validation ◽  
Nearest Neighbor ◽  
Principal Component ◽  
Component Analysis ◽  
K Nearest Neighbor ◽  
Fisher Discriminant Analysis ◽  
Fisher Discriminant ◽  
Kernel Fisher Discriminant Analysis

Pengenalan wajah manusia (face recognition) merupakan salah satu bidang penelitian yang penting dan belakangan ini banyak aplikasi yang menerapkannya, baik di bidang komersil ataupun di bidang penegakan hukum. Pengenalan wajah merupakan sebuah sistem yang berfungsikan untuk mengidentifikasi berdasarkan ciri-ciri dari wajah seseorang berbasis biometrik yang memiliki keakuratan tinggi. Pengenalan wajah dapat diterapkan pada sistem keamanan. Banyak metode yang dapat digunakan dalam aplikasi pengenalan wajah untuk keamanan sistem, namun pada artikel ini akan membahas tentang dua metode yaitu Two Dimensial Principal Component Analysis dan Kernel Fisher Discriminant Analysis dengan metode klasifikasi menggunakan K-Nearest Neigbor. Kedua metode ini diuji menggunakan metode cross validation. Hasil dari penelitian terdahulu terbukti bahwa sistem pengenalan wajah metode Two Dimensial Principal Component Analysis dengan 5-folds cross validation menghasilkan akurasi sebesar 88,73%, sedangkan dengan 2-folds validation akurasi yang dihasilkan sebesar 89,25%. Dan pengujian metode Kernel Fisher Discriminant dengan 2-folds cross validation menghasilkan akurasi rata rata sebesar 83,10%.

Combining Nearest Neighbor Classifiers Versus Cross-Validation Selection

2004 ◽  
Vol 3 (1) ◽  
pp. 1-19 ◽  
Author(s):  
Minhui Paik ◽  
Yuhong Yang
Keyword(s):  
Gene Expression ◽  
Cross Validation ◽  
Nearest Neighbor ◽  
Expression Profiles ◽  
Gene Expression Profiles ◽  
Data Sets ◽  
Weighting Method ◽  
Considerable Uncertainty ◽  
Combined Classifier ◽  
Nearest Neighbor Classifiers

Various discriminant methods have been applied for classification of tumors based on gene expression profiles, among which the nearest neighbor (NN) method has been reported to perform relatively well. Usually cross-validation (CV) is used to select the neighbor size as well as the number of variables for the NN method. However, CV can perform poorly when there is considerable uncertainty in choosing the best candidate classifier. As an alternative to selecting a single “winner," we propose a weighting method to combine the multiple NN rules. Four gene expression data sets are used to compare its performance with CV methods. The results show that when the CV selection is unstable, the combined classifier performs much better.

Sign in / Sign up

Export Citation Format

Share Document