Measuring the VC-Dimension Using Optimized Experimental Design

2000 ◽  
Vol 12 (8) ◽  
pp. 1969-1986 ◽  
Author(s):  
Xuhui Shao ◽  
Vladimir Cherkassky ◽  
William Li
Keyword(s):  
Sample Size ◽  
Uniform Design ◽  
Model Complexity ◽  
Accurate Estimation ◽  
Theoretical Formula ◽  
Vc Dimension ◽  
Experimental Procedure ◽  
Generalization Bounds ◽  
Complexity Control ◽  
Design Structure

VC-dimension is the measure of model complexity (capacity) used in VC-theory. The knowledge of the VC-dimension of an estimator is necessary for rigorous complexity control using analytic VC generalization bounds. Unfortunately, it is not possible to obtain the analytic estimates of the VC-dimension in most cases. Hence, a recent proposal is to measure the VC-dimension of an estimator experimentally by fitting the theoretical formula to a set of experimental measurements of the frequency of errors on artificially generated data sets of varying sizes (Vapnik, Levin, & Le Cun, 1994). However, it may be difficult to obtain an accurate estimate of the VC-dimension due to the variability of random samples in the experimental procedure proposed by Vapnik et al. (1994). We address this problem by proposing an improved design procedure for specifying the measurement points (i.e., the sample size and the number of repeated experiments at a given sample size). Our approach leads to a nonuniform design structure as opposed to the uniform design structure used in the original article (Vapnik et al., 1994). Our simulation results show that the proposed optimized design structure leads to a more accurate estimation of the VC-dimension using the experimental procedure. The results also show that a more accurate estimation of VC-dimension leads to improved complexity control using analytic VC-generalization bounds and, hence, better prediction accuracy.

Model complexity control for regression using VC generalization bounds

1999 ◽  
Vol 10 (5) ◽  
pp. 1075-1089 ◽  
Author(s):  
V. Cherkassky ◽  
Xuhui Shao ◽  
F.M. Mulier ◽  
V.N. Vapnik
Keyword(s):  
Model Complexity ◽  
Generalization Bounds ◽  
Complexity Control

scholarly journals Using Bootstrap Method to Evaluate the Power of Tests for Non-Linearity in Asymmetric Price Relationship

2013 ◽  
Vol 5 (4) ◽  
pp. 237-241
Author(s):  
Henry De-Graft Acquah
Keyword(s):  
Sample Size ◽  
Bootstrap Method ◽  
Complex Model ◽  
Model Complexity ◽  
Bootstrap Sample ◽  
Power Of The Test ◽  
Data Generating Process ◽  
Price Relationship ◽  
False Null Hypothesis ◽  
The Bootstrap Method

This paper introduces and applies the bootstrap method to compare the power of the test for asymmetry in the Granger and Lee (1989) and Von Cramon-Taubadel and Loy (1996) models. The results of the bootstrap simulations indicate that the power of the test for asymmetry depends on various conditions such as the bootstrap sample size, model complexity, difference in adjustment speeds and the amount of noise in the data generating process used in the application. The true model achieves greater power when compared with the complex model. With small bootstrap sample size or large noise, both models display low power in rejecting the (false) null hypothesis of symmetry.

Modelling the patient accrual with truncated Gaussian mixture distribution for the accurate estimation of sample size in survival trials

2020 ◽  
Vol 29 (10) ◽  
pp. 2972-2987
Author(s):  
Haixia Hu ◽  
Ling Wang ◽  
Chen Li ◽  
Wei Ge ◽  
Kejian Wu ◽  
...  
Keyword(s):  
Sample Size ◽  
Mixture Distribution ◽  
Gaussian Mixture ◽  
Accurate Estimation ◽  
Sample Size Estimation ◽  
Size Estimation ◽  
Accrual Rate ◽  
Truncated Gaussian ◽  
Patient Accrual ◽  
Gaussian Mixture Distribution

In survival trials with fixed trial length, the patient accrual rate has a significant impact on the sample size estimation or equivalently, on the power of trials. A larger sample size is required for the staggered patient entry. During enrollment, the patient accrual rate changes with the recruitment publicity effect, disease incidence and many other factors and fluctuations of the accrual rate occur frequently. However, the existing accrual models are either over-simplified for the constant rate assumption or complicated in calculation for the subdivision of the accrual period. A more flexible accrual model is required to represent the fluctuant patient accrual rate for accurate sample size estimation. In this paper, inspired by the flexibility of the Gaussian mixture distribution in approximating continuous densities, we propose the truncated Gaussian mixture distribution accrual model to represent different variations of accrual rate by different parameter configurations. The sample size calculation formula and the parameter setting of the proposed accrual model are discussed further.

scholarly journals Evaluating Model Fit in Bayesian Confirmatory Factor Analysis With Large Samples: Simulation Study Introducing the BRMSEA

2017 ◽  
Vol 78 (4) ◽  
pp. 537-568 ◽  
Author(s):  
Huub Hoofs ◽  
Rens van de Schoot ◽  
Nicole W. H. Jansen ◽  
IJmert Kant
Keyword(s):  
Factor Analysis ◽  
Confirmatory Factor Analysis ◽  
Sample Size ◽  
Simulation Study ◽  
Reliability And Validity ◽  
Model Fit ◽  
Model Complexity ◽  
Large Sample ◽  
Large Samples ◽  
Confirmatory Factor

Bayesian confirmatory factor analysis (CFA) offers an alternative to frequentist CFA based on, for example, maximum likelihood estimation for the assessment of reliability and validity of educational and psychological measures. For increasing sample sizes, however, the applicability of current fit statistics evaluating model fit within Bayesian CFA is limited. We propose, therefore, a Bayesian variant of the root mean square error of approximation (RMSEA), the BRMSEA. A simulation study was performed with variations in model misspecification, factor loading magnitude, number of indicators, number of factors, and sample size. This showed that the 90% posterior probability interval of the BRMSEA is valid for evaluating model fit in large samples ( N≥ 1,000), using cutoff values for the lower (<.05) and upper limit (<.08) as guideline. An empirical illustration further shows the advantage of the BRMSEA in large sample Bayesian CFA models. In conclusion, it can be stated that the BRMSEA is well suited to evaluate model fit in large sample Bayesian CFA models by taking sample size and model complexity into account.

scholarly journals Inferential Item-Fit Evaluation in Cognitive Diagnosis Modeling

2017 ◽  
Vol 41 (8) ◽  
pp. 614-631 ◽  
Author(s):  
Miguel A. Sorrel ◽  
Francisco J. Abad ◽  
Julio Olea ◽  
Jimmy de la Torre ◽  
Juan Ramón Barrada
Keyword(s):  
Sample Size ◽  
Goodness Of Fit ◽  
Type I Error ◽  
Cognitive Diagnosis ◽  
Model Complexity ◽  
Type I ◽  
Test Length ◽  
Item Fit ◽  
Lm Test ◽  
Item Quality

Research related to the fit evaluation at the item level involving cognitive diagnosis models (CDMs) has been scarce. According to the parsimony principle, balancing goodness of fit against model complexity is necessary. General CDMs require a larger sample size to be estimated reliably, and can lead to worse attribute classification accuracy than the appropriate reduced models when the sample size is small and the item quality is poor, which is typically the case in many empirical applications. The main purpose of this study was to systematically examine the statistical properties of four inferential item-fit statistics: [Formula: see text], the likelihood ratio (LR) test, the Wald (W) test, and the Lagrange multiplier (LM) test. To evaluate the performance of the statistics, a comprehensive set of factors, namely, sample size, correlational structure, test length, item quality, and generating model, is systematically manipulated using Monte Carlo methods. Results show that the [Formula: see text] statistic has unacceptable power. Type I error and power comparisons favor LR and W tests over the LM test. However, all the statistics are highly affected by the item quality. With a few exceptions, their performance is only acceptable when the item quality is high. In some cases, this effect can be ameliorated by an increase in sample size and test length. This implies that using the above statistics to assess item fit in practical settings when the item quality is low remains a challenge.

Sign in / Sign up

Export Citation Format

Share Document