scholarly journals Model Selection for Multilevel Mixture Rasch Models

2018 ◽  
Vol 43 (4) ◽  
pp. 272-289 ◽  
Author(s):  
Sedat Sen ◽  
Allan S. Cohen ◽  
Seock-Ho Kim
Keyword(s):  
Model Selection ◽  
Sample Size ◽  
Simulation Study ◽  
Akaike Information Criterion ◽  
Bayesian Information Criterion ◽  
Information Criterion ◽  
Total Sample ◽  
Size Number ◽  
Information Indices ◽  
Level 2

Mixture item response theory (MixIRT) models can sometimes be used to model the heterogeneity among the individuals from different subpopulations, but these models do not account for the multilevel structure that is common in educational and psychological data. Multilevel extensions of the MixIRT models have been proposed to address this shortcoming. Successful applications of multilevel MixIRT models depend in part on detection of the best fitting model. In this study, performance of information indices, Akaike information criterion (AIC), Bayesian information criterion (BIC), consistent Akaike information criterion (CAIC), and sample-size adjusted Bayesian information criterion (SABIC), were compared for use in model selection with a two-level mixture Rasch model in the context of a real data example and a simulation study. Level 1 consisted of students and Level 2 consisted of schools. The performances of the model selection criteria under different sample sizes were investigated in a simulation study. Total sample size (number of students) and Level 2 sample size (number of schools) were studied for calculation of information criterion indices to examine the performance of these fit indices. Simulation study results indicated that CAIC and BIC performed better than the other indices at detection of the true (i.e., generating) model. Furthermore, information indices based on total sample size yielded more accurate detections than indices at Level 2.

scholarly journals Model Selection Procedures in Bounds Test of Cointegration: Theoretical Comparison and Empirical Evidence

Economies ◽  
2020 ◽  
Vol 8 (2) ◽  
pp. 49 ◽  
Author(s):  
Waqar Badshah ◽  
Mehmet Bulut
Keyword(s):  
Model Selection ◽  
Akaike Information Criterion ◽  
Bayesian Information Criterion ◽  
Selection Process ◽  
Information Criterion ◽  
Small Sample ◽  
Information Criteria ◽  
Path Model ◽  
Sample Sizes ◽  
Bounds Test

Only unstructured single-path model selection techniques, i.e., Information Criteria, are used by Bounds test of cointegration for model selection. The aim of this paper was twofold; one was to evaluate the performance of these five routinely used information criteria {Akaike Information Criterion (AIC), Akaike Information Criterion Corrected (AICC), Schwarz/Bayesian Information Criterion (SIC/BIC), Schwarz/Bayesian Information Criterion Corrected (SICC/BICC), and Hannan and Quinn Information Criterion (HQC)} and three structured approaches (Forward Selection, Backward Elimination, and Stepwise) by assessing their size and power properties at different sample sizes based on Monte Carlo simulations, and second was the assessment of the same based on real economic data. The second aim was achieved by the evaluation of the long-run relationship between three pairs of macroeconomic variables, i.e., Energy Consumption and GDP, Oil Price and GDP, and Broad Money and GDP for BRICS (Brazil, Russia, India, China and South Africa) countries using Bounds cointegration test. It was found that information criteria and structured procedures have the same powers for a sample size of 50 or greater. However, BICC and Stepwise are better at small sample sizes. In the light of simulation and real data results, a modified Bounds test with Stepwise model selection procedure may be used as it is strongly theoretically supported and avoids noise in the model selection process.

scholarly journals Sensitivity and specificity of information criteria

2015 ◽  
Author(s):  
John J. Dziak ◽  
Donna L. Coffman ◽  
Stephanie T. Lanza ◽  
Runze Li
Keyword(s):  
Model Selection ◽  
Sample Size ◽  
Likelihood Ratio Test ◽  
Bayesian Information Criterion ◽  
Information Criterion ◽  
Information Criteria ◽  
Ratio Test ◽  
Relative Importance ◽  
Missed Opportunities ◽  
High Bias

Choosing a model with too few parameters can involve making unrealistically simple assumptions and lead to high bias, poor prediction, and missed opportunities for insight. Such models are not flexible enough to describe the sample or the population well. A model with too many parameters can t the observed data very well, but be too closely tailored to it. Such models may generalize poorly. Penalized-likelihood information criteria, such as Akaike's Information Criterion (AIC), the Bayesian Information Criterion (BIC), the Consistent AIC, and the Adjusted BIC, are widely used for model selection. However, different criteria sometimes support different models, leading to uncertainty about which criterion is the most trustworthy. In some simple cases the comparison of two models using information criteria can be viewed as equivalent to a likelihood ratio test, with the different models representing different alpha levels (i.e., different emphases on sensitivity or specificity; Lin & Dayton 1997). This perspective may lead to insights about how to interpret the criteria in less simple situations. For example, AIC or BIC could be preferable, depending on sample size and on the relative importance one assigns to sensitivity versus specificity. Understanding the differences among the criteria may make it easier to compare their results and to use them to make informed decisions.

Application of statistical model selection criteria to the Stock Synthesis assessment program

2000 ◽  
Vol 57 (9) ◽  
pp. 1784-1793 ◽  
Author(s):  
S Langitoto Helu ◽  
David B Sampson ◽  
Yanshui Yin
Keyword(s):  
Model Selection ◽  
Akaike Information Criterion ◽  
Bayesian Information Criterion ◽  
Selection Criteria ◽  
Information Criterion ◽  
Model Selection Criteria ◽  
Assessment Program ◽  
Correct Model ◽  
Complex Models ◽  
Multiple Data Sets

Statistical modeling involves building sufficiently complex models to represent the system being investigated. Overly complex models lead to imprecise parameter estimates, increase the subjective role of the modeler, and can distort the perceived characteristics of the system under investigation. One approach for controlling the tendency to increased complexity and subjectivity is to use model selection criteria that account for these factors. The effectiveness of two selection criteria was tested in an application with the stock assessment program known as Stock Synthesis. This program, which is often used on the U.S. west coast to assess the status of exploited marine fish stocks, can handle multiple data sets and mimic highly complex population dynamics. The Akaike information criterion and Schwarz's Bayesian information criterion are criteria that satisfy the fundamental principles of model selection: goodness-of-fit, parsimony, and objectivity. Their ability to select the correct model form and produce accurate estimates was evaluated in Monte Carlo experiments with the Stock Synthesis program. In general, the Akaike information criterion and the Bayesian information criterion had similar performance in selecting the correct model, and they produced comparable levels of accuracy in their estimates of ending stock biomass.

scholarly journals Sensitivity and specificity of information criteria

2017 ◽  
Author(s):  
John J. Dziak ◽  
Donna L. Coffman ◽  
Stephanie T. Lanza ◽  
Runze Li
Keyword(s):  
Model Selection ◽  
Sample Size ◽  
Likelihood Ratio Test ◽  
Bayesian Information Criterion ◽  
Information Criterion ◽  
Information Criteria ◽  
Ratio Test ◽  
Relative Importance ◽  
Missed Opportunities ◽  
High Bias

Choosing a model with too few parameters can involve making unrealistically simple assumptions and lead to high bias, poor prediction, and missed opportunities for insight. Such models are not flexible enough to describe the sample or the population well. A model with too many parameters can fit the observed data very well, but be too closely tailored to it. Such models may generalize poorly. Penalized-likelihood information criteria, such as Akaike's Information Criterion (AIC), the Bayesian Information Criterion (BIC), the Consistent AIC, and the Adjusted BIC, are widely used for model selection. However, different criteria sometimes support different models, leading to uncertainty about which criterion is the most trustworthy. In some simple cases the comparison of two models using information criteria can be viewed as equivalent to a likelihood ratio test, with the different models representing different alpha levels (i.e., different emphases on sensitivity or specificity; Lin & Dayton 1997). This perspective may lead to insights about how to interpret the criteria in less simple situations. For example, AIC or BIC could be preferable, depending on sample size and on the relative importance one assigns to sensitivity versus specificity. Understanding the differences among the criteria may make it easier to compare their results and to use them to make informed decisions.

scholarly journals Sensitivity and specificity of information criteria

2017 ◽  
Author(s):  
John J. Dziak ◽  
Donna L. Coffman ◽  
Stephanie T. Lanza ◽  
Runze Li
Keyword(s):  
Model Selection ◽  
Sample Size ◽  
Likelihood Ratio Test ◽  
Bayesian Information Criterion ◽  
Information Criterion ◽  
Information Criteria ◽  
Ratio Test ◽  
Relative Importance ◽  
Missed Opportunities ◽  
High Bias

Choosing a model with too few parameters can involve making unrealistically simple assumptions and lead to high bias, poor prediction, and missed opportunities for insight. Such models are not flexible enough to describe the sample or the population well. A model with too many parameters can fit the observed data very well, but be too closely tailored to it. Such models may generalize poorly. Penalized-likelihood information criteria, such as Akaike's Information Criterion (AIC), the Bayesian Information Criterion (BIC), the Consistent AIC, and the Adjusted BIC, are widely used for model selection. However, different criteria sometimes support different models, leading to uncertainty about which criterion is the most trustworthy. In some simple cases the comparison of two models using information criteria can be viewed as equivalent to a likelihood ratio test, with the different models representing different alpha levels (i.e., different emphases on sensitivity or specificity; Lin & Dayton 1997). This perspective may lead to insights about how to interpret the criteria in less simple situations. For example, AIC or BIC could be preferable, depending on sample size and on the relative importance one assigns to sensitivity versus specificity. Understanding the differences among the criteria may make it easier to compare their results and to use them to make informed decisions.

scholarly journals Sensitivity and specificity of information criteria

2015 ◽  
Author(s):  
John J. Dziak ◽  
Donna L. Coffman ◽  
Stephanie T. Lanza ◽  
Runze Li
Keyword(s):  
Model Selection ◽  
Sample Size ◽  
Likelihood Ratio Test ◽  
Bayesian Information Criterion ◽  
Information Criterion ◽  
Information Criteria ◽  
Ratio Test ◽  
Relative Importance ◽  
Missed Opportunities ◽  
High Bias

Choosing a model with too few parameters can involve making unrealistically simple assumptions and lead to high bias, poor prediction, and missed opportunities for insight. Such models are not flexible enough to describe the sample or the population well. A model with too many parameters can fit the observed data very well, but be too closely tailored to it. Such models may generalize poorly. Penalized-likelihood information criteria, such as Akaike's Information Criterion (AIC), the Bayesian Information Criterion (BIC), the Consistent AIC, and the Adjusted BIC, are widely used for model selection. However, different criteria sometimes support different models, leading to uncertainty about which criterion is the most trustworthy. In some simple cases the comparison of two models using information criteria can be viewed as equivalent to a likelihood ratio test, with the different models representing different alpha levels (i.e., different emphases on sensitivity or specificity; Lin & Dayton 1997). This perspective may lead to insights about how to interpret the criteria in less simple situations. For example, AIC or BIC could be preferable, depending on sample size and on the relative importance one assigns to sensitivity versus specificity. Understanding the differences among the criteria may make it easier to compare their results and to use them to make informed decisions.

scholarly journals Model Selection in Continuous Test Norming With GAMLSS

Assessment ◽  
2017 ◽  
Vol 26 (7) ◽  
pp. 1329-1346 ◽  
Author(s):  
Lieke Voncken ◽  
Casper J. Albers ◽  
Marieke E. Timmerman
Keyword(s):  
Model Selection ◽  
Sample Size ◽  
Akaike Information Criterion ◽  
Reference Group ◽  
Generalized Additive Models ◽  
Selection Procedure ◽  
Information Criterion ◽  
Exponential Model ◽  
Additive Models ◽  
Model Selection Procedure

To compute norms from reference group test scores, continuous norming is preferred over traditional norming. A suitable continuous norming approach for continuous data is the use of the Box–Cox Power Exponential model, which is found in the generalized additive models for location, scale, and shape. Applying the Box–Cox Power Exponential model for test norming requires model selection, but it is unknown how well this can be done with an automatic selection procedure. In a simulation study, we compared the performance of two stepwise model selection procedures combined with four model-fit criteria (Akaike information criterion, Bayesian information criterion, generalized Akaike information criterion (3), cross-validation), varying data complexity, sampling design, and sample size in a fully crossed design. The new procedure combined with one of the generalized Akaike information criterion was the most efficient model selection procedure (i.e., required the smallest sample size). The advocated model selection procedure is illustrated with norming data of an intelligence test.

scholarly journals Using Model Selection Criteria to Choose the Number of Principal Components

2021 ◽  
Vol 20 (3) ◽  
pp. 450-461
Author(s):  
Stanley L. Sclove
Keyword(s):  
Model Selection ◽  
Principal Components ◽  
Bayesian Information Criterion ◽  
Selection Criteria ◽  
Information Criterion ◽  
Information Criteria ◽  
Akaike's Information Criterion ◽  
Model Selection Criteria ◽  
Adequate Number ◽  
Number Of Principal Components

AbstractThe use of information criteria, especially AIC (Akaike’s information criterion) and BIC (Bayesian information criterion), for choosing an adequate number of principal components is illustrated.

Sign in / Sign up

Export Citation Format

Share Document