scholarly journals Using Model Selection Criteria to Choose the Number of Principal Components

2021 ◽  
Vol 20 (3) ◽  
pp. 450-461
Author(s):  
Stanley L. Sclove
Keyword(s):  
Model Selection ◽  
Principal Components ◽  
Bayesian Information Criterion ◽  
Selection Criteria ◽  
Information Criterion ◽  
Information Criteria ◽  
Akaike's Information Criterion ◽  
Model Selection Criteria ◽  
Adequate Number ◽  
Number Of Principal Components

AbstractThe use of information criteria, especially AIC (Akaike’s information criterion) and BIC (Bayesian information criterion), for choosing an adequate number of principal components is illustrated.

Application of statistical model selection criteria to the Stock Synthesis assessment program

2000 ◽  
Vol 57 (9) ◽  
pp. 1784-1793 ◽  
Author(s):  
S Langitoto Helu ◽  
David B Sampson ◽  
Yanshui Yin
Keyword(s):  
Model Selection ◽  
Akaike Information Criterion ◽  
Bayesian Information Criterion ◽  
Selection Criteria ◽  
Information Criterion ◽  
Model Selection Criteria ◽  
Assessment Program ◽  
Correct Model ◽  
Complex Models ◽  
Multiple Data Sets

Statistical modeling involves building sufficiently complex models to represent the system being investigated. Overly complex models lead to imprecise parameter estimates, increase the subjective role of the modeler, and can distort the perceived characteristics of the system under investigation. One approach for controlling the tendency to increased complexity and subjectivity is to use model selection criteria that account for these factors. The effectiveness of two selection criteria was tested in an application with the stock assessment program known as Stock Synthesis. This program, which is often used on the U.S. west coast to assess the status of exploited marine fish stocks, can handle multiple data sets and mimic highly complex population dynamics. The Akaike information criterion and Schwarz's Bayesian information criterion are criteria that satisfy the fundamental principles of model selection: goodness-of-fit, parsimony, and objectivity. Their ability to select the correct model form and produce accurate estimates was evaluated in Monte Carlo experiments with the Stock Synthesis program. In general, the Akaike information criterion and the Bayesian information criterion had similar performance in selecting the correct model, and they produced comparable levels of accuracy in their estimates of ending stock biomass.

scholarly journals Model Selection Procedures in Bounds Test of Cointegration: Theoretical Comparison and Empirical Evidence

Economies ◽  
2020 ◽  
Vol 8 (2) ◽  
pp. 49 ◽  
Author(s):  
Waqar Badshah ◽  
Mehmet Bulut
Keyword(s):  
Model Selection ◽  
Akaike Information Criterion ◽  
Bayesian Information Criterion ◽  
Selection Process ◽  
Information Criterion ◽  
Small Sample ◽  
Information Criteria ◽  
Path Model ◽  
Sample Sizes ◽  
Bounds Test

Only unstructured single-path model selection techniques, i.e., Information Criteria, are used by Bounds test of cointegration for model selection. The aim of this paper was twofold; one was to evaluate the performance of these five routinely used information criteria {Akaike Information Criterion (AIC), Akaike Information Criterion Corrected (AICC), Schwarz/Bayesian Information Criterion (SIC/BIC), Schwarz/Bayesian Information Criterion Corrected (SICC/BICC), and Hannan and Quinn Information Criterion (HQC)} and three structured approaches (Forward Selection, Backward Elimination, and Stepwise) by assessing their size and power properties at different sample sizes based on Monte Carlo simulations, and second was the assessment of the same based on real economic data. The second aim was achieved by the evaluation of the long-run relationship between three pairs of macroeconomic variables, i.e., Energy Consumption and GDP, Oil Price and GDP, and Broad Money and GDP for BRICS (Brazil, Russia, India, China and South Africa) countries using Bounds cointegration test. It was found that information criteria and structured procedures have the same powers for a sample size of 50 or greater. However, BICC and Stepwise are better at small sample sizes. In the light of simulation and real data results, a modified Bounds test with Stepwise model selection procedure may be used as it is strongly theoretically supported and avoids noise in the model selection process.

A Comparative Analysis and Evaluation of Model Selection Criteria

2017 ◽  
Author(s):  
Ahmed H. Kamel ◽  
Ali S. Shaqlaih ◽  
Arslan Rozyyev
Keyword(s):  
Experimental Data ◽  
Model Selection ◽  
Selection Criteria ◽  
Oil And Gas ◽  
Information Criterion ◽  
Oil And Gas Industry ◽  
Absolute Error ◽  
Ongoing Research ◽  
Model Selection Criteria ◽  
Innovative Strategy

The ongoing research for model choice and selection has generated a plethora of approaches. With such a wealth of methods, it can be difficult for a researcher to know what model selection approach is the proper way to proceed to select the appropriate model for prediction. The authors present an evaluation of various model selection criteria from decision-theoretic perspective using experimental data to define and recommend a criterion to select the best model. In this analysis, six of the most common selection criteria, nineteen friction factor correlations, and eight sets of experimental data are employed. The results show that while the use of the traditional correlation coefficient, R2 is inappropriate, root mean square error, RMSE can be used to rank models, but does not give much insight on their accuracy. Other criteria such as correlation ratio, mean absolute error, and standard deviation are also evaluated. The Akaike information criterion, AIC has shown its superiority to other selection criteria. The authors propose AIC as an alternative to use when fitting experimental data or evaluating existing correlations. Indeed, the AIC method is an information theory based, theoretically sound and stable. The paper presents a detailed discussion of the model selection criteria, their pros and cons, and how they can be utilized to allow proper comparison of different models for the best model to be inferred based on sound mathematical theory. In conclusion, model selection is an interesting problem and an innovative strategy to help alleviate similar challenges faced by the professionals in the oil and gas industry is introduced.

scholarly journals Model Selection Criteria on Beta Regression for Machine Learning

2019 ◽  
Vol 1 (1) ◽  
pp. 427-449
Author(s):  
Patrícia Espinheira ◽  
Luana da Silva ◽  
Alisson Silva ◽  
Raydonal Ospina
Keyword(s):  
Machine Learning ◽  
Model Selection ◽  
Selection Criteria ◽  
Real Data ◽  
Estimation Procedure ◽  
Computational Procedure ◽  
Information Criteria ◽  
Beta Regression ◽  
Finite Sample ◽  
Model Selection Criteria

Beta regression models are a class of supervised learning tools for regression problems with univariate and limited response. Current fitting procedures for beta regression require variable selection based on (potentially problematic) information criteria. We propose model selection criteria that take into account the leverage, residuals, and influence of the observations, both to systematic linear and nonlinear components. To that end, we propose a Predictive Residual Sum of Squares (PRESS)-like machine learning tool and a prediction coefficient, namely P 2 statistic, as a computational procedure. Monte Carlo simulation results on the finite sample behavior of prediction-based model selection criteria P 2 are provided. We also evaluated two versions of the R 2 criterion. Finally, applications to real data are presented. The new criterion proved to be crucial to choose models taking into account the robustness of the maximum likelihood estimation procedure in the presence of influential cases.

scholarly journals Improved Treatment of the Independent Variables for the Deployment of Model Selection Criteria in the Analysis of Complex Systems

Entropy ◽  
2021 ◽  
Vol 23 (9) ◽  
pp. 1202
Author(s):  
Luca Spolladore ◽  
Michela Gelfusa ◽  
Riccardo Rossi ◽  
Andrea Murari
Keyword(s):  
Model Selection ◽  
Complex Systems ◽  
Selection Criteria ◽  
A Priori ◽  
Synthetic Data ◽  
Information Criterion ◽  
Model Selection Criteria ◽  
Dependent Variables ◽  
Chi Squared ◽  
Highly Correlated

Model selection criteria are widely used to identify the model that best represents the data among a set of potential candidates. Amidst the different model selection criteria, the Bayesian information criterion (BIC) and the Akaike information criterion (AIC) are the most popular and better understood. In the derivation of these indicators, it was assumed that the model’s dependent variables have already been properly identified and that the entries are not affected by significant uncertainties. These are issues that can become quite serious when investigating complex systems, especially when variables are highly correlated and the measurement uncertainties associated with them are not negligible. More sophisticated versions of this criteria, capable of better detecting spurious relations between variables when non-negligible noise is present, are proposed in this paper. Their derivation is obtained starting from a Bayesian statistics framework and adding an a priori Chi-squared probability distribution function of the model, dependent on a specifically defined information theoretic quantity that takes into account the redundancy between the dependent variables. The performances of the proposed versions of these criteria are assessed through a series of systematic simulations, using synthetic data for various classes of functions and noise levels. The results show that the upgraded formulation of the criteria clearly outperforms the traditional ones in most of the cases reported.

scholarly journals On the Use of Entropy to Improve Model Selection Criteria

Entropy ◽  
2019 ◽  
Vol 21 (4) ◽  
pp. 394 ◽  
Author(s):  
Andrea Murari ◽  
Emmanuele Peluso ◽  
Francesco Cianfrani ◽  
Pasquale Gaudio ◽  
Michele Lungaroni
Keyword(s):  
Model Selection ◽  
Shannon Entropy ◽  
Selection Criteria ◽  
Mean Squared Error ◽  
Geodesic Distance ◽  
Information Criterion ◽  
Residual Distribution ◽  
Model Selection Criteria ◽  
Synthetic Indicators ◽  
Squared Error

The most widely used forms of model selection criteria, the Bayesian Information Criterion (BIC) and the Akaike Information Criterion (AIC), are expressed in terms of synthetic indicators of the residual distribution: the variance and the mean-squared error of the residuals respectively. In many applications in science, the noise affecting the data can be expected to have a Gaussian distribution. Therefore, at the same level of variance and mean-squared error, models, whose residuals are more uniformly distributed, should be favoured. The degree of uniformity of the residuals can be quantified by the Shannon entropy. Including the Shannon entropy in the BIC and AIC expressions improves significantly these criteria. The better performances have been demonstrated empirically with a series of simulations for various classes of functions and for different levels and statistics of the noise. In presence of outliers, a better treatment of the errors, using the Geodesic Distance, has proved essential.

scholarly journals Sensitivity and specificity of information criteria

2015 ◽  
Author(s):  
John J. Dziak ◽  
Donna L. Coffman ◽  
Stephanie T. Lanza ◽  
Runze Li
Keyword(s):  
Model Selection ◽  
Sample Size ◽  
Likelihood Ratio Test ◽  
Bayesian Information Criterion ◽  
Information Criterion ◽  
Information Criteria ◽  
Ratio Test ◽  
Relative Importance ◽  
Missed Opportunities ◽  
High Bias

Choosing a model with too few parameters can involve making unrealistically simple assumptions and lead to high bias, poor prediction, and missed opportunities for insight. Such models are not flexible enough to describe the sample or the population well. A model with too many parameters can t the observed data very well, but be too closely tailored to it. Such models may generalize poorly. Penalized-likelihood information criteria, such as Akaike's Information Criterion (AIC), the Bayesian Information Criterion (BIC), the Consistent AIC, and the Adjusted BIC, are widely used for model selection. However, different criteria sometimes support different models, leading to uncertainty about which criterion is the most trustworthy. In some simple cases the comparison of two models using information criteria can be viewed as equivalent to a likelihood ratio test, with the different models representing different alpha levels (i.e., different emphases on sensitivity or specificity; Lin & Dayton 1997). This perspective may lead to insights about how to interpret the criteria in less simple situations. For example, AIC or BIC could be preferable, depending on sample size and on the relative importance one assigns to sensitivity versus specificity. Understanding the differences among the criteria may make it easier to compare their results and to use them to make informed decisions.

scholarly journals Latent profile analysis of human values: What is the optimal number of clusters?

Methodology ◽  
2021 ◽  
Vol 17 (2) ◽  
pp. 127-148
Author(s):  
Mikkel N. Schmidt ◽  
Daniel Seddig ◽  
Eldad Davidov ◽  
Morten Mørup ◽  
Kristoffer Jon Albers ◽  
...  
Keyword(s):  
Model Selection ◽  
Selection Criteria ◽  
Latent Profile Analysis ◽  
Profile Analysis ◽  
Information Criterion ◽  
Optimal Number ◽  
Number Of Clusters ◽  
Model Selection Criteria ◽  
Optimal Number Of Clusters ◽  
Latent Profile

Latent Profile Analysis (LPA) is a method to extract homogeneous clusters characterized by a common response profile. Previous works employing LPA to human value segmentation tend to select a small number of moderately homogeneous clusters based on model selection criteria such as Akaike information criterion, Bayesian information criterion and Entropy. The question is whether a small number of clusters is all that can be gleaned from the data. While some studies have carefully compared different statistical model selection criteria, there is currently no established criteria to assess if an increased number of clusters generates meaningful theoretical insights. This article examines the content and meaningfulness of the clusters extracted using two algorithms: Variational Bayesian LPA and Maximum Likelihood LPA. For both methods, our results point towards eight as the optimal number of clusters for characterizing distinctive Schwartz value typologies that generate meaningful insights and predict several external variables.

scholarly journals Performance comparison of model selection criteria by generated experimental data

2018 ◽  
Vol 16 ◽  
pp. 02006
Author(s):  
Radoslav Mavrevski ◽  
Peter Milanov ◽  
Metodi Traykov ◽  
Nevena Pencheva
Keyword(s):  
Experimental Data ◽  
Model Selection ◽  
Mathematical Models ◽  
Bayesian Information Criterion ◽  
Selection Criteria ◽  
Information Criterion ◽  
Data Fitting ◽  
Performance Comparison ◽  
Artificial Data ◽  
Criteria For Evaluation

In Bioinformatics and other areas the model selection is a process of choosing a model from set of candidate models of different classes which will provide the best balance between goodness of fitting of the data and complexity of the model. There are many criteria for evaluation of mathematical models for data fitting. The main objectives of this study are: (1) to fitting artificial experimental data with different models with increasing complexity; (2) to test whether two known criteria as Akaike’s information criterion (AIC) and Bayesian information criterion (BIC) can correctly identify the model, used to generate the artificial data and (3) to assess and compare empirically the performance of AIC and BIC.

scholarly journals Sensitivity and specificity of information criteria

2017 ◽  
Author(s):  
John J. Dziak ◽  
Donna L. Coffman ◽  
Stephanie T. Lanza ◽  
Runze Li
Keyword(s):  
Model Selection ◽  
Sample Size ◽  
Likelihood Ratio Test ◽  
Bayesian Information Criterion ◽  
Information Criterion ◽  
Information Criteria ◽  
Ratio Test ◽  
Relative Importance ◽  
Missed Opportunities ◽  
High Bias

Choosing a model with too few parameters can involve making unrealistically simple assumptions and lead to high bias, poor prediction, and missed opportunities for insight. Such models are not flexible enough to describe the sample or the population well. A model with too many parameters can fit the observed data very well, but be too closely tailored to it. Such models may generalize poorly. Penalized-likelihood information criteria, such as Akaike's Information Criterion (AIC), the Bayesian Information Criterion (BIC), the Consistent AIC, and the Adjusted BIC, are widely used for model selection. However, different criteria sometimes support different models, leading to uncertainty about which criterion is the most trustworthy. In some simple cases the comparison of two models using information criteria can be viewed as equivalent to a likelihood ratio test, with the different models representing different alpha levels (i.e., different emphases on sensitivity or specificity; Lin & Dayton 1997). This perspective may lead to insights about how to interpret the criteria in less simple situations. For example, AIC or BIC could be preferable, depending on sample size and on the relative importance one assigns to sensitivity versus specificity. Understanding the differences among the criteria may make it easier to compare their results and to use them to make informed decisions.

Sign in / Sign up

Export Citation Format

Share Document