scholarly journals Rejoinder: More Limitations of Bayesian Leave-One-Out Cross-Validation

2018 ◽  
Author(s):  
Quentin Frederik Gronau ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Parameter Estimation ◽  
Model Selection ◽  
Model Comparison ◽  
Cross Validation ◽  
Mathematical Psychology ◽  
Bayes Rule ◽  
Epistemic Goal ◽  
Leave One Out

We recently discussed several limitations of Bayesian leave-one-out cross-validation (LOO) for model selection. Our contribution attracted three thought-provoking commentaries. In this rejoinder, we address each of the commentaries and identify several additional limitations of LOO-based methods such as Bayesian stacking. We focus on differences between LOO-based methods versus approaches that consistently use Bayes' rule for both parameter estimation and model comparison. We conclude that LOO-based methods do not align satisfactorily with the epistemic goal of mathematical psychology.

Fast leave-one-out methods for inference, model selection, and diagnostic checking

2020 ◽  
Vol 20 (4) ◽  
pp. 785-804
Author(s):  
Federico Belotti ◽  
Franco Peracchi
Keyword(s):  
Model Selection ◽  
Full Advantage ◽  
Cross Validation ◽  
Computing Time ◽  
Substantial Reduction ◽  
Inference Model ◽  
Diagnostic Measures ◽  
Diagnostic Checking ◽  
Linear Estimators ◽  
Leave One Out

In this article, we describe jackknife2, a new prefix command for jackknifing linear estimators. It takes full advantage of the available leave-one-out formula, thereby allowing for substantial reduction in computing time. Of special note is that jackknife2 allows the user to compute cross-validation and diagnostic measures that are currently not available after ivregress 2sls, xtreg, and xtivregress.

scholarly journals Modify Leave-One-Out Cross Validation by Moving Validation Samples around Random Normal Distributions: Move-One-Away Cross Validation

2020 ◽  
Vol 10 (7) ◽  
pp. 2448
Author(s):  
Liye Lv ◽  
Xueguan Song ◽  
Wei Sun
Keyword(s):  
Model Selection ◽  
Sample Size ◽  
Cross Validation ◽  
Engineering Problem ◽  
Coefficient Of Determination ◽  
Accuracy Rate ◽  
Mathematical Functions ◽  
Normal Distributions ◽  
Model Independent ◽  
Leave One Out

The leave-one-out cross validation (LOO-CV), which is a model-independent evaluate method, cannot always select the best of several models when the sample size is small. We modify the LOO-CV method by moving a validation point around random normal distributions—rather than leaving it out—naming it the move-one-away cross validation (MOA-CV), which is a model-dependent method. The key point of this method is to improve the accuracy rate of model selection that is unreliable in LOO-CV without enough samples. Errors from LOO-CV and MOA-CV, i.e., LOO-CVerror and MOA-CVerror, respectively, are employed to select the best one of four typical surrogate models through four standard mathematical functions and one engineering problem. The coefficient of determination (R-square, R2) is used to be a calibration of MOA-CVerror and LOO-CVerror. Results show that: (i) in terms of selecting the best models, MOA-CV and LOO-CV become better as sample size increases; (ii) MOA-CV has a better performance in selecting best models than LOO-CV; (iii) in the engineering problem, both the MOA-CV and LOO-CV can choose the worst models, and in most cases, MOA-CV has a higher probability to select the best model than LOO-CV.

scholarly journals Between the devil and the deep blue sea: Tensions between scientific judgement and statistical model selection

2018 ◽  
Author(s):  
Danielle Navarro
Keyword(s):  
Model Selection ◽  
Cross Validation ◽  
Bayes Factors ◽  
Prediction Problem ◽  
Problem Model ◽  
Deep Blue ◽  
Statistical Model Selection ◽  
Scientific Goal ◽  
Leave One Out ◽  
The Devil

Discussions of model selection in the psychological literature typically frame the issues as a question of statistical inference, with the goal being to determine which model makes the best predictions about data. Within this setting, advocates of leave-one-out cross-validation and Bayes factors disagree on precisely which prediction problem model selection questions should aim to answer. In this comment, I discuss some of these issues from a scientific perspective. What goal does model selection serve when all models are known to be systematically wrong? How might "toy problems" tell a misleading story? How does the scientific goal of explanation align with (or differ from) traditional statistical concerns? I do not offer answers to these questions, but hope to highlight the reasons why psychological researchers cannot avoid asking them.

scholarly journals Limitations of Bayesian Leave-One-Out Cross-Validation for Model Selection

2018 ◽  
Author(s):  
Quentin Frederik Gronau ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Simple Model ◽  
Model Selection ◽  
Cross Validation ◽  
Predictive Performance ◽  
Data Set ◽  
General Law ◽  
Out Of Sample ◽  
Measure Model ◽  
Leave One Out ◽  
And Behavior

Cross-validation (CV) is increasingly popular as a generic method to adjudicate between mathematical models of cognition and behavior. In order to measure model generalizability, CV quantifies out-of-sample predictive performance, and the CV preference goes to the model that predicted the out-of-sample data best. The advantages of CV include theoretic simplicity and practical feasibility. Despite its prominence, however, the limitations of CV are often underappreciated. Here we demonstrate the limitations of a particular form of CV --Bayesian leave-one-out cross-validation or LOO-- with three concrete examples. In each example, a data set of infinite size is perfectly in line with the predictions of a simple model (i.e., a general law or invariance). Nevertheless, LOO shows bounded and relatively modest support for the simple model. We conclude that CV is not a panacea for model selection.

scholarly journals Introduction to Bayesian Inference for Psychology

2017 ◽  
Author(s):  
Alexander Etz ◽  
Joachim Vandekerckhove
Keyword(s):  
Probability Theory ◽  
Parameter Estimation ◽  
Bayesian Inference ◽  
Model Comparison ◽  
Worked Examples ◽  
Bayes Rule ◽  
Special Issue ◽  
Psychonomic Bulletin ◽  
Set Up

We introduce the fundamental tenets of Bayesian inference, which derive from two basic laws of probability theory. We cover the interpretation of probabilities, discrete and continuous versions of Bayes' rule, parameter estimation, and model comparison. Using seven worked examples, we illustrate these principles and set up some of the technical background for the rest of this special issue of Psychonomic Bulletin & Review. Supplemental material is available via https://osf.io/wskex/.

Sign in / Sign up

Export Citation Format

Share Document