scholarly journals Modify Leave-One-Out Cross Validation by Moving Validation Samples around Random Normal Distributions: Move-One-Away Cross Validation

2020 ◽  
Vol 10 (7) ◽  
pp. 2448
Author(s):  
Liye Lv ◽  
Xueguan Song ◽  
Wei Sun

The leave-one-out cross validation (LOO-CV), which is a model-independent evaluate method, cannot always select the best of several models when the sample size is small. We modify the LOO-CV method by moving a validation point around random normal distributions—rather than leaving it out—naming it the move-one-away cross validation (MOA-CV), which is a model-dependent method. The key point of this method is to improve the accuracy rate of model selection that is unreliable in LOO-CV without enough samples. Errors from LOO-CV and MOA-CV, i.e., LOO-CVerror and MOA-CVerror, respectively, are employed to select the best one of four typical surrogate models through four standard mathematical functions and one engineering problem. The coefficient of determination (R-square, R2) is used to be a calibration of MOA-CVerror and LOO-CVerror. Results show that: (i) in terms of selecting the best models, MOA-CV and LOO-CV become better as sample size increases; (ii) MOA-CV has a better performance in selecting best models than LOO-CV; (iii) in the engineering problem, both the MOA-CV and LOO-CV can choose the worst models, and in most cases, MOA-CV has a higher probability to select the best model than LOO-CV.

Author(s):  
Federico Belotti ◽  
Franco Peracchi

In this article, we describe jackknife2, a new prefix command for jackknifing linear estimators. It takes full advantage of the available leave-one-out formula, thereby allowing for substantial reduction in computing time. Of special note is that jackknife2 allows the user to compute cross-validation and diagnostic measures that are currently not available after ivregress 2sls, xtreg, and xtivregress.


2020 ◽  
Vol 12 ◽  
pp. 175883592092958
Author(s):  
Run-Cong Nie ◽  
Shu-Qiang Yuan ◽  
Yun Wang ◽  
Xue-Bin Zou ◽  
Shi Chen ◽  
...  

Background: We assessed the surrogacy of objective response rate (ORR), disease control rate (DCR) and progression-free survival (PFS) for overall survival (OS) in anti-PD-1/PD-L1 trials of metastatic melanoma through a meta-analysis of randomized controlled trials (RCTs). Methods: PubMed and EMBASE were searched for phase II/III RCTs till June 2019 investigating anti-PD-1/PD-L1 agents. Treatment effect (hazard ratio or odds ratio) on potential surrogates (ORR/DCR/PFS) and OS were collected. At trial level, we assessed the correlation between treatment effect on potential surrogates and OS, weighted by sample size, fixed and random effect models, and calculated the surrogate threshold effect (STE). Sensitivity analyses and leave-one-out cross-validation approach were performed to evaluate the robustness of our findings. Results: We included 8 RCTs (4110 patients; 11 comparisons). We did not identify strong correlations between ORR [coefficient of determination ( R2): 0.09–0.25], DCR (0.41–0.57) and OS. However, we noted a strong correlation between PFS and OS, with R2 of 0.82 in sample size, 0.75 in fixed effect and 0.72 in random effect model weighting, the robustness of which was further verified by leave-one-out cross-validation approach. Sensitivity analyses with restriction to trials with less than 50% crossover, phase III trials, large trials and first-line trials strengthened the correlation (0.78–0.94). The STE for PFS was 0.78. Conclusions: PFS may be the appropriate surrogate for OS in anti-PD-1/PD-L1 trials of metastatic melanoma. A future anti-PD-1/PD-L1 trial would need less than 0.78 for PFS of the upper limit of confidence interval to predict an OS benefit.


2020 ◽  
Vol 38 (15_suppl) ◽  
pp. 10030-10030
Author(s):  
Run-Cong Nie ◽  
Shu-Qiang Yuan ◽  
Yuanfang Li ◽  
Yingbo Chen ◽  
Zhiwei Zhou

10030 Background: The mechanisms of action of anti-PD-1/PD-L1 agents are markedly distinct from those of cytotoxic agents, thus a critical issue that is under investigation is what is the optimal endpoint and how should tumor response be evaluated in anti-PD-1/PD-L1 trials for metastatic melanoma. Here, we assessed surrogacy of objective response rate (ORR), disease control rate (DCR) and progression-free survival (PFS) for overall survival (OS) in anti-PD-1/PD-L1 trials of metastatic melanoma through a meta-analysis of randomized controlled trials (RCTs). Methods: PubMed and EMBASE were searched for phase 2/3 RCTs till June 2019 investigating anti-PD-1/PD-L1 agents. Treatment effect (hazard ratio or odds ratio) on potential surrogates (ORR/DCR/PFS) and OS were collected. At trial level, we assessed the correlation between treatment effect on potential surrogates and OS, weighted by sample size, fixed and random effect models, and calculated the surrogate threshold effect (STE). Sensitivity analyses and leave-one-out cross-validation approach were performed to evaluate the robustness of our findings. Results: We included 8 RCTs (4,110 patients; 11 comparisons). We did not identify strong correlations between ORR (coefficient of determination [ R2]: 0.09 to 0.25), DCR (0.41 to 0.57) and OS. However, we noted a strong correlation between PFS and OS, with R2 of 0.82 in sample size, 0.75 in fixed effect, and 0.72 in random effect model weighting, the robustness of which was further verified by leave-one-out cross-validation approach. Sensitivity analyses with restriction to trials with less than 50% crossover ( R2: 0.94-0.94), phase 3 trials ( R2: 0.94-0.95), large trials ( R2: 0.78-0.86) and first-line trials ( R2: 0.83-0.91) strengthened the correlation. The STE for PFS was 0.78. Conclusions: PFS may be the appropriate surrogate for OS in anti-PD-1/PD-L1 trials of metastatic melanoma. A future anti-PD-1/PD-L1 trial would need less than 0.78 for PFS of the upper limit of confidence interval to predict an OS benefit.


2018 ◽  
Author(s):  
Danielle Navarro

Discussions of model selection in the psychological literature typically frame the issues as a question of statistical inference, with the goal being to determine which model makes the best predictions about data. Within this setting, advocates of leave-one-out cross-validation and Bayes factors disagree on precisely which prediction problem model selection questions should aim to answer. In this comment, I discuss some of these issues from a scientific perspective. What goal does model selection serve when all models are known to be systematically wrong? How might "toy problems" tell a misleading story? How does the scientific goal of explanation align with (or differ from) traditional statistical concerns? I do not offer answers to these questions, but hope to highlight the reasons why psychological researchers cannot avoid asking them.


2018 ◽  
Author(s):  
Quentin Frederik Gronau ◽  
Eric-Jan Wagenmakers

Cross-validation (CV) is increasingly popular as a generic method to adjudicate between mathematical models of cognition and behavior. In order to measure model generalizability, CV quantifies out-of-sample predictive performance, and the CV preference goes to the model that predicted the out-of-sample data best. The advantages of CV include theoretic simplicity and practical feasibility. Despite its prominence, however, the limitations of CV are often underappreciated. Here we demonstrate the limitations of a particular form of CV --Bayesian leave-one-out cross-validation or LOO-- with three concrete examples. In each example, a data set of infinite size is perfectly in line with the predictions of a simple model (i.e., a general law or invariance). Nevertheless, LOO shows bounded and relatively modest support for the simple model. We conclude that CV is not a panacea for model selection.


2018 ◽  
Vol 2 (1) ◽  
pp. 1-11 ◽  
Author(s):  
Quentin F. Gronau ◽  
Eric-Jan Wagenmakers

Sign in / Sign up

Export Citation Format

Share Document