ModL: exploring and restoring regularity when testing for positive selection

2018 ◽  
Vol 35 (15) ◽  
pp. 2545-2554 ◽  
Author(s):  
Joseph Mingrone ◽  
Edward Susko ◽  
Joseph P Bielawski
Keyword(s):  
Positive Selection ◽  
Likelihood Ratio ◽  
Type I Error ◽  
Error Rates ◽  
Ratio Test ◽  
Type I ◽  
Chi Square ◽  
Type I Error Rates ◽  
Modified Likelihood Ratio Test ◽  
Modified Likelihood

Abstract Motivation Likelihood ratio tests are commonly used to test for positive selection acting on proteins. They are usually applied with thresholds for declaring a protein under positive selection determined from a chi-square or mixture of chi-square distributions. Although it is known that such distributions are not strictly justified due to the statistical irregularity of the problem, the hope has been that the resulting tests are conservative and do not lose much power in comparison with the same test using the unknown, correct threshold. We show that commonly used thresholds need not yield conservative tests, but instead give larger than expected Type I error rates. Statistical regularity can be restored by using a modified likelihood ratio test. Results We give theoretical results to prove that, if the number of sites is not too small, the modified likelihood ratio test gives approximately correct Type I error probabilities regardless of the parameter settings of the underlying null hypothesis. Simulations show that modification gives Type I error rates closer to those stated without a loss of power. The simulations also show that parameter estimation for mixture models of codon evolution can be challenging in certain data-generation settings with very different mixing distributions giving nearly identical site pattern distributions unless the number of taxa and tree length are large. Because mixture models are widely used for a variety of problems in molecular evolution, the challenges and general approaches to solving them presented here are applicable in a broader context. Availability and implementation https://github.com/jehops/codeml_modl Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals Evaluating Meta-Analytic Methods to Detect Selective Reporting in the Presence of Dependent Effect Sizes

2019 ◽  
Author(s):  
Melissa Angelina Rodgers ◽  
James E Pustejovsky
Keyword(s):  
Effect Size ◽  
Type I Error ◽  
Error Rates ◽  
Effect Sizes ◽  
Selective Reporting ◽  
Ratio Test ◽  
Type I ◽  
Dependent Effect ◽  
Type I Error Rates ◽  
Dependent Effect Sizes

Selective reporting of results based on their statistical significance threatens the validity of meta-analytic findings. A variety of techniques for detecting selective reporting, publication bias, or small-study effects are available and are routinely used in research syntheses. Most such techniques are univariate, in that they assume that each study contributes a single, independent effect size estimate to the meta-analysis. In practice, however, studies often contribute multiple, statistically dependent effect size estimates, such as for multiple measures of a common outcome construct. Many methods are available for meta-analyzing dependent effect sizes, but methods for investigating selective reporting while also handling effect size dependencies require further investigation. Using Monte Carlo simulations, we evaluate three available univariate tests for small-study effects or selective reporting, including the Trim & Fill test, Egger's regression test, and a likelihood ratio test from a three-parameter selection model (3PSM), when dependence is ignored or handled using ad hoc techniques. We also examine two variants of Egger’s regression test that incorporate robust variance estimation (RVE) or multi-level meta-analysis (MLMA) to handle dependence. Simulation results demonstrate that ignoring dependence inflates Type I error rates for all univariate tests. Variants of Egger's regression maintain Type I error rates when dependent effect sizes are sampled or handled using RVE or MLMA. The 3PSM likelihood ratio test does not fully control Type I error rates. With the exception of the 3PSM, all methods have limited power to detect selection bias except under strong selection for statistically significant effects.

scholarly journals Robust Chi-Square in Extreme and Boundary Conditions: Comments on Jak et al. (2021)

Psych ◽  
2021 ◽  
Vol 3 (3) ◽  
pp. 542-551
Author(s):  
Tihomir Asparouhov ◽  
Bengt Muthén
Keyword(s):  
Boundary Conditions ◽  
Type I Error ◽  
Error Rates ◽  
Type I ◽  
Chi Square ◽  
Type I Error Rates ◽  
Chi Square Test

In this article we describe a modification of the robust chi-square test of fit that yields more accurate type I error rates when the estimated model is at the boundary of the admissible space.

The Effects of Purification and the Evaluation of Differential Item Functioning With the Likelihood Ratio Test

Methodology ◽  
2012 ◽  
Vol 8 (4) ◽  
pp. 134-145 ◽  
Author(s):  
Fabiola González-Betanzos ◽  
Francisco J. Abad
Keyword(s):  
Differential Item Functioning ◽  
Likelihood Ratio ◽  
Likelihood Ratio Test ◽  
Type I Error ◽  
Error Rates ◽  
Ratio Test ◽  
Type I ◽  
Two Stage ◽  
Item Functioning ◽  
Size Type

The current research compares the effects of several strategies to establish the anchor subtest when detecting for differential item functioning (DIF) using the IRT likelihood ratio test in one- and two-stage procedures. Two one-stage strategies were examined: (1) “One item” and (2) “All other items” used as anchor. Additionally, two two-stage strategies were tested: (3) “One anchor item with posterior anchor test augmentation” and (4) “All other items with purification.” The strategies were compared in a simulation study, where sample sizes, DIF size, type of DIF, and software implementation (MULTILOG vs. IRTLRDIF) were manipulated. Results indicated that Procedure (1) was more efficient than (2). Purification was found to improve Type I error rates substantially with the “all other items” strategy, while “posterior anchor test augmentation” did not yield a significant improvement. In relation to the effect of the software used, we found that MULTILOG generally offers better results than IRTLRDIF.

scholarly journals Chi-square and F Ratio: Which should be used when?

2018 ◽  
Vol 8 (2) ◽  
pp. 58-71
Author(s):  
Richard L. Gorsuch ◽  
Curtis Lehmann
Keyword(s):  
Count Data ◽  
Type I Error ◽  
Statistical Significance ◽  
Error Rates ◽  
P Value ◽  
Type I ◽  
Chi Square ◽  
P Values ◽  
Type I Error Rates ◽  
Criterion Variables

Approximations for Chi-square and F distributions can both be computed to provide a p-value, or probability of Type I error, to evaluate statistical significance. Although Chi-square has been used traditionally for tests of count data and nominal or categorical criterion variables (such as contingency tables) and F ratios for tests of non-nominal or continuous criterion variables (such as regression and analysis of variance), we demonstrate that either statistic can be applied in both situations. We used data simulation studies to examine when one statistic may be more accurate than the other for estimating Type I error rates across different types of analysis (count data/contingencies, dichotomous, and non-nominal) and across sample sizes (Ns) ranging from 20 to 160 (using 25,000 replications for simulating p-value derived from either Chi-squares or F-ratios). Our results showed that those derived from F ratios were generally closer to nominal Type I error rates than those derived from Chi-squares. The p-values derived from F ratios were more consistent for contingency table count data than those derived from Chi-squares. The smaller than 100 the N was, the more discrepant p-values derived from Chi-squares were from the nominal p-value. Only when the N was greater than 80 did the p-values from Chi-square tests become as accurate as those derived from F ratios in reproducing the nominal p-values. Thus, there was no evidence of any need for special treatment of dichotomous dependent variables. The most accurate and/or consistent p's were derived from F ratios. We conclude that Chi-square should be replaced generally with the F ratio as the statistic of choice and that the Chi-square test should only be taught as history.

The Robustness of the Likelihood Ratio Chi-Square Test for Structural Equation Models: A Meta-Analysis

2001 ◽  
Vol 26 (1) ◽  
pp. 105-132 ◽  
Author(s):  
Douglas A. Powell ◽  
William D. Schafer
Keyword(s):  
Structural Equation ◽  
Structural Equation Models ◽  
Type I Error ◽  
Meta Analysis ◽  
Generalized Least Squares ◽  
Error Rates ◽  
Type I ◽  
Chi Square ◽  
Distribution Free ◽  
Projection Techniques

The robustness literature for the structural equation model was synthesized following the method of Harwell which employs meta-analysis as developed by Hedges and Vevea. The study focused on the explanation of empirical Type I error rates for six principal classes of estimators: two that assume multivariate normality (maximum likelihood and generalized least squares), elliptical estimators, two distribution-free estimators (asymptotic and others), and latent projection. Generally, the chi-square tests for overall model fit were found to be sensitive to non-normality and the size of the model for all estimators (with the possible exception of the elliptical estimators with respect to model size and the latent projection techniques with respect to non-normality). The asymptotic distribution-free (ADF) and latent projection techniques were also found to be sensitive to sample sizes. Distribution-free methods other than ADF showed, in general, much less sensitivity to all factors considered.

Type I Error Rates for Parscale’s Fit Index

2005 ◽  
Vol 65 (1) ◽  
pp. 42-50 ◽  
Author(s):  
Christine E. Demars
Keyword(s):  
Type I Error ◽  
Error Rates ◽  
Type I ◽  
Type I Error Rates ◽  
Fit Index
Sign in / Sign up

Export Citation Format

Share Document