scholarly journals Chi-square and F Ratio: Which should be used when?

2018 ◽  
Vol 8 (2) ◽  
pp. 58-71
Author(s):  
Richard L. Gorsuch ◽  
Curtis Lehmann
Keyword(s):  
Count Data ◽  
Type I Error ◽  
Statistical Significance ◽  
Error Rates ◽  
P Value ◽  
Type I ◽  
Chi Square ◽  
P Values ◽  
Type I Error Rates ◽  
Criterion Variables

Approximations for Chi-square and F distributions can both be computed to provide a p-value, or probability of Type I error, to evaluate statistical significance. Although Chi-square has been used traditionally for tests of count data and nominal or categorical criterion variables (such as contingency tables) and F ratios for tests of non-nominal or continuous criterion variables (such as regression and analysis of variance), we demonstrate that either statistic can be applied in both situations. We used data simulation studies to examine when one statistic may be more accurate than the other for estimating Type I error rates across different types of analysis (count data/contingencies, dichotomous, and non-nominal) and across sample sizes (Ns) ranging from 20 to 160 (using 25,000 replications for simulating p-value derived from either Chi-squares or F-ratios). Our results showed that those derived from F ratios were generally closer to nominal Type I error rates than those derived from Chi-squares. The p-values derived from F ratios were more consistent for contingency table count data than those derived from Chi-squares. The smaller than 100 the N was, the more discrepant p-values derived from Chi-squares were from the nominal p-value. Only when the N was greater than 80 did the p-values from Chi-square tests become as accurate as those derived from F ratios in reproducing the nominal p-values. Thus, there was no evidence of any need for special treatment of dichotomous dependent variables. The most accurate and/or consistent p's were derived from F ratios. We conclude that Chi-square should be replaced generally with the F ratio as the statistic of choice and that the Chi-square test should only be taught as history.

scholarly journals Considering Both Statistical and Clinical Significance

2016 ◽  
Vol 5 (5) ◽  
pp. 16 ◽  
Author(s):  
Guolong Zhao
Keyword(s):  
Clinical Significance ◽  
Null Hypothesis ◽  
Type I Error ◽  
Statistical Significance ◽  
Error Rates ◽  
Type I ◽  
Data Set ◽  
Type I Error Rates ◽  
Empirical Coverage ◽  
Superiority Trials

To evaluate a drug, statistical significance alone is insufficient and clinical significance is also necessary. This paper explains how to analyze clinical data with considering both statistical and clinical significance. The analysis is practiced by combining a confidence interval under null hypothesis with that under non-null hypothesis. The combination conveys one of the four possible results: (i) both significant, (ii) only significant in the former, (iii) only significant in the latter or (iv) neither significant. The four results constitute a quadripartite procedure. Corresponding tests are mentioned for describing Type I error rates and power. The empirical coverage is exhibited by Monte Carlo simulations. In superiority trials, the four results are interpreted as clinical superiority, statistical superiority, non-superiority and indeterminate respectively. The interpretation is opposite in inferiority trials. The combination poses a deflated Type I error rate, a decreased power and an increased sample size. The four results may helpful for a meticulous evaluation of drugs. Of these, non-superiority is another profile of equivalence and so it can also be used to interpret equivalence. This approach may prepare a convenience for interpreting discordant cases. Nevertheless, a larger data set is usually needed. An example is taken from a real trial in naturally acquired influenza.

The Conundrum of P-Values: Statistical Significance is Unavoidable but Need Medical Significance Too

2020 ◽  
Author(s):  
Abhaya Indrayan
Keyword(s):  
Type I Error ◽  
Dominant Role ◽  
Statistical Significance ◽  
Empirical Studies ◽  
P Value ◽  
Selective Reporting ◽  
Type I ◽  
Practical Application ◽  
P Values ◽  
Zero Effect

Background: Small P-values have been conventionally considered as evidence to reject a null hypothesis in empirical studies. However, there is widespread criticism of P-values now and the threshold we use for statistical significance is questioned.Methods: This communication is on contrarian view and explains why P-value and its threshold are still useful for ruling out sampling fluctuation as a source of the findings.Results: The problem is not with P-values themselves but it is with their misuse, abuse, and over-use, including the dominant role they have assumed in empirical results. False results may be mostly because of errors in design, invalid data, inadequate analysis, inappropriate interpretation, accumulation of Type-I error, and selective reporting, and not because of P-values per se.Conclusion: A threshold of P-values such as 0.05 for statistical significance is helpful in making a binary inference for practical application of the result. However, a lower threshold can be suggested to reduce the chance of false results. Also, the emphasis should be on detecting a medically significant effect and not zero effect.

Directional-sum test for nonparametric Behrens-Fisher problem with applications to the dietary intervention trial

2021 ◽  
pp. 096228022110028
Author(s):  
Zhen Meng ◽  
Qinglong Yang ◽  
Qizhai Li ◽  
Baoxue Zhang
Keyword(s):  
Dietary Intervention ◽  
Type I Error ◽  
Statistical Significance ◽  
Error Rates ◽  
Type I ◽  
Simulation Studies ◽  
Intervention Trial ◽  
Combination Strategy ◽  
Type I Error Rates ◽  
Dietary Intervention Trial

For a nonparametric Behrens-Fisher problem, a directional-sum test is proposed based on division-combination strategy. A one-layer wild bootstrap procedure is given to calculate its statistical significance. We conduct simulation studies with data generated from lognormal, t and Laplace distributions to show that the proposed test can control the type I error rates properly and is more powerful than the existing rank-sum and maximum-type tests under most of the considered scenarios. Applications to the dietary intervention trial further show the performance of the proposed test.

scholarly journals Modifying and Evaluating the Alexander-Govern Test Using Real Data

2015 ◽  
Vol 9 (12) ◽  
pp. 1
Author(s):  
Tobi Kingsley Ochuko ◽  
Suhaida Abdullah ◽  
Zakiyah Binti Zain ◽  
Sharipah Syed Soaad Yahaya
Keyword(s):  
Type I Error ◽  
Parametric Method ◽  
Real Life ◽  
Good Control ◽  
Error Rates ◽  
P Value ◽  
Type I ◽  
Central Tendency ◽  
Central Tendency Measure ◽  
Type I Error Rates

This study examines the use of independent group test of comparing two or more means by using parametric method, such as the Alexander-Govern (<em>AG</em>) test. The Alexander-Govern test is used for comparing two or more groups and is a better alternative compared to the James test, the Welch test and the <em>ANOVA</em>. This test has a good control of Type I error rates and gives a high power under variance heterogeneity for a normal data, but it is not robust for non-normal data. As a result, trimmed mean was applied on the test under non-normal data for two group condition. But this test could not control the Type I error rates, when the number of groups exceed two groups. As a result, the <em>MOM</em> estimator was introduced on the test, as its central tendency measure and is not influenced by the number of groups. But this estimator fails to give a good control of Type I error rates, under skewed heavy tailed distribution. In this study, the <em>AGWMOM </em>test was applied in Alexander-Govern test as its central tendency measure. To evaluate the capacity of the test, a real life data was used. Descriptive statistics, Tests of Normality and boxplots were used to determine the normality and non-normality of the independent groups. The results show that only the group middle is not normally distributed due extreme value in the data distribution. The results from the test statistic show that the <em>AGWMOM</em> test has a smaller p-value of 0.0000002869 that is less than 0.05, compared to the <em>AG</em> test that produced a p-value of 0.06982, that is greater than 0.05. Therefore, the <em>AGWMOM</em> test is considered to be significant, compared to the <em>AG</em> test.

scholarly journals Robust Chi-Square in Extreme and Boundary Conditions: Comments on Jak et al. (2021)

Psych ◽  
2021 ◽  
Vol 3 (3) ◽  
pp. 542-551
Author(s):  
Tihomir Asparouhov ◽  
Bengt Muthén
Keyword(s):  
Boundary Conditions ◽  
Type I Error ◽  
Error Rates ◽  
Type I ◽  
Chi Square ◽  
Type I Error Rates ◽  
Chi Square Test

In this article we describe a modification of the robust chi-square test of fit that yields more accurate type I error rates when the estimated model is at the boundary of the admissible space.

ModL: exploring and restoring regularity when testing for positive selection

2018 ◽  
Vol 35 (15) ◽  
pp. 2545-2554 ◽  
Author(s):  
Joseph Mingrone ◽  
Edward Susko ◽  
Joseph P Bielawski
Keyword(s):  
Positive Selection ◽  
Likelihood Ratio ◽  
Type I Error ◽  
Error Rates ◽  
Ratio Test ◽  
Type I ◽  
Chi Square ◽  
Type I Error Rates ◽  
Modified Likelihood Ratio Test ◽  
Modified Likelihood

Abstract Motivation Likelihood ratio tests are commonly used to test for positive selection acting on proteins. They are usually applied with thresholds for declaring a protein under positive selection determined from a chi-square or mixture of chi-square distributions. Although it is known that such distributions are not strictly justified due to the statistical irregularity of the problem, the hope has been that the resulting tests are conservative and do not lose much power in comparison with the same test using the unknown, correct threshold. We show that commonly used thresholds need not yield conservative tests, but instead give larger than expected Type I error rates. Statistical regularity can be restored by using a modified likelihood ratio test. Results We give theoretical results to prove that, if the number of sites is not too small, the modified likelihood ratio test gives approximately correct Type I error probabilities regardless of the parameter settings of the underlying null hypothesis. Simulations show that modification gives Type I error rates closer to those stated without a loss of power. The simulations also show that parameter estimation for mixture models of codon evolution can be challenging in certain data-generation settings with very different mixing distributions giving nearly identical site pattern distributions unless the number of taxa and tree length are large. Because mixture models are widely used for a variety of problems in molecular evolution, the challenges and general approaches to solving them presented here are applicable in a broader context. Availability and implementation https://github.com/jehops/codeml_modl Supplementary information Supplementary data are available at Bioinformatics online.

scholarly journals Systematic Review of the use of “Magnitude-Based Inference” in Sports Science and Medicine

2020 ◽  
Author(s):  
Keith Lohse ◽  
Kristin Sainani ◽  
J. Andrew Taylor ◽  
Michael Lloyd Butson ◽  
Emma Knight ◽  
...  
Keyword(s):  
Sample Size ◽  
Multiple Testing ◽  
Type I Error ◽  
A Priori ◽  
Error Rates ◽  
Significance Testing ◽  
Type I ◽  
P Values ◽  
Type I Error Rates ◽  
Sports Science

Magnitude-based inference (MBI) is a controversial statistical method that has been used in hundreds of papers in sports science despite criticism from statisticians. To better understand how this method has been applied in practice, we systematically reviewed 232 papers that used MBI. We extracted data on study design, sample size, and choice of MBI settings and parameters. Median sample size was 10 per group (interquartile range, IQR: 8 – 15) for multi-group studies and 14 (IQR: 10 – 24) for single-group studies; few studies reported a priori sample size calculations (15%). Authors predominantly applied MBI’s default settings and chose “mechanistic/non-clinical” rather than “clinical” MBI even when testing clinical interventions (only 14 studies out of 232 used clinical MBI). Using these data, we can estimate the Type I error rates for the typical MBI study. Authors frequently made dichotomous claims about effects based on the MBI criterion of a “likely” effect and sometimes based on the MBI criterion of a “possible” effect. When the sample size is n=8 to 15 per group, these inferences have Type I error rates of 12%-22% and 22%-45%, respectively. High Type I error rates were compounded by multiple testing: Authors reported results from a median of 30 tests related to outcomes; and few studies specified a primary outcome (14%). We conclude that MBI has promoted small studies, promulgated a “black box” approach to statistics, and led to numerous papers where the conclusions are not supported by the data. Amidst debates over the role of p-values and significance testing in science, MBI also provides an important natural experiment: we find no evidence that moving researchers away from p-values or null hypothesis significance testing makes them less prone to dichotomization or over-interpretation of findings.

Sign in / Sign up

Export Citation Format

Share Document