scholarly journals Preprint of: "Redefine or Justify? Comments on the alpha debate"

2018 ◽  
Author(s):  
Jan Peter De Ruiter
Keyword(s):  
Empirical Evidence ◽  
Null Hypothesis ◽  
Psychological Research ◽  
P Value ◽  
Significance Tests ◽  
P Values ◽  
Alpha Level ◽  
Replication Crisis ◽  
Crisis In Psychology

Benjamin et al. (2017) proposed improving the reproducibility of findings in psychological research by lowering the alpha level of our conventional Null Hypothesis Significance Tests from .05 to .005, because findings with p-values close to .05 represent insufficient empirical evidence. They argued that findings with a p-value between 0.005 and 0.05 should still be published, but not called “significant” anymore.This proposal was criticized and rejected in a response by Lakens et al. (2018), who argued that instead of lowering the traditional alpha threshold to .005, we should stop using the term “statistically significant”, and require researchers to determine and justify their alpha levels before they collect data.In this contribution, I argue that the arguments presented by Lakens et al. against the proposal by Benjamin et al (2017) are not convincing. Thus, given that it is highly unlikely that our field will abandon the NHST paradigm any time soon, lowering our alpha level to .005 is at this moment the best way to combat the replication crisis in psychology.

scholarly journals Some comments on the “Justify Your Alpha” response to “Redefine Statistical Significance”

2017 ◽  
Author(s):  
Jan Peter De Ruiter
Keyword(s):  
Empirical Evidence ◽  
Null Hypothesis ◽  
Statistical Significance ◽  
Psychological Research ◽  
Negative Consequences ◽  
Insufficient Evidence ◽  
Significance Tests ◽  
P Values ◽  
Alpha Level ◽  
Alpha Response

Benjamin et al. (2017) proposed to improve the reproducibility of findings in psychological research by lowering the alpha level of our conventional Null Hypothesis Significance Tests from .05 to .005, because findings with p-values close to .05 represent insufficient evidence. This proposal was criticized and rejected by a commentary by Lakens et al. (2017), who argued that a) the empirical evidence for the effectiveness of such a policy is weak, b) the theoretical arguments for the effectiveness of such a policy are weak, and c) the proposal also has negative consequences for reproducibility. In this contribution, I argue that the arguments by Lakens et al. are either unconvincing, or in fact arguments in favor of the proposal by Benjamin et al.

scholarly journals Statcheck in Canada: What Proportion of CPA Journal Articles Contain Errors in the Reporting of p-Values?

2017 ◽  
Author(s):  
Christopher Green ◽  
Sahir Abbas ◽  
Arlie Belliveau ◽  
Nataly Beribisky ◽  
Ian Davidson ◽  
...  
Keyword(s):  
Error Rate ◽  
Null Hypothesis ◽  
Research Articles ◽  
P Value ◽  
Test Statistics ◽  
Journal Articles ◽  
Significance Tests ◽  
P Values ◽  
The Past ◽  
Digital Survey

Using a computer program called “Statcheck,” a 2016 digital survey of several prestigious American and European psychology journals showed that the p-values reported in research articles failed to agree with the corresponding test statistics (e.g., F, t, χ2) at surprisingly high rates: nearly half of all articles contained at least one such error, as did about 10% of all null hypothesis significance tests. We investigated whether this problem was present in Canadian psychology journals and, if so, at what frequency. We discovered similar rates of p-value errors in Canadian journals over the past 30 years. However, we also noticed, a large number of typographical errors in the electronic versions of the articles. When we hand corrected a sample of our articles, the per-article error rate remained about the same, but the per test rate of errors dropped to 6.3%. We recommend that, in future, journals include explicit checks of statistics in their editorial processes.

"MULTIPLE-COMPARISONS" PROBLEM

PEDIATRICS ◽  
1989 ◽  
Vol 84 (6) ◽  
pp. A30-A30
Author(s):  
Student
Keyword(s):  
Null Hypothesis ◽  
Multiple Comparisons ◽  
P Value ◽  
Expected Number ◽  
P Values ◽  
A Value ◽  
True Null Hypotheses ◽  
The Individual

Often investigators report many P values in the same study. The expected number of P values smaller than 0.05 is 1 in 20 tests of true null hypotheses; therefore the probability that at least one P value will be smaller than 0.05 increases with the number of tests, even when the null hypothesis is correct for each test. This increase is known as the "multiple-comparisons" problem...One reasonable way to correct for multiplicity is simply to multiply the P value by the number of tests. Thus, with five tests, an orignal 0.05 level for each is increased, perhaps to a value as high as 0.25 for the set. To achieve a level of not more than 0.05 for the set, we need to choose a level of 0.05/5 = 0.01 for the individual tests. This adjustment is conservative. We know only that the probability does not exceed 0.05 for the set.

scholarly journals Simple methods for estimating confidence levels, or tentative probabilities, for hypotheses instead of p values

2019 ◽  
Vol 12 (1) ◽  
pp. 205979911982651
Author(s):  
Michael Wood
Keyword(s):  
Statistical Inference ◽  
Confidence Intervals ◽  
Statistical Methods ◽  
Null Hypothesis ◽  
Significance Tests ◽  
P Values ◽  
Confidence Levels ◽  
Alternative Approach ◽  
Degree Of Certainty ◽  
Natural Way

In many fields of research, null hypothesis significance tests and p values are the accepted way of assessing the degree of certainty with which research results can be extrapolated beyond the sample studied. However, there are very serious concerns about the suitability of p values for this purpose. An alternative approach is to cite confidence intervals for a statistic of interest, but this does not directly tell readers how certain a hypothesis is. Here, I suggest how the framework used for confidence intervals could easily be extended to derive confidence levels, or “tentative probabilities,” for hypotheses. I also outline four quick methods for estimating these. This allows researchers to state their confidence in a hypothesis as a direct probability, instead of circuitously by p values referring to a hypothetical null hypothesis—which is usually not even stated explicitly. The inevitable difficulties of statistical inference mean that these probabilities can only be tentative, but probabilities are the natural way to express uncertainties, so, arguably, researchers using statistical methods have an obligation to estimate how probable their hypotheses are by the best available method. Otherwise, misinterpretations will fill the void.

scholarly journals When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing.

2021 ◽  
Author(s):  
Mark Rubin
Keyword(s):  
Present Article ◽  
Null Hypothesis ◽  
Multiple Testing ◽  
Multiple Comparisons ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Significance Threshold ◽  
Alpha Level ◽  
Replication Crisis ◽  
Test Result

Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically (mindlessly) assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses.

scholarly journals Visualization Strategies for Regression Estimates with Randomization Inference

2019 ◽  
Author(s):  
Marshall A. Taylor
Keyword(s):  
Confidence Interval ◽  
Confidence Intervals ◽  
Regression Models ◽  
Statistical Significance ◽  
Permutation Tests ◽  
P Value ◽  
P Values ◽  
Alpha Level ◽  
Significance Levels ◽  
Nonprobability Sample

Coefficient plots are a popular tool for visualizing regression estimates. The appeal of these plots is that they visualize confidence intervals around the estimates and generally center the plot around zero, meaning that any estimate that crosses zero is statistically non-significant at at least the alpha-level around which the confidence intervals are constructed. For models with statistical significance levels determined via randomization models of inference and for which there is no standard error or confidence intervals for the estimate itself, these plots appear less useful. In this paper, I illustrate a variant of the coefficient plot for regression models with p-values constructed using permutation tests. These visualizations plot each estimate's p-value and its associated confidence interval in relation to a specified alpha-level. These plots can help the analyst interpret and report both the statistical and substantive significance of their models. Illustrations are provided using a nonprobability sample of activists and participants at a 1962 anti-Communism school.

scholarly journals The frequent insignificance of a “significant” P-value

2021 ◽  
Author(s):  
David McGiffin ◽  
Geoff Cumming ◽  
Paul Myles
Keyword(s):  
Diagnostic Tests ◽  
Null Hypothesis ◽  
Open Science ◽  
Significance Testing ◽  
P Value ◽  
Conditional Probabilities ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Science Practices ◽  
Strength Of Evidence

Null hypothesis significance testing (NHST) and p-values are widespread in the cardiac surgical literature but are frequently misunderstood and misused. The purpose of the review is to discuss major disadvantages of p-values and suggest alternatives. We describe diagnostic tests, the prosecutor’s fallacy in the courtroom, and NHST, which involve inter-related conditional probabilities, to help clarify the meaning of p-values, and discuss the enormous sampling variability, or unreliability, of p-values. Finally, we use a cardiac surgical database and simulations to explore further issues involving p-values. In clinical studies, p-values provide a poor summary of the observed treatment effect, whereas the three- number summary provided by effect estimates and confidence intervals is more informative and minimises over-interpretation of a “significant” result. P-values are an unreliable measure of strength of evidence; if used at all they give only, at best, a very rough guide to decision making. Researchers should adopt Open Science practices to improve the trustworthiness of research and, where possible, use estimation (three-number summaries) or other better techniques.

scholarly journals Do p Values Lose Their Meaning in Exploratory Analyses? It Depends How You Define the Familywise Error Rate

2017 ◽  
Vol 21 (3) ◽  
pp. 269-275 ◽  
Author(s):  
Mark Rubin
Keyword(s):  
Present Article ◽  
Error Rate ◽  
Null Hypothesis ◽  
Research Report ◽  
Familywise Error Rate ◽  
P Values ◽  
Alpha Level

Several researchers have recently argued that p values lose their meaning in exploratory analyses due to an unknown inflation of the alpha level (e.g., Nosek & Lakens, 2014 ; Wagenmakers, 2016 ). For this argument to be tenable, the familywise error rate must be defined in relation to the number of hypotheses that are tested in the same study or article. Under this conceptualization, the familywise error rate is usually unknowable in exploratory analyses because it is usually unclear how many hypotheses have been tested on a spontaneous basis and then omitted from the final research report. In the present article, I argue that it is inappropriate to conceptualize the familywise error rate in relation to the number of hypotheses that are tested. Instead, it is more appropriate to conceptualize familywise error in relation to the number of different tests that are conducted on the same null hypothesis in the same study. Under this conceptualization, alpha-level adjustments in exploratory analyses are (a) less necessary and (b) objectively verifiable. As a result, p values do not lose their meaning in exploratory analyses.

scholarly journals Falacias sobre el valor p compartidas por profesores y estudiantes universitarios

2017 ◽  
Vol 16 (3) ◽  
pp. 1
Author(s):  
Laura Badenes-Ribera ◽  
Dolores Frias-Navarro
Keyword(s):  
College Students ◽  
Statistical Significance ◽  
Psychological Research ◽  
Practical Significance ◽  
P Value ◽  
P Values ◽  
Statistical Education ◽  
The Mean ◽  
Estudiantes Universitarios

Resumen La “Práctica Basada en la Evidencia” exige que los profesionales valoren de forma crítica los resultados de las investigaciones psicológicas. Sin embargo, las interpretaciones incorrectas de los valores p de probabilidad son abundantes y repetitivas. Estas interpretaciones incorrectas afectan a las decisiones profesionales y ponen en riesgo la calidad de las intervenciones y la acumulación de un conocimiento científico válido. Identificar el tipo de falacia que subyace a las decisiones estadísticas es fundamental para abordar y planificar estrategias de educación estadística dirigidas a intervenir sobre las interpretaciones incorrectas. En consecuencia, el objetivo de este estudio es analizar la interpretación del valor p en estudiantes y profesores universitarios de Psicología. La muestra estuvo formada por 161 participantes (43 profesores y 118 estudiantes). La antigüedad media como profesor fue de 16.7 años (DT = 10.07). La edad media de los estudiantes fue de 21.59 (DT = 1.3). Los hallazgos sugieren que los estudiantes y profesores universitarios no conocen la interpretación correcta del valor p. La falacia de la probabilidad inversa presenta mayores problemas de comprensión. Además, se confunde la significación estadística y la significación práctica o clínica. Estos resultados destacan la necesidad de la educación estadística y re-educación estadística. Abstract The "Evidence Based Practice" requires professionals to critically assess the results of psychological research. However, incorrect interpretations of p values of probability are abundant and repetitive. These misconceptions affect professional decisions and compromise the quality of interventions and the accumulation of a valid scientific knowledge. Identifying the types of fallacies that underlying statistical decisions is fundamental for approaching and planning statistical education strategies designed to intervene in incorrect interpretations. Therefore, the aim of this study is to analyze the interpretation of p value among college students of psychology and academic psychologist. The sample was composed of 161 participants (43 academic and 118 students). The mean number of years as academic was 16.7 (SD = 10.07). The mean age of college students was 21.59 years (SD = 1.3). The findings suggest that college students and academic do not know the correct interpretation of p values. The fallacy of the inverse probability presents major problems of comprehension. In addition, statistical significance and practical significance or clinical are confused. There is a need for statistical education and statistical re-education.

Sign in / Sign up

Export Citation Format

Share Document