scholarly journals Manipulating the alpha level cannot cure significance testing – comments on "Redefine statistical significance"

2017 ◽  
Author(s):  
David Trafimow ◽  
Valentin Amrhein ◽  
Corson N. Areshenkoff ◽  
Carlos Barrera-Causil ◽  
Eric J. Beh ◽  
...  
Keyword(s):  
Experimental Design ◽  
Statistical Significance ◽  
Significance Testing ◽  
P Value ◽  
Statistical Tools ◽  
P Values ◽  
Binary Decision ◽  
Clear Cut ◽  
Alpha Level ◽  
Cumulative Evidence

We argue that depending on p-values to reject null hypotheses, including a recent call for changing the canonical alpha level for statistical significance from .05 to .005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable criterion levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and determining sample sizes much more directly than significance testing does; but none of the statistical tools should replace significance testing as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, or implications for applications. To boil all this down to a binary decision based on a p-value threshold of .05, .01, .005, or anything else, is not acceptable.

scholarly journals Manipulating the alpha level cannot cure significance testing

2018 ◽  
Author(s):  
David Trafimow ◽  
Valentin Amrhein ◽  
Corson N. Areshenkoff ◽  
Carlos Barrera-Causil ◽  
Eric J. Beh ◽  
...  
Keyword(s):  
Experimental Design ◽  
Sample Size ◽  
Study Design ◽  
Significance Testing ◽  
Statistical Tools ◽  
Binary Decision ◽  
Clear Cut ◽  
Alpha Level ◽  
Cumulative Evidence ◽  
Scientific Hypotheses

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p= .05 to .005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of .05, .01, .005, or anything else, is not acceptable.

scholarly journals Manipulating the alpha level cannot cure significance testing

2018 ◽  
Author(s):  
David Trafimow ◽  
Valentin Amrhein ◽  
Corson N. Areshenkoff ◽  
Carlos Barrera-Causil ◽  
Eric J. Beh ◽  
...  
Keyword(s):  
Experimental Design ◽  
Sample Size ◽  
Study Design ◽  
Significance Testing ◽  
Statistical Tools ◽  
Binary Decision ◽  
Clear Cut ◽  
Alpha Level ◽  
Cumulative Evidence ◽  
Scientific Hypotheses

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p= .05 to .005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of .05, .01, .005, or anything else, is not acceptable.

scholarly journals Manipulating the alpha level cannot cure significance testing

2018 ◽  
Author(s):  
David Trafimow ◽  
Valentin Amrhein ◽  
Corson N. Areshenkoff ◽  
Carlos Barrera-Causil ◽  
Eric J. Beh ◽  
...  
Keyword(s):  
Experimental Design ◽  
Sample Size ◽  
Study Design ◽  
Significance Testing ◽  
Statistical Tools ◽  
Binary Decision ◽  
Clear Cut ◽  
Alpha Level ◽  
Cumulative Evidence ◽  
Scientific Hypotheses

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p= .05 to .005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of .05, .01, .005, or anything else, is not acceptable.

scholarly journals Visualization Strategies for Regression Estimates with Randomization Inference

2019 ◽  
Author(s):  
Marshall A. Taylor
Keyword(s):  
Confidence Interval ◽  
Confidence Intervals ◽  
Regression Models ◽  
Statistical Significance ◽  
Permutation Tests ◽  
P Value ◽  
P Values ◽  
Alpha Level ◽  
Significance Levels ◽  
Nonprobability Sample

Coefficient plots are a popular tool for visualizing regression estimates. The appeal of these plots is that they visualize confidence intervals around the estimates and generally center the plot around zero, meaning that any estimate that crosses zero is statistically non-significant at at least the alpha-level around which the confidence intervals are constructed. For models with statistical significance levels determined via randomization models of inference and for which there is no standard error or confidence intervals for the estimate itself, these plots appear less useful. In this paper, I illustrate a variant of the coefficient plot for regression models with p-values constructed using permutation tests. These visualizations plot each estimate's p-value and its associated confidence interval in relation to a specified alpha-level. These plots can help the analyst interpret and report both the statistical and substantive significance of their models. Illustrations are provided using a nonprobability sample of activists and participants at a 1962 anti-Communism school.

scholarly journals Visualization strategies for regression estimates with randomization inference

2020 ◽  
Vol 20 (2) ◽  
pp. 309-335
Author(s):  
Marshall A. Taylor
Keyword(s):  
Confidence Interval ◽  
Confidence Intervals ◽  
Regression Models ◽  
Statistical Significance ◽  
Permutation Tests ◽  
P Value ◽  
P Values ◽  
Alpha Level ◽  
Significance Levels ◽  
Nonprobability Sample

Coefficient plots are a popular tool for visualizing regression estimates. The appeal of these plots is that they visualize confidence intervals around the estimates and generally center the plot around zero, meaning that any estimate that crosses zero is statistically nonsignificant at least at the alpha level around which the confidence intervals are constructed. For models with statistical significance levels determined via randomization models of inference and for which there is no standard error or confidence intervals for the estimate itself, these plots appear less useful. In this article, I illustrate a variant of the coefficient plot for regression models with p-values constructed using permutation tests. These visualizations plot each estimate’s p-value and its associated confidence interval in relation to a specified alpha level. These plots can help the analyst interpret and report the statistical and substantive significances of their models. I illustrate using a nonprobability sample of activists and participants at a 1962 anticommunism school.

scholarly journals WHEN AND WHERE P-VALUE IS REQUIRED: MISNOMERS IN CLINICALAND STATISTICAL SIGNIFICANCE

2021 ◽  
pp. 1-2
Author(s):  
Sukhvinder Singh Oberoi ◽  
Mansi Atri
Keyword(s):  
Data Analysis ◽  
Confidence Intervals ◽  
Effect Size ◽  
Statistical Significance ◽  
Significance Testing ◽  
P Value ◽  
Practical Application ◽  
Statistical Software ◽  
P Values ◽  
Long Time

The interpretation of the p-value has been an arena for discussion making it difficult for many researchers. The p-value was introduced in 1900 by Pearson. Though, it is very difficult to comment about the demerits of the p-values and significance testing which has not been spoken in a long time because of the practical application of it as a measure of interpretation in clinical research. The usage of the confidence intervals around the sample statistics and effect size should be given more importance than relying solely upon the statistical significance. The researchers, should be consulting a statistician in the initial stages of the planning of the study for avoidance of the misinterpretation of the P-value especially if they are using statistical software for their data analysis.

Abstract MP11: Circulating Plasma Biomarkers Associated With Brain Arteriovenous Malformations

Stroke ◽  
2021 ◽  
Vol 52 (Suppl_1) ◽  
Author(s):  
Sarah E Wetzel-Strong ◽  
Shantel M Weinsheimer ◽  
Jeffrey Nelson ◽  
Ludmila Pawlikowska ◽  
Dewi Clark ◽  
...  
Keyword(s):  
Multiple Testing ◽  
Statistical Significance ◽  
Protein Profiling ◽  
P Value ◽  
P Values ◽  
Plasma Biomarkers ◽  
Standard Curve ◽  
Disease States ◽  
Heparin Plasma ◽  
Circulating Levels

Objective: Circulating plasma protein profiling may aid in the identification of cerebrovascular disease signatures. This study aimed to identify circulating angiogenic and inflammatory biomarkers that may serve as biomarkers to differentiate sporadic brain arteriovenous malformation (bAVM) patients from other conditions with brain AVMs, including hereditary hemorrhagic telangiectasia (HHT) patients. Methods: The Quantibody Human Angiogenesis Array 1000 (Raybiotech) is an ELISA multiplex panel that was used to assess the levels of 60 proteins related to angiogenesis and inflammation in heparin plasma samples from 13 sporadic unruptured bAVM patients (69% male, mean age 51 years) and 37 patients with HHT (40% male, mean age 47 years, n=19 (51%) with bAVM). The Quantibody Q-Analyzer tool was used to calculate biomarker concentrations based on the standard curve for each marker and log-transformed marker levels were evaluated for associations between disease states using a multivariable interval regression model adjusted for age, sex, ethnicity and collection site. Statistical significance was based on Bonferroni correction for multiple testing of 60 biomarkers (P< 8.3x10 - 4 ). Results: Circulating levels of two plasma proteins differed significantly between sporadic bAVM and HHT patients: PDGF-BB (P=2.6x10 -4 , PI= 3.37, 95% CI:1.76-6.46) and CCL5 (P=6.0x10 -6 , PI=3.50, 95% CI=2.04-6.03). When considering markers with a nominal p-value of less than 0.01, MMP1 and angiostatin levels also differed between patients with sporadic bAVM and HHT. Markers with nominal p-values less than 0.05 when comparing sporadic brain AVM and HHT patients also included angiostatin, IL2, VEGF, GRO, CXCL16, ITAC, and TGFB3. Among HHT patients, the circulating levels of UPAR and IL6 were elevated in patients with documented bAVMs when considering markers with nominal p-values less than 0.05. Conclusions: This study identified differential expression of two promising plasma biomarkers that differentiate sporadic bAVMs from patients with HHT. Furthermore, this study allowed us to evaluate markers that are associated with the presence of bAVMs in HHT patients, which may offer insight into mechanisms underlying bAVM pathophysiology.

The Conundrum of P-Values: Statistical Significance is Unavoidable but Need Medical Significance Too

2020 ◽  
Author(s):  
Abhaya Indrayan
Keyword(s):  
Type I Error ◽  
Dominant Role ◽  
Statistical Significance ◽  
Empirical Studies ◽  
P Value ◽  
Selective Reporting ◽  
Type I ◽  
Practical Application ◽  
P Values ◽  
Zero Effect

Background: Small P-values have been conventionally considered as evidence to reject a null hypothesis in empirical studies. However, there is widespread criticism of P-values now and the threshold we use for statistical significance is questioned.Methods: This communication is on contrarian view and explains why P-value and its threshold are still useful for ruling out sampling fluctuation as a source of the findings.Results: The problem is not with P-values themselves but it is with their misuse, abuse, and over-use, including the dominant role they have assumed in empirical results. False results may be mostly because of errors in design, invalid data, inadequate analysis, inappropriate interpretation, accumulation of Type-I error, and selective reporting, and not because of P-values per se.Conclusion: A threshold of P-values such as 0.05 for statistical significance is helpful in making a binary inference for practical application of the result. However, a lower threshold can be suggested to reduce the chance of false results. Also, the emphasis should be on detecting a medically significant effect and not zero effect.

scholarly journals P values: from suggestion to superstition

2016 ◽  
Vol 64 (7) ◽  
pp. 1166-1171 ◽  
Author(s):  
John Concato ◽  
John A Hartigan
Keyword(s):  
Clinical Research ◽  
Statistical Significance ◽  
Historical Context ◽  
P Value ◽  
Threshold Probability ◽  
Conceptual Approach ◽  
P Values ◽  
Clinical Investigations ◽  
Current Usage ◽  
Genomic Studies

A threshold probability value of ‘p≤0.05’ is commonly used in clinical investigations to indicate statistical significance. To allow clinicians to better understand evidence generated by research studies, this review defines the p value, summarizes the historical origins of the p value approach to hypothesis testing, describes various applications of p≤0.05 in the context of clinical research and discusses the emergence of p≤5×10−8 and other values as thresholds for genomic statistical analyses. Corresponding issues include a conceptual approach of evaluating whether data do not conform to a null hypothesis (ie, no exposure–outcome association). Importantly, and in the historical context of when p≤0.05 was first proposed, the 1-in-20 chance of a false-positive inference (ie, falsely concluding the existence of an exposure–outcome association) was offered only as a suggestion. In current usage, however, p≤0.05 is often misunderstood as a rigid threshold, sometimes with a misguided ‘win’ (p≤0.05) or ‘lose’ (p>0.05) approach. Also, in contemporary genomic studies, a threshold of p≤10−8 has been endorsed as a boundary for statistical significance when analyzing numerous genetic comparisons for each participant. A value of p≤0.05, or other thresholds, should not be employed reflexively to determine whether a clinical research investigation is trustworthy from a scientific perspective. Rather, and in parallel with conceptual issues of validity and generalizability, quantitative results should be interpreted using a combined assessment of strength of association, p values, CIs, and sample size.

Sign in / Sign up

Export Citation Format

Share Document