scholarly journals Significance, truth and proof of p values: reminders about common misconceptions regarding null hypothesis significance testing

2013 ◽  
Vol 23 (1) ◽  
pp. 5-7 ◽  
Author(s):  
Mathilde G. E. Verdam ◽  
Frans J. Oort ◽  
Mirjam A. G. Sprangers
Keyword(s):  
Null Hypothesis ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Common Misconceptions

The Use and Misuse of Classical Statistics: A Primer for Social Workers

2021 ◽  
pp. 104973152110082
Author(s):  
Daniel J. Dunleavy ◽  
Jeffrey R. Lacasse
Keyword(s):  
Social Workers ◽  
Null Hypothesis ◽  
Effect Sizes ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Research And Practice ◽  
Classical Statistics ◽  
Frequentist Statistics ◽  
Common Misconceptions

In this article, we offer a primer on “classical” frequentist statistics. In doing so, we aim to (1) provide social workers with a nuanced overview of common statistical concepts and tools, (2) clarify ways in which these ideas have oft been misused or misinterpreted in research and practice, and (3) help social workers better understand what frequentist statistics can and cannot offer. We begin broadly, starting with foundational issues in the philosophy of statistics. Then, we outline the Fisherian and Neyman–Pearson approaches to statistical inference and the practice of null hypothesis significance testing. We then discuss key statistical concepts including α, power, p values, effect sizes, and confidence intervals, exploring several common misconceptions about their use and interpretation. We close by considering some limitations of frequentist statistics and by offering an opinionated discussion on how social workers may promote more fruitful, responsible, and thoughtful statistical practice.

scholarly journals A Frequentist Alternative to Significance Testing, p-Values, and Confidence Intervals

Econometrics ◽  
2019 ◽  
Vol 7 (2) ◽  
pp. 26 ◽  
Author(s):  
David Trafimow
Keyword(s):  
Present Article ◽  
Confidence Intervals ◽  
Null Hypothesis ◽  
A Priori ◽  
Significance Testing ◽  
Population Parameters ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Statistical Procedures ◽  
Major Section

There has been much debate about null hypothesis significance testing, p-values without null hypothesis significance testing, and confidence intervals. The first major section of the present article addresses some of the main reasons these procedures are problematic. The conclusion is that none of them are satisfactory. However, there is a new procedure, termed the a priori procedure (APP), that validly aids researchers in obtaining sample statistics that have acceptable probabilities of being close to their corresponding population parameters. The second major section provides a description and review of APP advances. Not only does the APP avoid the problems that plague other inferential statistical procedures, but it is easy to perform too. Although the APP can be performed in conjunction with other procedures, the present recommendation is that it be used alone.

scholarly journals METHODS, THEORY, AND INNOVATION A BAYESIAN ANALYSIS OF EVIDENCE IN SUPPORT OF THE NULL HYPOTHESIS IN GERONTOLOGICAL PSYCHOLOGY (OR LACK THEREOF)

2019 ◽  
Vol 3 (Supplement_1) ◽  
pp. S773-S773
Author(s):  
Christopher Brydges ◽  
Allison A Bielak
Keyword(s):  
Strong Evidence ◽  
Null Hypothesis ◽  
Research Method ◽  
Statistical Technique ◽  
Bayes Factors ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Bayesian Analyses ◽  
P Values ◽  
Gerontological Research

Abstract Objective: Non-significant p values derived from null hypothesis significance testing do not distinguish between true null effects or cases where the data are insensitive in distinguishing the hypotheses. This study aimed to investigate the prevalence of Bayesian analyses in gerontological psychology, a statistical technique that can distinguish between conclusive and inconclusive non-significant results, by using Bayes factors (BFs) to reanalyze non-significant results from published gerontological research. Method: Non-significant results mentioned in abstracts of articles published in 2017 volumes of ten top gerontological psychology journals were extracted (N = 409) and categorized based on whether Bayesian analyses were conducted. BFs were calculated from non-significant t-tests within this sample to determine how frequently the null hypothesis was strongly supported. Results: Non-significant results were directly tested with Bayes factors in 1.22% of studies. Bayesian reanalyses of 195 non-significant t-tests found that only 7.69% of the findings provided strong evidence in support of the null hypothesis. Conclusions: Bayesian analyses are rarely used in gerontological research, and a large proportion of null findings were deemed inconclusive when reanalyzed with BFs. Researchers are encouraged to use BFs to test the validity of non-significant results, and ensure that sufficient sample sizes are used so that the meaningfulness of null findings can be evaluated.

Confidence Intervals

2009 ◽  
Vol 217 (1) ◽  
pp. 15-26 ◽  
Author(s):  
Geoff Cumming ◽  
Fiona Fidler
Keyword(s):  
Confidence Interval ◽  
Confidence Intervals ◽  
Null Hypothesis ◽  
Prediction Intervals ◽  
Effect Sizes ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Standardized Effect Sizes ◽  
Mainstream Science

Most questions across science call for quantitative answers, ideally, a single best estimate plus information about the precision of that estimate. A confidence interval (CI) expresses both efficiently. Early experimental psychologists sought quantitative answers, but for the last half century psychology has been dominated by the nonquantitative, dichotomous thinking of null hypothesis significance testing (NHST). The authors argue that psychology should rejoin mainstream science by asking better questions – those that demand quantitative answers – and using CIs to answer them. They explain CIs and a range of ways to think about them and use them to interpret data, especially by considering CIs as prediction intervals, which provide information about replication. They explain how to calculate CIs on means, proportions, correlations, and standardized effect sizes, and illustrate symmetric and asymmetric CIs. They also argue that information provided by CIs is more useful than that provided by p values, or by values of Killeen’s prep, the probability of replication.

scholarly journals Experiences with Using Bayes Factors for Regression Analysis in Biostatistical Setting

2017 ◽  
Vol 61 (3) ◽  
pp. 246
Author(s):  
Tamás Ferenci ◽  
Levente Kovács
Keyword(s):  
Null Hypothesis ◽  
Strong Relationship ◽  
Medical Problem ◽  
Bayes Factors ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Large Samples ◽  
Sampling Variation ◽  
The Impact

Null hypothesis significance testing dominates the current biostatistical practice. However, this routine has many flaws, in particular p-values are very often misused and misinterpreted. Several solutions has been suggested to remedy this situation, the application of Bayes Factors being perhaps the most well-known. Nevertheless, even Bayes Factors are very seldom applied in medical research. This paper investigates the application of Bayes Factors in the analysis of a realistic medical problem using actual data from a representative US survey, and compares the results to those obtained with traditional means. Linear regression is used as an example as it is one of the most basic tools in biostatistics. The effect of sample size and sampling variation is investigated (with resampling) as well as the impact of the choice of prior. Results show that there is a strong relationship between p-values and Bayes Factors, especially for large samples. The application of Bayes Factors should be encouraged evenin spite of this, as the message they convey is much more instructive and scientifically correct than the current typical practice.

scholarly journals The frequent insignificance of a “significant” P-value

2021 ◽  
Author(s):  
David McGiffin ◽  
Geoff Cumming ◽  
Paul Myles
Keyword(s):  
Diagnostic Tests ◽  
Null Hypothesis ◽  
Open Science ◽  
Significance Testing ◽  
P Value ◽  
Conditional Probabilities ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Science Practices ◽  
Strength Of Evidence

Null hypothesis significance testing (NHST) and p-values are widespread in the cardiac surgical literature but are frequently misunderstood and misused. The purpose of the review is to discuss major disadvantages of p-values and suggest alternatives. We describe diagnostic tests, the prosecutor’s fallacy in the courtroom, and NHST, which involve inter-related conditional probabilities, to help clarify the meaning of p-values, and discuss the enormous sampling variability, or unreliability, of p-values. Finally, we use a cardiac surgical database and simulations to explore further issues involving p-values. In clinical studies, p-values provide a poor summary of the observed treatment effect, whereas the three- number summary provided by effect estimates and confidence intervals is more informative and minimises over-interpretation of a “significant” result. P-values are an unreliable measure of strength of evidence; if used at all they give only, at best, a very rough guide to decision making. Researchers should adopt Open Science practices to improve the trustworthiness of research and, where possible, use estimation (three-number summaries) or other better techniques.

scholarly journals A Bayesian Analysis of Evidence in Support of the Null Hypothesis in Gerontological Psychology (or Lack Thereof)

2019 ◽  
Author(s):  
Christopher Brydges
Keyword(s):  
Bayesian Analysis ◽  
Strong Evidence ◽  
Null Hypothesis ◽  
Statistical Technique ◽  
Bayes Factors ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Bayesian Analyses ◽  
P Values ◽  
Gerontological Research

Objective: Non-significant p values derived from null hypothesis significance testing do not distinguish between true null effects or cases where the data are insensitive in distinguishing the hypotheses. This study aimed to investigate the prevalence of Bayesian analyses in gerontological psychology, a statistical technique that can distinguish between conclusive and inconclusive non-significant results, by using Bayes factors (BFs) to reanalyze non-significant results from published gerontological research.Method: Non-significant results mentioned in abstracts of articles published in 2017 volumes of ten top gerontological psychology journals were extracted (N = 409) and categorized based on whether Bayesian analyses were conducted. BFs were calculated from non-significant t-tests within this sample to determine how frequently the null hypothesis was strongly supported.Results: Non-significant results were directly tested with Bayes factors in 1.22% of studies. Bayesian reanalyses of 195 non-significant ¬t-tests found that only 7.69% of the findings provided strong evidence in support of the null hypothesis.Conclusions: Bayesian analyses are rarely used in gerontological research, and a large proportion of null findings were deemed inconclusive when reanalyzed with BFs. Researchers are encouraged to use BFs to test the validity of non-significant results, and ensure that sufficient sample sizes are used so that the meaningfulness of null findings can be evaluated.

A peculiar prevalence of p values just below .05

2012 ◽  
Vol 65 (11) ◽  
pp. 2271-2279 ◽  
Author(s):  
E.J. Masicampo ◽  
Daniel R. Lalande
Keyword(s):  
Publication Bias ◽  
Null Hypothesis ◽  
Degrees Of Freedom ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Large Subset ◽  
Potential Sources ◽  
Psychology Literature

In null hypothesis significance testing (NHST), p values are judged relative to an arbitrary threshold for significance (.05). The present work examined whether that standard influences the distribution of p values reported in the psychology literature. We examined a large subset of papers from three highly regarded journals. Distributions of p were found to be similar across the different journals. Moreover, p values were much more common immediately below .05 than would be expected based on the number of p values occurring in other ranges. This prevalence of p values just below the arbitrary criterion for significance was observed in all three journals. We discuss potential sources of this pattern, including publication bias and researcher degrees of freedom.

Invasive Plant Researchers Should Calculate Effect Sizes, Not P-Values

2010 ◽  
Vol 3 (2) ◽  
pp. 106-112 ◽  
Author(s):  
Matthew J. Rinella ◽  
Jeremy J. James
Keyword(s):  
Confidence Intervals ◽  
Null Hypothesis ◽  
Invasive Plant ◽  
Effect Sizes ◽  
Significance Testing ◽  
Research Articles ◽  
Plant Science ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Size Estimates

AbstractNull hypothesis significance testing (NHST) forms the backbone of statistical inference in invasive plant science. Over 95% of research articles in Invasive Plant Science and Management report NHST results such as P-values or statistics closely related to P-values such as least significant differences. Unfortunately, NHST results are less informative than their ubiquity implies. P-values are hard to interpret and are regularly misinterpreted. Also, P-values do not provide estimates of the magnitudes and uncertainties of studied effects, and these effect size estimates are what invasive plant scientists care about most. In this paper, we reanalyze four datasets (two of our own and two of our colleagues; studies put forth as examples in this paper are used with permission of their authors) to illustrate limitations of NHST. The re-analyses are used to build a case for confidence intervals as preferable alternatives to P-values. Confidence intervals indicate effect sizes, and compared to P-values, confidence intervals provide more complete, intuitively appealing information on what data do/do not indicate.

Sign in / Sign up

Export Citation Format

Share Document