Things We Still Haven’t Learned (So Far)

2015 ◽  
Vol 37 (4) ◽  
pp. 449-461 ◽  
Author(s):  
Andreas Ivarsson ◽  
Mark B. Andersen ◽  
Andreas Stenling ◽  
Urban Johnson ◽  
Magnus Lindwall
Keyword(s):  
Bayesian Statistics ◽  
Null Hypothesis ◽  
Historical Background ◽  
Effect Sizes ◽  
Significance Testing ◽  
Exercise Psychology ◽  
Null Hypothesis Significance Testing ◽  
The Past ◽  
The Common ◽  
Sport And Exercise

Null hypothesis significance testing (NHST) is like an immortal horse that some researchers have been trying to beat to death for over 50 years but without any success. In this article we discuss the flaws in NHST, the historical background in relation to both Fisher’s and Neyman and Pearson’s statistical ideas, the common misunderstandings of what p < 05 actually means, and the 2010 APA publication manual’s clear, but most often ignored, instructions to report effect sizes and to interpret what they all mean in the real world. In addition, we discuss how Bayesian statistics can be used to overcome some of the problems with NHST. We then analyze quantitative articles published over the past three years (2012–2014) in two top-rated sport and exercise psychology journals to determine whether we have learned what we should have learned decades ago about our use and meaningful interpretations of statistics.

scholarly journals Scientific Self-Correction: The Bayesian Way

2019 ◽  
Author(s):  
Felipe Romero ◽  
Jan Sprenger
Keyword(s):  
Bayesian Inference ◽  
Bayesian Statistics ◽  
Experimental Research ◽  
Null Hypothesis ◽  
Effect Sizes ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Scientific Disciplines ◽  
Replication Crisis ◽  
Reliable Knowledge

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to self-correct its findings and to produce reliable knowledge. Amongst a variety of possible methodological, social, and statistical reforms to address the crisis, we focus on replacing null hypothesis significance testing (NHST) with Bayesian inference. On the basis of a simulation study for meta-analytic aggregation of effect sizes, we study the relative advantages of this Bayesian reform, and its interaction with widespread limitations in experimental research. Moving to Bayesian statistics will not solve the replication crisis single-handely, but would eliminate important sources of effect size overestimation for the conditions we study.

scholarly journals Why Did It Take So Many Decades for the Behavioral Sciences to Develop a Sense of Crisis Around Methodology and Replication?

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Andrew Gelman ◽  
Simine Vazire
Keyword(s):  
Null Hypothesis ◽  
Empirical Support ◽  
Full Scale ◽  
Significance Testing ◽  
Behavioral Sciences ◽  
Null Hypothesis Significance Testing ◽  
The Past ◽  
Replication Crisis ◽  
The Common

For several decades, leading behavioral scientists have offered strong criticisms of the common practice of null hypothesis significance testing as producing spurious findings without strong theoretical or empirical support. But only in the past decade has this manifested as a full-scale replication crisis. We consider some possible reasons why, on or about December 2010, the behavioral sciences changed.

Confidence Intervals

2009 ◽  
Vol 217 (1) ◽  
pp. 15-26 ◽  
Author(s):  
Geoff Cumming ◽  
Fiona Fidler
Keyword(s):  
Confidence Interval ◽  
Confidence Intervals ◽  
Null Hypothesis ◽  
Prediction Intervals ◽  
Effect Sizes ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Standardized Effect Sizes ◽  
Mainstream Science

Most questions across science call for quantitative answers, ideally, a single best estimate plus information about the precision of that estimate. A confidence interval (CI) expresses both efficiently. Early experimental psychologists sought quantitative answers, but for the last half century psychology has been dominated by the nonquantitative, dichotomous thinking of null hypothesis significance testing (NHST). The authors argue that psychology should rejoin mainstream science by asking better questions – those that demand quantitative answers – and using CIs to answer them. They explain CIs and a range of ways to think about them and use them to interpret data, especially by considering CIs as prediction intervals, which provide information about replication. They explain how to calculate CIs on means, proportions, correlations, and standardized effect sizes, and illustrate symmetric and asymmetric CIs. They also argue that information provided by CIs is more useful than that provided by p values, or by values of Killeen’s prep, the probability of replication.

scholarly journals The Bayesian Inferential Paradigm in Archaeology

2021 ◽  
Author(s):  
Erik Otarola-Castillo ◽  
Meissa G Torquato ◽  
Caitlin E. Buck
Keyword(s):  
Bayesian Statistics ◽  
Null Hypothesis ◽  
Radiocarbon Dating ◽  
Relevant Literature ◽  
Significance Testing ◽  
Simple Explanation ◽  
Spatial Analyses ◽  
Null Hypothesis Significance Testing ◽  
Statistical Framework ◽  
Research Problems

Archaeologists often use data and quantitative statistical methods to evaluate their ideas. Although there are various statistical frameworks for decision-making in archaeology and science in general, in this chapter, we provide a simple explanation of Bayesian statistics. To contextualize the Bayesian statistical framework, we briefly compare it to the more widespread null hypothesis significance testing (NHST) approach. We also provide a simple example to illustrate how archaeologists use data and the Bayesian framework to compare hypotheses and evaluate their uncertainty. We then review how archaeologists have applied Bayesian statistics to solve research problems related to radiocarbon dating and chronology, lithic, ceramic, zooarchaeological, bioarchaeological, and spatial analyses. Because recent work has reviewed Bayesian applications in archaeology from the 1990s up to 2017, this work considers the relevant literature published since 2017.

scholarly journals The null hypothesis is always rejected with statistical tricks: Why do you need it?

2019 ◽  
Vol 53 (1) ◽  
pp. 17-27
Author(s):  
Freddy A. Paniagua
Keyword(s):  
Null Hypothesis ◽  
Statistical Significance ◽  
Effect Sizes ◽  
Practical Significance ◽  
Significance Testing ◽  
Behavioral Sciences ◽  
Null Hypothesis Significance Testing ◽  
Very High

Ferguson (2015) observed that the proportion of studies supporting the experimental hypothesis and rejecting the null hypothesis is very high. This paper argues that the reason for this scenario is that researchers in the behavioral sciences have learned that the null hypothesis can always be rejected if one knows the statistical tricks to reject it (e.g., the probability of rejecting the null hypothesis increases with p = 0.05 compare to p = 0.01). Examples of the advancement of science without the need to formulate the null hypothesis are also discussed, as well as alternatives to null hypothesis significance testing-NHST (e.g., effect sizes), and the importance to distinguish the statistical significance from the practical significance of results.  

Invasive Plant Researchers Should Calculate Effect Sizes, Not P-Values

2010 ◽  
Vol 3 (2) ◽  
pp. 106-112 ◽  
Author(s):  
Matthew J. Rinella ◽  
Jeremy J. James
Keyword(s):  
Confidence Intervals ◽  
Null Hypothesis ◽  
Invasive Plant ◽  
Effect Sizes ◽  
Significance Testing ◽  
Research Articles ◽  
Plant Science ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Size Estimates

AbstractNull hypothesis significance testing (NHST) forms the backbone of statistical inference in invasive plant science. Over 95% of research articles in Invasive Plant Science and Management report NHST results such as P-values or statistics closely related to P-values such as least significant differences. Unfortunately, NHST results are less informative than their ubiquity implies. P-values are hard to interpret and are regularly misinterpreted. Also, P-values do not provide estimates of the magnitudes and uncertainties of studied effects, and these effect size estimates are what invasive plant scientists care about most. In this paper, we reanalyze four datasets (two of our own and two of our colleagues; studies put forth as examples in this paper are used with permission of their authors) to illustrate limitations of NHST. The re-analyses are used to build a case for confidence intervals as preferable alternatives to P-values. Confidence intervals indicate effect sizes, and compared to P-values, confidence intervals provide more complete, intuitively appealing information on what data do/do not indicate.

Bayesian Statistics in Archaeology

2018 ◽  
Vol 47 (1) ◽  
pp. 435-453 ◽  
Author(s):  
Erik Otárola-Castillo ◽  
Melissa G. Torquato
Keyword(s):  
Bayesian Inference ◽  
Bayesian Statistics ◽  
Bayesian Methods ◽  
Null Hypothesis ◽  
Significance Testing ◽  
Archaeological Research ◽  
Historical Overview ◽  
Spatial Analyses ◽  
Null Hypothesis Significance Testing ◽  
Statistical Framework

Null hypothesis significance testing (NHST) is the most common statistical framework used by scientists, including archaeologists. Owing to increasing dissatisfaction, however, Bayesian inference has become an alternative to these methods. In this article, we review the application of Bayesian statistics to archaeology. We begin with a simple example to demonstrate the differences in applying NHST and Bayesian inference to an archaeological problem. Next, we formally define NHST and Bayesian inference, provide a brief historical overview of their development, and discuss the advantages and limitations of each method. A review of Bayesian inference and archaeology follows, highlighting the applications of Bayesian methods to chronological, bioarchaeological, zooarchaeological, ceramic, lithic, and spatial analyses. We close by considering the future applications of Bayesian statistics to archaeological research.

scholarly journals When the Numbers Do Not Add Up: The Practical Limits of Stochastologicals for Soft Psychology

2021 ◽  
pp. 174569162097055
Author(s):  
Nick J. Broers
Keyword(s):  
High Risk ◽  
Null Hypothesis ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Psychological Science ◽  
Body Of Knowledge ◽  
Psychological Theories ◽  
Time Continuum

One particular weakness of psychology that was left implicit by Meehl is the fact that psychological theories tend to be verbal theories, permitting at best ordinal predictions. Such predictions do not enable the high-risk tests that would strengthen our belief in the verisimilitude of theories but instead lead to the practice of null-hypothesis significance testing, a practice Meehl believed to be a major reason for the slow theoretical progress of soft psychology. The rising popularity of meta-analysis has led some to argue that we should move away from significance testing and focus on the size and stability of effects instead. Proponents of this reform assume that a greater emphasis on quantity can help psychology to develop a cumulative body of knowledge. The crucial question in this endeavor is whether the resulting numbers really have theoretical meaning. Psychological science lacks an undisputed, preexisting domain of observations analogous to the observations in the space-time continuum in physics. It is argued that, for this reason, effect sizes do not really exist independently of the adopted research design that led to their manifestation. Consequently, they can have no bearing on the verisimilitude of a theory.

Sign in / Sign up

Export Citation Format

Share Document