scholarly journals When the Numbers Do Not Add Up: The Practical Limits of Stochastologicals for Soft Psychology

2021 ◽  
pp. 174569162097055
Author(s):  
Nick J. Broers
Keyword(s):  
High Risk ◽  
Null Hypothesis ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Psychological Science ◽  
Body Of Knowledge ◽  
Psychological Theories ◽  
Time Continuum

One particular weakness of psychology that was left implicit by Meehl is the fact that psychological theories tend to be verbal theories, permitting at best ordinal predictions. Such predictions do not enable the high-risk tests that would strengthen our belief in the verisimilitude of theories but instead lead to the practice of null-hypothesis significance testing, a practice Meehl believed to be a major reason for the slow theoretical progress of soft psychology. The rising popularity of meta-analysis has led some to argue that we should move away from significance testing and focus on the size and stability of effects instead. Proponents of this reform assume that a greater emphasis on quantity can help psychology to develop a cumulative body of knowledge. The crucial question in this endeavor is whether the resulting numbers really have theoretical meaning. Psychological science lacks an undisputed, preexisting domain of observations analogous to the observations in the space-time continuum in physics. It is argued that, for this reason, effect sizes do not really exist independently of the adopted research design that led to their manifestation. Consequently, they can have no bearing on the verisimilitude of a theory.

scholarly journals When the numbers do not add up: the practical limits of stochastologicals for soft psychology

2021 ◽  
Author(s):  
Nick J. Broers
Keyword(s):  
High Risk ◽  
Null Hypothesis ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Psychological Science ◽  
Body Of Knowledge ◽  
Psychological Theories ◽  
Time Continuum

One particular weakness of psychology that was left implicit by Meehl (1978) is the fact that psychological theories tend to be verbal theories, permitting at best ordinal predictions. Such predictions do not enable the high risk tests that would strengthen our belief in the verisimilitude of theories but instead lead to the practice of null hypothesis significance testing, a practice Meehl believed to be a major reason for the slow theoretical progress of soft psychology. The rising popularity of meta-analysis has led some to argue that we should move away from significance testing and focus on the size and stability of effects instead. Proponents of this reform assume that a greater emphasis on quantity can help psychology to develop a cumulative body of knowledge. The crucial question in this endeavor is whether the resulting numbers really have theoretical meaning. Psychological science lacks an undisputed, pre-existing domain of observations analogous to the observations in the space-time continuum in physics. It is argued that for this reason effect sizes do not really exist independently of the adopted research design that led to their manifestation. Consequently, they can have no bearing on the verisimilitude of a theory.

The Rules of the Game Called Psychological Science

2012 ◽  
Vol 7 (6) ◽  
pp. 543-554 ◽  
Author(s):  
Marjan Bakker ◽  
Annette van Dijk ◽  
Jelte M. Wicherts
Keyword(s):  
Null Hypothesis ◽  
Research Strategy ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Questionable Research Practices ◽  
Psychological Science ◽  
Track Record ◽  
Journal Policies ◽  
Meta Analyses ◽  
Rules Of The Game

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.

Statistical significance testing, hypothetico-deductive method, and theory evaluation

2000 ◽  
Vol 23 (2) ◽  
pp. 292-293 ◽  
Author(s):  
Brian D. Haig
Keyword(s):  
Null Hypothesis ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Statistical Significance Testing ◽  
Theory Evaluation ◽  
Limited Role ◽  
Deductive Method

Chow's endorsement of a limited role for null hypothesis significance testing is a needed corrective of research malpractice, but his decision to place this procedure in a hypothetico-deductive framework of Popperian cast is unwise. Various failures of this version of the hypothetico-deductive method have negative implications for Chow's treatment of significance testing, meta-analysis, and theory evaluation.

Confidence Intervals

2009 ◽  
Vol 217 (1) ◽  
pp. 15-26 ◽  
Author(s):  
Geoff Cumming ◽  
Fiona Fidler
Keyword(s):  
Confidence Interval ◽  
Confidence Intervals ◽  
Null Hypothesis ◽  
Prediction Intervals ◽  
Effect Sizes ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Standardized Effect Sizes ◽  
Mainstream Science

Most questions across science call for quantitative answers, ideally, a single best estimate plus information about the precision of that estimate. A confidence interval (CI) expresses both efficiently. Early experimental psychologists sought quantitative answers, but for the last half century psychology has been dominated by the nonquantitative, dichotomous thinking of null hypothesis significance testing (NHST). The authors argue that psychology should rejoin mainstream science by asking better questions – those that demand quantitative answers – and using CIs to answer them. They explain CIs and a range of ways to think about them and use them to interpret data, especially by considering CIs as prediction intervals, which provide information about replication. They explain how to calculate CIs on means, proportions, correlations, and standardized effect sizes, and illustrate symmetric and asymmetric CIs. They also argue that information provided by CIs is more useful than that provided by p values, or by values of Killeen’s prep, the probability of replication.

scholarly journals Adopting a Meta-Generative Way of Thinking in the Field of Education via the Use of Bayesian Methods: A Multimethod Approach in a Post-Truth and COVID-19 Era1

2020 ◽  
pp. 4-19 ◽  
Author(s):  
Prathiba Natesan Batley ◽  
Peter Boedeker ◽  
Anthony J. Onwuegbuzie
Keyword(s):  
Bayesian Methods ◽  
Null Hypothesis ◽  
Quantitative Research ◽  
Meta Analysis ◽  
Research Synthesis ◽  
Research Data ◽  
Policy Implications ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Multimethod Approach

In this editorial, we introduce the multimethod concept of thinking meta-generatively, which we define as directly integrating findings from the extant literature during the data collection, analysis, and interpretation phases of primary studies. We demonstrate that meta-generative thinking goes further than do other research synthesis techniques (e.g., meta-analysis) because it involves meta-synthesis not only across studies but also within studies—thereby representing a multimethod approach. We describe how meta-generative thinking can be maximized/optimized with respect to quantitative research data/findings via the use of Bayesian methodology that has been shown to be superior to the inherently flawed null hypothesis significance testing. We contend that Bayesian meta-generative thinking is essential, given the potential for divisiveness and far-reaching sociopolitical, educational, and health policy implications of findings that lack generativity in a post-truth and COVID-19 era.

Things We Still Haven’t Learned (So Far)

2015 ◽  
Vol 37 (4) ◽  
pp. 449-461 ◽  
Author(s):  
Andreas Ivarsson ◽  
Mark B. Andersen ◽  
Andreas Stenling ◽  
Urban Johnson ◽  
Magnus Lindwall
Keyword(s):  
Bayesian Statistics ◽  
Null Hypothesis ◽  
Historical Background ◽  
Effect Sizes ◽  
Significance Testing ◽  
Exercise Psychology ◽  
Null Hypothesis Significance Testing ◽  
The Past ◽  
The Common ◽  
Sport And Exercise

Null hypothesis significance testing (NHST) is like an immortal horse that some researchers have been trying to beat to death for over 50 years but without any success. In this article we discuss the flaws in NHST, the historical background in relation to both Fisher’s and Neyman and Pearson’s statistical ideas, the common misunderstandings of what p < 05 actually means, and the 2010 APA publication manual’s clear, but most often ignored, instructions to report effect sizes and to interpret what they all mean in the real world. In addition, we discuss how Bayesian statistics can be used to overcome some of the problems with NHST. We then analyze quantitative articles published over the past three years (2012–2014) in two top-rated sport and exercise psychology journals to determine whether we have learned what we should have learned decades ago about our use and meaningful interpretations of statistics.

scholarly journals The null hypothesis is always rejected with statistical tricks: Why do you need it?

2019 ◽  
Vol 53 (1) ◽  
pp. 17-27
Author(s):  
Freddy A. Paniagua
Keyword(s):  
Null Hypothesis ◽  
Statistical Significance ◽  
Effect Sizes ◽  
Practical Significance ◽  
Significance Testing ◽  
Behavioral Sciences ◽  
Null Hypothesis Significance Testing ◽  
Very High

Ferguson (2015) observed that the proportion of studies supporting the experimental hypothesis and rejecting the null hypothesis is very high. This paper argues that the reason for this scenario is that researchers in the behavioral sciences have learned that the null hypothesis can always be rejected if one knows the statistical tricks to reject it (e.g., the probability of rejecting the null hypothesis increases with p = 0.05 compare to p = 0.01). Examples of the advancement of science without the need to formulate the null hypothesis are also discussed, as well as alternatives to null hypothesis significance testing-NHST (e.g., effect sizes), and the importance to distinguish the statistical significance from the practical significance of results.  

The logic of null hypothesis testing

1998 ◽  
Vol 21 (2) ◽  
pp. 197-198 ◽  
Author(s):  
Edward Erwin
Keyword(s):  
Hypothesis Testing ◽  
Null Hypothesis ◽  
Power Analysis ◽  
Meta Analysis ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Null Hypothesis Testing

In this commentary, I agree with Chow's treatment of null hypothesis significance testing as a noninferential procedure. However, I dispute his reconstruction of the logic of theory corroboration. I also challenge recent criticisms of NHSTP based on power analysis and meta-analysis.

Invasive Plant Researchers Should Calculate Effect Sizes, Not P-Values

2010 ◽  
Vol 3 (2) ◽  
pp. 106-112 ◽  
Author(s):  
Matthew J. Rinella ◽  
Jeremy J. James
Keyword(s):  
Confidence Intervals ◽  
Null Hypothesis ◽  
Invasive Plant ◽  
Effect Sizes ◽  
Significance Testing ◽  
Research Articles ◽  
Plant Science ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Size Estimates

AbstractNull hypothesis significance testing (NHST) forms the backbone of statistical inference in invasive plant science. Over 95% of research articles in Invasive Plant Science and Management report NHST results such as P-values or statistics closely related to P-values such as least significant differences. Unfortunately, NHST results are less informative than their ubiquity implies. P-values are hard to interpret and are regularly misinterpreted. Also, P-values do not provide estimates of the magnitudes and uncertainties of studied effects, and these effect size estimates are what invasive plant scientists care about most. In this paper, we reanalyze four datasets (two of our own and two of our colleagues; studies put forth as examples in this paper are used with permission of their authors) to illustrate limitations of NHST. The re-analyses are used to build a case for confidence intervals as preferable alternatives to P-values. Confidence intervals indicate effect sizes, and compared to P-values, confidence intervals provide more complete, intuitively appealing information on what data do/do not indicate.

Sign in / Sign up

Export Citation Format

Share Document