Why p-values are not measures of evidence

The recommendations by Muff and colleagues are an incoherent approach to statistical inferences, and should only be used if one wants to signal a misunderstanding of p-values. Coherent alternatives to quantify evidence exist, such as likelihoods and Bayes factors. Therefore, researchers should not follow the recommendation by Muff and colleagues to report p = 0.08 as ‘weak evidence’, p = 0.03 as ‘moderate evidence’, and p = 0.168 as ‘no evidence’.

Bayes factors for mixed-effects models

In psychology, we often want to know whether or not an effect exists. The traditional way of answering this question is to use frequentist statistics. However, a significance test against a null hypothesis of no effect cannot distinguish between two states of affairs: evidence of absence of an effect, and absence of evidence for or against an effect. Bayes factors can make this distinction; however, uptake of Bayes factors in psychology has so far been low for two reasons. Firstly, they require researchers to specify the range of effect sizes their theory predicts. Researchers are often unsure about how to do this, leading to the use of inappropriate default values which may give misleading results. Secondly, many implementations of Bayes factors have a substantial technical learning curve. We present a case study and simulations demonstrating a simple method for generating a range of plausible effect sizes based on the output from frequentist mixed-effects models. Bayes factors calculated using these estimates provide intuitively reasonable results across a range of real effect sizes. The approach provides a solution to the problem of how to come up with principled estimates of effect size, and produces comparable results to a state-of-the-art method without requiring researchers to learn novel statistical software.

The Importance of Random Slopes in Mixed Models for Bayesian Hypothesis Testing

Mixed models are gaining popularity in psychology. For frequentist mixed models, Barr, Levy, Scheepers, and Tily (2013) showed that excluding random slopes – differences between individuals in the direction and size of an effect – from a model when they are in the data can lead to a substantial increase in false-positive conclusions in null-hypothesis tests. Here I demonstrate through five simulations that the same is true for Bayesian hypothesis testing with mixed models, often yielding Bayes factors reflecting very strong evidence for a mean effect on the population level even if there was no such effect. Including random slopes in the model largely eliminates the risk of strong false positives, but reduces the chance of obtaining strong evidence for true effects. I recommend starting analysis with testing the support for random slopes in the data, and removing them from the models only if there is clear evidence against them.

The Use of Questionable Research Practices to Survive in Academia Examined With Expert Elicitation, Prior-Data Conflicts, Bayes Factors for Replication Effects, and the Bayes Truth Serum

The popularity and use of Bayesian methods have increased across many research domains. The current article demonstrates how some less familiar Bayesian methods can be used. Specifically, we applied expert elicitation, testing for prior-data conflicts, the Bayesian Truth Serum, and testing for replication effects via Bayes Factors in a series of four studies investigating the use of questionable research practices (QRPs). Scientifically fraudulent or unethical research practices have caused quite a stir in academia and beyond. Improving science starts with educating Ph.D. candidates: the scholars of tomorrow. In four studies concerning 765 Ph.D. candidates, we investigate whether Ph.D. candidates can differentiate between ethical and unethical or even fraudulent research practices. We probed the Ph.D.s’ willingness to publish research from such practices and tested whether this is influenced by (un)ethical behavior pressure from supervisors or peers. Furthermore, 36 academic leaders (deans, vice-deans, and heads of research) were interviewed and asked to predict what Ph.D.s would answer for different vignettes. Our study shows, and replicates, that some Ph.D. candidates are willing to publish results deriving from even blatant fraudulent behavior–data fabrication. Additionally, some academic leaders underestimated this behavior, which is alarming. Academic leaders have to keep in mind that Ph.D. candidates can be under more pressure than they realize and might be susceptible to using QRPs. As an inspiring example and to encourage others to make their Bayesian work reproducible, we published data, annotated scripts, and detailed output on the Open Science Framework (OSF).

Computing Bayes factors from data with missing values

The Bayes factor is increasingly used for the evaluation of hypotheses. These may betraditional hypotheses specified using equality constraints among the parameters of thestatistical model of interest or informative hypotheses specified using equality andinequality constraints. So far no attention has been given to the computation of Bayesfactors from data with missing values. A key property of such a Bayes factor should bethat it is only based on the information in the observed values. This paper will show thatsuch a Bayes factor can be obtained using multiple imputations of the missing values.

Staying informed without a cost: No effect of positive news media on stress reactivity, memory and affect in young adults

Introduction We read, see and hear news from various media sources every day. A large majority of the news is negative. A previous study from our laboratory showed that reading negative news is associated with both increased stress reactivity (measured via the stress hormone cortisol) and recall of the negative news segments in women. Objectives The present study investigated the effects of positive news on cortisol stress reactivity, memory and affect using a methodology highly similar to the study on negative news that was previously used by our team. Methods Sixty-two healthy participants aged between 18 and 35 years (81% women) were randomly exposed to either positive or neutral news segments, followed by a laboratory stressor. We assessed participants’ affect three times during the procedure and measured cortisol in saliva eight times (at 10-minute intervals). Twenty-four hours later, participants were contacted by phone to assess their recall of the news segments. Results Results showed that exposure to positive news, relative to neutral news, did not modulate participants’ cortisol levels in response to the laboratory stressor. Positive news had no impact on memory recall of the news and did not change participants’ positive or negative affect. Bayes factors suggested that these nonsignificant results are not attributable to low statistical power. Conclusion Contrary to negative news, positive and neutral news do not modulate stress reactivity, memory and affect. These results suggest that people can stay informed without physiological and psychological costs when the news to which they are exposed adopt a positive or neutral approach.

Bayesian tests of symmetry for the generalized Von Mises distribution

Bayesian tests on the symmetry of the generalized von Mises model for planar directions (Gatto and Jammalamadaka in Stat Methodol 4(3):341–353, 2007) are introduced. The generalized von Mises distribution is a flexible model that can be axially symmetric or asymmetric, unimodal or bimodal. A characterization of axial symmetry is provided and taken as null hypothesis for one of the proposed Bayesian tests. The Bayesian tests are obtained by the technique of probability perturbation. The prior probability measure is perturbed so to give a positive prior probability to the null hypothesis, which would be null otherwise. This allows for the derivation of simple computational formulae for the Bayes factors. Numerical results reveal that, whenever the simulation scheme of the samples supports the null hypothesis, the null posterior probabilities appear systematically larger than their prior counterpart.

CrossFit® Open Performance is Affected by the Nature of Past Competition Experiences

Purpose: To examine the relationships between past competition performances and 2020 CrossFit® Open (CFO) performance. Methods: A random selection from the top one thousand athletes (n = 220, 28.5 ± 4.4 years, 178 ± 7 cm, 87.5 ± 10.2 kg) were selected for this study. Overall and weekly performances (including ranks and scores) of the 2020 CFO, as well as overall ranks from previous CFO, regional, and Games™ competitions, were recorded from their publicly available online profile. The highest, lowest, average, and standard deviation (SD) of past rankings, as well as participation statistics (i.e., years since first appearance, total and consecutive appearances, and participation rate), were calculated for each competition stage. Relationships were then assessed between 2020 CFO performance and all past competition experience variables by calculating Kendall’s tau (τ) correlation coefficients and Bayes factors (BF10). Results: Overall and weekly ranking of the 2020 CFO was extremely favored (p < 0.001, BF10 > 100) to be related to the athlete’s highest previous CFO rank (τ = 0.26 – 0.39) and individual regional appearances (τ = –0.26 to –0.34), as well as individual Games™ appearances (overall and for weeks 1, 3, and 4; τ = –0.20 to –0.22, p < 0.001, BF10 > 100). Evidence for all other significant relationships ranged from moderate to very strong (p < 0.05, BF10 = 3 – 100) and varied among specific 2020 CFO workouts. Few associations were noted for team competition experience, and these were generally limited to Games™ appearances (τ = –0.12 to –0.18, p < 0.05, BF10 = 3.3 – 100). Conclusions: Although specific relationships were found between 2020 CFO performance and individual appearances at regional and Games™ competitions, the most consistent relationships were seen with participation and ranking in past CFO competitions.