P-Curve Analysis of the Köhler Motivation Gain Effect in Exercise Settings: A Demonstration of a Novel Technique to Estimate Evidential Value Across Multiple Studies

2020 ◽  
Author(s):  
Christopher R Hill ◽  
Stephen Samendinger ◽  
Amanda M Rymal
Keyword(s):  
Curve Analysis ◽  
Selective Reporting ◽  
Group Dynamic ◽  
Evidential Value ◽  
Dynamic Motivation ◽  
Single Effect ◽  
Kohler Effect ◽  
Motivation Effect ◽  
Motivation Gain ◽  
Size Estimates

Abstract Background Practitioners and researchers may not always be able to adequately evaluate the evidential value of findings from a series of independent studies. This is partially due to the possibility of inflated effect size estimates for these findings as a result of researcher manipulation or selective reporting of analyses (i.e., p-hacking). In light of the possible overestimation of effect sizes in the literature, the p-curve analysis has been proposed as a worthwhile tool that may help identify bias across a series of studies focused on a single effect. The p-curve analysis provides a measure of the evidential value in the published literature and might highlight p-hacking practices. Purpose Therefore, the purpose of this paper is to introduce the mechanics of the p-curve analysis to individuals researching phenomena in the psychosocial aspects of behavior and provide a substantive example of a p-curve analysis using findings from a series of studies examining a group dynamic motivation gain paradigm. Methods We performed a p-curve analysis on a sample of 13 studies that examined the Köhler motivation gain effect in exercise settings as a means to instruct readers how to conduct such an analysis on their own. Results The p-curve for studies examining the Köhler effect demonstrated evidential value and that this motivation effect is likely not a byproduct of p-hacking. The p-curve analysis is explained, as well as potential limitations of the analysis, interpretation of the results, and other uses where a p-curve analysis could be implemented.

Power-Enhanced Funnel Plots for Meta-Analysis

2020 ◽  
Vol 228 (1) ◽  
pp. 43-49 ◽  
Author(s):  
Michael Kossmeier ◽  
Ulrich S. Tran ◽  
Martin Voracek
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Relevant Information ◽  
Selective Reporting ◽  
Graphical Display ◽  
Funnel Plot ◽  
Evidential Value ◽  
Graphical Displays ◽  
Funnel Plots ◽  
Meta Analyses

Abstract. Currently, dedicated graphical displays to depict study-level statistical power in the context of meta-analysis are unavailable. Here, we introduce the sunset (power-enhanced) funnel plot to visualize this relevant information for assessing the credibility, or evidential value, of a set of studies. The sunset funnel plot highlights the statistical power of primary studies to detect an underlying true effect of interest in the well-known funnel display with color-coded power regions and a second power axis. This graphical display allows meta-analysts to incorporate power considerations into classic funnel plot assessments of small-study effects. Nominally significant, but low-powered, studies might be seen as less credible and as more likely being affected by selective reporting. We exemplify the application of the sunset funnel plot with two published meta-analyses from medicine and psychology. Software to create this variation of the funnel plot is provided via a tailored R function. In conclusion, the sunset (power-enhanced) funnel plot is a novel and useful graphical display to critically examine and to present study-level power in the context of meta-analysis.

scholarly journals P-Curve analysis of autonomous and controlling motivation priming effects supports their evidential value

2021 ◽  
Author(s):  
Stephen L. Murphy ◽  
Richard P. Steel
Keyword(s):  
Literature Search ◽  
Population Level ◽  
False Positives ◽  
Curve Analysis ◽  
Self Determination ◽  
Systematic Literature Search ◽  
Priming Effects ◽  
Extant Literature ◽  
Evidential Value ◽  
Prior Literature

AbstractExtant literature consistently demonstrates the level of self-determination individuals experience or demonstrate during an activity can be primed. However, considering most of this literature comes from a period wherein p-hacking was prevalent (pre-2015), it may be that these effects reflect false positives. The aim of the present study was to investigate whether published literature showing autonomous and controlling motivation priming effects contain evidential value or not. A systematic literature search was conducted to identify relevant priming research, while set rules determined which effects from each study would be used in p-curve analysis. Two p-curves including 33 effects each were constructed. P-curve analyses, even after excluding surprising effects (e.g., effects large in magnitude), demonstrated that literature showing autonomous and controlling motivation priming effects contained evidential value. The present findings support prior literature suggesting the effects of autonomous and controlling motivation primes exist at the population level. They also reduce (but do not eliminate) concerns from broader psychology that p-hacking may underlie reported effects.

scholarly journals Problems in using p-curve analysis and text-mining to detect rate of p-hacking

2015 ◽  
Author(s):  
Dorothy V Bishop ◽  
Paul A Thompson
Keyword(s):  
Text Mining ◽  
Curve Analysis ◽  
P Values ◽  
Evidential Value ◽  
Selection Of Variables ◽  
Dependent Variables ◽  
Selection Of ◽  
The Way

Background: The p-curve is a plot of the distribution of p-values below .05 reported in a set of scientific studies. Comparisons between ranges of p-values have been used to evaluate fields of research in terms of the extent to which studies have genuine evidential value, and the extent to which they suffer from bias in the selection of variables and analyses for publication, p-hacking. We argue that binomial tests on the p-curve are not robust enough to be used for this purpose. Methods: P-hacking can take various forms. Here we used R code to simulate the use of ghost variables, where an experimenter gathers data on several dependent variables but reports only those with statistically significant effects. We also examined a text-mined dataset used by Head et al. (2015) and assessed its suitability for investigating p-hacking. Results: We first show that a p-curve suggestive of p-hacking can be obtained if researchers misapply parametric tests to data that depart from normality, even when no p-hacking occurs. We go on to show that when there is ghost p-hacking, the shape of the p-curve depends on whether dependent variables are intercorrelated. For uncorrelated variables, simulated p-hacked data do not give the "p-hacking bump" just below .05 that is regarded as evidence of p-hacking, though there is a negative skew when simulated variables are inter-correlated. The way p-curves vary according to features of underlying data poses problems when automated text mining is used to detect p-values in heterogeneous sets of published papers. Conclusions: A significant bump in the p-curve just below .05 is not necessarily evidence of p-hacking, and lack of a bump is not indicative of lack of p-hacking. Furthermore, while studies with evidential value will usually generate a right-skewed p-curve, we cannot treat a right-skewed p-curve as an indicator of the extent of evidential value, unless we have a model specific to the type of p-values entered into the analysis. We conclude that it is not feasible to use the p-curve to estimate the extent of p-hacking and evidential value unless there is considerable control over the type of data entered into the analysis.

scholarly journals The differential calibration of the HPA axis as a function of trauma versus adversity: A systematic review and p-curve meta-analyses

2021 ◽  
Author(s):  
Niki H. Kamkar ◽  
Cassandra J Lowe ◽  
J. Bruce Morton
Keyword(s):  
Hpa Axis ◽  
Early Life ◽  
Life Experiences ◽  
Curve Analysis ◽  
Early Life Adversity ◽  
Cortisol Reactivity ◽  
Evidential Value ◽  
Dsm 5 ◽  
Meta Analyses ◽  
The Relationship

Although there is an abundance of evidence linking the function of the hypothalamic-pituitary-adrenal (HPA) axis to adverse early-life experiences, the precise nature of the association remains unclear. Some evidence suggests early-life adversity leads to cortisol hyper-reactivity, while other evidence suggests adversity leads to cortisol hypo-reactivity. Here, we distinguish between trauma and adversity, and use p-curves to interrogate the conflicting literature. In Study 1, trauma was operationalized according to DSM-5 criteria; the p-curve analysis included 68 articles and revealed that the literature reporting associations between trauma and blunted cortisol reactivity contains evidential value. Study 2 examined the relationship between adversity and cortisol reactivity. Thirty articles were included in the analysis, and p-curve demonstrated that adversity is related to heightened cortisol reactivity. These results support an inverted U-shaped function relating severity of adversity and cortisol reactivity, and underscore the importance of distinguishing between “trauma” and “adversity”.

scholarly journals The effect of reading a short passage of literary fiction on Theory of Mind: A replication of Kidd and Castano (2013)

2018 ◽  
Author(s):  
Iris van Kuijk ◽  
Peter Verkoeijen ◽  
Katinka Dijkstra ◽  
Rolf Antonius Zwaan
Keyword(s):  
Theory Of Mind ◽  
Meta Analysis ◽  
Good Practice ◽  
Reliability And Validity ◽  
Popular Fiction ◽  
Curve Analysis ◽  
Literary Fiction ◽  
Small Scale ◽  
Exclusion Criteria ◽  
Evidential Value

The results reported by Kidd and Castano (2013) indicated that reading a short passage of literary fiction improves theory of mind (ToM) relative to reading popular fiction. However, when we entered Kidd and Castano’s results in a p-curve analysis, it turned out that the evidential value of their findings is low. It is good practice to back up a p-curve analysis of a single paper with an adequately powered direct replication of at least one of the studies in the p-curve analysis. Therefore, we conducted a direct replication of the literary fiction condition and the popular fiction condition from Kidd and Castano’s Experiment 5 to scrutinize the effect of reading literary fiction on ToM. The results of this replication were largely consistent with Kidd and Castano’s original findings. Furthermore, we conducted a small-scale meta-analysis on the findings of the present study, those of Kidd and Castano and those reported in other published direct replications. The meta-analytic effect of reading literary fiction on ToM was small and non-significant but there was considerable heterogeneity between the included studies. The results of the present study and of the small-scale meta-analysis are discussed in the light of reading-times exclusion criteria as well as reliability and validity of ToM measured.

Assessing the evidence of perspective taking on stereotyping and negative evaluations: A p-curve analysis

2020 ◽  
pp. 136843022095708
Author(s):  
Qian Huang ◽  
Wei Peng ◽  
Jazmyne V. Simmons
Keyword(s):  
Social Science ◽  
Attitude Change ◽  
Perspective Taking ◽  
Statistical Power ◽  
Experimental Studies ◽  
Curve Analysis ◽  
Evidential Value ◽  
Intergroup Attitude ◽  
Science Disciplines

Perspective taking is conceptualized as the ability to consider or adopt the perspective of another individual who is perceived to be in need; it has shown mixed results in stereotype reduction and intergroup attitude change across many social science disciplines. The inconsistent results raise concerns about the robustness of the perspective-taking phenomenon. The present study uses p-curve analysis to examine whether evidential value existed among two sets of published experimental studies where perspective taking was operationalized in two different paradigms. Despite low statistical power, we found that both sets of studies revealed some evidential value of the effects of perspective taking. The theoretical and methodological implications of perspective-taking studies are discussed as well.

Sign in / Sign up

Export Citation Format

Share Document