OPEN A NEW WINDOW IN RATIONAL RESEARCH PLANNING: ADJUST ALPHA TO MAXIMIZE STATISTICAL POWER

Traditionally, statistical power was viewed as relevant to research planning but not evaluation of completed research. However, following discussions of high false finding rates (FFRs) associated with low statistical power, the assumed level of statistical power has become a key criterion for research acceptability. Yet, the links between power and false findings are not as straightforward as described. Assumptions underlying FFR calculations do not reflect research realities in personality and social psychology. Even granting the assumptions, the FFR calculations identify important limitations to any general influences of statistical power. Limits for statistical power in inflating false findings can also be illustrated through the use of FFR calculations to (a) update beliefs about the null or alternative hypothesis and (b) assess the relative support for the null versus alternative hypothesis when evaluating a set of studies. Taken together, statistical power should be de-emphasized in comparison to current uses in research evaluation.

Download Full-text

Power-Enhanced Funnel Plots for Meta-Analysis

Zeitschrift für Psychologie ◽

10.1027/2151-2604/a000392 ◽

2020 ◽

Vol 228 (1) ◽

pp. 43-49 ◽

Cited By ~ 1

Author(s):

Michael Kossmeier ◽

Ulrich S. Tran ◽

Martin Voracek

Keyword(s):

Statistical Power ◽

Meta Analysis ◽

Relevant Information ◽

Selective Reporting ◽

Graphical Display ◽

Funnel Plot ◽

Evidential Value ◽

Graphical Displays ◽

Funnel Plots ◽

Meta Analyses

Abstract. Currently, dedicated graphical displays to depict study-level statistical power in the context of meta-analysis are unavailable. Here, we introduce the sunset (power-enhanced) funnel plot to visualize this relevant information for assessing the credibility, or evidential value, of a set of studies. The sunset funnel plot highlights the statistical power of primary studies to detect an underlying true effect of interest in the well-known funnel display with color-coded power regions and a second power axis. This graphical display allows meta-analysts to incorporate power considerations into classic funnel plot assessments of small-study effects. Nominally significant, but low-powered, studies might be seen as less credible and as more likely being affected by selective reporting. We exemplify the application of the sunset funnel plot with two published meta-analyses from medicine and psychology. Software to create this variation of the funnel plot is provided via a tailored R function. In conclusion, the sunset (power-enhanced) funnel plot is a novel and useful graphical display to critically examine and to present study-level power in the context of meta-analysis.

Download Full-text

How to Detect Publication Bias in Psychological Research

Zeitschrift für Psychologie ◽

10.1027/2151-2604/a000386 ◽

2019 ◽

Vol 227 (4) ◽

pp. 261-279 ◽

Cited By ~ 2

Author(s):

Frank Renkewitz ◽

Melanie Keiner

Keyword(s):

Publication Bias ◽

Effect Size ◽

Statistical Power ◽

Type I Error ◽

Psychological Research ◽

Type I ◽

True Effect Size ◽

Questionable Research Practices ◽

True Effect ◽

Meta Analyses

Abstract. Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates. Both of these problems lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect such bias in meta-analytic results. We present an evaluation of the performance of six of these tools. To assess the Type I error rate and the statistical power of these methods, we simulated a large variety of literatures that differed with regard to true effect size, heterogeneity, number of available primary studies, and sample sizes of these primary studies; furthermore, simulated studies were subjected to different degrees of publication bias. Our results show that across all simulated conditions, no method consistently outperformed the others. Additionally, all methods performed poorly when true effect sizes were heterogeneous or primary studies had a small chance of being published, irrespective of their results. This suggests that in many actual meta-analyses in psychology, bias will remain undiscovered no matter which detection method is used.

Download Full-text

Replication of the Superstition and Performance Study by

Social Psychology ◽

10.1027/1864-9335/a000190 ◽

2014 ◽

Vol 45 (3) ◽

pp. 239-245 ◽

Cited By ~ 18

Author(s):

Robert J. Calin-Jageman ◽

Tracy L. Caldwell

Keyword(s):

Task Difficulty ◽

Statistical Power ◽

Meta Analysis ◽

A Priori ◽

Significant Heterogeneity ◽

Performance Study ◽

Improve Performance ◽

Research Designs ◽

Series Of Experiments ◽

And Performance

A recent series of experiments suggests that fostering superstitions can substantially improve performance on a variety of motor and cognitive tasks ( Damisch, Stoberock, & Mussweiler, 2010 ). We conducted two high-powered and precise replications of one of these experiments, examining if telling participants they had a lucky golf ball could improve their performance on a 10-shot golf task relative to controls. We found that the effect of superstition on performance is elusive: Participants told they had a lucky ball performed almost identically to controls. Our failure to replicate the target study was not due to lack of impact, lack of statistical power, differences in task difficulty, nor differences in participant belief in luck. A meta-analysis indicates significant heterogeneity in the effect of superstition on performance. This could be due to an unknown moderator, but no effect was observed among the studies with the strongest research designs (e.g., high power, a priori sampling plan).

Download Full-text

Power of Modified Brown-Forsythe and Mixed-Model Approaches in Split-Plot Designs

Methodology ◽

10.1027/1614-2241/a000124 ◽

2017 ◽

Vol 13 (1) ◽

pp. 9-22 ◽

Cited By ~ 1

Author(s):

Pablo Livacic-Rojas ◽

Guillermo Vallejo ◽

Paula Fernández ◽

Ellián Tuero-Herrero

Keyword(s):

Repeated Measures ◽

Statistical Power ◽

Mixed Model ◽

Covariance Structure ◽

Simulation Method ◽

Future Research ◽

Repeated Measures Design ◽

Fixed And Random Effects ◽

Split Plot ◽

High Level

Abstract. Low precision of the inferences of data analyzed with univariate or multivariate models of the Analysis of Variance (ANOVA) in repeated-measures design is associated to the absence of normality distribution of data, nonspherical covariance structures and free variation of the variance and covariance, the lack of knowledge of the error structure underlying the data, and the wrong choice of covariance structure from different selectors. In this study, levels of statistical power presented the Modified Brown Forsythe (MBF) and two procedures with the Mixed-Model Approaches (the Akaike’s Criterion, the Correctly Identified Model [CIM]) are compared. The data were analyzed using Monte Carlo simulation method with the statistical package SAS 9.2, a split-plot design, and considering six manipulated variables. The results show that the procedures exhibit high statistical power levels for within and interactional effects, and moderate and low levels for the between-groups effects under the different conditions analyzed. For the latter, only the Modified Brown Forsythe shows high level of power mainly for groups with 30 cases and Unstructured (UN) and Autoregressive Heterogeneity (ARH) matrices. For this reason, we recommend using this procedure since it exhibits higher levels of power for all effects and does not require a matrix type that underlies the structure of the data. Future research needs to be done in order to compare the power with corrected selectors using single-level and multilevel designs for fixed and random effects.

Download Full-text

The Limits of Ego Depletion

Social Psychology ◽

10.1027/1864-9335/a000365 ◽

2019 ◽

Vol 50 (5-6) ◽

pp. 292-304 ◽

Cited By ~ 2

Author(s):

Mario Wenzel ◽

Marina Lind ◽

Zarah Rowland ◽

Daniela Zahn ◽

Thomas Kubiak

Keyword(s):

Research Agenda ◽

Statistical Power ◽

Interindividual Variability ◽

Ego Depletion ◽

Crossover Design ◽

Effect Sizes ◽

Future Research ◽

Future Research Agenda ◽

Depletion Effect ◽

Within Subjects

Abstract. Evidence on the existence of the ego depletion phenomena as well as the size of the effects and potential moderators and mediators are ambiguous. Building on a crossover design that enables superior statistical power within a single study, we investigated the robustness of the ego depletion effect between and within subjects and moderating and mediating influences of the ego depletion manipulation checks. Our results, based on a sample of 187 participants, demonstrated that (a) the between- and within-subject ego depletion effects only had negligible effect sizes and that there was (b) large interindividual variability that (c) could not be explained by differences in ego depletion manipulation checks. We discuss the implications of these results and outline a future research agenda.

Download Full-text