scholarly journals Accuracy and Power Analysis of Social Interaction Networks

2021 ◽  
Author(s):  
Jordan D.A. Hart ◽  
Daniel W. Franks ◽  
Lauren J.N. Brent ◽  
Michael N. Weiss
Keyword(s):  
Power Analysis ◽  
Statistical Power ◽  
Power Level ◽  
Dynamic Network ◽  
Poisson Model ◽  
Complete Information ◽  
Interaction Networks ◽  
Post Hoc ◽  
Custom Power ◽  
Power Analyses

Power analysis is used to estimate the probability of correctly rejecting a null hypothesis for a given statistical model and dataset. Conventional power analyses assume complete information, but the stochastic nature of behavioural sampling can mean that true and estimated networks are poorly correlated. Power analyses of animal social networks do not currently take the effect of sampling into account. This could lead to inaccurate estimates of statistical power, potentially yielding misleading results. Here we develop a method for computing how well an estimated social network correlates with its true network using a Gamma-Poisson model of interaction rates. We use simulations to assess how the level of correlation between true and estimated networks affects the power of nodal regression analyses. We also develop a generic method of power analysis applicable to any statistical test, based on the concept of diminishing returns. We demonstrate that our network correlation estimator is both accurate and moderately robust to its assumptions being broken. We show that social differentiation, mean interaction rate, and the harmonic mean of sampling times positively impacts the strength of correlation between true and estimated networks. We also show that the required level of correlation between true and estimated networks to achieve a given power level depends on many factors, but that 80% correlation usually corresponded to around 80% power for nodal regression. We provide guidelines for using our network correlation estimator to verify the accuracy of interaction networks, and to conduct power analysis. This can be used prior to data collection, in post hoc analyses, or even for subsetting networks for use in dynamic network analysis. We make our code available so that custom power analysis can be used in future studies.

Differences in Lost Letter Responses from a Seaside City

2001 ◽  
Vol 88 (3_suppl) ◽  
pp. 1194-1198 ◽  
Author(s):  
F. Stephen Bridges ◽  
C. Bennett Williamson ◽  
Donna Rae Jarvis
Keyword(s):  
Power Analysis ◽  
Statistical Power ◽  
Power Level ◽  
Effect Sizes ◽  
Gay And Lesbian ◽  
Rates Of Return ◽  
Florida Panhandle

Of 75 letters “lost” in the Florida Panhandle, 33 (44%) were returned in the mail by the finders (the altruistic response). Addressees' affiliations were significantly associated with different rates of return; fewer emotive Intercontinental Gay and Lesbian Outdoors Organization addressees were returned than nonemotive ones. The technique for power analysis by Gillett (1996) was applied to data from an earlier study and indicated our sample of 75 subjects would still yield a desired power level, i.e., 80, for the likely effect sizes. Statistical power was .83, and the effect was medium in size at .34.

scholarly journals A Multi-faceted Mess: A Review of Statistical Power Analysis in Psychology Journal Articles

2019 ◽  
Author(s):  
Rob Cribbie ◽  
Nataly Beribisky ◽  
Udi Alter
Keyword(s):  
Sample Size ◽  
Effect Size ◽  
Power Analysis ◽  
Statistical Power ◽  
Type I Error ◽  
A Priori ◽  
Type I ◽  
Specific Level ◽  
Maximum Sample Size ◽  
Power Analyses

Many bodies recommend that a sample planning procedure, such as traditional NHST a priori power analysis, is conducted during the planning stages of a study. Power analysis allows the researcher to estimate how many participants are required in order to detect a minimally meaningful effect size at a specific level of power and Type I error rate. However, there are several drawbacks to the procedure that render it “a mess.” Specifically, the identification of the minimally meaningful effect size is often difficult but unavoidable for conducting the procedure properly, the procedure is not precision oriented, and does not guide the researcher to collect as many participants as feasibly possible. In this study, we explore how these three theoretical issues are reflected in applied psychological research in order to better understand whether these issues are concerns in practice. To investigate how power analysis is currently used, this study reviewed the reporting of 443 power analyses in high impact psychology journals in 2016 and 2017. It was found that researchers rarely use the minimally meaningful effect size as a rationale for the chosen effect in a power analysis. Further, precision-based approaches and collecting the maximum sample size feasible are almost never used in tandem with power analyses. In light of these findings, we offer that researchers should focus on tools beyond traditional power analysis when sample planning, such as collecting the maximum sample size feasible.

scholarly journals Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size

2019 ◽  
Author(s):  
Marjan Bakker ◽  
Coosje Lisabet Sterre Veldkamp ◽  
Olmo Van den Akker ◽  
Marcel A. L. M. van Assen ◽  
Elise Anne Victoire Crompvoets ◽  
...  
Keyword(s):  
Sample Size ◽  
Power Analysis ◽  
Statistical Power ◽  
Hypothesis Test ◽  
Open Science ◽  
Medium Effect ◽  
Main Hypothesis ◽  
Institutional Review ◽  
Formal Power ◽  
Power Analyses

In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistration Challenge (PCRs) and a sample of institutional review board (IRB) proposals from Tilburg School of Behavior and Social Sciences, which both include a recommendation to do a formal power analysis, and (ii) a sample of pre-registrations created according to the guidelines for Open Science Framework Standard Pre-Data Collection Registrations (SPRs) in which no guidance on sample size planning is given. We found that PCRs and IRBs (72%) more often included sample size decisions based on power analyses than the SPRs (45%). However, this did not result in larger planned sample sizes. The determined sample size of the PCRs and IRB proposals (Md = 90.50) was not higher than the determined sample size of the SPRs (Md = 126.00; W = 3389.5, p = 0.936). Typically, power analyses in the registrations were conducted with G*power, assuming a medium effect size, α = .05 and a power of .80. Only 20% of the power analyses contained enough information to fully reproduce the results and only 62% of these power analyses pertained to the main hypothesis test in the pre-registration. Therefore, we see ample room for improvements in the quality of the registrations and we offer several recommendations to do so.

scholarly journals Simulation-Based Power Analysis for Factorial Analysis of Variance Designs

2021 ◽  
Vol 4 (1) ◽  
pp. 251524592095150
Author(s):  
Daniël Lakens ◽  
Aaron R. Caldwell
Keyword(s):  
Analysis Of Variance ◽  
Power Analysis ◽  
Statistical Power ◽  
A Priori ◽  
R Package ◽  
Factorial Anova ◽  
Simulation Based ◽  
A Priori Power Analysis ◽  
Main Effects ◽  
Power Analyses

Researchers often rely on analysis of variance (ANOVA) when they report results of experiments. To ensure that a study is adequately powered to yield informative results with an ANOVA, researchers can perform an a priori power analysis. However, power analysis for factorial ANOVA designs is often a challenge. Current software solutions do not allow power analyses for complex designs with several within-participants factors. Moreover, power analyses often need [Formula: see text] or Cohen’s f as input, but these effect sizes are not intuitive and do not generalize to different experimental designs. We have created the R package Superpower and online Shiny apps to enable researchers without extensive programming experience to perform simulation-based power analysis for ANOVA designs of up to three within- or between-participants factors. Predicted effects are entered by specifying means, standard deviations, and, for within-participants factors, the correlations. The simulation provides the statistical power for all ANOVA main effects, interactions, and individual comparisons. The software can plot power across a range of sample sizes, can control for multiple comparisons, and can compute power when the homogeneity or sphericity assumption is violated. This Tutorial demonstrates how to perform a priori power analysis to design informative studies for main effects, interactions, and individual comparisons and highlights important factors that determine the statistical power for factorial ANOVA designs.

How Many Studies Do You Need?

2010 ◽  
Vol 35 (2) ◽  
pp. 215-247 ◽  
Author(s):  
Jeffrey C. Valentine ◽  
Therese D. Pigott ◽  
Hannah R. Rothstein
Keyword(s):  
Confidence Intervals ◽  
Random Effects ◽  
Power Analysis ◽  
Variance Component ◽  
Statistical Power ◽  
Meta Analysis ◽  
Inclusion Criteria ◽  
Fixed And Random Effects ◽  
Quantitative Synthesis ◽  
Power Analyses

In this article, the authors outline methods for using fixed and random effects power analysis in the context of meta-analysis. Like statistical power analysis for primary studies, power analysis for meta-analysis can be done either prospectively or retrospectively and requires assumptions about parameters that are unknown. The authors provide some suggestions for thinking about these parameters, in particular for the random effects variance component. The authors also show how the typically uninformative retrospective power analysis can be made more informative. The authors then discuss the value of confidence intervals, show how they could be used in addition to or instead of retrospective power analysis, and also demonstrate that confidence intervals can convey information more effectively in some situations than power analyses alone. Finally, the authors take up the question “How many studies do you need to do a meta-analysis?” and show that, given the need for a conclusion, the answer is “two studies,” because all other synthesis techniques are less transparent and/or are less likely to be valid. For systematic reviewers who choose not to conduct a quantitative synthesis, the authors provide suggestions for both highlighting the current limitations in the research base and for displaying the characteristics and results of studies that were found to meet inclusion criteria.

scholarly journals Simulation-Based Power-Analysis for Factorial ANOVA Designs

2019 ◽  
Author(s):  
Daniel Lakens ◽  
Aaron R Caldwell
Keyword(s):  
Power Analysis ◽  
Statistical Power ◽  
A Priori ◽  
Multiple Comparisons ◽  
Error Rates ◽  
Factorial Anova ◽  
Simulation Based ◽  
A Priori Power Analysis ◽  
Main Effects ◽  
Power Analyses

Researchers often rely on analysis of variance (ANOVA) when they report results of experiments. To ensure a study is adequately powered to yield informative results when performing an ANOVA, researchers can perform an a-priori power analysis. However, power analysis for factorial ANOVA designs is often a challenge. Current software solutions do not allow power analyses for complex designs with several within-subject factors. Moreover, power analyses often need partial eta-squared or Cohen's $f$ as input, but these effect sizes are not intuitive and do not generalize to different experimental designs. We have created the R package Superpower and online Shiny apps to enable researchers without extensive programming experience to perform simulation-based power analysis for ANOVA designs of up to three within- or between-subject factors. Predicted effects are entered by specifying means, standard deviations, and for within-subject factors the correlations. The simulation provides the statistical power for all ANOVA main effects, interactions, and individual comparisons, and allows researchers to correct for multiple comparisons. The software can plot power across a range of sample sizes, can control error rates for multiple comparisons, and can compute power when the homogeneity or sphericity assumptions are violated. This tutorial will demonstrate how to perform a-priori power analysis to design informative studies for main effects, interactions, and individual comparisons, and highlights important factors that determine the statistical power for factorial ANOVA designs.

Experimental design, power and sample size for animal reproduction experiments

2008 ◽  
Vol 20 (1) ◽  
pp. 33 ◽  
Author(s):  
Phillip L. Chapman ◽  
George E. Seidel, Jr.
Keyword(s):  
Sample Size ◽  
Statistical Power ◽  
Power Level ◽  
Statistical Design ◽  
Sample Size Determination ◽  
Power Calculation ◽  
Hypothesis Tests ◽  
Design Problems ◽  
Animal Reproduction ◽  
Post Hoc

The present paper concerns statistical issues in the design of animal reproduction experiments, with emphasis on the problems of sample size determination and power calculations. We include examples and non-technical discussions aimed at helping researchers avoid serious errors that may invalidate or seriously impair the validity of conclusions from experiments. Screen shots from interactive power calculation programs and basic SAS power calculation programs are presented to aid in understanding statistical power and computing power in some common experimental situations. Practical issues that are common to most statistical design problems are briefly discussed. These include one-sided hypothesis tests, power level criteria, equality of within-group variances, transformations of response variables to achieve variance equality, optimal specification of treatment group sizes, ‘post hoc’ power analysis and arguments for the increased use of confidence intervals in place of hypothesis tests.

Exploring power in response inhibition tasks using the bootstrap: The impact of number of participants, number of trials, effect magnitude, and study design

2019 ◽  
Author(s):  
Curtis David Von Gunten ◽  
Bruce D Bartholow
Keyword(s):  
Study Design ◽  
Effect Size ◽  
Statistical Power ◽  
Bootstrap Sampling ◽  
Reliable Measure ◽  
Internal Reliability ◽  
The Impact ◽  
Power Analyses ◽  
Effect Magnitude

A primary psychometric concern with laboratory-based inhibition tasks has been their reliability. However, a reliable measure may not be necessary or sufficient for reliably detecting effects (statistical power). The current study used a bootstrap sampling approach to systematically examine how the number of participants, the number of trials, the magnitude of an effect, and study design (between- vs. within-subject) jointly contribute to power in five commonly used inhibition tasks. The results demonstrate the shortcomings of relying solely on measurement reliability when determining the number of trials to use in an inhibition task: high internal reliability can be accompanied with low power and low reliability can be accompanied with high power. For instance, adding additional trials once sufficient reliability has been reached can result in large gains in power. The dissociation between reliability and power was particularly apparent in between-subject designs where the number of participants contributed greatly to power but little to reliability, and where the number of trials contributed greatly to reliability but only modestly (depending on the task) to power. For between-subject designs, the probability of detecting small-to-medium-sized effects with 150 participants (total) was generally less than 55%. However, effect size was positively associated with number of trials. Thus, researchers have some control over effect size and this needs to be considered when conducting power analyses using analytic methods that take such effect sizes as an argument. Results are discussed in the context of recent claims regarding the role of inhibition tasks in experimental and individual difference designs.

Sign in / Sign up

Export Citation Format

Share Document