scholarly journals Multiple stressor null models frequently fail to detect most interactions due to low statistical power

2021 ◽  
Author(s):  
Benjamin J Burgess ◽  
Michelle C Jackson ◽  
David J Murrell
Keyword(s):  
Sample Size ◽  
Statistical Power ◽  
Multiple Stressors ◽  
Statistical Significance ◽  
Null Model ◽  
Null Models ◽  
Data Uncertainty ◽  
Sample Sizes ◽  
Multiple Stressor ◽  
Null Expectation

1. Most ecosystems are subject to co-occurring, anthropogenically driven changes and understanding how these multiple stressors interact is a pressing concern. Stressor interactions are typically studied using null models, with the additive and multiplicative null expectation being those most widely applied. Such approaches classify interactions as being synergistic, antagonistic, reversal, or indistinguishable from the null expectation. Despite their wide-spread use, there has been no thorough analysis of these null models, nor a systematic test of the robustness of their results to sample size or sampling error in the estimates of the responses to stressors. 2. We use data simulated from food web models where the true stressor interactions are known, and analytical results based on the null model equations to uncover how (i) sample size, (ii) variation in biological responses to the stressors and (iii) statistical significance, affect the ability to detect non-null interactions. 3. Our analyses lead to three main results. Firstly, it is clear the additive and multiplicative null models are not directly comparable, and over one third of all simulated interactions had classifications that were model dependent. Secondly, both null models have weak power to correctly classify interactions at commonly implemented sample sizes (i.e., ≤6 replicates), unless data uncertainty is unrealistically low. This means all but the most extreme interactions are indistinguishable from the null model expectation. Thirdly, we show that increasing sample size increases the power to detect the true interactions but only very slowly. However, the biggest gains come from increasing replicates from 3 up to 25 and we provide an R function for users to determine sample sizes required to detect a critical effect size of biological interest for the additive model. 4. Our results will aid researchers in the design of their experiments and the subsequent interpretation of results. We find no clear statistical advantage of using one null model over the other and argue null model choice should be based on biological relevance rather than statistical properties. However, there is a pressing need to increase experiment sample sizes otherwise many biologically important synergistic and antagonistic stressor interactions will continue to be missed.

scholarly journals Methodological Reporting Behavior, Sample Sizes, and Statistical Power in Studies of Event- Related Potentials: Barriers to Reproducibility and Replicability

2019 ◽  
Author(s):  
Peter E Clayson ◽  
Kaylie Amanda Carbine ◽  
Scott Baldwin ◽  
Michael J. Larson
Keyword(s):  
Sample Size ◽  
Statistical Power ◽  
Event Related Potentials ◽  
Reporting Guidelines ◽  
Medium Effect ◽  
Sample Sizes ◽  
Reporting Behavior ◽  
Average Sample Size ◽  
Related Potentials ◽  
Average Sample

Methodological reporting guidelines for studies of event-related potentials (ERPs) were updated in Psychophysiology in 2014. These guidelines facilitate the communication of key methodological parameters (e.g., preprocessing steps). Failing to report key parameters represents a barrier to replication efforts, and difficultly with replicability increases in the presence of small sample sizes and low statistical power. We assessed whether guidelines are followed and estimated the average sample size and power in recent research. Reporting behavior, sample sizes, and statistical designs were coded for 150 randomly-sampled articles from five high-impact journals that frequently publish ERP research from 2011 to 2017. An average of 63% of guidelines were reported, and reporting behavior was similar across journals, suggesting that gaps in reporting is a shortcoming of the field rather than any specific journal. Publication of the guidelines paper had no impact on reporting behavior, suggesting that editors and peer reviewers are not enforcing these recommendations. The average sample size per group was 21. Statistical power was conservatively estimated as .72-.98 for a large effect size, .35-.73 for a medium effect, and .10-.18 for a small effect. These findings indicate that failing to report key guidelines is ubiquitous and that ERP studies are primarily powered to detect large effects. Such low power and insufficient following of reporting guidelines represent substantial barriers to replication efforts. The methodological transparency and replicability of studies can be improved by the open sharing of processing code and experimental tasks and by a priori sample size calculations to ensure adequately powered studies.

scholarly journals Statistical Power in Content Analysis Designs: How Effect Size, Sample Size and Coding Accuracy Jointly Affect Hypothesis Testing ‐ A Monte Carlo Simulation Approach.

2021 ◽  
Vol 3 (1) ◽  
pp. 61-89
Author(s):  
Stefan Geiß
Keyword(s):  
Monte Carlo Simulation ◽  
Monte Carlo ◽  
Content Analysis ◽  
Sample Size ◽  
Effect Size ◽  
Statistical Power ◽  
Effect Sizes ◽  
Sample Sizes ◽  
Expected Effect ◽  
Sample Size Effect

Abstract This study uses Monte Carlo simulation techniques to estimate the minimum required levels of intercoder reliability in content analysis data for testing correlational hypotheses, depending on sample size, effect size and coder behavior under uncertainty. The ensuing procedure is analogous to power calculations for experimental designs. In most widespread sample size/effect size settings, the rule-of-thumb that chance-adjusted agreement should be ≥.80 or ≥.667 corresponds to the simulation results, resulting in acceptable α and β error rates. However, this simulation allows making precise power calculations that can consider the specifics of each study’s context, moving beyond one-size-fits-all recommendations. Studies with low sample sizes and/or low expected effect sizes may need coder agreement above .800 to test a hypothesis with sufficient statistical power. In studies with high sample sizes and/or high expected effect sizes, coder agreement below .667 may suffice. Such calculations can help in both evaluating and in designing studies. Particularly in pre-registered research, higher sample sizes may be used to compensate for low expected effect sizes and/or borderline coding reliability (e.g. when constructs are hard to measure). I supply equations, easy-to-use tables and R functions to facilitate use of this framework, along with example code as online appendix.

scholarly journals Comparison of statistical methods for analysis of small sample sizes for detecting the differences in efficacy between treatments for knee osteoarthritis

2020 ◽  
Author(s):  
Chia-Lung Shih ◽  
Te-Yu Hung
Keyword(s):  
Knee Osteoarthritis ◽  
Sample Size ◽  
Statistical Methods ◽  
Statistical Power ◽  
Permutation Test ◽  
Small Sample Size ◽  
Small Sample ◽  
T Test ◽  
Sample Sizes ◽  
Knee Oa

Abstract Background A small sample size (n < 30 for each treatment group) is usually enrolled to investigate the differences in efficacy between treatments for knee osteoarthritis (OA). The objective of this study was to use simulation for comparing the power of four statistical methods for analysis of small sample size for detecting the differences in efficacy between two treatments for knee OA. Methods A total of 10,000 replicates of 5 sample sizes (n=10, 15, 20, 25, and 30 for each group) were generated based on the previous reported measures of treatment efficacy. Four statistical methods were used to compare the differences in efficacy between treatments, including the two-sample t-test (t-test), the Mann-Whitney U-test (M-W test), the Kolmogorov-Smirnov test (K-S test), and the permutation test (perm-test). Results The bias of simulated parameter means showed a decreased trend with sample size but the CV% of simulated parameter means varied with sample sizes for all parameters. For the largest sample size (n=30), the CV% could achieve a small level (<20%) for almost all parameters but the bias could not. Among the non-parametric tests for analysis of small sample size, the perm-test had the highest statistical power, and its false positive rate was not affected by sample size. However, the power of the perm-test could not achieve a high value (80%) even using the largest sample size (n=30). Conclusion The perm-test is suggested for analysis of small sample size to compare the differences in efficacy between two treatments for knee OA.

scholarly journals Necessary Sample Size for Method Comparison Studies Based on Regression Analysis

1999 ◽  
Vol 45 (6) ◽  
pp. 882-894 ◽  
Author(s):  
Kristian Linnet
Keyword(s):  
Regression Analysis ◽  
Sample Size ◽  
Statistical Power ◽  
Theoretical Calculations ◽  
Method Comparison ◽  
Sample Sizes ◽  
Analytical Error ◽  
Deming Regression ◽  
Standard Deviations ◽  
Range Of Values

Abstract Background: In method comparison studies, it is of importance to assure that the presence of a difference of medical importance is detected. For a given difference, the necessary number of samples depends on the range of values and the analytical standard deviations of the methods involved. For typical examples, the present study evaluates the statistical power of least-squares and Deming regression analyses applied to the method comparison data. Methods: Theoretical calculations and simulations were used to consider the statistical power for detection of slope deviations from unity and intercept deviations from zero. For situations with proportional analytical standard deviations, weighted forms of regression analysis were evaluated. Results: In general, sample sizes of 40–100 samples conventionally used in method comparison studies often must be reconsidered. A main factor is the range of values, which should be as wide as possible for the given analyte. For a range ratio (maximum value divided by minimum value) of 2, 544 samples are required to detect one standardized slope deviation; the number of required samples decreases to 64 at a range ratio of 10 (proportional analytical error). For electrolytes having very narrow ranges of values, very large sample sizes usually are necessary. In case of proportional analytical error, application of a weighted approach is important to assure an efficient analysis; e.g., for a range ratio of 10, the weighted approach reduces the requirement of samples by &gt;50%. Conclusions: Estimation of the necessary sample size for a method comparison study assures a valid result; either no difference is found or the existence of a relevant difference is confirmed.

scholarly journals A systematic review of sample size and power in leading neuroscience journals

2017 ◽  
Author(s):  
Alice Carter ◽  
Kate Tilling ◽  
Marcus R Munafò
Keyword(s):  
Systematic Review ◽  
Sample Size ◽  
Statistical Power ◽  
Reproducible Research ◽  
Sample Sizes ◽  
Significance Level ◽  
Power Calculations ◽  
Research Findings ◽  
Formal Power ◽  
Nature Neuroscience

AbstractAdequate sample size is key to reproducible research findings: low statistical power can increase the probability that a statistically significant result is a false positive. Journals are increasingly adopting methods to tackle issues of reproducibility, such as by introducing reporting checklists. We conducted a systematic review comparing articles submitted to Nature Neuroscience in the 3 months prior to checklists (n=36) that were subsequently published with articles submitted to Nature Neuroscience in the 3 months immediately after checklists (n=45), along with a comparison journal Neuroscience in this same 3-month period (n=123). We found that although the proportion of studies commenting on sample sizes increased after checklists (22% vs 53%), the proportion reporting formal power calculations decreased (14% vs 9%). Using sample size calculations for 80% power and a significance level of 5%, we found little evidence that sample sizes were adequate to achieve this level of statistical power, even for large effect sizes. Our analysis suggests that reporting checklists may not improve the use and reporting of formal power calculations.

Optimal Sample Sizes for Testing the Equivalence of Two Means

Methodology ◽  
2019 ◽  
Vol 15 (3) ◽  
pp. 128-136
Author(s):  
Jiin-Huarng Guo ◽  
Hubert J. Chen ◽  
Wei-Ming Luh
Keyword(s):  
Sample Size ◽  
Statistical Power ◽  
Search Algorithm ◽  
Sample Sizes ◽  
Unequal Variances ◽  
Population Means ◽  
Optimal Sample ◽  
Equivalence Tests ◽  
Shiny App ◽  
R Shiny

Abstract. Equivalence tests (also known as similarity or parity tests) have become more and more popular in addition to equality tests. However, in testing the equivalence of two population means, approximate sample sizes developed using conventional techniques found in the literature on this topic have usually been under-valued as having less statistical power than is required. In this paper, the authors first address the reason for this problem and then provide a solution using an exhaustive local search algorithm to find the optimal sample size. The proposed method is not only accurate but is also flexible so that unequal variances or sampling unit costs for different groups can be considered using different sample size allocations. Figures and a numerical example are presented to demonstrate various configurations. An R Shiny App is also available for easy use ( https://optimal-sample-size.shinyapps.io/equivalence-of-means/ ).

How Low Can You Go?

Methodology ◽  
2014 ◽  
Vol 10 (1) ◽  
pp. 1-11 ◽  
Author(s):  
Bethany A. Bell ◽  
Grant B. Morgan ◽  
Jason A. Schoeneberger ◽  
Jeffrey D. Kromrey ◽  
John M. Ferron
Keyword(s):  
Confidence Interval ◽  
Sample Size ◽  
Fixed Effects ◽  
Statistical Power ◽  
Multilevel Models ◽  
Type I Error ◽  
Model Complexity ◽  
Type I ◽  
Sample Sizes ◽  
Level 1

Whereas general sample size guidelines have been suggested when estimating multilevel models, they are only generalizable to a relatively limited number of data conditions and model structures, both of which are not very feasible for the applied researcher. In an effort to expand our understanding of two-level multilevel models under less than ideal conditions, Monte Carlo methods, through SAS/IML, were used to examine model convergence rates, parameter point estimates (statistical bias), parameter interval estimates (confidence interval accuracy and precision), and both Type I error control and statistical power of tests associated with the fixed effects from linear two-level models estimated with PROC MIXED. These outcomes were analyzed as a function of: (a) level-1 sample size, (b) level-2 sample size, (c) intercept variance, (d) slope variance, (e) collinearity, and (f) model complexity. Bias was minimal across nearly all conditions simulated. The 95% confidence interval coverage and Type I error rate tended to be slightly conservative. The degree of statistical power was related to sample sizes and level of fixed effects; higher power was observed with larger sample sizes and level-1 fixed effects.

scholarly journals Implications of sample size and acquired number of steps to investigate running biomechanics

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Anderson Souza Oliveira ◽  
Cristina Ioana Pirscoveanu
Keyword(s):  
Sample Size ◽  
Statistical Power ◽  
Loading Rate ◽  
Sequential Estimation ◽  
Human Movement ◽  
Effect Sizes ◽  
Medium Effect ◽  
Dependent Manner ◽  
Sample Sizes ◽  
Running Speed

AbstractLow reproducibility and non-optimal sample sizes are current concerns in scientific research, especially within human movement studies. Therefore, this study aimed to examine the implications of different sample sizes and number of steps on data variability and statistical outcomes from kinematic and kinetics running biomechanical variables. Forty-four participants ran overground using their preferred technique (normal) and minimizing the contact sound volume (silent). Running speed, peak vertical, braking forces, and vertical average loading rate were extracted from > 40 steps/runner. Data stability was computed using a sequential estimation technique. Statistical outcomes (p values and effect sizes) from the comparison normal vs silent running were extracted from 100,000 random samples, using various combinations of sample size (from 10 to 40 runners) and number of steps (from 5 to 40 steps). The results showed that only 35% of the study sample could reach average stability using up to 10 steps across all biomechanical variables. The loading rate was consistently significantly lower during silent running compared to normal running, with large effect sizes across all combinations. However, variables presenting small or medium effect sizes (running speed and peak braking force), required > 20 runners to reach significant differences. Therefore, varying sample sizes and number of steps are shown to influence the normal vs silent running statistical outcomes in a variable-dependent manner. Based on our results, we recommend that studies involving analysis of traditional running biomechanical variables use a minimum of 25 participants and 25 steps from each participant to provide appropriate data stability and statistical power.

scholarly journals Evaluation of satellite collar sample size requirements for mitigation of low-level military jet disturbance of the George River caribou herd

Rangifer ◽  
2003 ◽  
Vol 23 (5) ◽  
pp. 297 ◽  
Author(s):  
Robert D. Otto ◽  
Neal P.P. Simon ◽  
Serge Couturier ◽  
Isabelle Schmelzer
Keyword(s):  
Sample Size ◽  
Statistical Power ◽  
Rangifer Tarandus ◽  
Radio Telemetry ◽  
A Priori ◽  
Effective Sample Size ◽  
Sample Sizes ◽  
Low Level ◽  
Department Of National Defence ◽  
George River

Wildlife radio-telemetry and tracking projects often determine a priori required sample sizes by statistical means or default to the maximum number that can be maintained within a limited budget. After initiation of such projects, little attention is focussed on effective sample size requirements, resulting in lack of statistical power. The Department of National Defence operates a base in Labrador, Canada for low level jet fighter training activities, and maintain a sample of satellite collars on the George River caribou (Rangifer tarandus caribou) herd of the region for spatial avoidance mitiga&not;tion purposes. We analysed existing location data, in conjunction with knowledge of life history, to develop estimates of satellite collar sample sizes required to ensure adequate mitigation of GRCH. We chose three levels of probability in each of six annual caribou seasons. Estimated number of collars required ranged from 15 to 52, 23 to 68, and 36 to 184 for 50%, 75%, and 90% probability levels, respectively, depending on season. Estimates can be used to make more informed decisions about mitigation of GRCH, and, generally, our approach provides a means to adaptively assess radio collar sam&not;ple sizes for ongoing studies.

scholarly journals The consequences of null model selection for predicting mortality from multiple stressors

2021 ◽  
Vol 288 (1948) ◽  
Author(s):  
Cody J. Dey ◽  
Marten A. Koops
Keyword(s):  
Model Selection ◽  
Multiple Stressors ◽  
Meta Analysis ◽  
Null Model ◽  
Mortality Rates ◽  
Null Models ◽  
Model Framework ◽  
Joint Effects ◽  
Prediction Of Mortality ◽  
Selection For

Many ecological systems are now exposed to multiple stressors, and ecosystem management increasingly requires consideration of the joint effects of multiple stressors on focal populations, communities and ecosystems. In the absence of empirical data, ecosystem managers could use null models based on the combination of independently acting stressors to estimate the joint effects of multiple stressors. Here, we used a simulation study and a meta-analysis to explore the consequences of null model selection for the prediction of mortality resulting from exposure to two stressors. Comparing five existing null models, we show that some null models systematically predict lower mortality rates than others, with predicted mortality rates up to 67.5% higher or 50% lower than the commonly used Simple Addition model. However, the null model predicting the highest mortality rate differed across parameter sets, and therefore there is no general ‘precautionary null model’ for multiple stressors. Using a multi-model framework, we re-analysed data from two earlier meta-analyses and found that 54% of the observed joint effects fell within the range of predictions from the suite of null models. Furthermore, we found that most null models systematically underestimated the observed joint effects, with only the Stressor Addition model showing a bias for overestimation. Finally, we found that the intensity of individual stressors was the strongest predictor of the magnitude of the joint effect across all null models. As a result, studies characterizing the effects of individuals stressors are still required for accurate prediction of mortality resulting from multiple stressors.

Sign in / Sign up

Export Citation Format

Share Document