scholarly journals The lack of statistical power of subgroup analyses in meta-analyses: a cautionary note

2021 ◽  
Vol 30 ◽  
Author(s):  
Pim Cuijpers ◽  
Jason W. Griffin ◽  
Toshi A. Furukawa
Keyword(s):  
Subgroup Analysis ◽  
Statistical Power ◽  
Statistical Evaluation ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Cautionary Note ◽  
Core Component ◽  
Sufficient Power ◽  
Subgroup Analyses ◽  
Meta Analyses

Abstract One of the most used methods to examine sources of heterogeneity in meta-analyses is the so-called ‘subgroup analysis’. In a subgroup analysis, the included studies are divided into two or more subgroups, and it is tested whether the pooled effect sizes found in these subgroups differ significantly from each other. Subgroup analyses can be considered as a core component of most published meta-analyses. One important problem of subgroup analyses is the lack of statistical power to find significant differences between subgroups. In this paper, we explore the power problems of subgroup analyses in more detail, using ‘metapower’, a recently developed statistical package in R to examine power in meta-analyses, including subgroup analyses. We show that subgroup analyses require many more included studies in a meta-analysis than are needed for the main analyses. We work out an example of an ‘average’ meta-analysis, in which a subgroup analysis requires 3–4 times the number of studies that are needed for the main analysis to have sufficient power. This number of studies increases exponentially with decreasing effect sizes and when the studies are not evenly divided over the subgroups. Higher heterogeneity also requires increasing numbers of studies. We conclude that subgroup analyses remain an important method to examine potential sources of heterogeneity in meta-analyses, but that meta-analysts should keep in mind that power is very low for most subgroup analyses. As in any statistical evaluation, researchers should not rely on a test and p-value to interpret results, but should compare the confidence intervals and interpret results carefully.

scholarly journals The Relationship between Family Violence and Self-Control in Adolescence: A Multi-Level Meta-Analysis

2018 ◽  
Author(s):  
Yayouk E. Willems ◽  
Jian-bin Li ◽  
Anne M. Hendriks ◽  
Meike Bartels ◽  
Catrin Finkenauer
Keyword(s):  
Family Violence ◽  
Statistical Power ◽  
Meta Analysis ◽  
Self Control ◽  
Effect Sizes ◽  
Future Research ◽  
Cross Sectional ◽  
Multi Level ◽  
Meta Analyses ◽  
Low Self Control

Theoretical studies propose an association between family violence and low self-control in adolescence, yet empirical findings of this association are inconclusive. The aim of the present research was to systematically summarize available findings on the relation between family violence and self-control across adolescence. We included 27 studies with 143 effect sizes, representing more than 25,000 participants of eight countries from early to late adolescence. Applying a multi-level meta-analyses, taking dependency between effect sizes into account while retaining statistical power, we examined the magnitude and direction of the overall effect size. Additionally, we investigated whether theoretical moderators (e.g., age, gender, country), and methodological moderators (cross-sectional/longitudinal, informant) influenced the magnitude of the association between family violence and self-control. Our results revealed that family violence and self-control have a small to moderate significant negative association (r = -.191). This association did not vary across gender, country, and informants. The strength of the association, however, decreased with age and in longitudinal studies. This finding provides evidence that researchers and clinicians may expect low self-control in the wake of family violence, especially in early adolescence. Recommendations for future research in the area are discussed.

Power Statistics for Meta-analysis: Tests for Mean Effects and Homogeneity

2013 ◽  
Author(s):  
Marc J. Lajeunesse
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Experimental Sample ◽  
Sample Sizes ◽  
Homogeneity Tests ◽  
Sufficient Power ◽  
Experimental Effect ◽  
The Common

The common justification for meta-analysis is the increased statistical power to detect effects over what is obtained from individual studies. For ecologists and evolutionary biologists, the statistical power of meta-analysis is important because effect sizes are usually relatively small in these fields, and experimental sample sizes are often limited for logistic reasons. Consequently, many studies lack sufficient power to detect an experimental effect should it exist. This chapter provides a brief overview of the factors that determine the statistical power of meta-analysis. It presents statistics for calculating the power of pooled effect sizes to evaluate nonzero effects and the power of within- and between-study homogeneity tests. It also surveys ways to improve the statistical power of meta-analysis, and ends with a discussion on the overall utility of power statistics for meta-analysis.

scholarly journals The effects of repetitive transcranial magnetic stimulation on empathy: a systematic review and meta-analysis

2017 ◽  
Vol 48 (5) ◽  
pp. 737-750 ◽  
Author(s):  
C.-C. Yang ◽  
N. Khalifa ◽  
B. Völlm
Keyword(s):  
Transcranial Magnetic Stimulation ◽  
Magnetic Stimulation ◽  
Repetitive Transcranial Magnetic Stimulation ◽  
Meta Analysis ◽  
Therapeutic Interventions ◽  
Effect Sizes ◽  
Cochrane Library ◽  
Subgroup Analyses ◽  
Meta Analyses ◽  
Clinical Populations

Empathy is a multi-dimensional concept with affective and cognitive components, the latter often referred to as Theory of Mind (ToM). Impaired empathy is prevalent in people with neuropsychiatric disorders, such as personality disorder, psychopathy, and schizophrenia, highlighting the need to develop therapeutic interventions to address this. Repetitive transcranial magnetic stimulation (rTMS), a non-invasive therapeutic technique that has been effective in treating various neuropsychiatric conditions, can be potentially used to modulate empathy. To our knowledge, no systematic reviews or meta-analyses in this field have been conducted. The aim of the current study was to review the literature on the use of rTMS to modulate empathy in adults. Seven electronic databases (AMED, Cochrane library, EMBASE, Medline, Pubmed, PsycInfo, and Web of Science) were searched using appropriate search terms. Twenty-two studies were identified, all bar one study involved interventions in healthy rather than clinical populations, and 18 of them, providing results for 24 trials, were included in the meta-analyses. Results showed an overall small, but statistically significant, effect in favour of active rTMS in healthy individuals. Differential effects across cognitive and affective ToM were evident. Subgroup analyses for cognitive ToM revealed significant effect sizes on excitatory rTMS, offline paradigms, and non-randomised design trials. Subgroup analyses for affective ToM revealed significant effect sizes on excitatory rTMS, offline paradigms, and non-randomised design trials. Meta-regression revealed no significant sources of heterogeneity. In conclusion, rTMS may have discernible effects on different components of empathy. Further research is required to examine the effects of rTMS on empathy in clinical and non-clinical populations, using appropriate empathy tasks and rTMS protocols.

scholarly journals Most oxytocin administration studies are statistically underpowered to reliably detect (or reject) a wide range of effect sizes

2020 ◽  
Author(s):  
Daniel S Quintana
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Evidential Value ◽  
Statistical Equivalence ◽  
Wide Range ◽  
Noticeable Improvement ◽  
Meta Analyses ◽  
Study Designs ◽  
Analytical Approaches

The neuropeptide oxytocin has attracted substantial research interest for its role in behaviour and cognition; however, the evidence for its effects have been mixed. Meta-analysis is viewed as the gold-standard for synthesizing evidence, but the evidential value of a meta-analysis is dependent on the evidential value of the studies it synthesizes, and the analytical approaches used to derive conclusions. To assess the evidential value of oxytocin administration meta-analyses, this study calculated the statistical power of 107 studies from 35 meta-analyses and assessed the statistical equivalence of reported results. The mean statistical power across all studies was 12.2% and there has been no noticeable improvement in power over an eight-year period. None of the 26 non-significant meta-analyses were statistically equivalent, assuming a smallest effect size of interest of 0.1. Altogether, most oxytocin treatment study designs are statistically underpowered to either detect or reject a wide range of effect sizes that scholars may find worthwhile.

Metacognitive training for schizophrenia spectrum patients: a meta-analysis on outcome studies

2015 ◽  
Vol 46 (1) ◽  
pp. 47-57 ◽  
Author(s):  
B. van Oosterhout ◽  
F. Smit ◽  
L. Krabbendam ◽  
S. Castelein ◽  
A. B. P. Staring ◽  
...  
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Data Gathering ◽  
Effect Sizes ◽  
Sensitivity Analyses ◽  
Positive Symptoms ◽  
Metacognitive Training ◽  
Intention To Treat ◽  
Schizophrenia Spectrum ◽  
Meta Analyses

Background.Metacognitive training (MCT) for schizophrenia spectrum is widely implemented. It is timely to systematically review the literature and to conduct a meta-analysis.Method.Eligible studies were selected from several sources (databases and expert suggestions). Criteria included comparative studies with a MCT condition measuring positive symptoms and/or delusions and/or data-gathering bias. Three meta-analyses were conducted on data gathering (three studies; 219 participants), delusions (seven studies; 500 participants) and positive symptoms (nine studies; 436 participants). Hedges’ g is reported as the effect size of interest. Statistical power was sufficient to detect small to moderate effects.Results.All analyses yielded small non-significant effect sizes (0.26 for positive symptoms; 0.22 for delusions; 0.31 for data-gathering bias). Corrections for publication bias further reduced the effect sizes to 0.21 for positive symptoms and to 0.03 for delusions. In blinded studies, the corrected effect sizes were 0.22 for positive symptoms and 0.03 for delusions. In studies using proper intention-to-treat statistics the effect sizes were 0.10 for positive symptoms and −0.02 for delusions. The moderate to high heterogeneity in most analyses suggests that processes other than MCT alone have an impact on the results.Conclusions.The studies so far do not support a positive effect for MCT on positive symptoms, delusions and data gathering. The methodology of most studies was poor and sensitivity analyses to control for methodological flaws reduced the effect sizes considerably. More rigorous research would be helpful in order to create enough statistical power to detect small effect sizes and to reduce heterogeneity. Limitations and strengths are discussed.

Power-Enhanced Funnel Plots for Meta-Analysis

2020 ◽  
Vol 228 (1) ◽  
pp. 43-49 ◽  
Author(s):  
Michael Kossmeier ◽  
Ulrich S. Tran ◽  
Martin Voracek
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Relevant Information ◽  
Selective Reporting ◽  
Graphical Display ◽  
Funnel Plot ◽  
Evidential Value ◽  
Graphical Displays ◽  
Funnel Plots ◽  
Meta Analyses

Abstract. Currently, dedicated graphical displays to depict study-level statistical power in the context of meta-analysis are unavailable. Here, we introduce the sunset (power-enhanced) funnel plot to visualize this relevant information for assessing the credibility, or evidential value, of a set of studies. The sunset funnel plot highlights the statistical power of primary studies to detect an underlying true effect of interest in the well-known funnel display with color-coded power regions and a second power axis. This graphical display allows meta-analysts to incorporate power considerations into classic funnel plot assessments of small-study effects. Nominally significant, but low-powered, studies might be seen as less credible and as more likely being affected by selective reporting. We exemplify the application of the sunset funnel plot with two published meta-analyses from medicine and psychology. Software to create this variation of the funnel plot is provided via a tailored R function. In conclusion, the sunset (power-enhanced) funnel plot is a novel and useful graphical display to critically examine and to present study-level power in the context of meta-analysis.

The Orchard Plot: Cultivating a Forest Plot for Use in Ecology, Evolution and Beyond

2019 ◽  
Author(s):  
Shinichi Nakagawa ◽  
Malgorzata Lagisz ◽  
Rose E O'Dea ◽  
Joanna Rutkowska ◽  
Yefeng Yang ◽  
...  
Keyword(s):  
Meta Analysis ◽  
R Package ◽  
Effect Sizes ◽  
Forest Plot ◽  
Point Estimates ◽  
Aggregate Effect ◽  
The Individual ◽  
Meta Analyses ◽  
Heterogeneous Effect ◽  
Intuitive Interpretation

‘Classic’ forest plots show the effect sizes from individual studies and the aggregate effect from a meta-analysis. However, in ecology and evolution meta-analyses routinely contain over 100 effect sizes, making the classic forest plot of limited use. We surveyed 102 meta-analyses in ecology and evolution, finding that only 11% use the classic forest plot. Instead, most used a ‘forest-like plot’, showing point estimates (with 95% confidence intervals; CIs) from a series of subgroups or categories in a meta-regression. We propose a modification of the forest-like plot, which we name the ‘orchard plot’. Orchard plots, in addition to showing overall mean effects and CIs from meta-analyses/regressions, also includes 95% prediction intervals (PIs), and the individual effect sizes scaled by their precision. The PI allows the user and reader to see the range in which an effect size from a future study may be expected to fall. The PI, therefore, provides an intuitive interpretation of any heterogeneity in the data. Supplementing the PI, the inclusion of underlying effect sizes also allows the user to see any influential or outlying effect sizes. We showcase the orchard plot with example datasets from ecology and evolution, using the R package, orchard, including several functions for visualizing meta-analytic data using forest-plot derivatives. We consider the orchard plot as a variant on the classic forest plot, cultivated to the needs of meta-analysts in ecology and evolution. Hopefully, the orchard plot will prove fruitful for visualizing large collections of heterogeneous effect sizes regardless of the field of study.

Identification of and Correction for Publication Bias: Comment

2019 ◽  
Author(s):  
Amanda Kvarven ◽  
Eirik Strømland ◽  
Magnus Johannesson
Keyword(s):  
Publication Bias ◽  
False Positive ◽  
Large Scale ◽  
Meta Analysis ◽  
False Positive Rate ◽  
Effect Sizes ◽  
Replication Studies ◽  
Moderate Reduction ◽  
Positive Rate ◽  
Meta Analyses

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

scholarly journals Sarcopenia Is Associated with Mortality in Adults: A Systematic Review and Meta-Analysis

Gerontology ◽  
2021 ◽  
pp. 1-16
Author(s):  
Jane Xu ◽  
Ching S. Wan ◽  
Kiriakos Ktoris ◽  
Esmee M. Reijnierse ◽  
Andrea B. Maier
Keyword(s):  
Systematic Review ◽  
Working Group ◽  
Meta Analysis ◽  
Nursing Home Residents ◽  
Community Dwelling ◽  
Risk Of Mortality ◽  
European Working ◽  
Subgroup Analyses ◽  
Meta Analyses

<b><i>Background:</i></b> Sarcopenia can predispose individuals to falls, fractures, hospitalization, and mortality. The prevalence of sarcopenia depends on the population studied and the definition used for the diagnosis. <b><i>Objective:</i></b> This systematic review and meta-analysis aimed to investigate the association between sarcopenia and mortality and if it is dependent on the population and sarcopenia definition. <b><i>Methods:</i></b> A systematic search was conducted in MEDLINE, EMBASE, and Cochrane from 1 January 2010 to 6 April 2020 for articles relating to sarcopenia and mortality. Articles were included if they met the following criteria – cohorts with a mean or median age ≥18 years and either of the following sarcopenia definitions: Asian Working Group for Sarcopenia (AWGS and AWGS2019), European Working Group on Sarcopenia in Older People (EWGSOP and EWGSOP2), Foundation for the National Institutes of Health (FNIH), International Working Group for Sarcopenia (IWGS), or Sarcopenia Definition and Outcomes Consortium (SDOC). Hazard ratios (HR) and odds ratios (OR) were pooled separately in meta-analyses using a random-effects model, stratified by population (community-dwelling adults, outpatients, inpatients, and nursing home residents). Subgroup analyses were performed for sarcopenia definition and follow-up period. <b><i>Results:</i></b> Out of 3,025 articles, 57 articles were included in the systematic review and 56 in the meta-analysis (42,108 participants, mean age of 49.4 ± 11.7 to 86.6 ± 1.0 years, 40.3% females). Overall, sarcopenia was associated with a significantly higher risk of mortality (HR: 2.00 [95% CI: 1.71, 2.34]; OR: 2.35 [95% CI: 1.64, 3.37]), which was independent of population, sarcopenia definition, and follow-up period in subgroup analyses. <b><i>Conclusions:</i></b> Sarcopenia is associated with a significantly higher risk of mortality, independent of population and sarcopenia definition, which highlights the need for screening and early diagnosis in all populations.

Sign in / Sign up

Export Citation Format

Share Document