scholarly journals Most oxytocin administration studies are statistically underpowered to reliably detect (or reject) a wide range of effect sizes

2020 ◽  
Author(s):  
Daniel S Quintana
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Evidential Value ◽  
Statistical Equivalence ◽  
Wide Range ◽  
Noticeable Improvement ◽  
Meta Analyses ◽  
Study Designs ◽  
Analytical Approaches

The neuropeptide oxytocin has attracted substantial research interest for its role in behaviour and cognition; however, the evidence for its effects have been mixed. Meta-analysis is viewed as the gold-standard for synthesizing evidence, but the evidential value of a meta-analysis is dependent on the evidential value of the studies it synthesizes, and the analytical approaches used to derive conclusions. To assess the evidential value of oxytocin administration meta-analyses, this study calculated the statistical power of 107 studies from 35 meta-analyses and assessed the statistical equivalence of reported results. The mean statistical power across all studies was 12.2% and there has been no noticeable improvement in power over an eight-year period. None of the 26 non-significant meta-analyses were statistically equivalent, assuming a smallest effect size of interest of 0.1. Altogether, most oxytocin treatment study designs are statistically underpowered to either detect or reject a wide range of effect sizes that scholars may find worthwhile.

Power-Enhanced Funnel Plots for Meta-Analysis

2020 ◽  
Vol 228 (1) ◽  
pp. 43-49 ◽  
Author(s):  
Michael Kossmeier ◽  
Ulrich S. Tran ◽  
Martin Voracek
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Relevant Information ◽  
Selective Reporting ◽  
Graphical Display ◽  
Funnel Plot ◽  
Evidential Value ◽  
Graphical Displays ◽  
Funnel Plots ◽  
Meta Analyses

Abstract. Currently, dedicated graphical displays to depict study-level statistical power in the context of meta-analysis are unavailable. Here, we introduce the sunset (power-enhanced) funnel plot to visualize this relevant information for assessing the credibility, or evidential value, of a set of studies. The sunset funnel plot highlights the statistical power of primary studies to detect an underlying true effect of interest in the well-known funnel display with color-coded power regions and a second power axis. This graphical display allows meta-analysts to incorporate power considerations into classic funnel plot assessments of small-study effects. Nominally significant, but low-powered, studies might be seen as less credible and as more likely being affected by selective reporting. We exemplify the application of the sunset funnel plot with two published meta-analyses from medicine and psychology. Software to create this variation of the funnel plot is provided via a tailored R function. In conclusion, the sunset (power-enhanced) funnel plot is a novel and useful graphical display to critically examine and to present study-level power in the context of meta-analysis.

scholarly journals The Efficacy of Recasts in Language Intervention: A Systematic Review and Meta-Analysis

2015 ◽  
Vol 24 (2) ◽  
pp. 237-255 ◽  
Author(s):  
Patricia L. Cleave ◽  
Stephanie D. Becker ◽  
Maura K. Curran ◽  
Amanda J. Owen Van Horne ◽  
Marc E. Fey
Keyword(s):  
Systematic Review ◽  
Research Evidence ◽  
Meta Analysis ◽  
Language Intervention ◽  
Effect Sizes ◽  
Study Quality ◽  
Average Effect ◽  
Grammatical Development ◽  
Meta Analyses ◽  
Study Designs

Purpose This systematic review and meta-analysis critically evaluated the research evidence on the effectiveness of conversational recasts in grammatical development for children with language impairments. Method Two different but complementary reviews were conducted and then integrated. Systematic searches of the literature resulted in 35 articles for the systematic review. Studies that employed a wide variety of study designs were involved, but all examined interventions where recasts were the key component. The meta-analysis only included studies that allowed the calculation of effect sizes, but it did include package interventions in which recasts were a major part. Fourteen studies were included, 7 of which were also in the systematic review. Studies were grouped according to research phase and were rated for quality. Results Study quality and thus strength of evidence varied substantially. Nevertheless, across all phases, the vast majority of studies provided support for the use of recasts. Meta-analyses found average effect sizes of .96 for proximal measures and .76 for distal measures, reflecting a positive benefit of about 0.75 to 1.00 standard deviation. Conclusion The available evidence is limited, but it is supportive of the use of recasts in grammatical intervention. Critical features of recasts in grammatical interventions are discussed.

scholarly journals The Relationship between Family Violence and Self-Control in Adolescence: A Multi-Level Meta-Analysis

2018 ◽  
Author(s):  
Yayouk E. Willems ◽  
Jian-bin Li ◽  
Anne M. Hendriks ◽  
Meike Bartels ◽  
Catrin Finkenauer
Keyword(s):  
Family Violence ◽  
Statistical Power ◽  
Meta Analysis ◽  
Self Control ◽  
Effect Sizes ◽  
Future Research ◽  
Cross Sectional ◽  
Multi Level ◽  
Meta Analyses ◽  
Low Self Control

Theoretical studies propose an association between family violence and low self-control in adolescence, yet empirical findings of this association are inconclusive. The aim of the present research was to systematically summarize available findings on the relation between family violence and self-control across adolescence. We included 27 studies with 143 effect sizes, representing more than 25,000 participants of eight countries from early to late adolescence. Applying a multi-level meta-analyses, taking dependency between effect sizes into account while retaining statistical power, we examined the magnitude and direction of the overall effect size. Additionally, we investigated whether theoretical moderators (e.g., age, gender, country), and methodological moderators (cross-sectional/longitudinal, informant) influenced the magnitude of the association between family violence and self-control. Our results revealed that family violence and self-control have a small to moderate significant negative association (r = -.191). This association did not vary across gender, country, and informants. The strength of the association, however, decreased with age and in longitudinal studies. This finding provides evidence that researchers and clinicians may expect low self-control in the wake of family violence, especially in early adolescence. Recommendations for future research in the area are discussed.

scholarly journals Changing the logic of replication: A case from infant studies

2019 ◽  
Author(s):  
Francesco Margoni ◽  
Martin Shepperd
Keyword(s):  
Statistical Power ◽  
Sampling Error ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Small Sample ◽  
Infant Research ◽  
Replication Studies ◽  
Wide Range ◽  
Small Sample Sizes ◽  
Meta Analyses

Infant research is making considerable progresses. However, among infant researchers there is growing concern regarding the widespread habit of undertaking studies that have small sample sizes and employ tests with low statistical power (to detect a wide range of possible effects). For many researchers, issues of confidence may be partially resolved by relying on replications. Here, we bring further evidence that the classical logic of confirmation, according to which the result of a replication study confirms the original finding when it reaches statistical significance, could be usefully abandoned. With real examples taken from the infant literature and Monte Carlo simulations, we show that a very wide range of possible replication results would in a formal statistical sense constitute confirmation as they can be explained simply due to sampling error. Thus, often no useful conclusion can be derived from a single or small number of replication studies. We suggest that, in order to accumulate and generate new knowledge, the dichotomous view of replication as confirmatory/disconfirmatory can be replaced by an approach that emphasizes the estimation of effect sizes via meta-analysis. Moreover, we discuss possible solutions for reducing problems affecting the validity of conclusions drawn from meta-analyses in infant research.

scholarly journals The Empirical Status of Mindfulness-Based Interventions: A Systematic Review of 44 Meta-Analyses of Randomized Controlled Trials

2021 ◽  
pp. 174569162096877
Author(s):  
Simon B. Goldberg ◽  
Kevin M. Riordan ◽  
Shufang Sun ◽  
Richard J. Davidson
Keyword(s):  
Adverse Effects ◽  
Randomized Controlled Trials ◽  
Publication Bias ◽  
Statistical Power ◽  
Effect Sizes ◽  
Controlled Trials ◽  
Randomized Controlled ◽  
Wide Range ◽  
Meta Analyses ◽  
Active Controls

In response to questions regarding the scientific basis for mindfulness-based interventions (MBIs), we evaluated their empirical status by systematically reviewing meta-analyses of randomized controlled trials (RCTs). We searched six databases for effect sizes based on four or more trials that did not combine passive and active controls. Heterogeneity, moderators, tests of publication bias, risk of bias, and adverse effects were also extracted. Representative effect sizes based on the largest number of studies were identified across a wide range of populations, problems, interventions, comparisons, and outcomes (PICOS). A total of 160 effect sizes were reported in 44 meta-analyses ( k = 336 RCTs, N = 30,483 participants). MBIs showed superiority to passive controls across most PICOS ( ds = 0.10–0.89). Effects were typically smaller and less often statistically significant compared with active controls. MBIs were similar or superior to specific active controls and evidence-based treatments. Heterogeneity was typically moderate. Few consistent moderators were found. Results were generally robust to publication bias, although other important sources of bias were identified. Reporting of adverse effects was inconsistent. Statistical power may be lacking in meta-analyses, particularly for comparisons with active controls. Because MBIs show promise across some PICOS, future RCTs and meta-analyses should build on identified strengths and limitations of this literature.

scholarly journals A Quantitative Synthesis of Early Language Acquisition Using Meta-Analysis

2016 ◽  
Author(s):  
Molly Lewis ◽  
Mika Braginsky ◽  
Sho Tsuji ◽  
Christina Bergmann ◽  
Page Elizabeth Piccinini ◽  
...  
Keyword(s):  
Language Acquisition ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Analytic Method ◽  
Early Language ◽  
Evidential Value ◽  
System P ◽  
Quantitative Synthesis ◽  
Meta Analyses ◽  
High Degree

To acquire a language, children must learn a range of skills, from the sounds of their language to the meanings of words. These skills are typically studied in isolation in separate research programs, but there is a growing body of evidence that these skills may depend on each other in acquisition (e.g., Feldman, Myers, White, Griffiths, & Morgan, 2013; Johnson, Demuth, Jones, & Black, 2010; Shukla, White, & Aslin, 2011). We suggest that the meta-analytic method can support the process of building theories that take a systems-level perspective, as well as provide a tool for detecting bias in a literature. Here we present meta-analyses of 12 phenomena in language acquisition, with over 800 effect sizes. We find that the language acquisition literature overall has a high degree of evidential value. We then present a quantitative synthesis of language acquisition phenomena that suggests interactivity across the system.

scholarly journals The lack of statistical power of subgroup analyses in meta-analyses: a cautionary note

2021 ◽  
Vol 30 ◽  
Author(s):  
Pim Cuijpers ◽  
Jason W. Griffin ◽  
Toshi A. Furukawa
Keyword(s):  
Subgroup Analysis ◽  
Statistical Power ◽  
Statistical Evaluation ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Cautionary Note ◽  
Core Component ◽  
Sufficient Power ◽  
Subgroup Analyses ◽  
Meta Analyses

Abstract One of the most used methods to examine sources of heterogeneity in meta-analyses is the so-called ‘subgroup analysis’. In a subgroup analysis, the included studies are divided into two or more subgroups, and it is tested whether the pooled effect sizes found in these subgroups differ significantly from each other. Subgroup analyses can be considered as a core component of most published meta-analyses. One important problem of subgroup analyses is the lack of statistical power to find significant differences between subgroups. In this paper, we explore the power problems of subgroup analyses in more detail, using ‘metapower’, a recently developed statistical package in R to examine power in meta-analyses, including subgroup analyses. We show that subgroup analyses require many more included studies in a meta-analysis than are needed for the main analyses. We work out an example of an ‘average’ meta-analysis, in which a subgroup analysis requires 3–4 times the number of studies that are needed for the main analysis to have sufficient power. This number of studies increases exponentially with decreasing effect sizes and when the studies are not evenly divided over the subgroups. Higher heterogeneity also requires increasing numbers of studies. We conclude that subgroup analyses remain an important method to examine potential sources of heterogeneity in meta-analyses, but that meta-analysts should keep in mind that power is very low for most subgroup analyses. As in any statistical evaluation, researchers should not rely on a test and p-value to interpret results, but should compare the confidence intervals and interpret results carefully.

Metacognitive training for schizophrenia spectrum patients: a meta-analysis on outcome studies

2015 ◽  
Vol 46 (1) ◽  
pp. 47-57 ◽  
Author(s):  
B. van Oosterhout ◽  
F. Smit ◽  
L. Krabbendam ◽  
S. Castelein ◽  
A. B. P. Staring ◽  
...  
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Data Gathering ◽  
Effect Sizes ◽  
Sensitivity Analyses ◽  
Positive Symptoms ◽  
Metacognitive Training ◽  
Intention To Treat ◽  
Schizophrenia Spectrum ◽  
Meta Analyses

Background.Metacognitive training (MCT) for schizophrenia spectrum is widely implemented. It is timely to systematically review the literature and to conduct a meta-analysis.Method.Eligible studies were selected from several sources (databases and expert suggestions). Criteria included comparative studies with a MCT condition measuring positive symptoms and/or delusions and/or data-gathering bias. Three meta-analyses were conducted on data gathering (three studies; 219 participants), delusions (seven studies; 500 participants) and positive symptoms (nine studies; 436 participants). Hedges’ g is reported as the effect size of interest. Statistical power was sufficient to detect small to moderate effects.Results.All analyses yielded small non-significant effect sizes (0.26 for positive symptoms; 0.22 for delusions; 0.31 for data-gathering bias). Corrections for publication bias further reduced the effect sizes to 0.21 for positive symptoms and to 0.03 for delusions. In blinded studies, the corrected effect sizes were 0.22 for positive symptoms and 0.03 for delusions. In studies using proper intention-to-treat statistics the effect sizes were 0.10 for positive symptoms and −0.02 for delusions. The moderate to high heterogeneity in most analyses suggests that processes other than MCT alone have an impact on the results.Conclusions.The studies so far do not support a positive effect for MCT on positive symptoms, delusions and data gathering. The methodology of most studies was poor and sensitivity analyses to control for methodological flaws reduced the effect sizes considerably. More rigorous research would be helpful in order to create enough statistical power to detect small effect sizes and to reduce heterogeneity. Limitations and strengths are discussed.

scholarly journals Efficacy of ibuprofen in musculoskeletal post-traumatic pain in children: A systematic review

PLoS ONE ◽  
2020 ◽  
Vol 15 (12) ◽  
pp. e0243314
Author(s):  
Niccolò Parri ◽  
Simone Lazzeri
Keyword(s):  
Systematic Review ◽  
Pain Relief ◽  
Meta Analysis ◽  
Drug Of Choice ◽  
Pain Scores ◽  
Wide Range ◽  
Significant Difference ◽  
Meta Analyses ◽  
Study Designs ◽  
Initial Drug

Musculoskeletal (MSK) injuries are one of the most frequent reason for pain-related evaluation in the emergency department (ED) in children. There is still no consensus as to what constitutes the best analgesic for MSK pain in children. However, ibuprofen is reported to be the most commonly prescribed analgesic and is considered the standard first-line treatment for MSK injury pain in children, even if it is argued that it provides inadequate relief for many patients. The purpose of this study was to review the most recent literature to assess the efficacy of ibuprofen for pain relief in MSK injuries in children evaluated in the ED. We performed a systematic review of randomized controlled trials on pharmacological interventions in children and adolescents under 19 years of age with MSK injuries according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. The primary outcome was the risk ratio for successful reduction in pain scores. Six studies met the inclusion criteria and provided data on 1028 children. A meta-analysis was not performed since studies were not comparable due to the different analgesic treatment used. No significant difference in term of main pain score reduction between all the analgesics used in the included studies was noted. Patients who received oral opioids had side effects more frequently when compared to children who received ibuprofen. The combination of effect on pain relief and tolerability would suggest ibuprofen as the initial drug of choice in providing relief from mild-to-moderate MSK pain in children in the ED. The results obtained in this review and current research suggest that there’s no straightforward statistically significant evidence of the optimal analgesic agent to be used. However, ibuprofen may be preferable as the initial drug of choice in providing relief from MSK pain due to the favorable combination of effectiveness and safety profile. In fact, despite the non-significant pain reduction as compared to children who received opioids, there are less side effect associated to ibuprofen within studies. The wide range of primary outcomes measured in respect of pain scores and timing of recorded measures warrants a future standardization of study designs.

Sign in / Sign up

Export Citation Format

Share Document