Empirical Benchmarks for Interpreting Effect Size Variability in Meta-Analysis

2017 ◽  
Vol 10 (3) ◽  
pp. 472-479 ◽  
Author(s):  
Brenton M. Wiernik ◽  
Jack W. Kostal ◽  
Michael P. Wilmot ◽  
Stephan Dilchert ◽  
Deniz S. Ones
Keyword(s):  
Effect Size ◽  
Null Hypothesis ◽  
Meta Analysis ◽  
Hypothesis Tests ◽  
Size Variability ◽  
Meta Analyses

Generalization in meta-analyses is not a dichotomous decision (typically encountered in papers using the Q test for homogeneity, the 75% rule, or null hypothesis tests). Inattention to effect size variability in meta-analyses may stem from a lack of guidelines for interpreting credibility intervals. In this commentary, we describe two methods for making practical interpretations and determining whether a particular SDρ represents a meaningful level of variability.

scholarly journals Cluster Wild Bootstrapping to Handle Dependent Effect Sizes in Meta-Analysis with a Small Number of Studies

2021 ◽  
Author(s):  
Megha Joshi ◽  
James E Pustejovsky ◽  
S. Natasha Beretvas
Keyword(s):  
Effect Size ◽  
Type I Error ◽  
Meta Analysis ◽  
Error Rates ◽  
Small Sample ◽  
Type I ◽  
Hypothesis Tests ◽  
Type I Error Rates ◽  
Meta Analyses ◽  
Small Sample Correction

The most common and well-known meta-regression models work under the assumption that there is only one effect size estimate per study and that the estimates are independent. However, meta-analytic reviews of social science research often include multiple effect size estimates per primary study, leading to dependence in the estimates. Some meta-analyses also include multiple studies conducted by the same lab or investigator, creating another potential source of dependence. An increasingly popular method to handle dependence is robust variance estimation (RVE), but this method can result in inflated Type I error rates when the number of studies is small. Small-sample correction methods for RVE have been shown to control Type I error rates adequately but may be overly conservative, especially for tests of multiple-contrast hypotheses. We evaluated an alternative method for handling dependence, cluster wild bootstrapping, which has been examined in the econometrics literature but not in the context of meta-analysis. Results from two simulation studies indicate that cluster wild bootstrapping maintains adequate Type I error rates and provides more power than extant small sample correction methods, particularly for multiple-contrast hypothesis tests. We recommend using cluster wild bootstrapping to conduct hypothesis tests for meta-analyses with a small number of studies. We have also created an R package that implements such tests.

scholarly journals Predictability of rotational tooth movement with orthodontic aligners comparing software-based and achieved data: A systematic review and meta-analysis of observational studies

2021 ◽  
pp. 146531252110272
Author(s):  
Despina Koletsi ◽  
Anna Iliadi ◽  
Theodore Eliades
Keyword(s):  
Effect Size ◽  
Tooth Movement ◽  
Data Extraction ◽  
Meta Analysis ◽  
Risk Of Bias ◽  
Patient Treatment ◽  
Qualitative Synthesis ◽  
Meta Analyses ◽  
Enamel Reduction

Objective: To evaluate all available evidence on the prediction of rotational tooth movements with aligners. Data sources: Seven databases of published and unpublished literature were searched up to 4 August 2020 for eligible studies. Data selection: Studies were deemed eligible if they included evaluation of rotational tooth movement with any type of aligner, through the comparison of software-based and actually achieved data after patient treatment. Data extraction and data synthesis: Data extraction was done independently and in duplicate and risk of bias assessment was performed with the use of the QUADAS-2 tool. Random effects meta-analyses with effect sizes and their 95% confidence intervals (CIs) were performed and the quality of the evidence was assessed through GRADE. Results: Seven articles were included in the qualitative synthesis, of which three contributed to meta-analyses. Overall results revealed a non-accurate prediction of the outcome for the software-based data, irrespective of the use of attachments or interproximal enamel reduction (IPR). Maxillary canines demonstrated the lowest percentage accuracy for rotational tooth movement (three studies: effect size = 47.9%; 95% CI = 27.2–69.5; P < 0.001), although high levels of heterogeneity were identified (I2: 86.9%; P < 0.001). Contrary, mandibular incisors presented the highest percentage accuracy for predicted rotational movement (two studies: effect size = 70.7%; 95% CI = 58.9–82.5; P < 0.001; I2: 0.0%; P = 0.48). Risk of bias was unclear to low overall, while quality of the evidence ranged from low to moderate. Conclusion: Allowing for all identified caveats, prediction of rotational tooth movements with aligner treatment does not appear accurate, especially for canines. Careful selection of patients and malocclusions for aligner treatment decisions remain challenging.

scholarly journals A Statistical Method for Synthesizing Meta-Analyses

2013 ◽  
Vol 2013 ◽  
pp. 1-9 ◽  
Author(s):  
Liansheng Larry Tang ◽  
Michael Caudy ◽  
Faye Taxman
Keyword(s):  
Statistical Method ◽  
Effect Size ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Weighted Mean ◽  
Summary Effect ◽  
Different Types ◽  
Meta Analyses ◽  
Size Estimates ◽  
Similar Search

Multiple meta-analyses may use similar search criteria and focus on the same topic of interest, but they may yield different or sometimes discordant results. The lack of statistical methods for synthesizing these findings makes it challenging to properly interpret the results from multiple meta-analyses, especially when their results are conflicting. In this paper, we first introduce a method to synthesize the meta-analytic results when multiple meta-analyses use the same type of summary effect estimates. When meta-analyses use different types of effect sizes, the meta-analysis results cannot be directly combined. We propose a two-step frequentist procedure to first convert the effect size estimates to the same metric and then summarize them with a weighted mean estimate. Our proposed method offers several advantages over existing methods by Hemming et al. (2012). First, different types of summary effect sizes are considered. Second, our method provides the same overall effect size as conducting a meta-analysis on all individual studies from multiple meta-analyses. We illustrate the application of the proposed methods in two examples and discuss their implications for the field of meta-analysis.

scholarly journals Considering the methodological limitations in the evidence base of antidepressants for depression: a reanalysis of a network meta-analysis

BMJ Open ◽  
2019 ◽  
Vol 9 (6) ◽  
pp. e024886 ◽  
Author(s):  
Klaus Munkholm ◽  
Asger Sand Paludan-Müller ◽  
Kim Boesen
Keyword(s):  
Systematic Review ◽  
Clinical Study ◽  
Effect Size ◽  
Meta Analysis ◽  
Evidence Base ◽  
Risk Of Bias ◽  
Depression Symptom ◽  
Meta Analyses ◽  
The Impact ◽  
Clinical Study Reports

ObjectivesTo investigate whether the conclusion of a recent systematic review and network meta-analysis (Ciprianiet al) that antidepressants are more efficacious than placebo for adult depression was supported by the evidence.DesignReanalysis of a systematic review, with meta-analyses.Data sources522 trials (116 477 participants) as reported in the systematic review by Ciprianiet aland clinical study reports for 19 of these trials.AnalysisWe used the Cochrane Handbook’s risk of bias tool and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to evaluate the risk of bias and the certainty of evidence, respectively. The impact of several study characteristics and publication status was estimated using pairwise subgroup meta-analyses.ResultsSeveral methodological limitations in the evidence base of antidepressants were either unrecognised or underestimated in the systematic review by Ciprianiet al. The effect size for antidepressants versus placebo on investigator-rated depression symptom scales was higher in trials with a ‘placebo run-in’ study design compared with trials without a placebo run-in design (p=0.05). The effect size of antidepressants was higher in published trials compared with unpublished trials (p<0.0001). The outcome data reported by Ciprianiet aldiffered from the clinical study reports in 12 (63%) of 19 trials. The certainty of the evidence for the placebo-controlled comparisons should be very low according to GRADE due to a high risk of bias, indirectness of the evidence and publication bias. The mean difference between antidepressants and placebo on the 17-item Hamilton depression rating scale (range 0–52 points) was 1.97 points (95% CI 1.74 to 2.21).ConclusionsThe evidence does not support definitive conclusions regarding the benefits of antidepressants for depression in adults. It is unclear whether antidepressants are more efficacious than placebo.

scholarly journals Are Community Gardening and Horticultural Interventions Beneficial for Psychosocial Well-Being? A Meta-Analysis

2020 ◽  
Vol 17 (10) ◽  
pp. 3584
Author(s):  
Giuseppina Spano ◽  
Marina D’Este ◽  
Vincenzo Giannico ◽  
Giuseppe Carrus ◽  
Mario Elia ◽  
...  
Keyword(s):  
Human Health ◽  
Effect Size ◽  
Meta Analysis ◽  
Well Being ◽  
City Management ◽  
Community Gardening ◽  
Quantitative Studies ◽  
Bibliographic Records ◽  
Study Heterogeneity ◽  
Meta Analyses

Recent literature has revealed the positive effect of gardening on human health; however, empirical evidence on the effects of gardening-based programs on psychosocial well-being is scant. This meta-analysis aims to examine the scientific literature on the effect of community gardening or horticultural interventions on a variety of outcomes related to psychosocial well-being, such as social cohesion, networking, social support, and trust. From 383 bibliographic records retrieved (from 1975 to 2019), seven studies with a total of 22 effect sizes were selected on the basis of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Meta-analytic findings on 11 comparisons indicate a positive and moderate effect of horticultural or gardening interventions on psychosocial well-being. Moderation analysis shows a greater effect size in individualistic than collectivistic cultures. A greater effect size was also observed in studies involving community gardening compared to horticultural intervention. Nevertheless, an effect of publication bias and study heterogeneity has been detected. Despite the presence of a large number of qualitative studies on the effect of horticulture/gardening on psychosocial well-being, quantitative studies are lacking. There is a strong need to advance into further high-quality studies on this research topic given that gardening has promising applied implications for human health, the community, and sustainable city management.

Can Reliance be Placed on a Single Meta-Analysis?

1990 ◽  
Vol 24 (3) ◽  
pp. 405-415 ◽  
Author(s):  
Nathaniel McConaghy
Keyword(s):  
Literature Review ◽  
Effect Size ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Effect Sizes ◽  
Control Groups ◽  
Consistent Finding ◽  
Placebo Controls ◽  
Effect Of Treatment ◽  
Meta Analyses

Meta-analysis replaced statistical significance with effect size in the hope of resolving controversy concerning evaluation of treatment effects. Statistical significance measured reliability of the effect of treatment, not its efficacy. It was strongly influenced by the number of subjects investigated. Effect size as assessed originally, eliminated this influence but by standardizing the size of the treatment effect could distort it. Meta-analyses which combine the results of studies which employ different subject types, outcome measures, treatment aims, no-treatment rather than placebo controls or therapists with varying experience can be misleading. To ensure discussion of these variables meta-analyses should be used as an aid rather than a substitute for literature review. While meta-analyses produce contradictory findings, it seems unwise to rely on the conclusions of an individual analysis. Their consistent finding that placebo treatments obtain markedly higher effect sizes than no treatment hopefully will render the use of untreated control groups obsolete.

Meta-analysis of the impacts of water management on aquatic communities

2008 ◽  
Vol 65 (3) ◽  
pp. 437-447 ◽  
Author(s):  
Tim J Haxton ◽  
C Scott Findlay
Keyword(s):  
Water Management ◽  
Confidence Intervals ◽  
Effect Size ◽  
Meta Analysis ◽  
Ecological Impacts ◽  
Aquatic Communities ◽  
Flow Modification ◽  
Meta Analyses ◽  
Fail Safe ◽  
Habitat Generalists

Systematic meta-analyses were conducted on the ecological impacts of water management, including effects of (i) dewatering on macroinvertebrates, (ii) a hypolimnetic release on downstream aquatic fish and macro invertebrate communities, and (iii) flow modification on fluvial and habitat generalists. Our meta-analysis indicates, in general, that (i) macroinvertebrate abundance is lower in zones or areas that have been dewatered as a result of water fluctuations or low flows (overall effect size, –1.64; 95% confidence intervals (CIs), –2.51, –0.77), (ii) hypolimnetic draws are associated with reduced abundance of aquatic (fish and macroinvertebrates) communities (overall effect size, –0.84; 95% CIs, –1.38, –0.33) and macroinvertebrates (overall effect size, –0.73; 95% CIs, –1.24, –0.22) downstream of a dam, and (iii) altered flows are associated with reduced abundance of fluvial specialists (–0.42; 95% CIs, –0.81, –0.02) but not habitat generalists (overall effect size, –0.14; 95% CIs, –0.61, 0.32). Publication bias is evident in several of the meta-analyses; however, multiple experiments from a single study may be contributing to this bias. Fail-safe Ns suggest that many (>100) studies showing positive or no effects of water management on the selected endpoints would be required to qualitatively change the results of the meta-analysis, which in turn suggests that the conclusions are reasonably robust.

Differences in Helping Whites and Blacks: A Meta-Analysis

2005 ◽  
Vol 9 (1) ◽  
pp. 2-16 ◽  
Author(s):  
Donald A. Saucier ◽  
Carol T. Miller ◽  
Nicole Doucet
Keyword(s):  
Effect Size ◽  
Meta Analysis ◽  
Aversive Racism ◽  
Hypothesis Tests ◽  
Intergroup Anxiety

The amount of help given to Blacks versus Whites is often assumed to reflect underlying levels of racism (or lack thereof). This meta-analysis assessed discrimination against Blacks in helping studies. The overall effect size for the 48 hypothesis tests did not show universal discrimination against Blacks (d = .03, p = .103). However, consistent with the predictions of aversive racism, discrimination against Blacks was more likely when participants could rationalize decisions not to help with reasons having nothing to do with race. Specifically, when helping was lengthier, riskier, more difficult, more effortful, and when potential helpers were further away from targets, less help was given to Blacks than to Whites. Interestingly, discrimination against Blacks was shown when there were higher levels of emergency. This suggests that discrimination may occur when the ability to control prejudicial responding is inhibited, or when the arousal of the emergency is misattributed to intergroup anxiety.

scholarly journals A Meta–Analytic and Conceptual Update on the Associations between Procrastination and Multidimensional Perfectionism

2017 ◽  
Vol 31 (2) ◽  
pp. 137-159 ◽  
Author(s):  
Fuschia M. Sirois ◽  
Danielle S. Molnar ◽  
Jameson K. Hirsch ◽  
Mitja Back
Keyword(s):  
Effect Size ◽  
Meta Analysis ◽  
Self Regulation ◽  
Average Effect ◽  
Personality Psychology ◽  
Average Effect Size ◽  
Partial Correlations ◽  
Unidimensional Construct ◽  
Meta Analyses ◽  
Behavioural Dynamics

The equivocal and debated findings from a 2007 meta–analysis, which viewed perfectionism as a unidimensional construct, suggested that perfectionism was unrelated to procrastination. The present meta–analysis aimed to provide a conceptual update and reanalysis of the procrastination–perfectionism association guided by both a multidimensional view of perfectionism and self–regulation theory. The random–effects meta–analyses revealed a small to medium positive average effect size ( r = .23; k = 43, N = 10 000; 95% confidence interval (95% CI) [0.19, 0.27]) for trait procrastination and perfectionistic concerns and a small to medium negative average effect size ( r = −.22; k = 38, N = 9544; 95% CI [−0.26, −0.18]) for procrastination and perfectionistic strivings. The average correlations remained significant after statistically accounting for the joint variance between the two perfectionism dimensions via semi–partial correlations. For perfectionistic concerns, but not perfectionistic strivings, the effects depended on the perfectionism measure used. All effects did not vary by the trait procrastination measure used or the respondent's sex. Our findings confirm that from a multidimensional perspective, trait procrastination is both positively and negatively associated with higher–order perfectionism dimensions and further highlights the value of a self–regulation perspective for understanding the cognitive, affective and behavioural dynamics that characterise these traits. Copyright © 2017 European Association of Personality Psychology

scholarly journals A Comparison between Enriched and Nonenriched Enrollment Randomized Withdrawal Trials of Opioids for Chronic Noncancer Pain

2011 ◽  
Vol 16 (5) ◽  
pp. 337-351 ◽  
Author(s):  
Andrea D Furlan ◽  
Luis E Chaparro ◽  
Emma Irvin ◽  
Angela Mailis-Gagnon
Keyword(s):  
Adverse Effects ◽  
Effect Size ◽  
Meta Analysis ◽  
Primary Objective ◽  
Chronic Noncancer Pain ◽  
Enrichment Designs ◽  
Noncancer Pain ◽  
Strong Opioids ◽  
Meta Analyses ◽  
And Function

An enriched enrollment randomized withdrawal (EERW) trial design has been advocated to be useful for the study of drugs that are beneficial to only a fraction of the individuals who take them. Some investigators defend the use of enrichment designs for opioids in chronic noncancer pain (CNCP), reasoning that opioids may appear to underperform in clinically heterogeneous contexts, ie, that substantial efficacy in a particular patient subgroup may be diluted or masked by poor efficacy in another subgroup. The authors previously published a systematic review of opioids for CNCP in 2006; however, at that time, there were only a few EERW trials available for comparison. This more exhaustive, updated review compares the results between EERW and non-EERW trials of opioids for a variety of CNCP conditions.BACKGROUND: An enriched enrollment randomized withdrawal (EERW) design excludes potential participants who are nonresponders or who cannot tolerate the experimental drug before random assignment. It is unclear whether EERW design has an influence on the efficacy and safety of opioids for chronic noncancer pain (CNCP).OBJECTIVES: The primary objective was to compare the results from EERW and non-EERW trials of opioids for CNCP. Secondary objectives were to compare weak versus strong opioids, subgroups of patients with different types of pain, and the efficacy of opiods compared with placebo versus other drugs.METHODS: MEDLINE, EMBASE and CENTRAL were searched up to July 2009, for randomized controlled trials of any opioid for CNCP. Meta-analyses and meta-regressions were conducted to compare the results. Treatment efficacy was assessed by effect sizes (small, medium and large) and the incidence of adverse effects was assessed by a clinically relevant mean difference of 10% or greater.RESULTS: Sixty-two randomized trials were included. In 61 trials, the duration was less than 16 weeks. There was no difference in efficacy between EERW and non-EERW trials for both pain (P=0.6) and function (P=0.3). However, EERW trials failed to detect a clinically relevant difference for nausea, vomiting, somnolence, dizziness and dry skin/itching compared with non-EERW. Opioids were more effective than placebo in patients with nociceptive pain (effect size=0.60, 95% CI 0.49 to 0.72) and neuropathic pain (effect size=0.56, 95% CI 0.38 to 0.73).CONCLUSION: EERW trial designs appear not to bias the results of efficacy, but they underestimate the adverse effects. The present updated meta-analysis shows that weak and strong opioids are effective for CNCP of both nociceptive and neuropathic origin.

Sign in / Sign up

Export Citation Format

Share Document