Can Reliance be Placed on a Single Meta-Analysis?

1990 ◽  
Vol 24 (3) ◽  
pp. 405-415 ◽  
Author(s):  
Nathaniel McConaghy
Keyword(s):  
Literature Review ◽  
Effect Size ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Effect Sizes ◽  
Control Groups ◽  
Consistent Finding ◽  
Placebo Controls ◽  
Effect Of Treatment ◽  
Meta Analyses

Meta-analysis replaced statistical significance with effect size in the hope of resolving controversy concerning evaluation of treatment effects. Statistical significance measured reliability of the effect of treatment, not its efficacy. It was strongly influenced by the number of subjects investigated. Effect size as assessed originally, eliminated this influence but by standardizing the size of the treatment effect could distort it. Meta-analyses which combine the results of studies which employ different subject types, outcome measures, treatment aims, no-treatment rather than placebo controls or therapists with varying experience can be misleading. To ensure discussion of these variables meta-analyses should be used as an aid rather than a substitute for literature review. While meta-analyses produce contradictory findings, it seems unwise to rely on the conclusions of an individual analysis. Their consistent finding that placebo treatments obtain markedly higher effect sizes than no treatment hopefully will render the use of untreated control groups obsolete.

scholarly journals A Statistical Method for Synthesizing Meta-Analyses

2013 ◽  
Vol 2013 ◽  
pp. 1-9 ◽  
Author(s):  
Liansheng Larry Tang ◽  
Michael Caudy ◽  
Faye Taxman
Keyword(s):  
Statistical Method ◽  
Effect Size ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Weighted Mean ◽  
Summary Effect ◽  
Different Types ◽  
Meta Analyses ◽  
Size Estimates ◽  
Similar Search

Multiple meta-analyses may use similar search criteria and focus on the same topic of interest, but they may yield different or sometimes discordant results. The lack of statistical methods for synthesizing these findings makes it challenging to properly interpret the results from multiple meta-analyses, especially when their results are conflicting. In this paper, we first introduce a method to synthesize the meta-analytic results when multiple meta-analyses use the same type of summary effect estimates. When meta-analyses use different types of effect sizes, the meta-analysis results cannot be directly combined. We propose a two-step frequentist procedure to first convert the effect size estimates to the same metric and then summarize them with a weighted mean estimate. Our proposed method offers several advantages over existing methods by Hemming et al. (2012). First, different types of summary effect sizes are considered. Second, our method provides the same overall effect size as conducting a meta-analysis on all individual studies from multiple meta-analyses. We illustrate the application of the proposed methods in two examples and discuss their implications for the field of meta-analysis.

M. Meta-Analysis

2021 ◽  
pp. 163-165
Author(s):  
Noémie Laurens
Keyword(s):  
Literature Review ◽  
Empirical Research ◽  
Effect Size ◽  
Meta Analysis ◽  
Research Synthesis ◽  
Effect Sizes ◽  
Primary Research ◽  
Unit Of Analysis ◽  
Research Studies

This chapter illustrates meta-analysis, which is a specific type of literature review, and more precisely a type of research synthesis, alongside traditional narrative reviews. Unlike in primary research, the unit of analysis of a meta-analysis is the results of individual studies. And unlike traditional reviews, meta-analysis only applies to: empirical research studies with quantitative findings hat are conceptually comparable and configured in similar statistical forms. What further distinguishes meta-analysis from other research syntheses is the method of synthesizing the results of studies — i.e. the use of statistics and, in particular, of effect sizes. An effect size represents the degree to which the phenomenon under study exists.

scholarly journals Influence of Pilot and Small Trials in Meta-Analyses of Behavioral Interventions: A Meta-epidemiological Study

2020 ◽  
Author(s):  
Michael W. Beets ◽  
R. Glenn Weaver ◽  
John P.A. Ioannidis ◽  
Alexis Jones ◽  
Lauren von Klinggraeff ◽  
...  
Keyword(s):  
Sample Size ◽  
Behavioral Interventions ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Feasibility Studies ◽  
Small Sample ◽  
Effect Sizes ◽  
Absolute Difference ◽  
Meta Analyses ◽  
The Impact

Abstract Background: Pilot/feasibility or studies with small sample sizes may be associated with inflated effects. This study explores the vibration of effect sizes (VoE) in meta-analyses when considering different inclusion criteria based upon sample size or pilot/feasibility status. Methods: Searches were conducted for meta-analyses of behavioral interventions on topics related to the prevention/treatment of childhood obesity from 01-2016 to 10-2019. The computed summary effect sizes (ES) were extracted from each meta-analysis. Individual studies included in the meta-analyses were classified into one of the following four categories: self-identified pilot/feasibility studies or based upon sample size (N≤100, N>100, and N>370 the upper 75th of sample size). The VoE was defined as the absolute difference (ABS) between the re-estimations of summary ES restricted to study classifications compared to the originally reported summary ES. Concordance (kappa) of statistical significance between summary ES was assessed. Fixed and random effects models and meta-regressions were estimated. Three case studies are presented to illustrate the impact of including pilot/feasibility and N≤100 studies on the estimated summary ES.Results: A total of 1,602 effect sizes, representing 145 reported summary ES, were extracted from 48 meta-analyses containing 603 unique studies (avg. 22 avg. meta-analysis, range 2-108) and included 227,217 participants. Pilot/feasibility and N≤100 studies comprised 22% (0-58%) and 21% (0-83%) of studies. Meta-regression indicated the ABS between the re-estimated and original summary ES where summary ES were comprised of ≥40% of N≤100 studies was 0.29. The ABS ES was 0.46 when summary ES comprised of >80% of both pilot/feasibility and N≤100 studies. Where ≤40% of the studies comprising a summary ES had N>370, the ABS ES ranged from 0.20-0.30. Concordance was low when removing both pilot/feasibility and N≤100 studies (kappa=0.53) and restricting analyses only to the largest studies (N>370, kappa=0.35), with 20% and 26% of the originally reported statistically significant ES rendered non-significant. Reanalysis of the three case study meta-analyses resulted in the re-estimated ES rendered either non-significant or half of the originally reported ES. Conclusions: When meta-analyses of behavioral interventions include a substantial proportion of both pilot/feasibility and N≤100 studies, summary ES can be affected markedly and should be interpreted with caution.

scholarly journals Low statistical power in biomedical science: a review of three human research domains

2017 ◽  
Vol 4 (2) ◽  
pp. 160254 ◽  
Author(s):  
Estelle Dumas-Mallet ◽  
Katherine S. Button ◽  
Thomas Boraud ◽  
Francois Gonon ◽  
Marcus R. Munafò
Keyword(s):  
Effect Size ◽  
Statistical Power ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Average Power ◽  
Biomedical Science ◽  
Significant Finding ◽  
Biomedical Sciences ◽  
True Effect Size ◽  
Meta Analyses

Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result. We conducted a review of meta-analyses of studies investigating the association of biological, environmental or cognitive parameters with neurological, psychiatric and somatic diseases, excluding treatment studies, in order to estimate the average statistical power across these domains. Taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and assuming a threshold for declaring statistical significance of 5%, we found that approximately 50% of studies have statistical power in the 0–10% or 11–20% range, well below the minimum of 80% that is often considered conventional. Studies with low statistical power appear to be common in the biomedical sciences, at least in the specific subject areas captured by our search strategy. However, we also observe evidence that this depends in part on research methodology, with candidate gene studies showing very low average power and studies using cognitive/behavioural measures showing high average power. This warrants further investigation.

scholarly journals Cannons and sparrows II: the enhanced Bernoulli exact method for determining statistical significance and effect size in the meta-analysis of k 2 × 2 tables

2021 ◽  
Vol 18 (1) ◽  
Author(s):  
Lawrence M. Paul
Keyword(s):  
Effect Size ◽  
Type I Error ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Statistical Error ◽  
Exact Method ◽  
Type I ◽  
Exact Test ◽  
Inverse Variance ◽  
Meta Analyses

Abstract Background The use of meta-analysis to aggregate the results of multiple studies has increased dramatically over the last 40 years. For homogeneous meta-analysis, the Mantel–Haenszel technique has typically been utilized. In such meta-analyses, the effect size across the contributing studies of the meta-analysis differs only by statistical error. If homogeneity cannot be assumed or established, the most popular technique developed to date is the inverse-variance DerSimonian and Laird (DL) technique (DerSimonian and Laird, in Control Clin Trials 7(3):177–88, 1986). However, both of these techniques are based on large sample, asymptotic assumptions. At best, they are approximations especially when the number of cases observed in any cell of the corresponding contingency tables is small. Results This research develops an exact, non-parametric test for evaluating statistical significance and a related method for estimating effect size in the meta-analysis of k 2 × 2 tables for any level of heterogeneity as an alternative to the asymptotic techniques. Monte Carlo simulations show that even for large values of heterogeneity, the Enhanced Bernoulli Technique (EBT) is far superior at maintaining the pre-specified level of Type I Error than the DL technique. A fully tested implementation in the R statistical language is freely available from the author. In addition, a second related exact test for estimating the Effect Size was developed and is also freely available. Conclusions This research has developed two exact tests for the meta-analysis of dichotomous, categorical data. The EBT technique was strongly superior to the DL technique in maintaining a pre-specified level of Type I Error even at extremely high levels of heterogeneity. As shown, the DL technique demonstrated many large violations of this level. Given the various biases towards finding statistical significance prevalent in epidemiology today, a strong focus on maintaining a pre-specified level of Type I Error would seem critical. In addition, a related exact method for estimating the Effect Size was developed.

Experimental and Quasi-Experimental Studies of Inquiry-Based Science Teaching

2012 ◽  
Vol 82 (3) ◽  
pp. 300-329 ◽  
Author(s):  
Erin Marie Furtak ◽  
Tina Seidel ◽  
Heidi Iverson ◽  
Derek C. Briggs
Keyword(s):  
Science Education ◽  
Education Reform ◽  
Science Teaching ◽  
Effect Size ◽  
Meta Analysis ◽  
Experimental Studies ◽  
Effect Sizes ◽  
Science Education Reform ◽  
Quasi Experimental ◽  
Meta Analyses

Although previous meta-analyses have indicated a connection between inquiry-based teaching and improved student learning, the type of instruction characterized as inquiry based has varied greatly, and few have focused on the extent to which activities are led by the teacher or student. This meta-analysis introduces a framework for inquiry-based teaching that distinguishes between cognitive features of the activity and degree of guidance given to students. This framework is used to code 37 experimental and quasi-experimental studies published between 1996 and 2006, a decade during which inquiry was the main focus of science education reform. The overall mean effect size is .50. Studies that contrasted epistemic activities or the combination of procedural, epistemic, and social activities had the highest mean effect sizes. Furthermore, studies involving teacher-led activities had mean effect sizes about .40 larger than those with student-led conditions. The importance of establishing the validity of the treatment construct in meta-analyses is also discussed.

scholarly journals The Effects of Physical Education on Motor Competence in Children and Adolescents: A Systematic Review and Meta-Analysis

Sports ◽  
2020 ◽  
Vol 8 (6) ◽  
pp. 88 ◽  
Author(s):  
Håvard Lorås
Keyword(s):  
Physical Education ◽  
Children And Adolescents ◽  
Motor Skills ◽  
Effect Size ◽  
Meta Analysis ◽  
Control Groups ◽  
Motor Competence ◽  
Random Effects Models ◽  
Health Related ◽  
Meta Analyses

Appropriate levels of motor competence are an integrated part of individuals’ health-related fitness, and physical education is proposed as an important context for developing a broad range of motor skills. The aim of the current study was to apply meta-analyses to assess the effectiveness of curriculum-based physical education on the development of the overall motor competence of children and adolescents. Studies were located by searching seven databases and included according to predefined criteria. Random effects models using the standardized effect size (Hedges’ g) were used to aggregate results, including an examination of heterogeneity and inconsistency. The meta-analysis included 20 studies, and a total of 38 effect sizes were calculated. A statistically significant improvement in motor competence following curriculum-based physical education compared to active control groups was observed in children and adolescents (g = −0.69, 95% CI −0.91 to −0.46, n = 23). Participants’ ages, total time for physical education intervention, and type of motor competence assessment did not appear to be statistically significant moderators of effect size. Physical education with various curricula can, therefore, increase overall motor competence in children and adolescents.

scholarly journals Comparison of the Effectiveness of Whole Body Vibration in Stroke Patients: A Meta-Analysis

2018 ◽  
Vol 2018 ◽  
pp. 1-10 ◽  
Author(s):  
Yoo Jung Park ◽  
Sun Wook Park ◽  
Han Suk Lee
Keyword(s):  
Muscle Strength ◽  
Effect Size ◽  
Meta Analysis ◽  
Whole Body Vibration ◽  
Treatment Method ◽  
Effect Sizes ◽  
Whole Body ◽  
Safe Alternative ◽  
Body Vibration ◽  
Meta Analyses

Objectives. The goals of this study were to assess the effectiveness of WBV (whole body vibration) training through an analysis of effect sizes, identify advantages of WBV training, and suggest other effective treatment methods. Methods. Four databases, namely, EMBASE, PubMed, EBSCO, and Web of Science, were used to collect articles on vibration. Keywords such as “vibration” and “stroke” were used in the search for published articles. Consequently, eleven studies were selected in the second screening using meta-analyses. Results. The total effect size of patients with dementia in the studies was 0.25, which was small. The effect size of spasticity was the greatest at 1.24 (high), followed by metabolism at 0.99 (high), balance, muscle strength, gait, and circulation in the decreasing order of effect size. Conclusions. The effect sizes for muscle strength and balance and gait function, all of which play an important role in performance of daily activities, were small. In contrast, effect sizes for bone metabolism and spasticity were moderate. This suggests that WBV training may provide a safe, alternative treatment method for improving the symptoms of stroke in patients.

scholarly journals Psychological treatment of social anxiety disorder: a meta-analysis

2008 ◽  
Vol 39 (2) ◽  
pp. 241-254 ◽  
Author(s):  
C. Acarturk ◽  
P. Cuijpers ◽  
A. van Straten ◽  
R. de Graaf
Keyword(s):  
Anxiety Disorder ◽  
Social Anxiety ◽  
Social Anxiety Disorder ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Placebo Control ◽  
Control Groups ◽  
Psychological Treatments ◽  
Randomized Studies ◽  
Meta Analyses

BackgroundOlder meta-analyses of the effects of psychological treatments of social anxiety disorder have found that these treatments have moderate to large effects. However, these earlier meta-analyses also included non-randomized studies, and there are many featured studies in this area which were published after the recent meta-analysis.MethodWe conducted a systematic literature search and identified 29 randomized studies examining the effects of psychological treatments, with a total of 1628 subjects. The quality of studies varied. For the analyses, we used the computer program comprehensive meta-analysis (version 2.2.021; Biostat, Englewood, NJ, USA).ResultsThe mean effect size on social anxiety measures (47 contrast groups) was 0.70, 0.80 on cognitive measures (26 contrast groups) and 0.70 both on depression (19 contrast groups) and general anxiety measures (16 contrast groups). We found some heterogeneity, so we conducted a series of subgroup analyses for different variables of the studies. Studies with waiting-list control groups had significantly larger effect sizes than studies with placebo and treatment-as-usual control groups. Studies aimed at subjects who met Diagnostic and Statistical Manual of Mental Disorders (DSM) criteria for social anxiety disorder had smaller effect sizes than studies in which other inclusion criteria were used.ConclusionsThis study once more makes it clear that psychological treatments of social anxiety disorder are effective in adults, but that they may be less effective in more severe disorders and in studies in which care-as-usual and placebo control groups are used.

scholarly journals Canons and Sparrows II*: The Enhanced Bernoulli Exact Method for Determining Statistical Significance and Effect Size in the Meta-Analysis of k 2 x 2 Tables

2021 ◽  
Author(s):  
Lawrence Marc Paul
Keyword(s):  
Effect Size ◽  
Type I Error ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Statistical Error ◽  
Type I ◽  
Exact Test ◽  
Strong Focus ◽  
Inverse Variance ◽  
Meta Analyses

Abstract BackgroundThe use of meta-analysis to aggregate the results of multiple studies has increased dramatically over the last 40 years. For homogeneous meta-analysis, the Mantel-Haenszel technique has typically been utilized. In such meta-analyses, the effect size across the contributing studies of the meta-analysis differ only by statistical error. If homogeneity cannot be assumed or established, the most popular technique developed to date is the inverse-variance DerSimonian & Laird (DL) technique [1]. However, both of these techniques are based on large sample, asymptotic assumptions. At best, they are approximations especially when the number of cases observed in any cell of the corresponding contingency tables is small.ResultsThis research develops an exact, non-parametric test for evaluating statistical significance and a related method for estimating effect size in the meta-analysis of k 2 x 2 tables for any level of heterogeneity as an alternative to the asymptotic techniques. Monte Carlo simulations show that even for large values of heterogeneity, the Enhanced Bernoulli Technique (EBT) is far superior at maintaining the pre-specified level of Type I Error than the DL technique. A fully tested implementation in the R statistical language is freely available from the author. In addition, a second related exact test for estimating the Effect Size was developed and is also freely available.ConclusionsThis research has developed two exact tests for the meta-analysis of dichotomous, categorical data. The EBT technique was strongly superior to the DL technique in maintaining a pre-specified level of Type I Error even at extremely high levels of heterogeneity. As shown, the DL technique demonstrated many large violations of this level. Given the various biases towards finding statistical significance prevalent in epidemiology today, a strong focus on maintaining a pre-specified level of Type I Error would seem critical.

Sign in / Sign up

Export Citation Format

Share Document