Supplemental Material for Meta-Analysis to Integrate Effect Sizes Within an Article: Possible Misuse and Type I Error Inflation

2016 ◽  
Keyword(s):  
Type I Error ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Type I

scholarly journals Power Approximations for Meta-Analysis of Dependent Effect Sizes

2022 ◽  
Author(s):  
Mikkel Helding Vembye ◽  
James E Pustejovsky ◽  
Terri Pigott
Keyword(s):  
Variance Estimation ◽  
Type I Error ◽  
Meta Analysis ◽  
A Priori ◽  
Effect Sizes ◽  
Type I ◽  
Dependent Effect ◽  
Dependent Effect Sizes ◽  
Robust Variance Estimation ◽  
Analytic Models

Meta-analytic models for dependent effect sizes have grown increasingly sophisticated over the last few decades, which has created challenges for a priori power calculations. We introduce power approximations for tests of average effect sizes based upon the most common models for handling dependent effect sizes. In a Monte Carlo simulation, we show that the new power formulas can accurately approximate the true power of common meta-analytic models for dependent effect sizes. Lastly, we investigate the Type I error rate and power for several common models, finding that tests using robust variance estimation provide better Type I error calibration than tests with model-based variance estimation. We consider implications for practice with respect to selecting a working model and an inferential approach.

scholarly journals Model Misspecification and Assumption Violations With the Linear Mixed Model: A Meta-Analysis

SAGE Open ◽  
2018 ◽  
Vol 8 (4) ◽  
pp. 215824401882038 ◽  
Author(s):  
Brandon LeBeau ◽  
Yoon Ah Song ◽  
Wei Cheng Liu
Keyword(s):  
Random Effects ◽  
Error Rate ◽  
Mixed Model ◽  
Linear Mixed Model ◽  
Type I Error ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Type I ◽  
Type I Error Rate ◽  
Fixed And Random Effects

This meta-analysis attempts to synthesize the Monte Carlo (MC) literature for the linear mixed model under a longitudinal framework. The meta-analysis aims to inform researchers about conditions that are important to consider when evaluating model assumptions and adequacy. In addition, the meta-analysis may be helpful to those wishing to design future MC simulations in identifying simulation conditions. The current meta-analysis will use the empirical type I error rate as the effect size and MC simulation conditions will be coded to serve as moderator variables. The type I error rate for the fixed and random effects will be explored as the primary dependent variable. Effect sizes were coded from 13 studies, resulting in a total of 4,002 and 621 effect sizes for fixed and random effects respectively. Meta-regression and proportional odds models were used to explore variation in the empirical type I error rate effect sizes. Implications for applied researchers and researchers planning new MC studies will be explored.

The Robustness of the Likelihood Ratio Chi-Square Test for Structural Equation Models: A Meta-Analysis

2001 ◽  
Vol 26 (1) ◽  
pp. 105-132 ◽  
Author(s):  
Douglas A. Powell ◽  
William D. Schafer
Keyword(s):  
Structural Equation ◽  
Structural Equation Models ◽  
Type I Error ◽  
Meta Analysis ◽  
Generalized Least Squares ◽  
Error Rates ◽  
Type I ◽  
Chi Square ◽  
Distribution Free ◽  
Projection Techniques

The robustness literature for the structural equation model was synthesized following the method of Harwell which employs meta-analysis as developed by Hedges and Vevea. The study focused on the explanation of empirical Type I error rates for six principal classes of estimators: two that assume multivariate normality (maximum likelihood and generalized least squares), elliptical estimators, two distribution-free estimators (asymptotic and others), and latent projection. Generally, the chi-square tests for overall model fit were found to be sensitive to non-normality and the size of the model for all estimators (with the possible exception of the elliptical estimators with respect to model size and the latent projection techniques with respect to non-normality). The asymptotic distribution-free (ADF) and latent projection techniques were also found to be sensitive to sample sizes. Distribution-free methods other than ADF showed, in general, much less sensitivity to all factors considered.

scholarly journals The Relationship Between Study Quality and the Effects of Supplemental Reading Interventions: A Meta-Analysis

2018 ◽  
Vol 85 (3) ◽  
pp. 347-366
Author(s):  
Christy R. Austin ◽  
Jeanne Wanzek ◽  
Nancy K. Scammacca ◽  
Sharon Vaughn ◽  
Samantha A. Gesel ◽  
...  
Keyword(s):  
Reading Skills ◽  
Reading Interventions ◽  
Meta Analysis ◽  
Statistical Treatment ◽  
Effect Sizes ◽  
Type I ◽  
Supplemental Reading ◽  
Study Quality ◽  
Standardized Measures ◽  
Treatment Type

Empirical studies investigating supplemental reading interventions for students with or at risk for reading disabilities in the early elementary grades have demonstrated a range of effect sizes. Identifying the findings from high-quality research can provide greater certainty of findings related to the effectiveness of supplemental reading interventions. This meta-analysis investigated how four variables of study quality (study design, statistical treatment, Type I error, and fidelity of implementation) were related to effect sizes from standardized measures of foundational reading skills and language and comprehension. The results from 88 studies indicated that year of publication was a significant predictor of effect sizes for both standardized measures of foundational reading skills and language and comprehension, with more recent studies demonstrating smaller effect sizes. Results also demonstrated that with the exception of research design predicting effect sizes on foundational reading skills measures, study quality was not related to the effects of supplemental reading interventions. Implications for research and practice are discussed.

scholarly journals Cluster Wild Bootstrapping to Handle Dependent Effect Sizes in Meta-Analysis with a Small Number of Studies

2021 ◽  
Author(s):  
Megha Joshi ◽  
James E Pustejovsky ◽  
S. Natasha Beretvas
Keyword(s):  
Effect Size ◽  
Type I Error ◽  
Meta Analysis ◽  
Error Rates ◽  
Small Sample ◽  
Type I ◽  
Hypothesis Tests ◽  
Type I Error Rates ◽  
Meta Analyses ◽  
Small Sample Correction

The most common and well-known meta-regression models work under the assumption that there is only one effect size estimate per study and that the estimates are independent. However, meta-analytic reviews of social science research often include multiple effect size estimates per primary study, leading to dependence in the estimates. Some meta-analyses also include multiple studies conducted by the same lab or investigator, creating another potential source of dependence. An increasingly popular method to handle dependence is robust variance estimation (RVE), but this method can result in inflated Type I error rates when the number of studies is small. Small-sample correction methods for RVE have been shown to control Type I error rates adequately but may be overly conservative, especially for tests of multiple-contrast hypotheses. We evaluated an alternative method for handling dependence, cluster wild bootstrapping, which has been examined in the econometrics literature but not in the context of meta-analysis. Results from two simulation studies indicate that cluster wild bootstrapping maintains adequate Type I error rates and provides more power than extant small sample correction methods, particularly for multiple-contrast hypothesis tests. We recommend using cluster wild bootstrapping to conduct hypothesis tests for meta-analyses with a small number of studies. We have also created an R package that implements such tests.

scholarly journals A Likelihood Ratio Test For The Homogeneity of Between-Study Variance in Network Meta-Analysis

2021 ◽  
Author(s):  
Dapeng Hu ◽  
Chong Wang ◽  
Annette O'Connor
Keyword(s):  
Likelihood Ratio ◽  
Likelihood Ratio Test ◽  
Random Effects ◽  
Type I Error ◽  
Meta Analysis ◽  
Indirect Evidence ◽  
Point Estimate ◽  
Ratio Test ◽  
Type I ◽  
Statistical Heterogeneity

Abstract Background: Network meta-analysis (NMA) is a statistical method used to combine results from several clinical trials and simultaneously compare multiple treatments using direct and indirect evidence. Statistical heterogeneity is a characteristic describing the variability in the intervention effects being evaluated in the different studies in network meta-analysis. One approach to dealing with statistical heterogeneity is to perform a random effects network meta-analysis that incorporates a between-study variance into the statistical model. A common assumption in the random effects model for network meta-analysis is the homogeneity of between-study variance across all interventions. However, there are applications of NMA where the single between-study assumption is potentially incorrect and instead the model should incorporate more than one between-study variances. Methods: In this paper, we develop an approach to testing the homogeneity of between-study variance assumption based on a likelihood ratio test. A simulation study was conducted to assess the type I error and power of the proposed test. This method is then applied to a network meta-analysis of antibiotic treatments for Bovine respiratory disease (BRD). Results: The type I error rate was well controlled in the Monte Carlo simulation. The homogeneous between-study variance assumption is unrealistic both statistically and practically in the network meta-analysis BRD. The point estimate and conffdence interval of relative effect sizes are strongly inuenced by this assumption. Conclusions: Since homogeneous between-study variance assumption is a strong assumption, it is crucial to test the validity of this assumption before conducting a network meta-analysis. Here we propose and validate a method for testing this single between-study variance assumption which is widely used for many NMA.

Is It Really Robust?

Methodology ◽  
2010 ◽  
Vol 6 (4) ◽  
pp. 147-151 ◽  
Author(s):  
Emanuel Schmider ◽  
Matthias Ziegler ◽  
Erik Danay ◽  
Luzi Beyer ◽  
Markus Bühner
Keyword(s):  
Goodness Of Fit ◽  
Type I Error ◽  
Effect Sizes ◽  
Random Numbers ◽  
Type I ◽  
Type Ii ◽  
Type Ii Error ◽  
Normality Assumption ◽  
Different Types ◽  
Factor Type

Empirical evidence to the robustness of the analysis of variance (ANOVA) concerning violation of the normality assumption is presented by means of Monte Carlo methods. High-quality samples underlying normally, rectangularly, and exponentially distributed basic populations are created by drawing samples which consist of random numbers from respective generators, checking their goodness of fit, and allowing only the best 10% to take part in the investigation. A one-way fixed-effect design with three groups of 25 values each is chosen. Effect-sizes are implemented in the samples and varied over a broad range. Comparing the outcomes of the ANOVA calculations for the different types of distributions, gives reason to regard the ANOVA as robust. Both, the empirical type I error α and the empirical type II error β remain constant under violation. Moreover, regression analysis identifies the factor “type of distribution” as not significant in explanation of the ANOVA results.

scholarly journals Calibrate your confidence in research findings: A tutorial on improving research methods and practices

2020 ◽  
Vol 14 ◽  
Author(s):  
Aline da Silva Frost ◽  
Alison Ledgerwood
Keyword(s):  
Research Methods ◽  
Statistical Power ◽  
Type I Error ◽  
Effect Sizes ◽  
Online Media ◽  
Type I ◽  
Journal Articles ◽  
Psychological Science ◽  
Different Types ◽  
Research Findings

Abstract This article provides an accessible tutorial with concrete guidance for how to start improving research methods and practices in your lab. Following recent calls to improve research methods and practices within and beyond the borders of psychological science, resources have proliferated across book chapters, journal articles, and online media. Many researchers are interested in learning more about cutting-edge methods and practices but are unsure where to begin. In this tutorial, we describe specific tools that help researchers calibrate their confidence in a given set of findings. In Part I, we describe strategies for assessing the likely statistical power of a study, including when and how to conduct different types of power calculations, how to estimate effect sizes, and how to think about power for detecting interactions. In Part II, we provide strategies for assessing the likely type I error rate of a study, including distinguishing clearly between data-independent (“confirmatory”) and data-dependent (“exploratory”) analyses and thinking carefully about different forms and functions of preregistration.

Sign in / Sign up

Export Citation Format

Share Document