Consequences of effect size heterogeneity for meta-analysis: a Monte Carlo study

The inclusion of partial regression coefficients in the Ferguson (2015a) meta-analysis published in Perspectives on Psychological Science caught our eye. While some anecdotal support was given in this paper to justify this practice, no empirical rationale was offered. This is odd, because a great deal of methodological literature cautions against this practice. To better address this concern, we present evidence through both computational examples and Monte Carlo simulations to suggest that the inclusion of partial relationships in meta-analytic models represents a statistical misspecification that obfuscates the ability to estimate the true population effect size. Additionally, we discuss three means of remediating this issue that specifically address the possibility of statistical control in meta-analysis.

Download Full-text

Cross-Temporal Meta-Analysis: A Conceptual and Empirical Critique

10.31234/osf.io/exskp ◽

2018 ◽

Author(s):

Cort Rudolph ◽

David Costanza ◽

Charlotte Wright ◽

Hannes Zacher

Keyword(s):

Monte Carlo ◽

Meta Analysis ◽

Monte Carlo Study ◽

Cohort Effects ◽

Social Phenomena ◽

Period Effects ◽

Intractable Problems ◽

Research Questions ◽

Proper Estimation

The proper estimation of age, period, and cohort (APC) effects is a pervasive concern for the study of a variety of psychological and social phenomena, inside and outside of organizations. One analytic technique that has been used to estimate APC effects is cross-temporal meta- analysis (CTMA). While CTMA has some appealing qualities (e.g., ease of interpretability), it has also been criticized on theoretical and methodological grounds. Furthermore, CTMA makes strong assumptions about the nature and operation of cohort effects relative to age and period effects that have not been empirically tested. Accordingly, the goal of this paper was to explore CTMA, its history, and these assumptions. Using a Monte Carlo study, we demonstrate that in many cases, cohort effects are misestimated (i.e., systematically over- or underestimated) by CTMA. This work provides further evidence that APC effects pose intractable problems for research questions where APC effects are of interest.

Download Full-text

Posterior Probabilities of Effect Sizes and Heterogeneity in Meta-Analysis: An Intuitive Approach of Dealing with Publication Bias

10.31219/osf.io/avkgj ◽

2021 ◽

Author(s):

Hilde Elisabeth Maria Augusteijn ◽

Robbie Cornelis Maria van Aert ◽

Marcel A. L. M. van Assen

Keyword(s):

Publication Bias ◽

Effect Size ◽

Web Application ◽

Meta Analysis ◽

Real Data ◽

Effect Sizes ◽

Posterior Probabilities ◽

True Effect Size ◽

Heterogeneity Parameter ◽

Size Heterogeneity

Publication bias remains to be a great challenge when conducting a meta-analysis. It may result in overestimated effect sizes, increased frequency of false positives, and over- or underestimation of the effect size heterogeneity parameter. A new method is introduced, Bayesian Meta-Analytic Snapshot (BMAS), which evaluates both effect size and its heterogeneity and corrects for potential publication bias. It evaluates the probability of the true effect size being zero, small, medium or large, and the probability of true heterogeneity being zero, small, medium or large. This approach, which provides an intuitive evaluation of uncertainty in the evaluation of effect size and heterogeneity, is illustrated with a real-data example, a simulation study, and a Shiny web application of BMAS.

Download Full-text