A Meta-Analysis of the 2004 Campaign Polls: An Analogy to Practice and Publication in Psychology

2007 ◽  
Vol 100 (3) ◽  
pp. 847-856
Author(s):  
Raymond D. Collings ◽  
Leslie G. Eaton ◽  
Miranda Hendrickson
Keyword(s):  
Presidential Election ◽  
Null Hypothesis ◽  
Current Practice ◽  
Meta Analysis ◽  
Presidential Campaign ◽  
Single Sample ◽  
Conventional Analysis ◽  
Margin Of Error ◽  
Statistical Approaches ◽  
Meta Analyses

The current practice of relying on single sample null hypothesis tests is being re-evaluated in the behavioral sciences. To highlight the issues raised by both sides of this discussion, a meta-analysis of The Gallup Organization's most recent U.S. Presidential election polling data was conducted. During the 2004 Presidential campaign, most pre-election polling percentages reported Bush ahead of Kerry, although the differences between the voters' preferences were typically within the margin of error. The meta-analyses used in this study showed significant differences between the two candidates' polling percentages, thus yielding a more accurate prediction than the conventional analysis which was based on single samples. These improved predictions provide support for a continued discussion about potential changes in statistical approaches to psychology.

First Steps in Beginning a Meta-analysis

2013 ◽  
Author(s):  
Gavin B. Stewart ◽  
Isabelle M. Côté ◽  
Hannah R. Rothstein ◽  
Peter S. Curtis
Keyword(s):  
Field Experiment ◽  
Systematic Approach ◽  
Sampling Error ◽  
Meta Analysis ◽  
Research Synthesis ◽  
Systematic Research ◽  
Different Types ◽  
Statistical Approaches ◽  
Definition Of ◽  
Meta Analyses

This chapter discusses the initiation of the process of systematic research synthesis. Without a systematic approach to defining, obtaining, and collating data, meta-analyses may yield precise but erroneous results, with different types of sampling error (biases) and excess subjectivity in choice of methods and definition of thresholds; these devalue the rigor of any statistical approaches employed. The chapter considers exactly the same issues that face an ecologist designing a field experiment. What's the question? How can I define my sampling universe? How should I collect my data? What analyses should I undertake? How should I interpret my results robustly? These questions are considered in the context of research synthesis.

Statistical Models and Approaches to Inference

2013 ◽  
Author(s):  
Kerrie Mengersen ◽  
Christopher H. Schmid ◽  
Michael D. Jennions ◽  
Jessica Gurevitch
Keyword(s):  
Statistical Model ◽  
Statistical Models ◽  
Meta Analysis ◽  
General Structure ◽  
Male Mating ◽  
Female Fecundity ◽  
Mating History ◽  
Statistical Approaches ◽  
Meta Analyses

This chapter provides an introduction and overview of the three statistical components of the meta-analysis: (1) the statistical model that describes how the study-specific estimates of interest will be combined; (2) the key statistical approaches for meta-analysis; and (3) the corresponding estimates, inferences, and decisions that arise from a meta-analysis. First, it describes common statistical models used in ecological meta-analyses and the relationships between these models, showing how they are all variations of the same general structure. It then discusses the three main approaches to analysis and inference, again with the aim of providing a general understanding of these methods. Finally, it briefly considers a number of statistical considerations which arise in meta-analysis. In order to illustrate the concepts described, the chapter considers the Lepidoptera mating example described in Appendix 8.1. This is a meta-analysis of 25 studies of the association between male mating history and female fecundity in Lepidoptera.

Empirical Benchmarks for Interpreting Effect Size Variability in Meta-Analysis

2017 ◽  
Vol 10 (3) ◽  
pp. 472-479 ◽  
Author(s):  
Brenton M. Wiernik ◽  
Jack W. Kostal ◽  
Michael P. Wilmot ◽  
Stephan Dilchert ◽  
Deniz S. Ones
Keyword(s):  
Effect Size ◽  
Null Hypothesis ◽  
Meta Analysis ◽  
Hypothesis Tests ◽  
Size Variability ◽  
Meta Analyses

Generalization in meta-analyses is not a dichotomous decision (typically encountered in papers using the Q test for homogeneity, the 75% rule, or null hypothesis tests). Inattention to effect size variability in meta-analyses may stem from a lack of guidelines for interpreting credibility intervals. In this commentary, we describe two methods for making practical interpretations and determining whether a particular SDρ represents a meaningful level of variability.

“Life Doesn't Happen at the Between-Person Level,” or a Cautionary Note on Generating Scientific Inferences Through Meta-Analyses

2017 ◽  
Vol 10 (3) ◽  
pp. 459-464
Author(s):  
Hannah L. Samuelson ◽  
Jessica R. Fernandez ◽  
James A. Grand
Keyword(s):  
Lived Experiences ◽  
Current Practice ◽  
Meta Analysis ◽  
Sense Making ◽  
Von Bertalanffy ◽  
Cautionary Note ◽  
Research And Practice ◽  
Scientific Inference ◽  
Validity Generalization ◽  
Meta Analyses

The implicit philosophy for how research and practice in industrial and organizational (I-O) psychology has pursued inferences about our field's core phenomena has largely been based on a nomothetic, variable-based, and aggregate/“large-sample” ideal. As Tett, Hundley, and Christiansen (2017) expertly highlight, there are more insightful means for drawing inferences about the nature of such aggregate relationships based on meta-analytic techniques than the current practice in the organizational sciences. However, the motivating force behind our commentary has less to do with the issues raised by Tett et al. (2017) concerning the practice of using meta-analysis for purposes of validity generalization and more to do with the practice of using meta-analysis for purposes of scientific inference. Between-person philosophies in which the end-goal is to identify general conclusions that apply to the aggregate (cf. Hanges & Wang, 2012) have historically guided our scientific inferences and have supported the proliferation of meta-analytic techniques (including what Tett et al. describe as tertiary analyses based on such findings). These philosophies have led to a dearth of understanding at the within-person and social system levels—the levels at which most of our meaningful phenomena exist (e.g., Hamaker, 2012; Von Bertalanffy, 1950). Learning, performing, decision making, communicating, sense-making, feeling/expressing emotion: These are the concepts that drive the lived experiences of individuals both inside and outside of the workplace, and all are vulnerable to being misunderstood or misinterpreted by focusing only on aggregate evidence at the between-person level. Consequently, we wish to first supplement Tett et al.’s recommendations for drawing generalizability inferences in meta-analysis and suggest a “pre-emptive” question (i.e., Question 0) to the list of four they advance in their focal article.

scholarly journals Implicit and explicit influences of religious cognition on Dictator Game transfers

2018 ◽  
Vol 5 (8) ◽  
pp. 170238 ◽  
Author(s):  
Joseph Billingsley ◽  
Cristina M. Gomes ◽  
Michael E. McCullough
Keyword(s):  
Cognitive Processes ◽  
Null Hypothesis ◽  
Meta Analysis ◽  
Dictator Game ◽  
Prosocial Behaviour ◽  
Online Environment ◽  
Bayesian Analyses ◽  
Reliable Effect ◽  
Meta Analyses ◽  
Implicit And Explicit

Does religion promote prosocial behaviour? Despite numerous publications that seem to answer this question affirmatively, divergent results from recent meta-analyses and pre-registered replication efforts suggest that the issue is not yet settled. Uncertainty lingers around (i) whether the effects of religious cognition on prosocial behaviour were obtained through implicit cognitive processes, explicit cognitive processes or both and (ii) whether religious cognition increases generosity only among people disinclined to share with anonymous strangers. Here, we report two experiments designed to address these concerns. In Experiment 1, we sought to replicate Shariff and Norenzayan's demonstration of the effects of implicit religious priming on Dictator Game transfers to anonymous strangers; unlike Shariff and Norenzayan, however, we used an online environment where anonymity was virtually assured. In Experiment 2, we introduced a ‘taking’ option to allow greater expression of baseline selfishness. In both experiments, we sought to activate religious cognition implicitly and explicitly, and we investigated the possibility that religious priming depends on the extent to which subjects view God as a punishing, authoritarian figure. Results indicated that in both experiments, religious subjects transferred more money on average than did non-religious subjects. Bayesian analyses supported the null hypothesis that implicit religious priming did not increase Dictator Game transfers in either experiment, even among religious subjects. Collectively, the two experiments furnished support for a small but reliable effect of explicit priming, though among religious subjects only. Neither experiment supported the hypothesis that the effect of religious priming depends on viewing God as a punishing figure. Finally, in a meta-analysis of relevant studies, we found that the overall effect of implicit religious priming on Dictator Game transfers was small and did not statistically differ from zero.

scholarly journals Implicit and Explicit Influences of Religious Cognition on Dictator Game Transfers

2018 ◽  
Author(s):  
Michael E. McCullough ◽  
Joseph Billingsley ◽  
Cristina Gomes
Keyword(s):  
Cognitive Processes ◽  
Strong Evidence ◽  
Null Hypothesis ◽  
Meta Analysis ◽  
Dictator Game ◽  
Prosocial Behaviour ◽  
Online Environment ◽  
Reliable Effect ◽  
Meta Analyses ◽  
Implicit And Explicit

Does religious cognition motivate generosity toward strangers? Divergent results from recent meta-analyses and pre-registered replication efforts suggest the issue is not yet settled. Additional uncertainty lingers around whether (a) the effects of religious cognition on prosocial behaviour obtain through implicit cognitive processes, explicit cognitive processes, or both; (b) whether religious cognition might increase generosity only among religious people; and (c) whether religious cognition might increase generosity only among people otherwise disinclined to share with anonymous strangers. Here we report the results of two experiments designed to address these concerns. In Experiment 1 we sought to replicate the classic demonstration of the effect of implicit religious priming on Dictator Game transfers, but in an online environment that maximises anonymity. In Experiment 2, we gave subjects the option to take as well as to give money, allowing greater expression of baseline selfishness. In both experiments, we sought to activate religious cognition implicitly and explicitly, and we investigated the possibility that religious priming depends upon the extent to which subjects view God as a punishing, authoritarian figure. Bayesian statistical methods supported the null hypothesis that implicit religious priming did not increase Dictator Game transfers in either experiment, even among religious subjects. Collectively, the two experiments provided support for a small but reliable effect of explicit priming, though among religious subjects only. Neither experiment offered strong evidence to support the hypothesis that the effect of religious priming depends upon viewing God as a punishing figure. Finally, in a random-effects meta-analysis of relevant studies, we found that the overall effect of implicit religious priming on Dictator Game transfers was small and not statistically different from zero.

A Meta-Analysis of Interviews and Cognitive Ability

2013 ◽  
Vol 12 (4) ◽  
pp. 157-169 ◽  
Author(s):  
Philip L. Roth ◽  
Allen I. Huffcutt
Keyword(s):  
Cognitive Ability ◽  
Meta Analysis ◽  
Incremental Validity ◽  
Employment Interviews ◽  
Moderator Analysis ◽  
The Future ◽  
Research Questions ◽  
Meta Analyses ◽  
The Relationship

The topic of what interviews measure has received a great deal of attention over the years. One line of research has investigated the relationship between interviews and the construct of cognitive ability. A previous meta-analysis reported an overall corrected correlation of .40 ( Huffcutt, Roth, & McDaniel, 1996 ). A more recent meta-analysis reported a noticeably lower corrected correlation of .27 ( Berry, Sackett, & Landers, 2007 ). After reviewing both meta-analyses, it appears that the two studies posed different research questions. Further, there were a number of coding judgments in Berry et al. that merit review, and there was no moderator analysis for educational versus employment interviews. As a result, we reanalyzed the work by Berry et al. and found a corrected correlation of .42 for employment interviews (.15 higher than Berry et al., a 56% increase). Further, educational interviews were associated with a corrected correlation of .21, supporting their influence as a moderator. We suggest a better estimate of the correlation between employment interviews and cognitive ability is .42, and this takes us “back to the future” in that the better overall estimate of the employment interviews – cognitive ability relationship is roughly .40. This difference has implications for what is being measured by interviews and their incremental validity.

Power-Enhanced Funnel Plots for Meta-Analysis

2020 ◽  
Vol 228 (1) ◽  
pp. 43-49 ◽  
Author(s):  
Michael Kossmeier ◽  
Ulrich S. Tran ◽  
Martin Voracek
Keyword(s):  
Statistical Power ◽  
Meta Analysis ◽  
Relevant Information ◽  
Selective Reporting ◽  
Graphical Display ◽  
Funnel Plot ◽  
Evidential Value ◽  
Graphical Displays ◽  
Funnel Plots ◽  
Meta Analyses

Abstract. Currently, dedicated graphical displays to depict study-level statistical power in the context of meta-analysis are unavailable. Here, we introduce the sunset (power-enhanced) funnel plot to visualize this relevant information for assessing the credibility, or evidential value, of a set of studies. The sunset funnel plot highlights the statistical power of primary studies to detect an underlying true effect of interest in the well-known funnel display with color-coded power regions and a second power axis. This graphical display allows meta-analysts to incorporate power considerations into classic funnel plot assessments of small-study effects. Nominally significant, but low-powered, studies might be seen as less credible and as more likely being affected by selective reporting. We exemplify the application of the sunset funnel plot with two published meta-analyses from medicine and psychology. Software to create this variation of the funnel plot is provided via a tailored R function. In conclusion, the sunset (power-enhanced) funnel plot is a novel and useful graphical display to critically examine and to present study-level power in the context of meta-analysis.

Which Data to Meta-Analyze, and How?

2019 ◽  
Vol 227 (1) ◽  
pp. 64-82 ◽  
Author(s):  
Martin Voracek ◽  
Michael Kossmeier ◽  
Ulrich S. Tran
Keyword(s):  
Empirical Research ◽  
Meta Analysis ◽  
Curve Analysis ◽  
Research Progress ◽  
Primary Data ◽  
Statistical Procedures ◽  
Graphical Displays ◽  
Primary Data Analysis ◽  
Meta Analyses ◽  
Data Analytic

Abstract. Which data to analyze, and how, are fundamental questions of all empirical research. As there are always numerous flexibilities in data-analytic decisions (a “garden of forking paths”), this poses perennial problems to all empirical research. Specification-curve analysis and multiverse analysis have recently been proposed as solutions to these issues. Building on the structural analogies between primary data analysis and meta-analysis, we transform and adapt these approaches to the meta-analytic level, in tandem with combinatorial meta-analysis. We explain the rationale of this idea, suggest descriptive and inferential statistical procedures, as well as graphical displays, provide code for meta-analytic practitioners to generate and use these, and present a fully worked real example from digit ratio (2D:4D) research, totaling 1,592 meta-analytic specifications. Specification-curve and multiverse meta-analysis holds promise to resolve conflicting meta-analyses, contested evidence, controversial empirical literatures, and polarized research, and to mitigate the associated detrimental effects of these phenomena on research progress.

The Orchard Plot: Cultivating a Forest Plot for Use in Ecology, Evolution and Beyond

2019 ◽  
Author(s):  
Shinichi Nakagawa ◽  
Malgorzata Lagisz ◽  
Rose E O'Dea ◽  
Joanna Rutkowska ◽  
Yefeng Yang ◽  
...  
Keyword(s):  
Meta Analysis ◽  
R Package ◽  
Effect Sizes ◽  
Forest Plot ◽  
Point Estimates ◽  
Aggregate Effect ◽  
The Individual ◽  
Meta Analyses ◽  
Heterogeneous Effect ◽  
Intuitive Interpretation

‘Classic’ forest plots show the effect sizes from individual studies and the aggregate effect from a meta-analysis. However, in ecology and evolution meta-analyses routinely contain over 100 effect sizes, making the classic forest plot of limited use. We surveyed 102 meta-analyses in ecology and evolution, finding that only 11% use the classic forest plot. Instead, most used a ‘forest-like plot’, showing point estimates (with 95% confidence intervals; CIs) from a series of subgroups or categories in a meta-regression. We propose a modification of the forest-like plot, which we name the ‘orchard plot’. Orchard plots, in addition to showing overall mean effects and CIs from meta-analyses/regressions, also includes 95% prediction intervals (PIs), and the individual effect sizes scaled by their precision. The PI allows the user and reader to see the range in which an effect size from a future study may be expected to fall. The PI, therefore, provides an intuitive interpretation of any heterogeneity in the data. Supplementing the PI, the inclusion of underlying effect sizes also allows the user to see any influential or outlying effect sizes. We showcase the orchard plot with example datasets from ecology and evolution, using the R package, orchard, including several functions for visualizing meta-analytic data using forest-plot derivatives. We consider the orchard plot as a variant on the classic forest plot, cultivated to the needs of meta-analysts in ecology and evolution. Hopefully, the orchard plot will prove fruitful for visualizing large collections of heterogeneous effect sizes regardless of the field of study.

Sign in / Sign up

Export Citation Format

Share Document