scholarly journals Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

2018 ◽  
Author(s):  
Michele B. Nuijten ◽  
Marcel A. L. M. van Assen ◽  
Hilde Augusteijn ◽  
Elise Anne Victoire Crompvoets ◽  
Jelte M. Wicherts
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Meta Analysis ◽  
Convincing Evidence ◽  
Small Study ◽  
Effect Sizes ◽  
Medium Effect ◽  
Average Effect ◽  
Average Effect Size ◽  
Meta Analyses

In this meta-study, we analyzed 2,442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson’s correlation of .26, and the median sample size was 60. Furthermore, across primary studies, we found a median power of 11.9% to detect a small effect, 54.5% to detect a medium effect, and 93.9% to detect a large effect. We documented differences in average effect size and median estimated power between different types of in intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). On average, across all meta-analyses (but not in every meta-analysis), we found evidence for small study effects, potentially indicating publication bias and overestimated effects. We found no differences in small study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We conclude that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

scholarly journals Effect Sizes, Power, and Biases in Intelligence Research: A Meta-Meta-Analysis

2020 ◽  
Vol 8 (4) ◽  
pp. 36
Author(s):  
Michèle B. Nuijten ◽  
Marcel A. L. M. van Assen ◽  
Hilde E. M. Augusteijn ◽  
Elise A. V. Crompvoets ◽  
Jelte M. Wicherts
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Meta Analysis ◽  
Convincing Evidence ◽  
Small Study ◽  
Effect Sizes ◽  
Medium Effect ◽  
Average Effect ◽  
Average Effect Size ◽  
Meta Analyses

In this meta-study, we analyzed 2442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson’s correlation of 0.26, and the median sample size was 60. Furthermore, across primary studies, we found a median power of 11.9% to detect a small effect, 54.5% to detect a medium effect, and 93.9% to detect a large effect. We documented differences in average effect size and median estimated power between different types of intelligence studies (correlational studies, studies of group differences, experiments, toxicology, and behavior genetics). On average, across all meta-analyses (but not in every meta-analysis), we found evidence for small-study effects, potentially indicating publication bias and overestimated effects. We found no differences in small-study effects between different study types. We also found no convincing evidence for the decline effect, US effect, or citation bias across meta-analyses. We concluded that intelligence research does show signs of low power and publication bias, but that these problems seem less severe than in many other scientific fields.

scholarly journals A Meta–Analytic and Conceptual Update on the Associations between Procrastination and Multidimensional Perfectionism

2017 ◽  
Vol 31 (2) ◽  
pp. 137-159 ◽  
Author(s):  
Fuschia M. Sirois ◽  
Danielle S. Molnar ◽  
Jameson K. Hirsch ◽  
Mitja Back
Keyword(s):  
Effect Size ◽  
Meta Analysis ◽  
Self Regulation ◽  
Average Effect ◽  
Personality Psychology ◽  
Average Effect Size ◽  
Partial Correlations ◽  
Unidimensional Construct ◽  
Meta Analyses ◽  
Behavioural Dynamics

The equivocal and debated findings from a 2007 meta–analysis, which viewed perfectionism as a unidimensional construct, suggested that perfectionism was unrelated to procrastination. The present meta–analysis aimed to provide a conceptual update and reanalysis of the procrastination–perfectionism association guided by both a multidimensional view of perfectionism and self–regulation theory. The random–effects meta–analyses revealed a small to medium positive average effect size ( r = .23; k = 43, N = 10 000; 95% confidence interval (95% CI) [0.19, 0.27]) for trait procrastination and perfectionistic concerns and a small to medium negative average effect size ( r = −.22; k = 38, N = 9544; 95% CI [−0.26, −0.18]) for procrastination and perfectionistic strivings. The average correlations remained significant after statistically accounting for the joint variance between the two perfectionism dimensions via semi–partial correlations. For perfectionistic concerns, but not perfectionistic strivings, the effects depended on the perfectionism measure used. All effects did not vary by the trait procrastination measure used or the respondent's sex. Our findings confirm that from a multidimensional perspective, trait procrastination is both positively and negatively associated with higher–order perfectionism dimensions and further highlights the value of a self–regulation perspective for understanding the cognitive, affective and behavioural dynamics that characterise these traits. Copyright © 2017 European Association of Personality Psychology

scholarly journals P-uniform*

2018 ◽  
Author(s):  
Robbie Cornelis Maria van Aert ◽  
Marcel A. L. M. van Assen
Keyword(s):  
Publication Bias ◽  
Random Effects ◽  
Effect Size ◽  
Meta Analysis ◽  
Statistical Properties ◽  
Selection Model ◽  
Effect Sizes ◽  
Random Effects Model ◽  
Average Effect ◽  
Model Approach

Publication bias is a major threat to the validity of a meta-analysis resulting in overestimated effect sizes. P-uniform is a meta-analysis method that corrects estimates for publication bias but overestimates average effect size if heterogeneity in true effect sizes (i.e., between-study variance) is present. We propose an extension and improvement of p-uniform called p-uniform*. P-uniform* improves upon p-uniform in three important ways, as it (i) entails a more efficient estimator, (ii) eliminates the overestimation of effect size in case of between-study variance in true effect sizes, and (iii) enables estimating and testing for the presence of the between-study variance. We compared the statistical properties of p-uniform* with p-uniform, the selection model approach of Hedges (1992), and the random-effects model. Statistical properties of p-uniform* and the selection model approach were comparable and generally outperformed p-uniform and the random-effects model if publication bias was present. We demonstrate that p-uniform* and the selection model approach estimate average effect size and between-study variance rather well with ten or more studies in the meta-analysis when publication bias is not extreme. P-uniform* generally provides more accurate estimates of the between-study variance in meta-analyses containing many studies (e.g., 60 or more) and if publication bias is present. However, both methods do not perform well if the meta-analysis only includes statistically significant studies. P-uniform performed best in this case but only when between-study variance was zero or small. We offer recommendations for applied researchers, and provide an R package and an easy-to-use web application for applying p-uniform*.

scholarly journals Interpreting Average Effect Sizes: Never a Center Without a Spread

2019 ◽  
Vol 103 (4) ◽  
pp. 273-280
Author(s):  
Thomas R. Guskey
Keyword(s):  
Effect Size ◽  
School Leaders ◽  
Effect Sizes ◽  
Average Effect ◽  
Average Effect Size ◽  
Education Leaders ◽  
Meta Analyses

School leaders today are making important decisions regarding education innovations based on published average effect sizes, even though few understand exactly how effect sizes are calculated or what they mean. This article explains how average effect sizes are determined in meta-analyses and the importance of including measures of variability with any average effect size. By considering the variation in effect sizes among studies of the same innovation, education leaders can make better decisions about innovations and greatly increase the likelihood of achieving optimal results from implementation.

Unresolved Heterogeneity in Meta-Analysis: Combined Construct Invalidity, Confounding, and Other Challenges to Understanding Mean Effect Sizes

2020 ◽  
Vol 46 (2-3) ◽  
pp. 343-354 ◽  
Author(s):  
Timothy R Levine ◽  
René Weber
Keyword(s):  
Content Analysis ◽  
Publication Bias ◽  
Effect Size ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Size Distributions ◽  
Questionable Research Practices ◽  
Graphical Displays ◽  
Meta Analyses ◽  
Effect Heterogeneity

Abstract We examined the interplay between how communication researchers use meta-analyses to make claims and the prevalence, causes, and implications of unresolved heterogeneous findings. Heterogeneous findings can result from substantive moderators, methodological artifacts, and combined construct invalidity. An informal content analysis of meta-analyses published in four elite communication journals revealed that unresolved between-study effect heterogeneity was ubiquitous. Communication researchers mainly focus on computing mean effect sizes, to the exclusion of how effect sizes in primary studies are distributed and of what might be driving effect size distributions. We offer four recommendations for future meta-analyses. Researchers are advised to be more diligent and sophisticated in testing for heterogeneity. We encourage greater description of how effects are distributed, coupled with greater reliance on graphical displays. We council greater recognition of combined construct invalidity and advocate for content expertise. Finally, we endorse greater awareness and improved tests for publication bias and questionable research practices.

scholarly journals Dissertation R.C.M. van Aert

2018 ◽  
Author(s):  
Robbie Cornelis Maria van Aert
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Meta Analysis ◽  
Original Study ◽  
The Other ◽  
Effect Sizes ◽  
True Effect Size ◽  
True Effect ◽  
Meta Analyses ◽  
Q Statistic

More and more scientific research gets published nowadays, asking for statistical methods that enable researchers to get an overview of the literature in a particular research field. For that purpose, meta-analysis methods were developed that can be used for statistically combining the effect sizes from independent primary studies on the same topic. My dissertation focuses on two issues that are crucial when conducting a meta-analysis: publication bias and heterogeneity in primary studies’ true effect sizes. Accurate estimation of both the meta-analytic effect size as well as the between-study variance in true effect size is crucial since the results of meta-analyses are often used for policy making. Publication bias distorts the results of a meta-analysis since it refers to situations where publication of a primary study depends on its results. We developed new meta-analysis methods, p-uniform and p-uniform*, which estimate effect sizes corrected for publication bias and also test for publication bias. Although the methods perform well in many conditions, these and the other existing methods are shown not to perform well when researchers use questionable research practices. Additionally, when publication bias is absent or limited, traditional methods that do not correct for publication bias outperform p¬-uniform and p-uniform*. Surprisingly, we found no strong evidence for the presence of publication bias in our pre-registered study on the presence of publication bias in a large-scale data set consisting of 83 meta-analyses and 499 systematic reviews published in the fields of psychology and medicine. We also developed two methods for meta-analyzing a statistically significant published original study and a replication of that study, which reflects a situation often encountered by researchers. One method is a frequentist whereas the other method is a Bayesian statistical method. Both methods are shown to perform better than traditional meta-analytic methods that do not take the statistical significance of the original study into account. Analytical studies of both methods also show that sometimes the original study is better discarded for optimal estimation of the true effect size. Finally, we developed a program for determining the required sample size in a replication analogous to power analysis in null hypothesis testing. Computing the required sample size with the method revealed that large sample sizes (approximately 650 participants) are required to be able to distinguish a zero from a small true effect.Finally, in the last two chapters we derived a new multi-step estimator for the between-study variance in primary studies’ true effect sizes, and examined the statistical properties of two methods (Q-profile and generalized Q-statistic method) to compute the confidence interval of the between-study variance in true effect size. We proved that the multi-step estimator converges to the Paule-Mandel estimator which is nowadays one of the recommended methods to estimate the between-study variance in true effect sizes. Two Monte-Carlo simulation studies showed that the coverage probabilities of Q-profile and generalized Q-statistic method can be substantially below the nominal coverage rate if the assumptions underlying the random-effects meta-analysis model were violated.

Identification of and Correction for Publication Bias: Comment

2019 ◽  
Author(s):  
Amanda Kvarven ◽  
Eirik Strømland ◽  
Magnus Johannesson
Keyword(s):  
Publication Bias ◽  
False Positive ◽  
Large Scale ◽  
Meta Analysis ◽  
False Positive Rate ◽  
Effect Sizes ◽  
Replication Studies ◽  
Moderate Reduction ◽  
Positive Rate ◽  
Meta Analyses

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

scholarly journals Cross-linguistic influence in simultaneous and early sequential bilingual children: a meta-analysis

2021 ◽  
pp. 1-33
Author(s):  
Chantal VAN DIJK ◽  
Elise VAN WONDEREN ◽  
Elly KOUTAMANIS ◽  
Gerrit Jan KOOTSTRA ◽  
Ton DIJKSTRA ◽  
...  
Keyword(s):  
Effect Size ◽  
Meta Analysis ◽  
Experimental Studies ◽  
Uniform Method ◽  
Bilingual Children ◽  
Average Effect ◽  
Language Dominance ◽  
Average Effect Size ◽  
Comprehensive Theory ◽  
Bilingual Development

Abstract Although cross-linguistic influence at the level of morphosyntax is one of the most intensively studied topics in child bilingualism, the circumstances under which it occurs remain unclear. In this meta-analysis, we measured the effect size of cross-linguistic influence and systematically assessed its predictors in 750 simultaneous and early sequential bilingual children in 17 unique language combinations across 26 experimental studies. We found a significant small to moderate average effect size of cross-linguistic influence, indicating that cross-linguistic influence is part and parcel of bilingual development. Language dominance, operationalized as societal language, was a significant predictor of cross-linguistic influence, whereas surface overlap, language domain and age were not. Perhaps an even more important finding was that definitions and operationalisations of cross-linguistic influence and its predictors varied considerably between studies. This could explain the absence of a comprehensive theory in the field. To solve this issue, we argue for a more uniform method of studying cross-linguistic influence.

scholarly journals A Statistical Method for Synthesizing Meta-Analyses

2013 ◽  
Vol 2013 ◽  
pp. 1-9 ◽  
Author(s):  
Liansheng Larry Tang ◽  
Michael Caudy ◽  
Faye Taxman
Keyword(s):  
Statistical Method ◽  
Effect Size ◽  
Meta Analysis ◽  
Effect Sizes ◽  
Weighted Mean ◽  
Summary Effect ◽  
Different Types ◽  
Meta Analyses ◽  
Size Estimates ◽  
Similar Search

Multiple meta-analyses may use similar search criteria and focus on the same topic of interest, but they may yield different or sometimes discordant results. The lack of statistical methods for synthesizing these findings makes it challenging to properly interpret the results from multiple meta-analyses, especially when their results are conflicting. In this paper, we first introduce a method to synthesize the meta-analytic results when multiple meta-analyses use the same type of summary effect estimates. When meta-analyses use different types of effect sizes, the meta-analysis results cannot be directly combined. We propose a two-step frequentist procedure to first convert the effect size estimates to the same metric and then summarize them with a weighted mean estimate. Our proposed method offers several advantages over existing methods by Hemming et al. (2012). First, different types of summary effect sizes are considered. Second, our method provides the same overall effect size as conducting a meta-analysis on all individual studies from multiple meta-analyses. We illustrate the application of the proposed methods in two examples and discuss their implications for the field of meta-analysis.

Can Reliance be Placed on a Single Meta-Analysis?

1990 ◽  
Vol 24 (3) ◽  
pp. 405-415 ◽  
Author(s):  
Nathaniel McConaghy
Keyword(s):  
Literature Review ◽  
Effect Size ◽  
Meta Analysis ◽  
Statistical Significance ◽  
Effect Sizes ◽  
Control Groups ◽  
Consistent Finding ◽  
Placebo Controls ◽  
Effect Of Treatment ◽  
Meta Analyses

Meta-analysis replaced statistical significance with effect size in the hope of resolving controversy concerning evaluation of treatment effects. Statistical significance measured reliability of the effect of treatment, not its efficacy. It was strongly influenced by the number of subjects investigated. Effect size as assessed originally, eliminated this influence but by standardizing the size of the treatment effect could distort it. Meta-analyses which combine the results of studies which employ different subject types, outcome measures, treatment aims, no-treatment rather than placebo controls or therapists with varying experience can be misleading. To ensure discussion of these variables meta-analyses should be used as an aid rather than a substitute for literature review. While meta-analyses produce contradictory findings, it seems unwise to rely on the conclusions of an individual analysis. Their consistent finding that placebo treatments obtain markedly higher effect sizes than no treatment hopefully will render the use of untreated control groups obsolete.

Sign in / Sign up

Export Citation Format

Share Document