A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes

Purpose: The increasing need for school-based mental health services has altered teachers’ involvement in mental health services. Methods: This study presents a meta-analysis from a previous systematic review to identify which study characteristics result in effective treatment outcomes. Specific treatment characteristics analyzed in this study include type of intervention, treatment modality, length of treatment, and type of measurement. Effect sizes were coded by internalizing and externalizing disorders, depending on the symptoms the corresponding treatments were intended to address. A final sample size included 9 independent effect sizes of internalizing behaviors and 21 effect sizes of externalizing behaviors. Results: Internalizing disorders, social skill interventions, classroom modalities, and medium treatment length were moderating treatment characteristics. No significant effects were found for externalizing disorders. Conclusions: These results further add to the research on teacher’s role in school-based mental health services and provide important information for social workers who work in schools.

Download Full-text

A Meta-Analysis on Computer Technology Intervention Effects on Mathematics Achievement for Low-Performing Students in K-12 Classrooms

Journal of Educational Computing Research ◽

10.1177/0735633120952063 ◽

2020 ◽

pp. 073563312095206

Author(s):

Hua Ran ◽

Murat Kasli ◽

Walter G. Secada

Keyword(s):

Instructional Design ◽

Mathematics Achievement ◽

Computer Technology ◽

Meta Analysis ◽

Empirical Studies ◽

Effect Sizes ◽

Independent Effect ◽

Technology Intervention ◽

K 12 ◽

Positive Effect

This meta-analysis extended the current literature regarding the effects of computer technology (CT) on mathematics achievement, with a particular focus on low-performing students. A total of 45 independent effect sizes extracted from 31 empirical studies based on a total of 2,044 low-performing students in K-12 classrooms were included in this meta-analysis. Consistent with previous reviews, this study suggested a statistically significant and positive effect of CT ([Formula: see text] = 0.56) on low-performing students’ mathematics achievement. Of four CT types, the largest CT effect was found with problem-solving system ([Formula: see text] = 0.86), followed by tutoring [Formula: see text] = 0.80), game-based intervention ([Formula: see text] = .58), and computerized practice ([Formula: see text] = .23). Furthermore, other moderators were found to explain variation in CT effects on low-performing students’ mathematics achievement. Study findings contributed to clarifying the effect of CT for low-performing students’ mathematics achievement. Implications for instructional design and practices are also discussed.

Download Full-text

A Guide to Conducting a Meta-Analysis with Non-Independent Effect Sizes

10.31234/osf.io/5p7dj ◽

2019 ◽

Author(s):

Mike W.-L. Cheung

Keyword(s):

Ad Hoc ◽

Meta Analysis ◽

Real Data ◽

Effect Sizes ◽

Independent Effect ◽

New Techniques ◽

Missed Opportunities ◽

Research Questions ◽

Conventional Models ◽

Traditional Approaches

Conventional meta-analytic procedures assume that effect sizes are independent. When effect sizes are non-independent, conclusions based on these conventional models can be misleading or even wrong. Traditional approaches, such as averaging the effect sizes and selecting one effect size per study, are usually used to remove the dependence of the effect sizes. These ad-hoc approaches, however, may lead to missed opportunities to utilize all available data to address the relevant research questions. Both multivariate meta-analysis and three-level meta-analysis have been proposed to handle non-independent effect sizes. This paper gives a brief introduction to these new techniques for applied researchers. The first objective is to highlight the benefits of using these methods to address non-independent effect sizes. The second objective is to illustrate how to apply these techniques with real data in R and Mplus. Researchers may modify the sample R and Mplus code to fit their data.

Download Full-text

Toward Optimized Effectiveness of Employee Training Programs

Journal of Personnel Psychology ◽

10.1027/1866-5888/a000290 ◽

2021 ◽

Author(s):

Dilek Uslu ◽

Justin Marcus ◽

Yasemin Kisbu-Sakarya

Keyword(s):

Training Programs ◽

Meta Analysis ◽

Employee Training ◽

Effect Sizes ◽

Independent Effect ◽

Workplace Training ◽

Affective Outcomes ◽

Training Outcomes ◽

Training Interventions ◽

Training Delivery

Abstract. Although organizations invest heavily on employee training, the effectiveness of employee training programs has not been well-established. In the current study, we examine the training delivery features of employee training programs to derive a better understanding of features that may be of best benefit in the improvement of employee affective outcomes. Specifically, and via the use of meta-analysis ( k = 79 studies totaling 107 independent effect sizes), we focus on two broad classes of affective employee training outcomes including attitudinal and motivational outcomes. Results evidence support for the effectiveness of employee workplace training interventions and indicate that employee training programs associated with attitudinal versus motivational outcomes require different features while being delivered to reach optimal effectiveness.

Download Full-text

A Meta-analysis of Employment Discrimination Against Muslims and Arabs

10.31234/osf.io/6nw9h ◽

2018 ◽

Author(s):

Timothy Bartkoski ◽

Ellen Herrmann ◽

Chelsea Witt ◽

Cort Rudolph

Keyword(s):

Meta Analysis ◽

Experimental Studies ◽

Research Literature ◽

Effect Sizes ◽

Independent Effect ◽

Work Setting ◽

Employment Decisions ◽

Quantitative Review ◽

Almost All ◽

Future Work

Muslim and Arab individuals are discriminated against in almost all domains. Recently, there hasbeen a focus on examining the treatment of these groups in the work setting. Despite the great number of primary studies examining this issue, there has not yet been a quantitative review of the research literature. To fill this gap, this meta-analysis examined the presence and magnitude of hiring discrimination against Muslim and Arab individuals. Using 46 independent effect sizes from 26 sources, we found evidence of discrimination against Muslim and Arab people in employment judgments, behaviors, and decisions across multiple countries. Moderator analyses revealed that discrimination is stronger in field settings, when actual employment decisions are made, and when experimental studies used “Arab” (vs. “Muslim”) targets. However, primary studies provide inconsistent and inaccurate distinctions between Arabs and Muslims, therefore future work should be cautious in categorizing the exact aspect of identity being studied.

Download Full-text

A Meta-Analysis of the Effects of Mental Contrasting With Implementation Intentions on Goal Attainment

Frontiers in Psychology ◽

10.3389/fpsyg.2021.565202 ◽

2021 ◽

Vol 12 ◽

Author(s):

Guoxia Wang ◽

Yi Wang ◽

Xiaosong Gai

Keyword(s):

Goal Attainment ◽

Implementation Intentions ◽

Meta Analysis ◽

Empirical Studies ◽

Self Regulation ◽

Effect Sizes ◽

Medium Effect ◽

Independent Effect ◽

Mental Contrasting ◽

Medium Effect Size

Mental contrasting with implementation intentions (MCII) is a self-regulation strategy that enhances goal attainment. This meta-analysis evaluated the efficacy of MCII for goal attainment and explored potential moderators. A total of 21 empirical studies with 24 independent effect sizes (15,907 participants) were included in the analysis. Results showed that MCII to be effective for goal attainment with a small to medium effect size (g = 0.336). The effect was mainly moderated by intervention style. Specifically, studies with interventions based on interactions between participants and experimenters (g = 0.465) had stronger effects than studies with interventions based on interactions between participants and documents (g = 0.277). The results revealed that MCII is a brief and effective strategy for goal attainment with a small to moderate effect; however, because of some publication bias, the actual effect sizes may be smaller. Due to small number of studies in this meta-analysis, additional studies are needed to determine the role of moderator variables.

Download Full-text

A meta-analytic review of the relation between impulsivity-related traits and family history of alcoholism

10.32469/10355/69814 ◽

2018 ◽

Author(s):

◽

Angela Maria Haeny

Keyword(s):

Family History ◽

Effect Size ◽

Sensation Seeking ◽

Meta Analysis ◽

Alcohol Problems ◽

Effect Sizes ◽

Independent Effect ◽

Increased Risk ◽

Analytic Review ◽

History Of

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI AT AUTHOR'S REQUEST.] Extensive research provides evidence that people with a family history of alcoholism are at risk for developing alcohol use disorder (AUD). Similarly, people with impulsivity-related traits are at increased risk for developing alcohol problems. Importantly, research suggests that impulsivity mediates the relation between family history of alcoholism and the development of alcohol problems. However, impulsivity is a heterogenous construct and has been assessed with a myriad of measures. The present work is a quantitative synthesis of the literature on the relation between family history of alcoholism and impulsivity-related traits and that also examines various potential moderators of this association. Sixty-nine independent effect sizes from 65 studies (N = 11,127) qualified for the meta-analysis. The overall effect size was small-to-moderate (d = .32 [95% CI: 0.25, 0.39], k = 69), and was moderated by offspring age (Z = 3.73, p less than .001), with the effect size increasing with age. When examining specific facets of impulsivity, a small effect was found for harm avoidance (d = -.26 [95% CI: -.41, -.11], k = 10) and was moderated by family history density (Q (1) = 4.12, p = .04) such that the effect was much larger among those with more than one alcoholic family member (d = -.66 [95% CI: -1.10, -.22], k = 3). A small-to-moderate effect size was found for sensation seeking (d = .30 [95% CI: .21, .40], k = 29) and was moderated by age (Z = 3.09, p = .002), with the effect increasing with age. The effect sizes for all other facets of impulsivity were not significant. Notably, there were much fewer studies investigating other facets of impulsivity (e.g., reward dependence, lack of perseverance, lack of planning) compared to sensation seeking, limiting power to detect larger effect sizes. Findings from this review suggest the need for additional studies investigating the relation between specific facets of impulsivity (e.g., positive and negative urgency) and family history of alcoholism. In addition, this review suggests that, to some degree, we can identify phenotypic risk beyond mere family history status and, thus, inform the development of interventions for individuals with a family history of alcoholism, targeting the specific types of impulsivity manifested.

Download Full-text

Supplemental Material for Meta-Analysis to Integrate Effect Sizes Within an Article: Possible Misuse and Type I Error Inflation

Journal of Experimental Psychology General ◽

10.1037/xge0000159.supp ◽

2016 ◽

Keyword(s):

Type I Error ◽

Meta Analysis ◽

Effect Sizes ◽

Type I

Download Full-text

The Orchard Plot: Cultivating a Forest Plot for Use in Ecology, Evolution and Beyond

10.32942/osf.io/epqa7 ◽

2019 ◽

Author(s):

Shinichi Nakagawa ◽

Malgorzata Lagisz ◽

Rose E O'Dea ◽

Joanna Rutkowska ◽

Yefeng Yang ◽

...

Keyword(s):

Meta Analysis ◽

R Package ◽

Effect Sizes ◽

Forest Plot ◽

Point Estimates ◽

Aggregate Effect ◽

The Individual ◽

Meta Analyses ◽

Heterogeneous Effect ◽

Intuitive Interpretation

‘Classic’ forest plots show the effect sizes from individual studies and the aggregate effect from a meta-analysis. However, in ecology and evolution meta-analyses routinely contain over 100 effect sizes, making the classic forest plot of limited use. We surveyed 102 meta-analyses in ecology and evolution, finding that only 11% use the classic forest plot. Instead, most used a ‘forest-like plot’, showing point estimates (with 95% confidence intervals; CIs) from a series of subgroups or categories in a meta-regression. We propose a modification of the forest-like plot, which we name the ‘orchard plot’. Orchard plots, in addition to showing overall mean effects and CIs from meta-analyses/regressions, also includes 95% prediction intervals (PIs), and the individual effect sizes scaled by their precision. The PI allows the user and reader to see the range in which an effect size from a future study may be expected to fall. The PI, therefore, provides an intuitive interpretation of any heterogeneity in the data. Supplementing the PI, the inclusion of underlying effect sizes also allows the user to see any influential or outlying effect sizes. We showcase the orchard plot with example datasets from ecology and evolution, using the R package, orchard, including several functions for visualizing meta-analytic data using forest-plot derivatives. We consider the orchard plot as a variant on the classic forest plot, cultivated to the needs of meta-analysts in ecology and evolution. Hopefully, the orchard plot will prove fruitful for visualizing large collections of heterogeneous effect sizes regardless of the field of study.

Download Full-text

Identification of and Correction for Publication Bias: Comment

10.31222/osf.io/dh87m ◽

2019 ◽

Author(s):

Amanda Kvarven ◽

Eirik Strømland ◽

Magnus Johannesson

Keyword(s):

Publication Bias ◽

False Positive ◽

Large Scale ◽

Meta Analysis ◽

False Positive Rate ◽

Effect Sizes ◽

Replication Studies ◽

Moderate Reduction ◽

Positive Rate ◽

Meta Analyses

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

Download Full-text