scholarly journals Multi-level meta-analysis of single-case experimental designs using robust variance estimation

2021 ◽  
Author(s):  
Man Chen ◽  
James E Pustejovsky
Keyword(s):  
Effect Size ◽  
Variance Estimation ◽  
Single Case ◽  
Meta Analysis ◽  
Data Series ◽  
Robust Variance ◽  
Auto Correlation ◽  
Robust Variance Estimation ◽  
Multi Level ◽  
Size Estimates

Single-case experimental designs (SCEDs) are used to study the effects of interventions on the behavior of individual cases, by making comparisons between repeated measurements of an outcome under different conditions. In research areas where SCEDs are prevalent, there is a need for methods to synthesize results across multiple studies. One approach to synthesis uses a multi-level meta-analysis (MLMA) model to describe the distribution of effect sizes across studies and across cases within studies. However, MLMA relies on having accurate sampling variances of effect size estimates for each case, which may not be possible due to auto-correlation in the raw data series. One possible solution is to combine MLMA with robust variance estimation (RVE), which provides valid assessments of uncertainty even if the sampling variances of effect size estimates are inaccurate. Another possible solution is to forgo MLMA and use simpler, ordinary least squares (OLS) methods, with RVE. This study evaluates the performance of effect size estimators and methods of synthesizing SCEDs in the presence of auto-correlation, for several different effect size metrics, via a Monte Carlo simulation designed to emulate the features of real data series. Results demonstrate that the MLMA model with RVE performs properly in terms of bias, accuracy, and confidence interval coverage for estimating overall average log response ratios. The OLS estimator corrected with RVE performs the best in estimating overall average Tau effect sizes. None of the available methods perform adequately for meta-analysis of within-case standardized mean differences.

scholarly journals An assessment of statistical methods for non-independent data in ecological meta-analyses: Comment

2021 ◽  
Author(s):  
Shinichi Nakagawa ◽  
Alistair M Senior ◽  
Wolfgang Viechtbauer ◽  
Daniel W.A. Noble
Keyword(s):  
Random Effects ◽  
Effect Size ◽  
Variance Estimation ◽  
Meta Analysis ◽  
Estimation Method ◽  
Random Effects Model ◽  
Step Method ◽  
Robust Variance ◽  
Robust Variance Estimation ◽  
Meta Analyses

Recently, Song et al. (2020) conducted a simulation study using different methods to deal with non-independence resulting from effect sizes originating from the same paper – a common occurrence in ecological meta-analyses. The main methods that were of interest in their simulations were: 1) a standard random-effects model used in combination with a weighted average effect size for each paper (i.e., a two-step method), 2) a standard random-effects model after randomly choosing one effect size per paper, 3) a multilevel (hierarchical) meta-analysis model, modelling paper identity as a random factor, and 4) a meta-analysis making use of a robust variance estimation method. Based on their simulation results, they recommend that meta-analysts should either use the two-step method, which involves taking a weighted paper mean followed by analysis with a random-effects model, or the robust variance estimation method.

Listening Ears or Reading Eyes: A Meta-Analysis of Reading and Listening Comprehension Comparisons

2021 ◽  
pp. 003465432110608
Author(s):  
Virginia Clinton-Lisell
Keyword(s):  
Listening Comprehension ◽  
Variance Estimation ◽  
Meta Analysis ◽  
Age Groups ◽  
Robust Variance ◽  
Reading Condition ◽  
P 16 ◽  
Robust Variance Estimation ◽  
Literal Comprehension

In this study, a meta-analysis of reading and listening comprehension comparisons across age groups was conducted. Based on robust variance estimation (46 studies; N = 4,687), the overall difference between reading and listening comprehension was not reliably different (g = 0.07, p = .23). Reading was beneficial over listening when the reading condition was self-paced (g = 0.13, p = .049) rather than experimenter-paced (g = −0.32, p = .16). Reading also had a benefit when inferential and general comprehension rather than literal comprehension was assessed (g = 0.36, p = .02; g = .15, p = .05; g = −0.01, p = .93, respectively). There was some indication that reading and listening were more similar in languages with transparent orthographies than opaque orthographies (g = 0.001, p = .99; g = 0.10, p = .19, respectively). The findings may be used to inform theories of comprehension about modality influences in that both lower-level skill and affordances vary comparisons of reading and listening comprehension. Moreover, the findings may guide choices of modality; however, both audio and written options are needed for accessible instruction.

scholarly journals Meta-Analysis with Robust Variance Estimation: Expanding the Range of Working Models

2020 ◽  
Author(s):  
James E Pustejovsky ◽  
Elizabeth Tipton
Keyword(s):  
Variance Estimation ◽  
Meta Analysis ◽  
Estimation Methods ◽  
Single Type ◽  
Dependence Structure ◽  
Dependent Effect ◽  
Working Models ◽  
Robust Variance ◽  
Robust Variance Estimation ◽  
Meta Regression

In prevention science and related fields, large meta-analyses are common, and these analyses often involve dependent effect size estimates. Robust variance estimation (RVE) methods provide a way to include all dependent effect sizes in a single meta-regression model, even when the nature of the dependence is unknown. RVE uses a working model of the dependence structure, but the two currently available working models are limited to each describing a single type of dependence. Drawing on flexible tools from multivariate meta-analysis, this paper describes an expanded range of working models, along with accompanying estimation methods, which offer benefits in terms of better capturing the types of data structures that occur in practice and improving the efficiency of meta-regression estimates. We describe how the methods can be implemented using existing software (the ‘metafor’ and ‘clubSandwich’ packages for R) and illustrate the approach in a meta-analysis of randomized trials examining the effects of brief alcohol interventions for adolescents and young adults.

Does Self-Control Training Improve Self-Control? A Meta-Analysis

2017 ◽  
Vol 12 (6) ◽  
pp. 1077-1099 ◽  
Author(s):  
Malte Friese ◽  
Julius Frankenbach ◽  
Veronika Job ◽  
David D. Loschelder
Keyword(s):  
Variance Estimation ◽  
Meta Analysis ◽  
Self Control ◽  
Medium Effect ◽  
Critical Element ◽  
Training Effects ◽  
Control Groups ◽  
Robust Variance Estimation ◽  
Size Estimates ◽  
Study Bias

Self-control is positively associated with a host of beneficial outcomes. Therefore, psychological interventions that reliably improve self-control are of great societal value. A prominent idea suggests that training self-control by repeatedly overriding dominant responses should lead to broad improvements in self-control over time. Here, we conducted a random-effects meta-analysis based on robust variance estimation of the published and unpublished literature on self-control training effects. Results based on 33 studies and 158 effect sizes revealed a small-to-medium effect of g = 0.30, confidence interval (CI95) [0.17, 0.42]. Moderator analyses found that training effects tended to be larger for (a) self-control stamina rather than strength, (b) studies with inactive compared to active control groups, (c) males than females, and (d) when proponents of the strength model of self-control were (co)authors of a study. Bias-correction techniques suggested the presence of small-study effects and/or publication bias and arrived at smaller effect size estimates (range: gcorrected = .13 to .24). The mechanisms underlying the effect are poorly understood. There is not enough evidence to conclude that the repeated control of dominant responses is the critical element driving training effects.

No Evidence of Creative Benefit Accompanying Dyslexia: A Meta-Analysis

2021 ◽  
pp. 002221942110103
Author(s):  
Florina Erbeli ◽  
Peng Peng ◽  
Marianne Rice
Keyword(s):  
Random Effects ◽  
Variance Estimation ◽  
Meta Analysis ◽  
Robust Variance ◽  
Summary Effect ◽  
Significant Performance ◽  
Mean And Variance ◽  
Robust Variance Estimation ◽  
The Mean ◽  
Creativity Task

Research on the question of creative benefit accompanying dyslexia has produced conflicting findings. In this meta-analysis, we determined summary effects of mean and variance differences in creativity between groups with and without dyslexia. Twenty studies were included ( n = 770 individuals with dyslexia, n = 1,671 controls). A random-effects robust variance estimation (RVE) analysis indicated no mean ( g = −0.02, p = .84) or variance differences ( g = −0.0004, p = .99) in creativity between groups. The mean summary effect was moderated by age, gender, and creativity domain. Compared with adolescents, adults with dyslexia showed an advantage over nondyslexic adults in creativity. In addition, a higher proportion of males in the dyslexia group was associated with poorer performance compared with the controls. Finally, the dyslexia group showed a significant performance disadvantage in verbal versus figural creativity. Regarding variance differences, they varied across age and creativity domains. Compared with adults, adolescents showed smaller variability in the dyslexia group. If the creativity task measured verbal versus figural or combined creativity, then the dyslexia group exhibited smaller variability. Altogether, our results suggest that individuals with dyslexia as a group are no more creative or show greater variability in creativity than peers without dyslexia.

Sign in / Sign up

Export Citation Format

Share Document