scholarly journals Small effect size leads to reproducibility failure in resting-state fMRI studies

2018 ◽  
Author(s):  
Xi-Ze Jia ◽  
Na Zhao ◽  
Barek Barton ◽  
Roxana Burciu ◽  
Nicolas Carrière ◽  
...  
Keyword(s):  
Effect Size ◽  
Resting State ◽  
Meta Analysis ◽  
False Negative ◽  
Autism Spectrum ◽  
Multiple Comparison ◽  
Brain Disorders ◽  
Group Design ◽  
Small Effect Size ◽  
Multiple Comparison Correction

AbstractThousands of papers using resting-state functional magnetic resonance imaging (RS-fMRI) have been published on brain disorders. Results in each paper may have survived correction for multiple comparison. However, since there have been no robust results from large scale meta-analysis, we do not know how many of published results are truly positives. The present meta-analytic work included 60 original studies, with 57 studies (4 datasets, 2266 participants) that used a between-group design and 3 studies (1 dataset, 107 participants) that employed a within-group design. To evaluate the effect size of brain disorders, a very large neuroimaging dataset ranging from neurological to psychiatric isorders together with healthy individuals have been analyzed. Parkinson’s disease off levodopa (PD-off) included 687 participants from 15 studies. PD on levodopa (PD-on) included 261 participants from 9 studies. Autism spectrum disorder (ASD) included 958 participants from 27 studies. The meta-analyses of a metric named amplitude of low frequency fluctuation (ALFF) showed that the effect size (Hedges’ g) was 0.19 - 0.39 for the 4 datasets using between-group design and 0.46 for the dataset using within-group design. The effect size of PD-off, PD-on and ASD were 0.23, 0.39, and 0.19, respectively. Using the meta-analysis results as the robust results, the between-group design results of each study showed high false negative rates (median 99%), high false discovery rates (median 86%), and low accuracy (median 1%), regardless of whether stringent or liberal multiple comparison correction was used. The findings were similar for 4 RS-fMRI metrics including ALFF, regional homogeneity, and degree centrality, as well as for another widely used RS-fMRI metric namely seed-based functional connectivity. These observations suggest that multiple comparison correction does not control for false discoveries across multiple studies when the effect sizes are relatively small. Meta-analysis on un-thresholded t-maps is critical for the recovery of ground truth. We recommend that to achieve high reproducibility through meta-analysis, the neuroimaging research field should share raw data or, at minimum, provide un-thresholded statistical images.

Exploring the Morning Morality Effect in the context of moral utilitarianism: a post-hoc analysis

2021 ◽  
Author(s):  
Bastien Trémolière ◽  
Corentin J Gosling
Keyword(s):  
Effect Size ◽  
Autonomous Vehicles ◽  
Meta Analysis ◽  
Individual Study ◽  
Post Hoc Analysis ◽  
Cognitive Reflection ◽  
Confounding Variables ◽  
Small Effect Size ◽  
Large Heterogeneity ◽  
Post Hoc

Recent research has shown mixed evidence for the morning morality effect (i.e., the observation that individuals are less immoral in the morning than in the afternoon). In the present research, we target the morning morality effect in the context of moral utilitarianism, by reanalyzing observational data previously collected by our lab. These data include different tasks capturing moral utilitarianism (i.e., standard sacrificial dilemmas, an ecological utilitarian scale, and/or dilemmas involving the morality of autonomous vehicles). We report a meta-analysis of 6 studies which showed that participants became less utilitarian as the day goes on, but with a small effect size (r = -0.14) and a large heterogeneity. Exploration of this heterogeneity showed that such a conclusion was statistically significant for classic sacrificial dilemmas only. Notably, even when restricting the analysis to the classic sacrificial dilemmas, a moderate inconsistency remained. Post-hoc analysis of an individual study showed that this small effect did not survive the inclusion of potentially confounding variables, such as psychopathy trait and cognitive reflection. Implications and limitations are discussed.

scholarly journals Graph Ricci Curvatures Reveal Disease-related Changes in Autism Spectrum Disorder

2021 ◽  
Author(s):  
Pavithra Elumalai ◽  
Yasharth Yadav ◽  
Nitin Williams ◽  
Emil Saucan ◽  
Jürgen Jost ◽  
...  
Keyword(s):  
Autism Spectrum Disorder ◽  
Resting State ◽  
Neurodevelopmental Disorders ◽  
Data Exchange ◽  
Meta Analysis ◽  
Resting State Fmri ◽  
Brain Regions ◽  
Autism Spectrum ◽  
Spectrum Disorder ◽  
The Brain

Autism Spectrum Disorder (ASD) is a set of neurodevelopmental disorders that pose a significant global health burden. Measures from graph theory have been used to characterise ASD-related changes in resting-state fMRI functional connectivity networks (FCNs), but recently developed geometry-inspired measures have not been applied so far. In this study, we applied geometry-inspired graph Ricci curvatures to investigate ASD-related changes in resting-state fMRI FCNs. To do this, we applied Forman-Ricci and Ollivier-Ricci curvatures to compare networks of ASD and healthy controls (N = 1112) from the Autism Brain Imaging Data Exchange I (ABIDE-I) dataset. We performed these comparisons at the brain-wide level as well as at the level of individual brain regions, and further, determined the behavioral relevance of region-specific differences with Neurosynth meta-analysis decoding. We found brain-wide ASD-related differences for both Forman-Ricci and Ollivier-Ricci curvatures. For Forman-Ricci curvature, these differences were distributed across 83 of the 200 brain regions studied, and concentrated within the Default Mode, Somatomotor and Ventral Attention Network. Meta-analysis decoding identified the brain regions showing curvature differences as involved in social cognition, memory, language and movement. Notably, comparison with results from previous non-invasive stimulation (TMS/tDCS) experiments revealed that the set of brain regions showing curvature differences overlapped with the set of brain regions whose stimulation resulted in positive cognitive or behavioural outcomes in ASD patients. These results underscore the utility of geometry-inspired graph Ricci curvatures in characterising disease-related changes in ASD, and possibly, other neurodevelopmental disorders.

Effectiveness of Hydrodilatation in Adhesive Capsulitis of Shoulder: A Systematic Review and Meta-Analysis

2018 ◽  
Vol 107 (4) ◽  
pp. 285-293 ◽  
Author(s):  
M. Saltychev ◽  
K. Laimi ◽  
P. Virolainen ◽  
M. Fredericson
Keyword(s):  
Confidence Interval ◽  
Effect Size ◽  
Meta Analysis ◽  
Adhesive Capsulitis ◽  
Treatment Method ◽  
Pain Severity ◽  
Current Evidence ◽  
Number Needed To Treat ◽  
Small Effect Size ◽  
Shoulder Motion

Background and Aims: Even though hydrodilatation has been used for 50 years, the evidence on its effectiveness is not clear. Only one earlier review has strictly focused on this treatment method. The aims of this study are to evaluate the evidence on the effectiveness of hydrodilatation in treatment of adhesive capsulitis and, if appropriate, to assess the correlation between the effects of this procedure and the amount of fluid injected. Materials and Methods: A literature search on MEDLINE, Embase, Scopus, Cochrane Central, Web of Science, and CINAHL databases was done; random-effects meta-analysis and meta-regression were employed; and cost-effectiveness and safeness analyses were left outside the scope of the review. Results: Of the 270 records identified through search, 12 studies were included in qualitative and quantitative analysis and seven were included in a meta-analysis. The lower 95% confidence interval for the effect of hydrodilatation on pain severity was 0.12 indicating small effect size and mean number needed to treat 12. The pooled effect of hydrodilatation on disability level was insignificant 0.2 (95% confidence interval: −0.04 to 0.44). The lower 95% confidence interval for the effect of hydrodilatation on the range of shoulder motion was close to zero (0.07) indicating small effect size with mean number needed to treat 12. The amount of injected solution did not have a substantial effect on pain severity or range of shoulder motion. The heterogeneity level I2 was acceptable from 0% to 60%. Conclusion: According to current evidence, hydrodilatation has only a small, clinically insignificant effect when treating adhesive capsulitis.

scholarly journals Strength of functional signature correlates with effect size in autism

2016 ◽  
Author(s):  
Sara Ballouz ◽  
Jesse Gillis
Keyword(s):  
Study Design ◽  
Effect Size ◽  
Statistical Power ◽  
Fragile X ◽  
Meta Analysis ◽  
Autism Spectrum ◽  
Protein Interaction Data ◽  
Functional Convergence ◽  
Study Designs ◽  
The Impact

AbstractBackgroundDisagreements over genetic signatures associated with disease have been particularly prominent in the field of psychiatric genetics, creating a sharp divide between disease burdens attributed to common and rare variation, with study designs independently targeting each. Meta-analysis within each of these study designs is routine, whether using raw data or summary statistics, but combining results across study designs is atypical. However, tests of functional convergence are used across all study designs, where candidate gene sets are assessed for overlaps with previously known properties. This suggests one possible avenue for combining not study data, but the functional conclusions that they reach.MethodIn this work, we test for functional convergence in autism spectrum disorder (ASD) across different study types, and specifically whether the degree to which a gene is implicated in autism is correlated with the degree to which it drives functional convergence. Because different study designs are distinguishable by their differences in effect size, this also provides a unified means of incorporating the impact of study design into the analysis of convergence.ResultsWe detected remarkably significant positive trends in aggregate (p < 2.2e-16) with 14 individually significant properties (FDR<0.01), many in areas researchers have targeted based on different reasoning, such as the fragile X mental retardation protein (FMRP) interactor enrichment (FDR 0.003). We are also able to detect novel technical effects and we see that network enrichment from protein-protein interaction data is heavily confounded with study design, arising readily in control data.ConclusionsWe see a convergent functional signal for a subset of known and novel functions in ASD from all sources of genetic variation. Meta-analytic approaches explicitly accounting for different study designs can be adapted to other diseases to discover novel functional associations and increase statistical power.

scholarly journals Effectiveness of Digital Education on Communication Skills Among Medical Students: Systematic Review and Meta-Analysis by the Digital Health Education Collaboration

10.2196/12967 ◽  
2019 ◽  
Vol 21 (8) ◽  
pp. e12967 ◽  
Author(s):  
Bhone Myint Kyaw ◽  
Pawel Posadzki ◽  
Sophie Paddock ◽  
Josip Car ◽  
James Campbell ◽  
...  
Keyword(s):  
Medical Students ◽  
Communication Skills ◽  
Effect Size ◽  
Meta Analysis ◽  
Traditional Learning ◽  
Digital Education ◽  
Significant Difference ◽  
Small Effect Size ◽  
Quality Evidence

Background Effective communication skills are essential in diagnosis and treatment processes and in building the doctor-patient relationship. Objective Our aim was to evaluate the effectiveness of digital education in medical students for communication skills development. Broadly, we assessed whether digital education could improve the quality of future doctors’ communication skills. Methods We performed a systematic review and searched seven electronic databases and two trial registries for randomized controlled trials (RCTs) and cluster RCTs (cRCTs) published between January 1990 and September 2018. Two reviewers independently screened the citations, extracted data from the included studies, and assessed the risk of bias. We also assessed the quality of evidence using the Grading of Recommendations, Assessment, Development, and Evaluations assessment (GRADE). Results We included 12 studies with 2101 medical students, of which 10 were RCTs and two were cRCTs. The digital education included online modules, virtual patient simulations, and video-assisted oral feedback. The control groups included didactic lectures, oral feedback, standard curriculum, role play, and no intervention as well as less interactive forms of digital education. The overall risk of bias was high, and the quality of evidence ranged from moderate to very low. For skills outcome, meta-analysis of three studies comparing digital education to traditional learning showed no statistically significant difference in postintervention skills scores between the groups (standardized mean difference [SMD]=–0.19; 95% CI –0.9 to 0.52; I2=86%, N=3 studies [304 students]; small effect size; low-quality evidence). Similarly, a meta-analysis of four studies comparing the effectiveness of blended digital education (ie, online or offline digital education plus traditional learning) and traditional learning showed no statistically significant difference in postintervention skills between the groups (SMD=0.15; 95% CI –0.26 to 0.56; I2=86%; N=4 studies [762 students]; small effect size; low-quality evidence). The additional meta-analysis of four studies comparing more interactive and less interactive forms of digital education also showed little or no difference in postintervention skills scores between the two groups (SMD=0.12; 95% CI: –0.09 to 0.33; I2=40%; N=4 studies [893 students]; small effect size; moderate-quality evidence). For knowledge outcome, two studies comparing the effectiveness of blended online digital education and traditional learning reported no difference in postintervention knowledge scores between the groups (SMD=0.18; 95% CI: –0.2 to 0.55; I2=61%; N=2 studies [292 students]; small effect size; low-quality evidence). The findings on attitudes, satisfaction, and patient-related outcomes were limited or mixed. None of the included studies reported adverse outcomes or economic evaluation of the interventions. Conclusions We found low-quality evidence showing that digital education is as effective as traditional learning in medical students’ communication skills training. Blended digital education seems to be at least as effective as and potentially more effective than traditional learning for communication skills and knowledge. We also found no difference in postintervention skills between more and less interactive forms of digital education. There is a need for further research to evaluate the effectiveness of other forms of digital education such as virtual reality, serious gaming, and mobile learning on medical students’ attitude, satisfaction, and patient-related outcomes as well as the adverse effects and cost-effectiveness of digital education.

scholarly journals Do sounds near the hand facilitate tactile reaction times? Four experiments and a meta-analysis provide mixed support and suggest a small effect size

2020 ◽  
Vol 238 (4) ◽  
pp. 995-1009 ◽  
Author(s):  
Nicholas Paul Holmes ◽  
Dennis Martin ◽  
William Mitchell ◽  
Zeeshan Noorani ◽  
Amber Thorne
Keyword(s):  
Effect Size ◽  
Meta Analysis ◽  
Reaction Times ◽  
Mixed Support ◽  
Small Effect Size

The Effects of Social Skills Interventions for Students With EBD and ASD: A Single-Case Meta-Analysis

2019 ◽  
Vol 44 (5) ◽  
pp. 773-794 ◽  
Author(s):  
Nancy S. Hutchins ◽  
Mack D. Burke ◽  
Lisa Bowman-Perrott ◽  
Kevin R. Tarlow ◽  
Heather Hatton
Keyword(s):  
At Risk ◽  
Social Skills ◽  
Effect Size ◽  
Single Case ◽  
Meta Analysis ◽  
Autism Spectrum ◽  
Social Emotional ◽  
Weighted Mean ◽  
Social Skills Interventions ◽  
Students With Ebd

Social skills interventions are critical for promoting social, emotional, and behavioral competence for students with or at risk of emotional and behavioral disorders (EBD) and autism spectrum disorders (ASD). This single-case meta-analysis examined the effects of social skills interventions (SSIs) for students with EBD and ASD. Effect sizes were calculated for 78 cases across 25 included studies using a nonparametric effect size, Baseline Corrected Tau. The overall weighted mean effect size of 0.54 suggested a moderate effect across the 25 studies. The overall weighted mean effects for studies reporting maintenance and generalization data were 0.68 and 0.37, respectively. Potential moderators examined (disability, intervention design, intervention delivery, methodological quality) were not significant. As such, they did not moderate the outcomes for participants. We conducted a post hoc analysis and hypothesized that between-study differences may be more meaningful than the similarities shared by participants in the same moderator groups. Implications are discussed on using SSIs to address the social, emotional, and behavioral challenges of students with or at risk of EBD and ASD.

scholarly journals Do animated triangles reveal a marked difficulty among autistic people with reading minds?

Autism ◽  
2021 ◽  
pp. 136236132198915
Author(s):  
Alexander C Wilson
Keyword(s):  
Effect Size ◽  
Meta Analysis ◽  
General Information ◽  
Medium Effect ◽  
Group Differences ◽  
Social Challenges ◽  
Autistic People ◽  
The Social ◽  
Small Effect Size ◽  
The Difference

This meta-analysis tested whether autistic people show a marked, isolated difficulty with mentalising when assessed using the Frith -Happé Animations, an advanced test of mentalising (or ‘theory of mind’). Effect sizes were aggregated in multivariate meta-analysis from 33 papers reporting data for over 3000 autistic and non-autistic people. Relative to non-autistic individuals, autistic people underperformed, with a small effect size on the non-mentalising control conditions and a medium effect size on the mentalising condition. This indicates that studies have reliably found mentalising to be an area of challenge for autistic people, although the group differences were not large. It remains to be seen how important mentalising difficulties are in accounting for the social difficulties diagnostic of autism. As autistic people underperformed on the control conditions as well as the mentalising condition, it is likely that group differences on the test are partly due to domain-general information processing differences. Finally, there was evidence of publication bias, suggesting that true effects on the Frith -Happé Animations may be somewhat smaller than reported in the literature. Lay abstract Autistic people are thought to have difficulty with mentalising (our drive to track and understand the minds of other people). Mentalising is often measured by the Frith -Happé Animations task, where individuals need to interpret the interactions of abstract shapes. This review article collated results from over 3000 people to assess how autistic people performed on the task. Analysis showed that autistic people tended to underperform compared to non-autistic people on the task, although the scale of the difference was moderate rather than large. Also, autistic people showed some difficulty with the non-mentalising as well as mentalising aspects of the task. These results raise questions about the scale and specificity of mentalising difficulties in autism. It also remains unclear how well mentalising difficulties account for the social challenges diagnostic of autism.

scholarly journals The Effects of the Early Start Denver Model for Children with Autism Spectrum Disorder: A Meta-Analysis

2020 ◽  
Vol 10 (6) ◽  
pp. 368 ◽  
Author(s):  
Elizabeth A. Fuller ◽  
Kelsey Oliver ◽  
Sarah F. Vejnoska ◽  
Sally J. Rogers
Keyword(s):  
Effect Size ◽  
Variance Estimation ◽  
Meta Analysis ◽  
Children With Autism ◽  
Autism Spectrum ◽  
Repetitive Behaviors ◽  
Early Start ◽  
Early Start Denver Model ◽  
Unique Effect ◽  
Moderate Effect Size

This meta-analysis examined the effects of the Early Start Denver Model (ESDM) for young children with autism on developmental outcome measures. The 12 included studies reported results from 640 children with autism across 44 unique effect sizes. The aggregated effect size, calculated using a robust variance estimation meta-analysis, was 0.357 (p = 0.024), which is a moderate effect size with a statistically significant overall weighted averaged that favored participants who received the ESDM compared to children in control groups, with moderate heterogeneity across studies. This result was largely driven by improvements in cognition (g = 0.412) and language (g = 0.408). There were no significant effects observed for measures of autism symptomology, adaptive behavior, social communication, or restrictive and repetitive behaviors.

Sign in / Sign up

Export Citation Format

Share Document