scholarly journals Confidence Sets for Cohen’s d Effect Size Images

2020 ◽  
Author(s):  
Alexander Bowring ◽  
Fabian Telschow ◽  
Armin Schwartzman ◽  
Thomas E. Nichols
Keyword(s):  
Effect Size ◽  
Null Hypothesis ◽  
Large Scale ◽  
Brain Regions ◽  
Spatial Uncertainty ◽  
Effect Sizes ◽  
Effect Estimate ◽  
Confidence Sets ◽  
Human Connectome Project ◽  
Inference Methods

AbstractCurrent statistical inference methods for task-fMRI suffer from two fundamental limitations. First, the focus is solely on detection of non-zero signal or signal change, a problem that is exasperated for large scale studies (e.g. UK Biobank, N = 40, 000+) where the ‘null hypothesis fallacy’ causes even trivial effects to be determined as significant. Second, for any sample size, widely used cluster inference methods only indicate regions where a null hypothesis can be rejected, without providing any notion of spatial uncertainty about the activation. In this work, we address these issues by developing spatial Confidence Sets (CSs) on clusters found in thresholded Cohen’s d effect size images. We produce an upper and lower CS to make confidence statements about brain regions where Cohen’s d effect sizes have exceeded and fallen short of a non-zero threshold, respectively. The CSs convey information about the magnitude and reliability of effect sizes that is usually given separately in a t-statistic and effect estimate map. We expand the theory developed in our previous work on CSs for %BOLD change effect maps (Bowring et al., 2019) using recent results from the bootstrapping literature. By assessing the empirical coverage with 2D and 3D Monte Carlo simulations resembling fMRI data, we find our method is accurate in sample sizes as low as N = 60. We compute Cohen’s d CSs for the Human Connectome Project working memory taskfMRI data, illustrating the brain regions with a reliable Cohen’s d response for a given threshold. By comparing the CSs with results obtained from a traditional statistical voxelwise inference, we highlight the improvement in activation localization that can be gained with the Confidence Sets.

scholarly journals Spatial Confidence Sets for Raw Effect Size Images

2019 ◽  
Author(s):  
Alexander Bowring ◽  
Fabian Telschow ◽  
Armin Schwartzman ◽  
Thomas E. Nichols
Keyword(s):  
Sample Size ◽  
Effect Size ◽  
Null Hypothesis ◽  
Statistical Power ◽  
Memory Task ◽  
Small Sample ◽  
Spatial Uncertainty ◽  
Practical Implementation ◽  
Confidence Sets ◽  
Human Connectome Project

AbstractThe mass-univariate approach for functional magnetic resonance imagery (fMRI) analysis remains a widely used and fundamental statistical tool within neuroimaging. However, this method suffers from at least two fundamental limitations: First, with sample sizes growing to 4, 5 or even 6 digits, the entire approach is undermined by the null hypothesis fallacy, i.e. with sufficient sample size, there is high enough statistical power to reject the null hypothesis everywhere, making it difficult if not impossible to localize effects of interest. Second, with any sample size, when cluster-size inference is used a significant p-value only indicates that a cluster is larger than chance, and no notion of spatial uncertainty is provided. Therefore, no perception of confidence is available to express the size or location of a cluster that could be expected with repeated sampling from the population.In this work, we address these issues by extending on a method proposed by Sommerfeld, Sain, and Schwartzman (2018) to develop spatial Confidence Sets (CSs) on clusters found in thresholded raw effect size maps. While hypothesis testing indicates where the null, i.e. a raw effect size of zero, can be rejected, the CSs give statements on the locations where raw effect sizes exceed, and fall short of, a non-zero threshold, providing both an upper and lower CS.While the method can be applied to any parameter in a mass-univariate General Linear Model, we motivate the method in the context of BOLD fMRI contrast maps for inference on percentage BOLD change raw effects. We propose several theoretical and practical implementation advancements to the original method in order to deliver an improved performance in small-sample settings. We validate the method with 3D Monte Carlo simulations that resemble fMRI data. Finally, we compute CSs for the Human Connectome Project working memory task contrast images, illustrating the brain regions that show a reliable %BOLD change for a given %BOLD threshold.

Effect Sizes: Conventional Choices and Calculations

2013 ◽  
Author(s):  
Michael S. Rosenberg ◽  
Hannah R. Rothstein ◽  
Jessica Gurevitch
Keyword(s):  
Critical Point ◽  
Effect Size ◽  
Evolutionary Biology ◽  
Meta Analysis ◽  
Statistical Parameter ◽  
Effect Sizes ◽  
Common Effect ◽  
Inference Methods ◽  
Conventional Effect ◽  
Estimate Effect Size

One of the fundamental concepts in meta-analysis is that of the effect size. An effect size is a statistical parameter that can be used to compare, on the same scale, the results of different studies in which a common effect of interest has been measured. This chapter describes the conventional effect sizes most commonly encountered in ecology and evolutionary biology, and the types of data associated with them. While choice of a specific measure of effect size may influence the interpretation of results, it does not influence the actual inference methods of meta-analysis. One critical point to remember is that one cannot combine different measures of effect size in a single meta-analysis: once you have chosen how you are going to estimate effect size, you need to use it for all of the studies to be analyzed.

The Ouroboros of Psychological Methodology: The Case of Effect Sizes (Mechanical Objectivity vs. Expertise)

2018 ◽  
Vol 22 (4) ◽  
pp. 469-476 ◽  
Author(s):  
Ian J. Davidson
Keyword(s):  
Effect Size ◽  
Null Hypothesis ◽  
Significance Test ◽  
Effect Sizes ◽  
Circular Path ◽  
Bottom Up ◽  
Psychological Science ◽  
Null Hypothesis Significance Test ◽  
Theoretical Tool ◽  
Mechanical Objectivity

The reporting and interpretation of effect sizes is often promoted as a panacea for the ramifications of institutionalized statistical rituals associated with the null-hypothesis significance test. Mechanical objectivity—conflating the use of a method with the obtainment of truth—is a useful theoretical tool for understanding the possible failure of effect size reporting ( Porter, 1995 ). This article helps elucidate the ouroboros of psychological methodology. This is the cycle of improved tools to produce trustworthy knowledge, leading to their institutionalization and adoption as forms of thinking, leading to methodologists eventually admonishing researchers for relying too heavily on rituals, finally leading to the production of more new improved quantitative tools that may follow along this circular path. Despite many critiques and warnings, research psychologists’ superficial adoption of effect sizes might preclude expert interpretation much like in the null-hypothesis significance test as widely received. One solution to this situation is bottom-up: promoting a balance of mechanical objectivity and expertise in the teaching of methods and research. This would require the acceptance and encouragement of expert interpretation within psychological science.

scholarly journals The Developing Human Connectome Project: typical and disrupted perinatal functional connectivity

2020 ◽  
Author(s):  
Michael Eyre ◽  
Sean P Fitzgibbon ◽  
Judit Ciarrusta ◽  
Lucilio Cordero-Grande ◽  
Anthony N Price ◽  
...  
Keyword(s):  
Preterm Birth ◽  
Functional Connectivity ◽  
Large Scale ◽  
Developmental Coordination Disorder ◽  
Positive Association ◽  
Brain Regions ◽  
Higher Order ◽  
Dependent Manner ◽  
Normal Term ◽  
Human Connectome Project

AbstractThe Developing Human Connectome Project (dHCP) is an Open Science project which provides the first large sample of neonatal functional MRI (fMRI) data with high temporal and spatial resolution. This data enables mapping of intrinsic functional connectivity between spatially distributed brain regions under normal and adverse perinatal circumstances, offering a framework to study the ontogeny of large-scale brain organisation in humans. Here, we characterise in unprecedented detail the maturation and integrity of resting-state networks (RSNs) at normal term age in 337 infants (including 65 born preterm).First, we applied group independent component analysis (ICA) to define 11 RSNs in term-born infants scanned at 43.5-44.5 weeks postmenstrual age (PMA). Adult-like topography was observed in RSNs encompassing primary sensorimotor, visual and auditory cortices. Among six higher-order, association RSNs, analogues of the adult networks for language and ocular control were identified, but a complete default mode network precursor was not. Next, we regressed the subject-level datasets from an independent cohort of infants scanned at 37-43.5 weeks PMA against the group-level RSNs to test for the effects of age, sex and preterm birth. Brain mapping in term-born infants revealed areas of positive association with age across four of six association RSNs, indicating active maturation in functional connectivity from 37 to 43.5 weeks PMA. Female infants showed increased connectivity in inferotemporal regions of the visual association network. Preterm birth was associated with striking impairments of functional connectivity across all RSNs in a dose-dependent manner; conversely, connectivity of the superior parietal lobules within the lateral motor network was abnormally increased in preterm infants, suggesting a possible mechanism for specific difficulties such as developmental coordination disorder which occur frequently in preterm children.Overall, we find a robust, modular, symmetrical functional brain organisation at normal term age. A complete set of adult-equivalent primary RSNs is already instated, alongside emerging connectivity in immature association RSNs, consistent with a primary-to-higher-order ontogenetic sequence of brain development. The early developmental disruption imposed by preterm birth is associated with extensive alterations in functional connectivity.

scholarly journals Robust prediction of individual personality from brain functional connectome

2020 ◽  
Vol 15 (3) ◽  
pp. 359-369 ◽  
Author(s):  
Huanhuan Cai ◽  
Jiajia Zhu ◽  
Yongqiang Yu
Keyword(s):  
Personality Traits ◽  
Large Scale ◽  
Five Factor Model ◽  
Brain Regions ◽  
Individual Variability ◽  
Personality Factors ◽  
Imaging Features ◽  
Imaging Data ◽  
Functional Connectome ◽  
Human Connectome Project

Abstract Neuroimaging studies have linked inter-individual variability in the brain to individualized personality traits. However, only one or several aspects of personality have been effectively predicted based on brain imaging features. The objective of this study was to construct a reliable prediction model of personality in a large sample by using connectome-based predictive modeling (CPM), a recently developed machine learning approach. High-quality resting-state functional magnetic resonance imaging data of 810 healthy young participants from the Human Connectome Project dataset were used to construct large-scale brain networks. Personality traits of the five-factor model (FFM) were assessed by the NEO Five Factor Inventory. We found that CPM successfully and reliably predicted all the FFM personality factors (agreeableness, openness, conscientiousness and neuroticism) other than extraversion in novel individuals. At the neural level, we found that the personality-associated functional networks mainly included brain regions within default mode, frontoparietal executive control, visual and cerebellar systems. Although different feature selection thresholds and parcellation strategies did not significantly influence the prediction results, some findings lost significance after controlling for confounds including age, gender, intelligence and head motion. Our finding of robust personality prediction from an individual’s unique functional connectome may help advance the translation of ‘brain connectivity fingerprinting’ into real-world personality psychological settings.

scholarly journals Effect Size

Psychology ◽  
2019 ◽  
Author(s):  
David B. Flora
Keyword(s):  
Effect Size ◽  
Null Hypothesis ◽  
Power Analysis ◽  
Meta Analysis ◽  
Significance Test ◽  
Effect Sizes ◽  
Research Reporting ◽  
The Difference ◽  
Strength Of Association ◽  
True Population

Simply put, effect size (ES) is the magnitude or strength of association between or among variables. Effect sizes (ESs) are commonly represented numerically (i.e., as parameters for population ESs and statistics for sample estimates of population ESs) but also may be communicated graphically. Although the word “effect” may imply that an ES quantifies the strength of a causal association (“cause and effect”), ESs are used more broadly to represent any empirical association between variables. Effect sizes serve three general purposes: research results reporting, power analysis, and meta-analysis. Even under the same research design, an ES that is appropriate for one of these purposes may not be ideal for another. Effect size can be conveyed graphically or numerically using either unstandardized metrics, which are interpreted relative to the original scales of the variables involved (e.g., the difference between two means or an unstandardized regression slope), or standardized metrics, which are interpreted in relative terms (e.g., Cohen’s d or multiple R2). Whereas unstandardized ESs and graphs illustrating ES are typically most effective for research reporting, that is, communicating the original findings of an empirical study, many standardized ES measures have been developed for use in power analysis and especially meta-analysis. Although the concept of ES is clearly fundamental to data analysis, ES reporting has been advocated as an essential complement to null hypothesis significance testing (NHST), or even as a replacement for NHST. A null hypothesis significance test involves making a dichotomous judgment about whether to reject a hypothesis that a true population effect equals zero. Even in the context of a traditional NHST paradigm, ES is a critical concept because of its central role in power analysis.

The Counternull Value of an Effect Size: A New Statistic

1994 ◽  
Vol 5 (6) ◽  
pp. 329-334 ◽  
Author(s):  
Robert Rosenthal ◽  
Donald B. Rubin
Keyword(s):  
Confidence Interval ◽  
Effect Size ◽  
Null Hypothesis ◽  
Degree Of Freedom ◽  
Effect Sizes ◽  
P Value ◽  
Summary Effect ◽  
Null Value ◽  
Meta Analyses

We introduce a new, readily computed statistic, the counternull value of an obtained effect size, which is the nonnull magnitude of effect size that is supported by exactly the same amount of evidence as supports the null value of the effect size In other words, if the counternull value were taken as the null hypothesis, the resulting p value would be the same as the obtained p value for the actual null hypothesis Reporting the counternull, in addition to the p value, virtually eliminates two common errors (a) equating failure to reject the null with the estimation of the effect size as equal to zero and (b) taking the rejection of a null hypothesis on the basis of a significant p value to imply a scientifically important finding In many common situations with a one-degree-of-freedom effect size, the value of the counternull is simply twice the magnitude of the obtained effect size, but the counternull is defined in general, even with multi-degree-of-freedom effect sizes, and therefore can be applied when a confidence interval cannot be The use of the counter-null can be especially useful in meta-analyses when evaluating the scientific importance of summary effect sizes

scholarly journals Predicting replication outcomes in the Many Labs 2 study

2018 ◽  
Author(s):  
Eskil Forsell ◽  
Domenico Viganola ◽  
Thomas Pfeiffer ◽  
Johan Almenberg ◽  
Brad Wilson ◽  
...  
Keyword(s):  
Effect Size ◽  
Large Scale ◽  
Scientific Progress ◽  
Relative Effect ◽  
Effect Sizes ◽  
Prediction Markets ◽  
Prediction Errors ◽  
Original Direction ◽  
Highly Correlated ◽  
The Many

Understanding and improving reproducibility is crucial for scientific progress. Prediction markets and related methods of eliciting peer beliefs are promising tools to predict replication outcomes. We invited researchers in the field of psychology to judge the replicability of 24 studies replicated in the large scale Many Labs 2 project. We elicited peer beliefs in prediction markets and surveys about two replication success metrics: the probability that the replication yields a statistically significant effect in the original direction (p<0.001), and the relative effect size of the replication. The prediction markets correctly predicted 75% of the replication outcomes, and were highly correlated with the replication outcomes. Survey beliefs were also significantly correlated with replication outcomes, but had higher prediction errors. The prediction markets for relative effect sizes attracted little trading and thus did not work well. The survey beliefs about relative effect sizes performed better and were significantly correlated with observed relative effect sizes. These results suggest that replication outcomes can be predicted and that the elicitation of peer beliefs can increase our knowledge about scientific reproducibility and the dynamics of hypothesis testing.

Effect Sizes and F Ratios < 1.0

Methodology ◽  
2007 ◽  
Vol 3 (1) ◽  
pp. 35-46 ◽  
Author(s):  
Manuel C. Voelkle ◽  
Phillip L. Ackerman ◽  
Werner W. Wittmann
Keyword(s):  
Simulation Study ◽  
Effect Size ◽  
Null Hypothesis ◽  
Fixed Effects ◽  
Effect Sizes ◽  
Association Strength ◽  
Measures Of Association ◽  
Journal Editors ◽  
The Face ◽  
Size Estimates

Abstract. Standard statistics texts indicate that the expected value of the F ratio is 1.0 (more precisely: N/(N-2)) in a completely balanced fixed-effects ANOVA, when the null hypothesis is true. Even though some authors suggest that the null hypothesis is rarely true in practice (e.g., Meehl, 1990 ), F ratios < 1.0 are reported quite frequently in the literature. However, standard effect size statistics (e.g., Cohen's f) often yield positive values when F < 1.0, which appears to create confusion about the meaningfulness of effect size statistics when the null hypothesis may be true. Given the repeated emphasis on reporting effect sizes, it is shown that in the face of F < 1.0 it is misleading to only report sample effect size estimates as often recommended. Causes of F ratios < 1.0 are reviewed, illustrated by a short simulation study. The calculation and interpretation of corrected and uncorrected effect size statistics under these conditions are discussed. Computing adjusted measures of association strength and incorporating effect size confidence intervals are helpful in an effort to reduce confusion surrounding results when sample sizes are small. Detailed recommendations are directed to authors, journal editors, and reviewers.

scholarly journals When and how to calculate the Bayes factor with an interval null hypothesis

2019 ◽  
Author(s):  
Bence Palfi ◽  
Zoltan Dienes
Keyword(s):  
Big Data ◽  
Effect Size ◽  
Standard Error ◽  
Null Hypothesis ◽  
Bayes Factor ◽  
Null Model ◽  
Negligible Effect ◽  
Effect Sizes ◽  
Point Null Hypothesis

Existing calculators of the Bayes factor typically represent the prediction of the null-hypothesis with a point null model (e.g., Dienes &amp; Mclatchie, 2018; Morey &amp; Rouder, 2015; Wagenmakers et al., 2018). The point null model remains a good approximation of the null hypothesis as long as the standard error of the estimate of interest is large relative to the range of theoretically negligible effect sizes. However, this assumption may be violated, for instance, when there is big data and the standard error becomes comparable to or smaller than the range of effect sizes that is meaningless according to the theory. In this case, the conclusion that was based on the point null model may not be scientifically valid. Here, we introduce a case study to demonstrate why is it critical to calculate the Bayes factor with an interval null hypothesis rather than a point null hypothesis when the smallest meaningful effect size is not approximately zero relative to the standard error.

Sign in / Sign up

Export Citation Format

Share Document