scholarly journals The importance of heterogeneity in large-scale replications

2020 ◽  
Vol 8 (1) ◽  
pp. 25-29 ◽  
Author(s):  
Matthew H. Goldberg ◽  
Sander van der Linden
Keyword(s):  
Effect Size ◽  
Statistical Power ◽  
Large Scale ◽  
Theory Development ◽  
Scientific Research ◽  
Psychological Science ◽  
Study Results ◽  
Research Questions ◽  
Size Heterogeneity

In a large-scale replication effort, Klein et al. (2018, https://doi.org/10.1177/2515245918810225) investigate the variation in replicability and effect size across many different samples and settings. The authors concluded that, for any given effect being studied, heterogeneity across samples and settings does not explain failures to replicate. In the current commentary, we argue that the heterogeneity observed indeed has implications for replication failures, as well as for statistical power and theory development. We argue that psychological scientific research questions should be contextualized—considering how historical, political, or cultural circumstances might affect study results. We discuss how a perspectivist approach to psychological science is a fruitful way for designing research that aims to explain effect size heterogeneity.

scholarly journals The Importance of Heterogeneity in Large-Scale Replications

2019 ◽  
Author(s):  
Matthew Goldberg ◽  
Sander van der Linden
Keyword(s):  
Effect Size ◽  
Statistical Power ◽  
Large Scale ◽  
Theory Development ◽  
Scientific Research ◽  
Psychological Science ◽  
Study Results ◽  
Research Questions ◽  
Size Heterogeneity

In a large-scale replication effort, Klein et al., (2018) investigate the variation in replicability and effect size across many different samples and settings. The authors concluded that, for any given effect being studied, heterogeneity across samples and settings does not explain failures to replicate. In the current commentary, we argue that the heterogeneity observed indeed has implications for replication failures, as well as for statistical power and theory development. We argue that psychological scientific research questions should be contextualized—considering how historical, political, or cultural circumstances might affect study results. We discuss how a perspectivist approach to psychological science is a fruitful way for designing research that aims to explain effect size heterogeneity.

scholarly journals The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network

2018 ◽  
Vol 1 (4) ◽  
pp. 501-515 ◽  
Author(s):  
Hannah Moshontz ◽  
Lorne Campbell ◽  
Charles R. Ebersole ◽  
Hans IJzerman ◽  
Heather L. Urry ◽  
...  
Keyword(s):  
Statistical Power ◽  
Large Scale ◽  
Psychological Research ◽  
Distributed Network ◽  
Research Projects ◽  
Mental Processes ◽  
Background Structure ◽  
Psychological Science ◽  
Rigorous Research ◽  
Research Questions

Concerns about the veracity of psychological research have been growing. Many findings in psychological science are based on studies with insufficient statistical power and nonrepresentative samples, or may otherwise be limited to specific, ungeneralizable settings or populations. Crowdsourced research, a type of large-scale collaboration in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. These projects can focus on novel research questions or replicate prior research in large, diverse samples. The PSA’s mission is to accelerate the accumulation of reliable and generalizable evidence in psychological science. Here, we describe the background, structure, principles, procedures, benefits, and challenges of the PSA. In contrast to other crowdsourced research networks, the PSA is ongoing (as opposed to time limited), efficient (in that structures and principles are reused for different projects), decentralized, diverse (in both subjects and researchers), and inclusive (of proposals, contributions, and other relevant input from anyone inside or outside the network). The PSA and other approaches to crowdsourced psychological science will advance understanding of mental processes and behaviors by enabling rigorous research and systematic examination of its generalizability.

scholarly journals The Psychological Science Accelerator: Advancing Psychology through a Distributed Collaborative Network

2018 ◽  
Author(s):  
Hannah Moshontz ◽  
Lorne Campbell ◽  
Charles R. Ebersole ◽  
Hans IJzerman ◽  
Heather L. Urry ◽  
...  
Keyword(s):  
Statistical Power ◽  
Large Scale ◽  
Psychological Research ◽  
Distributed Network ◽  
Research Projects ◽  
Mental Processes ◽  
Background Structure ◽  
Psychological Science ◽  
Rigorous Research ◽  
Research Questions

Concerns have been growing about the veracity of psychological research. Many findings in psychological science are based on studies with insufficient statistical power and nonrepresentative samples, or may otherwise be limited to specific, ungeneralizable settings or populations. Crowdsourced research, a type of large-scale collaboration in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. These projects can focus on novel research questions, or attempt to replicate prior research, in large, diverse samples. The PSA’s mission is to accelerate the accumulation of reliable and generalizable evidence in psychological science. Here, we describe the background, structure, principles, procedures, benefits, and challenges of the PSA. In contrast to other crowdsourced research networks, the PSA is ongoing (as opposed to time-limited), efficient (in terms of re-using structures and principles for different projects), decentralized, diverse (in terms of participants and researchers), and inclusive (of proposals, contributions, and other relevant input from anyone inside or outside of the network). The PSA and other approaches to crowdsourced psychological science will advance our understanding of mental processes and behaviors by enabling rigorous research and systematically examining its generalizability.

scholarly journals From Text to Thought: How Analyzing Language Can Advance Psychological Science

2020 ◽  
Author(s):  
Joshua Conrad Jackson ◽  
Joseph Watts ◽  
Johann-Mattis List ◽  
Ryan Drabble ◽  
Kristen Lindquist
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Statistical Power ◽  
Culturally Diverse ◽  
Large Scale ◽  
Human Cognition ◽  
Comparative Linguistics ◽  
Psychological Science ◽  
Language Analysis ◽  
The Mind

Humans have been using language for thousands of years, but psychologists seldom consider what natural language can tell us about the mind. Here we propose that language offers a unique window into human cognition. After briefly summarizing the legacy of language analyses in psychological science, we show how methodological advances have made these analyses more feasible and insightful than ever before. In particular, we describe how two forms of language analysis—comparative linguistics and natural language processing—are already contributing to how we understand emotion, creativity, and religion, and overcoming methodological obstacles related to statistical power and culturally diverse samples. We summarize resources for learning both of these methods, and highlight the best way to combine language analysis techniques with behavioral paradigms. Applying language analysis to large-scale and cross-cultural datasets promises to provide major breakthroughs in psychological science.

scholarly journals From Text to Thought: How Analyzing Language Can Advance Psychological Science

2020 ◽  
Author(s):  
Joshua Conrad Jackson ◽  
Joseph Watts ◽  
Johann-Mattis List ◽  
Curtis Puryear ◽  
Ryan Drabble ◽  
...  
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Statistical Power ◽  
Culturally Diverse ◽  
Large Scale ◽  
Human Cognition ◽  
Comparative Linguistics ◽  
Psychological Science ◽  
Language Analysis ◽  
The Mind

Humans have been using language for thousands of years, but psychologists seldom consider what natural language can tell us about the mind. Here we propose that language offers a unique window into human cognition. After briefly summarizing the legacy of language analyses in psychological science, we show how methodological advances have made these analyses more feasible and insightful than ever before. In particular, we describe how two forms of language analysis—comparative linguistics and natural language processing—are already contributing to how we understand emotion, creativity, and religion, and overcoming methodological obstacles related to statistical power and culturally diverse samples. We summarize resources for learning both of these methods, and highlight the best way to combine language analysis techniques with behavioral paradigms. Applying language analysis to large-scale and cross-cultural datasets promises to provide major breakthroughs in psychological science.

scholarly journals The Psychological Science Accelerator: Advancing Psychology through a Distributed Collaborative Network

2018 ◽  
Author(s):  
Gerit Pfuhl
Keyword(s):  
Statistical Power ◽  
Large Scale ◽  
Collaborative Network ◽  
Distributed Network ◽  
Research Projects ◽  
Mental Processes ◽  
Background Structure ◽  
Psychological Science ◽  
Rigorous Research ◽  
Representative Samples

Concerns have been growing about the veracity of psychological findings. Many findings in psychological science are based on studies with insufficient statistical power and non-representative samples, or may otherwise be limited to specific, ungeneralizable settings or populations. Large-scale collaboration, in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. The PSA’s mission is to accelerate the accumulation of reliable and generalizable evidence in psychological science. Here, we describe the background, structure, principles, procedures, benefits, and challenges of the PSA. In contrast to other crowdsourced research networks, the PSA is ongoing (as opposed to time-limited), efficient (in terms of re-using structures and principles for different projects), decentralized, diverse (in terms of participants and researchers), and inclusive (of proposals, contributions, and other relevant input from anyone inside or outside of the network). The PSA and other approaches to crowdsourced psychological science will advance our understanding of mental processes and behaviors by enabling rigorous research and systematically examining its generalizability.

scholarly journals How to detect publication bias in psychological research? A comparative evaluation of six statistical methods

2018 ◽  
Author(s):  
Frank Renkewitz ◽  
Melanie Keiner
Keyword(s):  
Publication Bias ◽  
Effect Size ◽  
Statistical Power ◽  
Detection Methods ◽  
Type I ◽  
Optimal Method ◽  
True Effect Size ◽  
True Effect ◽  
Size Heterogeneity ◽  
Meta Analyses

Publication biases and questionable research practices are assumed to be two of the main causes of low replication rates observed in the social sciences. Both of these problems do not only increase the proportion of false positives in the literature but can also lead to severely inflated effect size estimates in meta-analyses. Methodologists have proposed a number of statistical tools to detect and correct such bias in meta-analytic results. We present an evaluation of the performance of six of these tools in detecting bias. To assess the Type I error rate and the statistical power of these tools we simulated a large variety of literatures that differed with regard to underlying true effect size, heterogeneity, number of available primary studies and variation of sample sizes in these primary studies. Furthermore, simulated primary studies were subjected to different degrees of publication bias. Our results show that the power of the detection methods follows a complex pattern. Across all simulated conditions, no method consistently outperformed all others. Hence, choosing an optimal method would require knowledge about parameters (e.g., true effect size, heterogeneity) that meta-analysts cannot have. Additionally, all methods performed badly when true effect sizes were heterogeneous or primary studies had a small chance of being published irrespective of their results. This suggests, that in many actual meta-analyses in psychology bias will remain undiscovered no matter which detection method is used.

scholarly journals Performing High-Powered Studies Efficiently With Sequential Analyses

2017 ◽  
Author(s):  
Daniel Lakens
Keyword(s):  
Data Collection ◽  
Effect Size ◽  
Statistical Power ◽  
Large Scale ◽  
Analysis Data ◽  
Bayesian Approaches ◽  
Type 1 Error ◽  
Experimenter Bias ◽  
Practical Challenge ◽  
Sequential Analyses

Running studies with high statistical power, while effect size estimates in psychology are often inaccurate, leads to a practical challenge when designing an experiment. This challenge can be addressed by performing sequential analyses while the data collection is still in progress. At an interim analysis, data collection can be stopped whenever the results are convincing enough to conclude an effect is present, more data can be collected, or the study can be terminated whenever it is extremely unlikely the predicted effect will be observed if data collection would be continued. Such interim analyses can be performed while controlling the Type 1 error rate. Sequential analyses can greatly improve the efficiency with which data is collected. Additional flexibility is provided by adaptive designs where sample sizes are increased based on the observed effect size. The need for pre-registration, ways to prevent experimenter bias, and a comparison between Bayesian approaches and NHST are discussed. Sequential analyses, which are widely used in large scale medical trials, provide an efficient way to perform high-powered informative experiments. I hope this introduction will provide a practical primer that allows researchers to incorporate sequential analyses in their research.

scholarly journals From Text to Thought: How Analyzing Language Can Advance Psychological Science

2021 ◽  
pp. 174569162110048
Author(s):  
Joshua Conrad Jackson ◽  
Joseph Watts ◽  
Johann-Mattis List ◽  
Curtis Puryear ◽  
Ryan Drabble ◽  
...  
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Statistical Power ◽  
Culturally Diverse ◽  
Large Scale ◽  
Comparative Linguistics ◽  
Psychological Science ◽  
Language Analysis ◽  
The Mind ◽  
Psychological Paradigms

Humans have been using language for millennia but have only just begun to scratch the surface of what natural language can reveal about the mind. Here we propose that language offers a unique window into psychology. After briefly summarizing the legacy of language analyses in psychological science, we show how methodological advances have made these analyses more feasible and insightful than ever before. In particular, we describe how two forms of language analysis—natural-language processing and comparative linguistics—are contributing to how we understand topics as diverse as emotion, creativity, and religion and overcoming obstacles related to statistical power and culturally diverse samples. We summarize resources for learning both of these methods and highlight the best way to combine language analysis with more traditional psychological paradigms. Applying language analysis to large-scale and cross-cultural datasets promises to provide major breakthroughs in psychological science.

scholarly journals Progress toward openness, transparency, and reproducibility in cognitive neuroscience

2017 ◽  
Author(s):  
Rick Owen Gilmore ◽  
Michele Diaz ◽  
Brad Wyble ◽  
Tal Yarkoni
Keyword(s):  
Brain Imaging ◽  
Cognitive Neuroscience ◽  
Rapid Growth ◽  
Statistical Power ◽  
Large Scale ◽  
Research Groups ◽  
Imaging Studies ◽  
Psychological Science ◽  
Open Research ◽  
Data Archives

Accumulating evidence suggests that many findings in psychological science and cognitive neuroscience may prove difficult to reproduce; statistical power in brain imaging studies is low, and has not improved recently; software errors in common analysis tools are common, and can go undetected for many years; and, a few large scale studies notwithstanding, open sharing of data, code, and materials remains the rare exception. At the same time, there is a renewed focus on reproducibility, transparency, and openness as essential core values in cognitive neuroscience. The emergence and rapid growth of data archives, meta-analytic tools, software pipelines, and research groups devoted to improved methodology reflects this new sensibility. We review evidence that the field has begun to embrace new open research practices, and illustrate how these can begin to address problems of reproducibility, statistical power, and transparency in ways that will ultimately accelerate discovery.tr

Sign in / Sign up

Export Citation Format

Share Document