scholarly journals Progress toward openness, transparency, and reproducibility in cognitive neuroscience

2017 ◽  
Author(s):  
Rick Owen Gilmore ◽  
Michele Diaz ◽  
Brad Wyble ◽  
Tal Yarkoni
Keyword(s):  
Brain Imaging ◽  
Cognitive Neuroscience ◽  
Rapid Growth ◽  
Statistical Power ◽  
Large Scale ◽  
Research Groups ◽  
Imaging Studies ◽  
Psychological Science ◽  
Open Research ◽  
Data Archives

Accumulating evidence suggests that many findings in psychological science and cognitive neuroscience may prove difficult to reproduce; statistical power in brain imaging studies is low, and has not improved recently; software errors in common analysis tools are common, and can go undetected for many years; and, a few large scale studies notwithstanding, open sharing of data, code, and materials remains the rare exception. At the same time, there is a renewed focus on reproducibility, transparency, and openness as essential core values in cognitive neuroscience. The emergence and rapid growth of data archives, meta-analytic tools, software pipelines, and research groups devoted to improved methodology reflects this new sensibility. We review evidence that the field has begun to embrace new open research practices, and illustrate how these can begin to address problems of reproducibility, statistical power, and transparency in ways that will ultimately accelerate discovery.tr

scholarly journals Opportunities for increased reproducibility and replicability of developmental neuroimaging

2019 ◽  
Author(s):  
Eduard Klapwijk ◽  
Wouter van den Bos ◽  
Christian K. Tamnes ◽  
Nora Maria Raschle ◽  
Kathryn L. Mills
Keyword(s):  
Data Collection ◽  
Cognitive Neuroscience ◽  
Statistical Power ◽  
Large Scale ◽  
Small Samples ◽  
Developmental Studies ◽  
Data Analyses ◽  
Methodological Rigor ◽  
Research Findings ◽  
Tools And Techniques

Many workflows and tools that aim to increase the reproducibility and replicability of research findings have been suggested. In this review, we discuss the opportunities that these efforts offer for the field of developmental cognitive neuroscience, in particular developmental neuroimaging. We focus on issues broadly related to statistical power and to flexibility and transparency in data analyses. Critical considerations relating to statistical power include challenges in recruitment and testing of young populations, how to increase the value of studies with small samples, and the opportunities and challenges related to working with large-scale datasets. Developmental studies involve challenges such as choices about age groupings, lifespan modelling, analyses of longitudinal changes, and data that can be processed and analyzed in a multitude of ways. Flexibility in data acquisition, analyses and description may thereby greatly impact results. We discuss methods for improving transparency in developmental neuroimaging, and how preregistration can improve methodological rigor. While outlining challenges and issues that may arise before, during, and after data collection, solutions and resources are highlighted aiding to overcome some of these. Since the number of useful tools and techniques is ever-growing, we highlight the fact that many practices can be implemented stepwise.

scholarly journals From Text to Thought: How Analyzing Language Can Advance Psychological Science

2020 ◽  
Author(s):  
Joshua Conrad Jackson ◽  
Joseph Watts ◽  
Johann-Mattis List ◽  
Ryan Drabble ◽  
Kristen Lindquist
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Statistical Power ◽  
Culturally Diverse ◽  
Large Scale ◽  
Human Cognition ◽  
Comparative Linguistics ◽  
Psychological Science ◽  
Language Analysis ◽  
The Mind

Humans have been using language for thousands of years, but psychologists seldom consider what natural language can tell us about the mind. Here we propose that language offers a unique window into human cognition. After briefly summarizing the legacy of language analyses in psychological science, we show how methodological advances have made these analyses more feasible and insightful than ever before. In particular, we describe how two forms of language analysis—comparative linguistics and natural language processing—are already contributing to how we understand emotion, creativity, and religion, and overcoming methodological obstacles related to statistical power and culturally diverse samples. We summarize resources for learning both of these methods, and highlight the best way to combine language analysis techniques with behavioral paradigms. Applying language analysis to large-scale and cross-cultural datasets promises to provide major breakthroughs in psychological science.

Brain Imaging Revisited

2007 ◽  
Vol 13 (6) ◽  
pp. 1071-1072
Author(s):  
Cheryl L. Grady
Keyword(s):  
Brain Imaging ◽  
Cognitive Neuroscience ◽  
Rapid Growth ◽  
Functional Neuroimaging ◽  
Previous Edition ◽  
History Of ◽  
Cognitive Areas

Handbook of Functional Neuroimaging of Cognition, Second Edition. 2006. Roberto Cabeza and Alan Kingstone (Eds.), Cambridge, MA, The MIT Press, 480 pp., $65.00 (HB)The first edition of the Handbook of Functional Neuroimaging of Cognition, edited by Roberto Cabeza and Alan Kingstone, was a welcome addition to the cognitive neuroscience field when it was published in 2001. There were chapters on the history of neuroimaging and analysis, and all the major cognitive areas that had been studied at the time, written by senior people in their respective areas. That a second edition has appeared so soon after the first is a testament to the rapid growth of the cognitive neuroscience field, which is both gratifying and somewhat daunting to those of us who vainly attempt to keep up with this burgeoning literature. The same authors as in the previous edition write some chapters, but many have been penned by different authors, equally well known in the field, which is also a sign that the field is healthy and growing.

scholarly journals From Text to Thought: How Analyzing Language Can Advance Psychological Science

2020 ◽  
Author(s):  
Joshua Conrad Jackson ◽  
Joseph Watts ◽  
Johann-Mattis List ◽  
Curtis Puryear ◽  
Ryan Drabble ◽  
...  
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Statistical Power ◽  
Culturally Diverse ◽  
Large Scale ◽  
Human Cognition ◽  
Comparative Linguistics ◽  
Psychological Science ◽  
Language Analysis ◽  
The Mind

Humans have been using language for thousands of years, but psychologists seldom consider what natural language can tell us about the mind. Here we propose that language offers a unique window into human cognition. After briefly summarizing the legacy of language analyses in psychological science, we show how methodological advances have made these analyses more feasible and insightful than ever before. In particular, we describe how two forms of language analysis—comparative linguistics and natural language processing—are already contributing to how we understand emotion, creativity, and religion, and overcoming methodological obstacles related to statistical power and culturally diverse samples. We summarize resources for learning both of these methods, and highlight the best way to combine language analysis techniques with behavioral paradigms. Applying language analysis to large-scale and cross-cultural datasets promises to provide major breakthroughs in psychological science.

scholarly journals The Psychological Science Accelerator: Advancing Psychology through a Distributed Collaborative Network

2018 ◽  
Author(s):  
Gerit Pfuhl
Keyword(s):  
Statistical Power ◽  
Large Scale ◽  
Collaborative Network ◽  
Distributed Network ◽  
Research Projects ◽  
Mental Processes ◽  
Background Structure ◽  
Psychological Science ◽  
Rigorous Research ◽  
Representative Samples

Concerns have been growing about the veracity of psychological findings. Many findings in psychological science are based on studies with insufficient statistical power and non-representative samples, or may otherwise be limited to specific, ungeneralizable settings or populations. Large-scale collaboration, in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. The PSA’s mission is to accelerate the accumulation of reliable and generalizable evidence in psychological science. Here, we describe the background, structure, principles, procedures, benefits, and challenges of the PSA. In contrast to other crowdsourced research networks, the PSA is ongoing (as opposed to time-limited), efficient (in terms of re-using structures and principles for different projects), decentralized, diverse (in terms of participants and researchers), and inclusive (of proposals, contributions, and other relevant input from anyone inside or outside of the network). The PSA and other approaches to crowdsourced psychological science will advance our understanding of mental processes and behaviors by enabling rigorous research and systematically examining its generalizability.

scholarly journals The importance of heterogeneity in large-scale replications

2020 ◽  
Vol 8 (1) ◽  
pp. 25-29 ◽  
Author(s):  
Matthew H. Goldberg ◽  
Sander van der Linden
Keyword(s):  
Effect Size ◽  
Statistical Power ◽  
Large Scale ◽  
Theory Development ◽  
Scientific Research ◽  
Psychological Science ◽  
Study Results ◽  
Research Questions ◽  
Size Heterogeneity

In a large-scale replication effort, Klein et al. (2018, https://doi.org/10.1177/2515245918810225) investigate the variation in replicability and effect size across many different samples and settings. The authors concluded that, for any given effect being studied, heterogeneity across samples and settings does not explain failures to replicate. In the current commentary, we argue that the heterogeneity observed indeed has implications for replication failures, as well as for statistical power and theory development. We argue that psychological scientific research questions should be contextualized—considering how historical, political, or cultural circumstances might affect study results. We discuss how a perspectivist approach to psychological science is a fruitful way for designing research that aims to explain effect size heterogeneity.

scholarly journals From Text to Thought: How Analyzing Language Can Advance Psychological Science

2021 ◽  
pp. 174569162110048
Author(s):  
Joshua Conrad Jackson ◽  
Joseph Watts ◽  
Johann-Mattis List ◽  
Curtis Puryear ◽  
Ryan Drabble ◽  
...  
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Statistical Power ◽  
Culturally Diverse ◽  
Large Scale ◽  
Comparative Linguistics ◽  
Psychological Science ◽  
Language Analysis ◽  
The Mind ◽  
Psychological Paradigms

Humans have been using language for millennia but have only just begun to scratch the surface of what natural language can reveal about the mind. Here we propose that language offers a unique window into psychology. After briefly summarizing the legacy of language analyses in psychological science, we show how methodological advances have made these analyses more feasible and insightful than ever before. In particular, we describe how two forms of language analysis—natural-language processing and comparative linguistics—are contributing to how we understand topics as diverse as emotion, creativity, and religion and overcoming obstacles related to statistical power and culturally diverse samples. We summarize resources for learning both of these methods and highlight the best way to combine language analysis with more traditional psychological paradigms. Applying language analysis to large-scale and cross-cultural datasets promises to provide major breakthroughs in psychological science.

scholarly journals Extracting representations of cognition across neuroimaging studies improves brain decoding

2021 ◽  
Vol 17 (5) ◽  
pp. e1008795
Author(s):  
Arthur Mensch ◽  
Julien Mairal ◽  
Bertrand Thirion ◽  
Gaël Varoquaux
Keyword(s):  
Brain Imaging ◽  
Statistical Power ◽  
Brain Structures ◽  
Data Driven ◽  
Cognitive Tasks ◽  
Imaging Studies ◽  
Mental Processes ◽  
Brain Responses ◽  
Neural Substrate ◽  
Neuro Imaging

Cognitive brain imaging is accumulating datasets about the neural substrate of many different mental processes. Yet, most studies are based on few subjects and have low statistical power. Analyzing data across studies could bring more statistical power; yet the current brain-imaging analytic framework cannot be used at scale as it requires casting all cognitive tasks in a unified theoretical framework. We introduce a new methodology to analyze brain responses across tasks without a joint model of the psychological processes. The method boosts statistical power in small studies with specific cognitive focus by analyzing them jointly with large studies that probe less focal mental processes. Our approach improves decoding performance for 80% of 35 widely-different functional-imaging studies. It finds commonalities across tasks in a data-driven way, via common brain representations that predict mental processes. These are brain networks tuned to psychological manipulations. They outline interpretable and plausible brain structures. The extracted networks have been made available; they can be readily reused in new neuro-imaging studies. We provide a multi-study decoding tool to adapt to new data.

scholarly journals The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network

2018 ◽  
Vol 1 (4) ◽  
pp. 501-515 ◽  
Author(s):  
Hannah Moshontz ◽  
Lorne Campbell ◽  
Charles R. Ebersole ◽  
Hans IJzerman ◽  
Heather L. Urry ◽  
...  
Keyword(s):  
Statistical Power ◽  
Large Scale ◽  
Psychological Research ◽  
Distributed Network ◽  
Research Projects ◽  
Mental Processes ◽  
Background Structure ◽  
Psychological Science ◽  
Rigorous Research ◽  
Research Questions

Concerns about the veracity of psychological research have been growing. Many findings in psychological science are based on studies with insufficient statistical power and nonrepresentative samples, or may otherwise be limited to specific, ungeneralizable settings or populations. Crowdsourced research, a type of large-scale collaboration in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. These projects can focus on novel research questions or replicate prior research in large, diverse samples. The PSA’s mission is to accelerate the accumulation of reliable and generalizable evidence in psychological science. Here, we describe the background, structure, principles, procedures, benefits, and challenges of the PSA. In contrast to other crowdsourced research networks, the PSA is ongoing (as opposed to time limited), efficient (in that structures and principles are reused for different projects), decentralized, diverse (in both subjects and researchers), and inclusive (of proposals, contributions, and other relevant input from anyone inside or outside the network). The PSA and other approaches to crowdsourced psychological science will advance understanding of mental processes and behaviors by enabling rigorous research and systematic examination of its generalizability.

Sign in / Sign up

Export Citation Format

Share Document