scholarly journals Attentionally modulated subjective images reconstructed from brain activity

2020 ◽  
Author(s):  
Tomoyasu Horikawa ◽  
Yukiyasu Kamitani
Keyword(s):  
Image Reconstruction ◽  
Visual Experience ◽  
Brain Activity ◽  
Visual Image ◽  
Neural Representation ◽  
Visual Images ◽  
Top Down ◽  
Neural Representations ◽  
Deep Image ◽  
The Brain

SummaryVisual image reconstruction from brain activity produces images whose features are consistent with the neural representations in the visual cortex given arbitrary visual instances [1–3], presumably reflecting the person’s visual experience. Previous reconstruction studies have been concerned either with how stimulus images are faithfully reconstructed or with whether mentally imagined contents can be reconstructed in the absence of external stimuli. However, many lines of vision research have demonstrated that even stimulus perception is shaped both by stimulus-induced processes and top-down processes. In particular, attention (or the lack of it) is known to profoundly affect visual experience [4–8] and brain activity [9–21]. Here, to investigate how top-down attention impacts the neural representation of visual images and the reconstructions, we use a state-of-the-art method (deep image reconstruction [3]) to reconstruct visual images from fMRI activity measured while subjects attend to one of two images superimposed with equally weighted contrasts. Deep image reconstruction exploits the hierarchical correspondence between the brain and a deep neural network (DNN) to translate (decode) brain activity into DNN features of multiple layers, and then create images that are consistent with the decoded DNN features [3, 22, 23]. Using the deep image reconstruction model trained on fMRI responses to single natural images, we decode brain activity during the attention trials. Behavioral evaluations show that the reconstructions resemble the attended rather than the unattended images. The reconstructions can be modeled by superimposed images with contrasts biased to the attended one, which are comparable to the appearance of the stimuli under attention measured in a separate session. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain–DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses and modulate neural representations to render reconstructions in accordance with subjective appearance. The reconstructions appear to reflect the content of visual experience and volitional control, opening a new possibility of brain-based communication and creation.

scholarly journals Attention modulates neural representation to render reconstructions according to subjective appearance

2022 ◽  
Vol 5 (1) ◽  
Author(s):  
Tomoyasu Horikawa ◽  
Yukiyasu Kamitani
Keyword(s):  
Deep Neural Network ◽  
State Of The Art ◽  
Brain Activity ◽  
Visual Representations ◽  
Neural Representation ◽  
Induced Responses ◽  
Top Down ◽  
Neural Representations ◽  
Visual Cortical ◽  
The Brain

AbstractStimulus images can be reconstructed from visual cortical activity. However, our perception of stimuli is shaped by both stimulus-induced and top-down processes, and it is unclear whether and how reconstructions reflect top-down aspects of perception. Here, we investigate the effect of attention on reconstructions using fMRI activity measured while subjects attend to one of two superimposed images. A state-of-the-art method is used for image reconstruction, in which brain activity is translated (decoded) to deep neural network (DNN) features of hierarchical layers then to an image. Reconstructions resemble the attended rather than unattended images. They can be modeled by superimposed images with biased contrasts, comparable to the appearance during attention. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain–DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses, modulating neural representations to render reconstructions in accordance with subjective appearance.

scholarly journals Insights into Early Word Comprehension: Tracking the Neural Representations of Word Semantics in Infants.

2021 ◽  
Author(s):  
Rohan Saha ◽  
Jennifer Campbell ◽  
Janet F. Werker ◽  
Alona Fyshe
Keyword(s):  
Brain Activity ◽  
Word Meaning ◽  
Age Groups ◽  
Event Related Potential ◽  
Neural Representation ◽  
Machine Learning Techniques ◽  
Semantic Representations ◽  
Word Comprehension ◽  
Neural Representations ◽  
The Brain

Infants start developing rudimentary language skills and can start understanding simple words well before their first birthday. This development has also been shown primarily using Event Related Potential (ERP) techniques to find evidence of word comprehension in the infant brain. While these works validate the presence of semantic representations of words (word meaning) in infants, they do not tell us about the mental processes involved in the manifestation of these semantic representations or the content of the representations. To this end, we use a decoding approach where we employ machine learning techniques on Electroencephalography (EEG) data to predict the semantic representations of words found in the brain activity of infants. We perform multiple analyses to explore word semantic representations in two groups of infants (9-month-old and 12-month-old). Our analyses show significantly above chance decodability of overall word semantics, word animacy, and word phonetics. As we analyze brain activity, we observe that participants in both age groups show signs of word comprehension immediately after word onset, marked by our model's significantly above chance word prediction accuracy. We also observed strong neural representations of word phonetics in the brain data for both age groups, some likely correlated to word decoding accuracy and others not. Lastly, we discover that the neural representations of word semantics are similar in both infant age groups. Our results on word semantics, phonetics, and animacy decodability, give us insights into the evolution of neural representation of word meaning in infants.

scholarly journals Spontaneous perception: a framework for task-free, self-paced perception

2021 ◽  
Vol 2021 (2) ◽  
Author(s):  
Shira Baror ◽  
Biyu J He
Keyword(s):  
Time Course ◽  
Visual Experience ◽  
Brain Activity ◽  
Human Perception ◽  
Future Research ◽  
Task Constraints ◽  
Spontaneous Brain Activity ◽  
The Brain ◽  
Coarse To Fine ◽  
Organizing Principles

Abstract Flipping through social media feeds, viewing exhibitions in a museum, or walking through the botanical gardens, people consistently choose to engage with and disengage from visual content. Yet, in most laboratory settings, the visual stimuli, their presentation duration, and the task at hand are all controlled by the researcher. Such settings largely overlook the spontaneous nature of human visual experience, in which perception takes place independently from specific task constraints and its time course is determined by the observer as a self-governing agent. Currently, much remains unknown about how spontaneous perceptual experiences unfold in the brain. Are all perceptual categories extracted during spontaneous perception? Does spontaneous perception inherently involve volition? Is spontaneous perception segmented into discrete episodes? How do different neural networks interact over time during spontaneous perception? These questions are imperative to understand our conscious visual experience in daily life. In this article we propose a framework for spontaneous perception. We first define spontaneous perception as a task-free and self-paced experience. We propose that spontaneous perception is guided by four organizing principles that grant it temporal and spatial structures. These principles include coarse-to-fine processing, continuity and segmentation, agency and volition, and associative processing. We provide key suggestions illustrating how these principles may interact with one another in guiding the multifaceted experience of spontaneous perception. We point to testable predictions derived from this framework, including (but not limited to) the roles of the default-mode network and slow cortical potentials in underlying spontaneous perception. We conclude by suggesting several outstanding questions for future research, extending the relevance of this framework to consciousness and spontaneous brain activity. In conclusion, the spontaneous perception framework proposed herein integrates components in human perception and cognition, which have been traditionally studied in isolation, and opens the door to understand how visual perception unfolds in its most natural context.

scholarly journals End-to-End Deep Image Reconstruction From Human Brain Activity

2019 ◽  
Vol 13 ◽  
Author(s):  
Guohua Shen ◽  
Kshitij Dwivedi ◽  
Kei Majima ◽  
Tomoyasu Horikawa ◽  
Yukiyasu Kamitani
Keyword(s):  
Image Reconstruction ◽  
Human Brain ◽  
Brain Activity ◽  
Deep Image ◽  
End To End

scholarly journals Visual image reconstruction from human brain activity: A modular decoding approach

2009 ◽  
Vol 197 ◽  
pp. 012021 ◽  
Author(s):  
Yoichi Miyawaki ◽  
Hajime Uchida ◽  
Okito Yamashita ◽  
Masa-aki Sato ◽  
Yusuke Morito ◽  
...  
Keyword(s):  
Image Reconstruction ◽  
Human Brain ◽  
Brain Activity ◽  
Visual Image

Methods of Predicting the Brain Activity Based on Noun

2013 ◽  
Vol 347-350 ◽  
pp. 2516-2520
Author(s):  
Jian Hua Jiang ◽  
Xu Yu ◽  
Zhi Xing Huang
Keyword(s):  
Brain Activity ◽  
Past Research ◽  
Neural Representation ◽  
Fmri Data ◽  
Semantic Features ◽  
Functional Magnetic Resonance ◽  
Resonance Imaging ◽  
The Past ◽  
Machine Learning Methods ◽  
The Brain

Over the last decade, functional magnetic resonance imaging (fMRI) has become a primary tool to predict the brain activity.During the past research, researchers transfer the focus from the picture to the word.The results of these researches are relatively successful. In this paper, several typical methods which are machine learning methods are introduced. And most of the methods are by using fMRI data associated with words features. The semantic features (properties or factors) support words neural representation, and have a certain commonality in the people.The purpose of the application of these methods is used for prediction or classification.

scholarly journals The dimensionality of neural representations for control

2020 ◽  
Author(s):  
David Badre ◽  
Apoorva Bhandari ◽  
Haley Keglovits ◽  
Atsushi Kikumoto
Keyword(s):  
Prefrontal Cortex ◽  
Cognitive Control ◽  
Neural Computation ◽  
Neural Representation ◽  
Future Research ◽  
Trade Off ◽  
Neural Representations ◽  
Open Questions ◽  
The Brain ◽  
Review Current

Cognitive control allows us to think and behave flexibly based on our context and goals. At the heart of theories of cognitive control is a control representation that enables the same input to produce different outputs contingent on contextual factors. In this review, we focus on an important property of the control representation’s neural code: its representational dimensionality. Dimensionality of a neural representation balances a basic separability/generalizability trade-off in neural computation. We will discuss the implications of this trade-off for cognitive control. We will then briefly review current neuroscience findings regarding the dimensionality of control representations in the brain, particularly the prefrontal cortex. We conclude by highlighting open questions and crucial directions for future research.

scholarly journals Computational mechanisms for neural representation of words

2021 ◽  
Author(s):  
Ze Fu ◽  
Xiaosha Wang ◽  
Xiaoying Wang ◽  
Huichao Yang ◽  
Jiahuan Wang ◽  
...  
Keyword(s):  
Neural Network ◽  
Brain Activity ◽  
Inferior Frontal Gyrus ◽  
Activity Patterns ◽  
Explanatory Power ◽  
Word Meaning ◽  
Brain Regions ◽  
Neural Representation ◽  
Space Representation ◽  
Neural Representations

A critical way for humans to acquire, represent and communicate information is through language, yet the underlying computation mechanisms through which language contributes to our word meaning representations are poorly understood. We compared three major types of word computation mechanisms from large language corpus (simple co-occurrence, graph-space relations and neural-network-vector-embedding relations) in terms of the association of words’ brain activity patterns, measured by two functional magnetic resonance imaging (fMRI) experiments. Word relations derived from a graph-space representation, and not neural-network-vector-embedding, had unique explanatory power for the neural activity patterns in brain regions that have been shown to be particularly sensitive to language processes, including the anterior temporal lobe (capturing graph-common-neighbors), inferior frontal gyrus, and posterior middle/inferior temporal gyrus (capturing graph-shortest-path). These results were robust across different window sizes and graph sizes and were relatively specific to language inputs. These findings highlight the role of cumulative language inputs in organizing word meaning neural representations and provide a mathematical model to explain how different brain regions capture different types of language-derived information.

scholarly journals Decoding the Brain: Neural Representation and the Limits of Multivariate Pattern Analysis in Cognitive Neuroscience

2017 ◽  
Author(s):  
J. Brendan Ritchie ◽  
David Michael Kaplan ◽  
Colin Klein
Keyword(s):  
Cognitive Neuroscience ◽  
Strong Evidence ◽  
Neural Activity ◽  
Pattern Analysis ◽  
Neural Representation ◽  
Neural Decoding ◽  
Multivariate Pattern Analysis ◽  
Neural Representations ◽  
Multivariate Pattern ◽  
The Brain

AbstractSince its introduction, multivariate pattern analysis (MVPA), or “neural decoding”, has transformed the field of cognitive neuroscience. Underlying its influence is a crucial inference, which we call the Decoder’s Dictum: if information can be decoded from patterns of neural activity, then this provides strong evidence about what information those patterns represent. Although the Dictum is a widely held and well-motivated principle in decoding research, it has received scant philosophical attention. We critically evaluate the Dictum, arguing that it is false: decodability is a poor guide for revealing the content of neural representations. However, we also suggest how the Dictum can be improved on, in order to better justify inferences about neural representation using MVPA.

scholarly journals Inter-individual deep image reconstruction

2022 ◽  
Author(s):  
Jun Kai Ho ◽  
Tomoyasu Horikawa ◽  
Kei Majima ◽  
Yukiyasu Kamitani
Keyword(s):  
Image Reconstruction ◽  
Brain Activity ◽  
Activity Patterns ◽  
Visual Image ◽  
Training Data ◽  
Visual Features ◽  
Fine Grained ◽  
Visual Areas ◽  
Brain Activity Pattern ◽  
Visual Cortical

The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. While anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual contents. In this study, we evaluated machine learning models called neural code converters that predict one's brain activity pattern (target) from another's (source) given the same stimulus by the decoding of hierarchical visual features and the reconstruction of perceived images. The training data for converters consisted of fMRI data obtained with identical sets of natural images presented to pairs of individuals. Converters were trained using the whole visual cortical voxels from V1 through the ventral object areas, without explicit labels of visual areas. We decoded the converted brain activity patterns into hierarchical visual features of a deep neural network (DNN) using decoders pre-trained on the target brain and then reconstructed images via the decoded features. Without explicit information about visual cortical hierarchy, the converters automatically learned the correspondence between the visual areas of the same levels. DNN feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The viewed images were faithfully reconstructed with recognizable silhouettes of objects even with relatively small amounts of data for converter training. The conversion also allows pooling data across multiple individuals, leading to stably high reconstruction accuracy compared to those converted between individuals. These results demonstrate that the conversion learns hierarchical correspondence and preserves the fine-grained representations of visual features, enabling visual image reconstruction using decoders trained on other individuals.

Sign in / Sign up

Export Citation Format

Share Document