Modeling movie-evoked human brain activity using motion-energy and space-time vision transformer features

Mapping Intimacies ◽

10.1101/2021.08.22.457251 ◽

2021 ◽

Author(s):

Shinji Nishimoto

Keyword(s):

Human Brain ◽

Model Building ◽

Brain Activity ◽

Space Time ◽

Visual Features ◽

Hierarchical Representation ◽

Motion Energy ◽

Video Viewing ◽

Visual Areas ◽

The Brain

SummaryIn this paper, the process of building a model for predicting human brain activity under video viewing conditions was described as a part of an entry into the Algonauts Project 2021 Challenge. The model was designed to predict brain activity measured using functional MRI (fMRI) by weighted linear summations of the spatiotemporal visual features that appear in the video stimuli (video features). Two types of video features were used: (1) motion-energy features designed based on neurophysiological findings, and (2) features derived from a space-time vision transformer (TimeSformer). To utilize the features of various video domains, the features of the TimeSformer models pre-trained using several different movie sets were combined. Through these model building and validation processes, results showed that there is a certain correspondence between the hierarchical representation of the TimeSformer model and the hierarchical representation of the visual system in the brain. The motion-energy features are effective in predicting brain activity in the early visual areas, while TimeSformer-derived features are effective in higher-order visual areas, and a hybrid model that uses motion energy and TimeSformer features is effective for predicting whole brain activity.

Download Full-text

Brain hierarchy score: Which deep neural networks are hierarchically brain-like?

10.1101/2020.07.22.216713 ◽

2020 ◽

Author(s):

Soma Nonaka ◽

Kei Majima ◽

Shuntaro C. Aoki ◽

Yukiyasu Kamitani

Keyword(s):

Neural Networks ◽

Image Recognition ◽

High Performance ◽

Deep Neural Networks ◽

Recognition Performance ◽

Brain Activity ◽

Spatial Integration ◽

Hierarchical Processing ◽

Visual Areas ◽

The Brain

SummaryAchievement of human-level image recognition by deep neural networks (DNNs) has spurred interest in whether and how DNNs are brain-like. Both DNNs and the visual cortex perform hierarchical processing, and correspondence has been shown between hierarchical visual areas and DNN layers in representing visual features. Here, we propose the brain hierarchy (BH) score as a metric to quantify the degree of hierarchical correspondence based on the decoding of individual DNN unit activations from human brain activity. We find that BH scores for 29 pretrained DNNs with varying architectures are negatively correlated with image recognition performance, indicating that recently developed high-performance DNNs are not necessarily brain-like. Experimental manipulations of DNN models suggest that relatively simple feedforward architecture with broad spatial integration is critical to brain-like hierarchy. Our method provides new ways for designing DNNs and understanding the brain in consideration of their representational homology.

Download Full-text

Estimation of the Directions of Alpha Rhythm Elementary Sources Using the Method of Human Brain Functional Tomography Based On the Magnetic Encephalography Data

Математическая биология и биоинформатика ◽

10.17537/2018.13.426 ◽

2018 ◽

Vol 13 (2) ◽

pp. 426-436 ◽

Cited By ~ 3

Author(s):

M.N. Ustinin ◽

S.D. Rykunov ◽

A.I. Boyko ◽

O.A. Maslova ◽

K.D. Walton ◽

...

Keyword(s):

Inverse Problem ◽

Time Series ◽

Human Brain ◽

Alpha Rhythm ◽

Brain Activity ◽

Problem Solution ◽

New York University ◽

Magnetic Encephalography ◽

The Brain ◽

Calculation Grid

New method for the magnetic encephalography data analysis was proposed. The method transforms multichannel time series into the spatial structure of the human brain activity. In this paper we further develop this method to determine the dominant direction of the electrical sources of brain activity at each node of the calculation grid. We have considered the experimental data, obtained with three 275-channel magnetic encephalographs in New York University, McGill University and Montreal University. The human alpha rhythm phenomenon was selected as a model object. Magnetic encephalograms of the brain spontaneous activity were registered for 5-7 minutes in magnetically shielded room. Detailed multichannel spectra were obtained by the Fourier transform of the whole time series. For all spectral components, the inverse problem was solved in elementary current dipole model and the functional structure of the brain activity was calculated in the frequency band 8-12 Hz. In order to estimate the local activity direction, at the each node of calculation grid the vector of the inverse problem solution was selected, having the maximal spectral power. So, the 3D-map of the brain activity vector field was produced – the directional functional tomogram. Such maps were generated for 15 subjects and some common patterns were revealed in the directions of the alpha rhythm elementary sources. The proposed method can be used to study the local properties of the brain activity in any spectral band and in any brain compartment.

Download Full-text

Sleeping While Awake: The Intrusion of Neural Activity Associated with Sleep Onset in the Awake Human Brain

10.1101/2020.06.04.133603 ◽

2020 ◽

Cited By ~ 2

Author(s):

Stephanie Hawes ◽

Carrie R. H. Innes ◽

Nicholas Parsons ◽

Sean P.A. Drummond ◽

Karen Caeyensberghs ◽

...

Keyword(s):

Human Brain ◽

Basal Forebrain ◽

Brain Activity ◽

Cognitive Task ◽

Sleep Onset ◽

Theoretical Modelling ◽

Related Activity ◽

Independent Dataset ◽

Time Graph ◽

The Brain

AbstractSleep can intrude into the awake human brain when sleep deprived or fatigued, even while performing cognitive tasks. However, how the brain activity associated with sleep onset can co-exist with the activity associated with cognition in the awake humans remains unexplored. Here, we used simultaneous fMRI and EEG to generate fMRI activity maps associated with EEG theta (4-7 Hz) activity associated with sleep onset. We implemented a method to track these fMRI activity maps in individuals performing a cognitive task after well-rested and sleep-deprived nights. We found frequent intrusions of the fMRI maps associated with sleep-onset in the task-related fMRI data. These sleep events elicited a pattern of transient fMRI activity, which was spatially distinct from the task-related activity in the frontal and parietal areas of the brain. They were concomitant with reduced arousal as indicated by decreased pupil size and increased response time. Graph theoretical modelling showed that the activity associated with sleep onset emerges from the basal forebrain and spreads anterior-posteriorly via the brain’s structural connectome. We replicated the key findings in an independent dataset, which suggests that the approach can be reliably used in understanding the neuro-behavioural consequences of sleep and circadian disturbances in humans.

Download Full-text

Is Artistic Composition in Abstract Art Detected Automatically?

10.1093/oso/9780197513620.003.0021 ◽

2021 ◽

pp. 102-106

Author(s):

Claudia Menzel ◽

Gyula Kovács ◽

Gregor U. Hayn-Leichsenring ◽

Christoph Redies

Keyword(s):

Human Brain ◽

Significant Role ◽

Mismatch Negativity ◽

Brain Activity ◽

Perceptual Processing ◽

Abstract Art ◽

Electrical Brain Activity ◽

Image Set ◽

The Brain

Most artists who create abstract paintings place the pictorial elements not at random, but arrange them intentionally in a specific artistic composition. This arrangement results in a pattern of image properties that differs from image versions in which the same pictorial elements are randomly shuffled. In the article under discussion, the original abstract paintings of the author’s image set were rated as more ordered and harmonious but less interesting than their shuffled counterparts. The authors tested whether the human brain distinguishes between these original and shuffled images by recording electrical brain activity in a particular paradigm that evokes a so-called visual mismatch negativity. The results revealed that the brain detects the differences between the two types of images fast and automatically. These findings are in line with models that postulate a significant role of early (low-level) perceptual processing of formal image properties in aesthetic evaluations.

Download Full-text

“Grammar box” in the brain

Behavioral and Brain Sciences ◽

10.1017/s0140525x03270151 ◽

2003 ◽

Vol 26 (6) ◽

pp. 672-673

Author(s):

Valéria Csépe

Keyword(s):

Human Brain ◽

Brain Activity ◽

Neural Correlates ◽

Modular Architecture ◽

Activity Data ◽

The Brain ◽

Semantic Domains

Brain activity data prove the existence of qualitatively different structures in the brain. However, the question is whether the human brain acts as linguists assume in their models. The modular architecture of grammar that has been claimed by many linguists raises some empirical questions. One of the main questions is whether the threefold abstract partition of language (into syntactic, phonological, and semantic domains) has distinct neural correlates.

Download Full-text

Is the human brain only responsive?

Behavioral and Brain Sciences ◽

10.1017/s0140525x00038899 ◽

1995 ◽

Vol 18 (2) ◽

pp. 365-366

Author(s):

Rumyana Kristeva-Feige ◽

Bernd Feige

Keyword(s):

Human Brain ◽

Spontaneous Activity ◽

Cognitive Functions ◽

Evoked Potential ◽

Imaging Techniques ◽

Brain Activity ◽

External Events ◽

Potential Methods ◽

The Brain

AbstractPosner & Raichle's (1994) book is a fascinating and readable account of the studies the authors have conducted on the localization of cognitive functions in the brain mainly using PET and EEC evoked potential methods. Our criticism concerns the underrepresentation of some imaging techniques (magnetoencephalography) and some forms of brain activity (spontaneous activity). Furthermore, the book leaves the reader with the impression that the brain only responds to external events.

Download Full-text

Dissociable Neural Representations of Adversarially Perturbed Images in Convolutional Neural Networks and the Human Brain

Frontiers in Neuroinformatics ◽

10.3389/fninf.2021.677925 ◽

2021 ◽

Vol 15 ◽

Author(s):

Chi Zhang ◽

Xiao-Han Duan ◽

Lin-Yuan Wang ◽

Yong-Li Li ◽

Bin Yan ◽

...

Keyword(s):

Neural Networks ◽

Human Brain ◽

Convolutional Neural Networks ◽

Brain Activity ◽

Neural Responses ◽

Neural Representations ◽

Artificial Neurons ◽

Visual Areas ◽

Two Systems ◽

Visual Tasks

Despite the remarkable similarities between convolutional neural networks (CNN) and the human brain, CNNs still fall behind humans in many visual tasks, indicating that there still exist considerable differences between the two systems. Here, we leverage adversarial noise (AN) and adversarial interference (AI) images to quantify the consistency between neural representations and perceptual outcomes in the two systems. Humans can successfully recognize AI images as the same categories as their corresponding regular images but perceive AN images as meaningless noise. In contrast, CNNs can recognize AN images similar as corresponding regular images but classify AI images into wrong categories with surprisingly high confidence. We use functional magnetic resonance imaging to measure brain activity evoked by regular and adversarial images in the human brain, and compare it to the activity of artificial neurons in a prototypical CNN—AlexNet. In the human brain, we find that the representational similarity between regular and adversarial images largely echoes their perceptual similarity in all early visual areas. In AlexNet, however, the neural representations of adversarial images are inconsistent with network outputs in all intermediate processing layers, providing no neural foundations for the similarities at the perceptual level. Furthermore, we show that voxel-encoding models trained on regular images can successfully generalize to the neural responses to AI images but not AN images. These remarkable differences between the human brain and AlexNet in representation-perception association suggest that future CNNs should emulate both behavior and the internal neural presentations of the human brain.

Download Full-text

Searching through functional space reveals distributed visual, auditory, and semantic coding in the human brain

10.1101/2020.04.20.052175 ◽

2020 ◽

Author(s):

Sreejan Kumar ◽

Cameron T. Ellis ◽

Thomas O’Connell ◽

Marvin M Chun ◽

Nicholas B. Turk-Browne

Keyword(s):

Human Brain ◽

Language Processing ◽

Brain Function ◽

Computational Models ◽

Brain Activity ◽

Functional Space ◽

Semantic Features ◽

Brain Functions ◽

Auditory Features ◽

The Brain

AbstractThe extent to which brain functions are localized or distributed is a foundational question in neuroscience. In the human brain, common fMRI methods such as cluster correction, atlas parcellation, and anatomical searchlight are biased by design toward finding localized representations. Here we introduce the functional searchlight approach as an alternative to anatomical searchlight analysis, the most commonly used exploratory multivariate fMRI technique. Functional searchlight removes any anatomical bias by grouping voxels based only on functional similarity and ignoring anatomical proximity. We report evidence that visual and auditory features from deep neural networks and semantic features from a natural language processing model are more widely distributed across the brain than previously acknowledged. This approach provides a new way to evaluate and constrain computational models with brain activity and pushes our understanding of human brain function further along the spectrum from strict modularity toward distributed representation.

Download Full-text

The modular and integrative functional architecture of the human brain

Proceedings of the National Academy of Sciences ◽

10.1073/pnas.1510619112 ◽

2015 ◽

Vol 112 (49) ◽

pp. E6798-E6807 ◽

Cited By ~ 228

Author(s):

Maxwell A. Bertolero ◽

B. T. Thomas Yeo ◽

Mark D’Esposito

Keyword(s):

Human Brain ◽

Cognitive Functions ◽

Topic Model ◽

Brain Activity ◽

Modular Function ◽

Imaging Data ◽

Functional Architecture ◽

Brain Imaging Data ◽

The Brain

Network-based analyses of brain imaging data consistently reveal distinct modules and connector nodes with diverse global connectivity across the modules. How discrete the functions of modules are, how dependent the computational load of each module is to the other modules’ processing, and what the precise role of connector nodes is for between-module communication remains underspecified. Here, we use a network model of the brain derived from resting-state functional MRI (rs-fMRI) data and investigate the modular functional architecture of the human brain by analyzing activity at different types of nodes in the network across 9,208 experiments of 77 cognitive tasks in the BrainMap database. Using an author–topic model of cognitive functions, we find a strong spatial correspondence between the cognitive functions and the network’s modules, suggesting that each module performs a discrete cognitive function. Crucially, activity at local nodes within the modules does not increase in tasks that require more cognitive functions, demonstrating the autonomy of modules’ functions. However, connector nodes do exhibit increased activity when more cognitive functions are engaged in a task. Moreover, connector nodes are located where brain activity is associated with many different cognitive functions. Connector nodes potentially play a role in between-module communication that maintains the modular function of the brain. Together, these findings provide a network account of the brain’s modular yet integrated implementation of cognitive functions.

Download Full-text