Analysing linear multivariate pattern transformations in neuroimaging data

AbstractMost connectivity metrics in neuroimaging research reduce multivariate activity patterns in regions-of-interests (ROIs) to one dimension, which leads to a loss of information. Importantly, it prevents us from investigating the transformations between patterns in different ROIs. Here, we applied linear estimation theory in order to robustly estimate the linear transformations between multivariate fMRI patterns with a cross-validated Tikhonov regularisation approach. We derived three novel metrics that describe different features of these voxel-by-voxel mappings: goodness-of-fit, sparsity and pattern deformation. The goodness-of-fit describes the degree to which the patterns in an input region can be described as a linear transformation of patterns in an output region. The sparsity metric, which relies on a Monte Carlo procedure, was introduced in order to test whether the transformation mostly consists of one-to-one mappings between voxels in different regions. Furthermore, we defined a metric for pattern deformation, i.e. the degree to which the transformation rotates or rescales the input patterns. As a proof of concept, we applied these metrics to an event-related fMRI data set consisting of four subjects that has been used in previous studies. We focused on the transformations from early visual cortex (EVC) to inferior temporal cortex (ITC), fusiform face area (FFA) and parahippocampal place area (PPA). Our results suggest that the estimated linear mappings are able to explain a significant amount of variance of the three output ROIs. The transformation from EVC to ITC shows the highest goodness-of-fit, and those from EVC to FFA and PPA show the expected preference for faces and places as well as animate and inanimate objects, respectively. The pattern transformations are sparse, but sparsity is lower than would have been expected for one-to-one mappings, thus suggesting the presence of one-to-few voxel mappings. ITC, FFA and PPA patterns are not simple rotations of an EVC pattern, indicating that the corresponding transformations amplify or dampen certain dimensions of the input patterns. While our results are only based on a small number of subjects, they show that our pattern transformation metrics can describe novel aspects of multivariate functional connectivity in neuroimaging data.

Download Full-text

Preparatory Activity in Posterior Temporal Cortex Causally Contributes to Object Detection in Scenes

Journal of Cognitive Neuroscience ◽

10.1162/jocn_a_00845 ◽

2015 ◽

Vol 27 (11) ◽

pp. 2117-2125 ◽

Cited By ~ 11

Author(s):

Reshanne R. Reeder ◽

Francesca Perini ◽

Marius V. Peelen

Keyword(s):

Visual Cortex ◽

Object Detection ◽

Activity Patterns ◽

Temporal Cortex ◽

Relevant Information ◽

Visual Selective Attention ◽

Object Categories ◽

Preparatory Attention ◽

Preparatory Activity ◽

Early Visual Cortex

Theories of visual selective attention propose that top–down preparatory attention signals mediate the selection of task-relevant information in cluttered scenes. Neuroimaging and electrophysiology studies have provided correlative evidence for this hypothesis, finding increased activity in target-selective neural populations in visual cortex in the period between a search cue and target onset. In this study, we used online TMS to test whether preparatory neural activity in visual cortex is causally involved in naturalistic object detection. In two experiments, participants detected the presence of object categories (cars, people) in a diverse set of photographs of real-world scenes. TMS was applied over a region in posterior temporal cortex identified by fMRI as carrying category-specific preparatory activity patterns. Results showed that TMS applied over posterior temporal cortex before scene onset (−200 and −100 msec) impaired the detection of object categories in subsequently presented scenes, relative to vertex and early visual cortex stimulation. This effect was specific to category level detection and was related to the type of attentional template participants adopted, with the strongest effects observed in participants adopting category level templates. These results provide evidence for a causal role of preparatory attention in mediating the detection of objects in cluttered daily-life environments.

Download Full-text

Dynamic Population Coding of Category Information in Inferior Temporal and Prefrontal Cortex

Journal of Neurophysiology ◽

10.1152/jn.90248.2008 ◽

2008 ◽

Vol 100 (3) ◽

pp. 1407-1419 ◽

Cited By ~ 211

Author(s):

Ethan M. Meyers ◽

David J. Freedman ◽

Gabriel Kreiman ◽

Earl K. Miller ◽

Tomaso Poggio

Keyword(s):

Prefrontal Cortex ◽

Visual Information ◽

Activity Patterns ◽

Temporal Cortex ◽

Relevant Information ◽

Population Coding ◽

Inferior Temporal Cortex ◽

Category Information ◽

Almost All ◽

Dynamic Population

Most electrophysiology studies analyze the activity of each neuron separately. While such studies have given much insight into properties of the visual system, they have also potentially overlooked important aspects of information coded in changing patterns of activity that are distributed over larger populations of neurons. In this work, we apply a population decoding method to better estimate what information is available in neuronal ensembles and how this information is coded in dynamic patterns of neural activity in data recorded from inferior temporal cortex (ITC) and prefrontal cortex (PFC) as macaque monkeys engaged in a delayed match-to-category task. Analyses of activity patterns in ITC and PFC revealed that both areas contain “abstract” category information (i.e., category information that is not directly correlated with properties of the stimuli); however, in general, PFC has more task-relevant information, and ITC has more detailed visual information. Analyses examining how information coded in these areas show that almost all category information is available in a small fraction of the neurons in the population. Most remarkably, our results also show that category information is coded by a nonstationary pattern of activity that changes over the course of a trial with individual neurons containing information on much shorter time scales than the population as a whole.

Download Full-text

Large-scale hyperparameter search for predicting human brain responses in the Algonauts challenge

10.1101/689844 ◽

2019 ◽

Author(s):

Kamila M. Jozwik ◽

Michael Lee ◽

Tiago Marques ◽

Martin Schrimpf ◽

Pouya Bashivan

Keyword(s):

Large Scale ◽

Visual Stimuli ◽

Temporal Cortex ◽

Image Features ◽

Inferior Temporal Cortex ◽

Network Architectures ◽

Image Preprocessing ◽

Neural Representations ◽

Brain Responses ◽

Early Visual Cortex

Image features computed by specific convolutional artificial neural networks (ANNs) can be used to make state-of-the-art predictions of primate ventral stream responses to visual stimuli.However, in addition to selecting the specific ANN and layer that is used, the modeler makes other choices in preprocessing the stimulus image and generating brain predictions from ANN features. The effect of these choices on brain predictivity is currently underexplored.Here, we directly evaluated many of these choices by performing a grid search over network architectures, layers, image preprocessing strategies, feature pooling mechanisms, and the use of dimensionality reduction. Our goal was to identify model configurations that produce responses to visual stimuli that are most similar to the human neural representations, as measured by human fMRI and MEG responses. In total, we evaluated more than 140,338 model configurations. We found that specific configurations of CORnet-S best predicted fMRI responses in early visual cortex, and CORnet-R and SqueezeNet models best predicted fMRI responses in inferior temporal cortex. We found specific configurations of VGG-16 and CORnet-S models that best predicted the MEG responses.We also observed that downsizing input images to ~50-75% of the input tensor size lead to better performing models compared to no downsizing (the default choice in most brain models for vision). Taken together, we present evidence that brain predictivity is sensitive not only to which ANN architecture and layer is used, but choices in image preprocessing and feature postprocessing, and these choices should be further explored.

Download Full-text

Learning to Recognize Visual Objects With Microstimulation in Inferior Temporal Cortex

Journal of Neurophysiology ◽

10.1152/jn.90247.2008 ◽

2008 ◽

Vol 100 (1) ◽

pp. 197-211 ◽

Cited By ~ 7

Author(s):

Keisuke Kawasaki ◽

David L. Sheinberg

Keyword(s):

Activity Patterns ◽

Visual Learning ◽

Temporal Cortex ◽

Inferior Temporal Cortex ◽

Classification Learning ◽

Critical Question ◽

Direct Stimulation ◽

Visual Classification ◽

Association Learning ◽

New Associations

The malleability of object representations by experience is essential for adaptive behavior. It has been hypothesized that neurons in inferior temporal cortex (IT) in monkeys are pivotal in visual association learning, evidenced by experiments revealing changes in neural selectivity following visual learning, as well as by lesion studies, wherein functional inactivation of IT impairs learning. A critical question remaining to be answered is whether IT neuronal activity is sufficient for learning. To address this question directly, we conducted experiments combining visual classification learning with microstimulation in IT. We assessed the effects of IT microstimulation during learning in cases where the stimulation was exclusively informative, conditionally informative, and informative but not necessary for the classification task. The results show that localized microstimulation in IT can be used to establish visual classification learning, and the same stimulation applied during learning can predictably bias judgments on subsequent recognition. The effect of induced activity can be explained neither by direct stimulation-motor association nor by simple detection of cortical stimulation. We also found that the learning effects are specific to IT stimulation as they are not observed by microstimulation in an adjacent auditory area. Our results add the evidence that the differential activity in IT during visual association learning is sufficient for establishing new associations. The results suggest that experimentally manipulated activity patterns within IT can be effectively combined with ongoing visually induced activity during the formation of new associations.

Download Full-text

Identifying Decision Structures Underlying Activity Patterns: An Exploration of Data Mining Algorithms

Transportation Research Record Journal of the Transportation Research Board ◽

10.3141/1718-01 ◽

2000 ◽

Vol 1718 (1) ◽

pp. 1-9 ◽

Cited By ~ 39

Author(s):

Geert Wets ◽

Koen Vanhoof ◽

Theo Arentze ◽

Harry Timmermans

Keyword(s):

Data Mining ◽

Decision Tree ◽

Logit Model ◽

Goodness Of Fit ◽

Travel Demand ◽

Activity Patterns ◽

Future Research ◽

Data Set ◽

Data Mining Algorithms ◽

Mining Algorithms

The utility-maximizing framework—in particular, the logit model—is the dominantly used framework in transportation demand modeling. Computational process modeling has been introduced as an alternative approach to deal with the complexity of activity-based models of travel demand. Current rule-based systems, however, lack a methodology to derive rules from data. The relevance and performance of data-mining algorithms that potentially can provide the required methodology are explored. In particular, the C4 algorithm is applied to derive a decision tree for transport mode choice in the context of activity scheduling from a large activity diary data set. The algorithm is compared with both an alternative method of inducing decision trees (CHAID) and a logit model on the basis of goodness-of-fit on the same data set. The ratio of correctly predicted cases of a holdout sample is almost identical for the three methods. This suggests that for data sets of comparable complexity, the accuracy of predictions does not provide grounds for either rejecting or choosing the C4 method. However, the method may have advantages related to robustness. Future research is required to determine the ability of decision tree-based models in predicting behavioral change.

Download Full-text

Temporal encoding of two-dimensional patterns by single units in primate primary visual cortex. II. Information transmission

Journal of Neurophysiology ◽

10.1152/jn.1990.64.2.370 ◽

1990 ◽

Vol 64 (2) ◽

pp. 370-380 ◽

Cited By ~ 88

Author(s):

B. J. Richmond ◽

L. M. Optican

Keyword(s):

Cortical Neurons ◽

Activity Patterns ◽

Temporal Cortex ◽

Principal Component ◽

Response Strength ◽

Inferior Temporal Cortex ◽

Temporal Code ◽

Response Code ◽

Stimulus Response ◽

Black And White

1. Previously, we studied how picture information was processed by neurons in inferior temporal cortex. We found that responses varying in both response strength and temporal waveform carried information about briefly flashed stationary black-and-white patterns. Now, we have applied that same paradigm to the study of striate cortical neurons. 2. In this approach the responses to a set of basic black and white pictures were quantified through use of a set of basic waveforms, the principal components (extracted from all the responses of each neuron). We found that the first principal component, which corresponds to the response strength, and others, which correspond to different basic temporal activity patterns, were significantly related to the stimuli, i.e., the stimulus drove both the response strength and its temporal pattern. 3. Our previous study had shown that, when information theory was used to quantify the stimulus-response relation, inferior temporal neurons convey over twice as much information in a response code that includes temporal modulation as in a response code that includes only the response strength. This study shows that striate cortical neurons also carry twice as much information in a temporal code as in a response strength code. Thus single visual neurons at both ends of a cortical processing chain for visual pattern use a multidimensional temporal code to carry stimulus-related information. 4. These results support our multiplex-filter hypothesis, which states that single visual system neurons can be regarded as several simultaneously active parallel channels, each of which conveys independent information about the stimulus.

Download Full-text

fMRI Activity Patterns in Human LOC Carry Information about Object Exemplars within Category

Journal of Cognitive Neuroscience ◽

10.1162/jocn.2008.20019 ◽

2008 ◽

Vol 20 (2) ◽

pp. 356-370 ◽

Cited By ~ 117

Author(s):

Evelyn Eger ◽

John Ashburner ◽

John-Dylan Haynes ◽

Raymond J. Dolan ◽

Geraint Rees

Keyword(s):

Pattern Recognition ◽

Visual Cortex ◽

Activity Patterns ◽

Temporal Cortex ◽

Support Vector ◽

Specific Information ◽

Object Identity ◽

Stimulus Similarity ◽

Lateral Occipital Complex ◽

Early Visual Cortex

The lateral occipital complex (LOC) is a set of areas in the human occipito-temporal cortex responding to objects as opposed to low-level control stimuli. Conventional functional magnetic resonance imaging (fMRI) analysis methods based on regional averages could not detect signals discriminative of different types of objects in this region. Here, we examined fMRI signals using multivariate pattern recognition (support vector classification) to systematically explore the nature of object-related information available in fine-grained activity patterns in the LOC. Distributed fMRI signals from the LOC allowed for above-chance discrimination not only of the category but also of within-category exemplars of everyday man-made objects, and such exemplar-specific information generalized across changes in stimulus size and viewpoint, particularly in posterior subregions. Object identity could also be predicted from responses of the early visual cortex, even significantly across the changes in size and viewpoint used here. However, a dissociation was observed between these two regions of interest in the degree of discrimination for objects relative to size: In the early visual cortex, two different sizes of the same object were even better discriminated than two different objects (in accordance with measures of pixelwise stimulus similarity), whereas the opposite was true in the LOC. These findings provide the first evidence that direct evoked fMRI activity patterns in the LOC can be different for individual object exemplars (within a single category). We propose that pattern recognition methods as used here may provide an alternative approach to study mechanisms of neuronal representation based on aspects of the fMRI response independent of those assessed in adaptation paradigms.

Download Full-text

Category-specific representational patterns in left inferior frontal and temporal cortex reflect similarities and differences in the sensorimotor and distributional properties of concepts

10.1101/2021.09.03.458378 ◽

2021 ◽

Author(s):

Francesca Carota ◽

Nikolaus Kriegeskorte ◽

Hamed Nili ◽

Friedemann Pulvermüller

Keyword(s):

Activity Patterns ◽

Temporal Cortex ◽

Inferior Temporal Cortex ◽

Angular Gyrus ◽

Activation Patterns ◽

Neuronal Populations ◽

Distributional Information ◽

Motor Information ◽

Category Discrimination ◽

Sensorimotor Knowledge

AbstractNeuronal populations code similar concepts by similar activity patterns across the human brain’s networks supporting language comprehension. However, it is unclear to what extent such meaning-to-symbol mapping reflects statistical distributions of symbol meanings in language use, as quantified by word co-occurrence frequencies, or, rather, experiential information thought to be necessary for grounding symbols in sensorimotor knowledge. Here we asked whether integrating distributional semantics with human judgments of grounded sensorimotor semantics better approximates the representational similarity of conceptual categories in the brain, as compared with each of these methods used separately. We examined the similarity structure of activation patterns elicited by action- and object-related concepts using multivariate representational similarity analysis (RSA) of fMRI data. The results suggested that a semantic vector integrating both sensorimotor and distributional information yields best category discrimination on the cognitive-linguistic level, and explains the corresponding activation patterns in left posterior inferior temporal cortex. In turn, semantic vectors based on detailed visual and motor information uncovered category-specific similarity patterns in fusiform and angular gyrus for object-related concepts, and in motor cortex, left inferior frontal cortex (BA 44), and supramarginal gyrus for action-related concepts.

Download Full-text