scholarly journals Similarity judgments and cortical visual responses reflect different properties of object and scene categories in naturalistic images

2018 ◽  
Author(s):  
Marcie L. King ◽  
Iris I. A. Groen ◽  
Adam Steel ◽  
Dwight J. Kravitz ◽  
Chris I. Baker
Keyword(s):  
Neural Network ◽  
Visual Cortex ◽  
Real World ◽  
Deep Neural Network ◽  
Visual Features ◽  
Visual Responses ◽  
Good Correspondence ◽  
Human Visual Cortex ◽  
Similarity Judgments ◽  
High Level

AbstractNumerous factors have been reported to underlie the representation of complex images in high-level human visual cortex, including categories (e.g. faces, objects, scenes), animacy, and real-world size, but the extent to which this organization is reflected in behavioral judgments of real-world stimuli is unclear. Here, we compared representations derived from explicit similarity judgments and ultra-high field (7T) fMRI of human visual cortex for multiple exemplars of a diverse set of naturalistic images from 48 object and scene categories. Behavioral judgements revealed a coarse division between man-made (including humans) and natural (including animals) images, with clear groupings of conceptually-related categories (e.g. transportation, animals), while these conceptual groupings were largely absent in the fMRI representations. Instead, fMRI responses tended to reflect a separation of both human and non-human faces/bodies from all other categories. This pattern yielded a statistically significant, but surprisingly limited correlation between the two representational spaces. Further, comparison of the behavioral and fMRI representational spaces with those derived from the layers of a deep neural network (DNN) showed a strong correspondence with behavior in the top-most layer and with fMRI in the mid-level layers. These results suggest that there is no simple mapping between responses in high-level visual cortex and behavior – each domain reflects different visual properties of the images and responses in high-level visual cortex may correspond to intermediate stages of processing between basic visual features and the conceptual categories that dominate the behavioral response.Significance StatementIt is commonly assumed there is a correspondence between behavioral judgments of complex visual stimuli and the response of high-level visual cortex. We directly compared these representations across a diverse set of naturalistic object and scene categories and found a surprisingly and strikingly different representational structure. Further, both types of representation showed good correspondence with a deep neural network, but each correlated most strongly with different layers. These results show that behavioral judgments reflect more conceptual properties and visual cortical fMRI responses capture more general visual features. Collectively, our findings highlight that great care must be taken in mapping the response of visual cortex onto behavior, which clearly reflect different information.

scholarly journals Computational mechanisms underlying cortical responses to the affordance properties of visual scenes

2017 ◽  
Author(s):  
Michael F. Bonner ◽  
Russell A. Epstein
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Visual Cortex ◽  
Deep Neural Network ◽  
Deep Neural Networks ◽  
Cortical Responses ◽  
Internal Operations ◽  
High Level ◽  
The Brain ◽  
Sensory Functions

ABSTRACTBiologically inspired deep convolutional neural networks (CNNs), trained for computer vision tasks, have been found to predict cortical responses with remarkable accuracy. However, the complex internal operations of these models remain poorly understood, and the factors that account for their success are unknown. Here we developed a set of techniques for using CNNs to gain insights into the computational mechanisms underlying cortical responses. We focused on responses in the occipital place area (OPA), a scene-selective region of dorsal occipitoparietal cortex. In a previous study, we showed that fMRI activation patterns in the OPA contain information about the navigational affordances of scenes: that is, information about where one can and cannot move within the immediate environment. We hypothesized that this affordance information could be extracted using a set of purely feedforward computations. To test this idea, we examined a deep CNN with a feedforward architecture that had been previously trained for scene classification. We found that the CNN was highly predictive of OPA representations, and, importantly, that it accounted for the portion of OPA variance that reflected the navigational affordances of scenes. The CNN could thus serve as an image-computable candidate model of affordance-related responses in the OPA. We then ran a series of in silico experiments on this model to gain insights into its internal computations. These analyses showed that the computation of affordance-related features relied heavily on visual information at high-spatial frequencies and cardinal orientations, both of which have previously been identified as low-level stimulus preferences of scene-selective visual cortex. These computations also exhibited a strong preference for information in the lower visual field, which is consistent with known retinotopic biases in the OPA. Visualizations of feature selectivity within the CNN suggested that affordance-based responses encoded features that define the layout of the spatial environment, such as boundary-defining junctions and large extended surfaces. Together, these results map the sensory functions of the OPA onto a fully quantitative model that provides insights into its visual computations. More broadly, they advance integrative techniques for understanding visual cortex across multiple level of analysis: from the identification of cortical sensory functions to the modeling of their underlying algorithmic implementations.AUTHOR SUMMARYHow does visual cortex compute behaviorally relevant properties of the local environment from sensory inputs? For decades, computational models have been able to explain only the earliest stages of biological vision, but recent advances in the engineering of deep neural networks have yielded a breakthrough in the modeling of high-level visual cortex. However, these models are not explicitly designed for testing neurobiological theories, and, like the brain itself, their complex internal operations remain poorly understood. Here we examined a deep neural network for insights into the cortical representation of the navigational affordances of visual scenes. In doing so, we developed a set of high-throughput techniques and statistical tools that are broadly useful for relating the internal operations of neural networks with the information processes of the brain. Our findings demonstrate that a deep neural network with purely feedforward computations can account for the processing of navigational layout in high-level visual cortex. We next performed a series of experiments and visualization analyses on this neural network, which characterized a set of stimulus input features that may be critical for computing navigationally related cortical representations and identified a set of high-level, complex scene features that may serve as a basis set for the cortical coding of navigational layout. These findings suggest a computational mechanism through which high-level visual cortex might encode the spatial structure of the local navigational environment, and they demonstrate an experimental approach for leveraging the power of deep neural networks to understand the visual computations of the brain.

scholarly journals Contour junctions underlie neural representations of scene categories in high-level human visual cortex

2016 ◽  
Author(s):  
Heeyoung Choo ◽  
Dirk B Walther
Keyword(s):  
Visual Cortex ◽  
Real World ◽  
Brain Regions ◽  
Neural Representation ◽  
Contour Length ◽  
Behavioral Experiments ◽  
Line Drawings ◽  
Human Visual Cortex ◽  
Neural Representations ◽  
High Level

Humans efficiently grasp complex visual environments, making highly consistent judgments of entry-level category despite their high variability in visual appearance. How does the human brain arrive at the invariant neural representations underlying categorization of real-world environments? We here show that the neural representation of visual environments in scenes-selective human visual cortex relies on statistics of contour junctions, which provide cues for the three-dimensional arrangement of surfaces in a scene. We manipulated line drawings of real-world environments such that statistics of contour orientations or junctions were disrupted. Manipulated and intact line drawings were presented to participants in an fMRI experiment. Scene categories were decoded from neural activity patterns in the parahippocampal place area (PPA), the occipital place area (OPA) and other visual brain regions. Disruption of junctions but not orientations led to a drastic decrease in decoding accuracy in the PPA and OPA, indicating the reliance of these areas on intact junction statistics. Accuracy of decoding from early visual cortex, on the other hand, was unaffected by either image manipulation. We further show that the correlation of error patterns between decoding from the scene-selective brain areas and behavioral experiments is contingent on intact contour junctions. Finally, a searchlight analysis exposes the reliance of visually active brain regions on different sets of contour properties. Statistics of contour length and curvature dominate neural representations of scene categories in early visual areas and contour junctions in high-level scene-selective brain regions.

scholarly journals The Effect of High-Level Illumination on Brain Activity in the Human Visual Cortex

2014 ◽  
Vol 98 (2) ◽  
pp. 87-91
Author(s):  
Yasuhiro Kawashima ◽  
Hiroyuki Yamashiro ◽  
Hiroki Yamamoto ◽  
Tomokazu Murase ◽  
Yoshikatsu Ichimura ◽  
...  
Keyword(s):  
Visual Cortex ◽  
Brain Activity ◽  
Human Visual Cortex ◽  
High Level

Deep neural network to extract high-level features and labels in multi-label classification problems

2020 ◽  
Vol 413 ◽  
pp. 259-270
Author(s):  
Marilyn Bello ◽  
Gonzalo Nápoles ◽  
Ricardo Sánchez ◽  
Rafael Bello ◽  
Koen Vanhoof
Keyword(s):  
Neural Network ◽  
Deep Neural Network ◽  
Classification Problems ◽  
High Level

scholarly journals Feature-Based Interpretation of the Deep Neural Network

Electronics ◽  
2021 ◽  
Vol 10 (21) ◽  
pp. 2687
Author(s):  
Eun-Hun Lee ◽  
Hyeoncheol Kim
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Neural Network ◽  
Lower Layer ◽  
Network Models ◽  
Neural Network Models ◽  
Network Pruning ◽  
The Neural Network ◽  
Single Instance ◽  
High Level

The significant advantage of deep neural networks is that the upper layer can capture the high-level features of data based on the information acquired from the lower layer by stacking layers deeply. Since it is challenging to interpret what knowledge the neural network has learned, various studies for explaining neural networks have emerged to overcome this problem. However, these studies generate the local explanation of a single instance rather than providing a generalized global interpretation of the neural network model itself. To overcome such drawbacks of the previous approaches, we propose the global interpretation method for the deep neural network through features of the model. We first analyzed the relationship between the input and hidden layers to represent the high-level features of the model, then interpreted the decision-making process of neural networks through high-level features. In addition, we applied network pruning techniques to make concise explanations and analyzed the effect of layer complexity on interpretability. We present experiments on the proposed approach using three different datasets and show that our approach could generate global explanations on deep neural network models with high accuracy and fidelity.

scholarly journals Layer-specific modulation of visual responses in human visual cortex by emotional faces

2020 ◽  
Vol 20 (11) ◽  
pp. 587
Author(s):  
Tina Liu ◽  
Jason Fu ◽  
Shruti Japee ◽  
Yuhui Chai ◽  
Leslie Ungerleider ◽  
...  
Keyword(s):  
Visual Cortex ◽  
Visual Responses ◽  
Human Visual Cortex ◽  
Emotional Faces

scholarly journals Evolving super stimuli for real neurons using deep generative networks

2019 ◽  
Author(s):  
Carlos R. Ponce ◽  
Will Xiao ◽  
Peter F. Schade ◽  
Till S. Hartmann ◽  
Gabriel Kreiman ◽  
...  
Keyword(s):  
Neural Network ◽  
Genetic Algorithm ◽  
Visual Cortex ◽  
Neural Coding ◽  
Deep Neural Network ◽  
Natural Images ◽  
List Type ◽  
Neuronal Responses ◽  
Diagnostic Features ◽  
Neural Selectivity

AbstractFinding the best stimulus for a neuron is challenging because it is impossible to test all possible stimuli. Here we used a vast, unbiased, and diverse hypothesis space encoded by a generative deep neural network model to investigate neuronal selectivity in inferotemporal cortex without making any assumptions about natural features or categories. A genetic algorithm, guided by neuronal responses, searched this space for optimal stimuli. Evolved synthetic images evoked higher firing rates than even the best natural images and revealed diagnostic features, independently of category or feature selection. This approach provides a way to investigate neural selectivity in any modality that can be represented by a neural network and challenges our understanding of neural coding in visual cortex.HighlightsA generative deep neural network interacted with a genetic algorithm to evolve stimuli that maximized the firing of neurons in alert macaque inferotemporal and primary visual cortex.The evolved images activated neurons more strongly than did thousands of natural images.Distance in image space from the evolved images predicted responses of neurons to novel images.

Sign in / Sign up

Export Citation Format

Share Document