Activations of Deep Convolutional Neural Network are Aligned with Gamma Band Activity of Human Visual Cortex

Previous work demonstrated a direct correspondence between the hierarchy of the human visual areas and layers of deep convolutional neural networks (DCNN) trained on visual object recognition. We used DCNNs to investigate which frequency bands correlate with feature transformations of increasing complexity along the ventral visual pathway. By capitalizing on intracranial depth recordings from 100 patients and 11293 electrodes we assessed the alignment between the DCNN and signals at different frequency bands in different time windows. We found that gamma activity, especially in the low gamma-band (30 – 70 Hz), matched the increasing complexity of visual feature representations in the DCNN. These findings show that the activity of the DCNN captures the essential characteristics of biological object recognition not only in space and time, but also in the frequency domain. These results also demonstrate the potential that modern artificial intelligence algorithms have in advancing our understanding of the brain.Significance StatementRecent advances in the field of artificial intelligence have revealed principles about neural processing, in particular about vision. Previous works have demonstrated a direct correspondence between the hierarchy of human visual areas and layers of deep convolutional neural networks (DCNNs), suggesting that DCNN is a good model of visual object recognition in primate brain. Studying intracranial depth recordings allowed us to extend previous works by assessing when and at which frequency bands the activity of the visual system corresponds to the DCNN. Our key finding is that signals in gamma frequencies along the ventral visual pathway are aligned with the layers of DCNN. Gamma frequencies play a major role in transforming visual input to coherent object representations.

Download Full-text

Comparing Object Recognition in Humans and Deep Convolutional Neural Networks—An Eye Tracking Study

Frontiers in Neuroscience ◽

10.3389/fnins.2021.750639 ◽

2021 ◽

Vol 15 ◽

Author(s):

Leonard Elia van Dyck ◽

Roland Kwitt ◽

Sebastian Jochen Denzler ◽

Walter Roland Gruber

Keyword(s):

Neural Networks ◽

Object Recognition ◽

Eye Tracking ◽

Convolutional Neural Networks ◽

Visual Pathway ◽

Proof Of Concept ◽

Deep Convolutional Neural Networks ◽

Saliency Maps ◽

Ventral Visual Pathway ◽

Specific Factors

Deep convolutional neural networks (DCNNs) and the ventral visual pathway share vast architectural and functional similarities in visual challenges such as object recognition. Recent insights have demonstrated that both hierarchical cascades can be compared in terms of both exerted behavior and underlying activation. However, these approaches ignore key differences in spatial priorities of information processing. In this proof-of-concept study, we demonstrate a comparison of human observers (N = 45) and three feedforward DCNNs through eye tracking and saliency maps. The results reveal fundamentally different resolutions in both visualization methods that need to be considered for an insightful comparison. Moreover, we provide evidence that a DCNN with biologically plausible receptive field sizes called vNet reveals higher agreement with human viewing behavior as contrasted with a standard ResNet architecture. We find that image-specific factors such as category, animacy, arousal, and valence have a direct link to the agreement of spatial object recognition priorities in humans and DCNNs, while other measures such as difficulty and general image properties do not. With this approach, we try to open up new perspectives at the intersection of biological and computer vision research.

Download Full-text

Temporal Limitations in Object Processing Across the Human Ventral Visual Pathway

Journal of Neurophysiology ◽

10.1152/jn.00568.2006 ◽

2007 ◽

Vol 98 (1) ◽

pp. 382-393 ◽

Cited By ~ 35

Author(s):

Thomas J. McKeeff ◽

David A. Remus ◽

Frank Tong

Keyword(s):

Object Recognition ◽

Temporal Processing ◽

Visual Processing ◽

Temporal Frequency ◽

Visual Pathway ◽

Processing Capacity ◽

Complex Objects ◽

Visual Areas ◽

Ventral Visual Pathway ◽

High Level

Behavioral studies have shown that object recognition becomes severely impaired at fast presentation rates, indicating a limitation in temporal processing capacity. Here, we studied whether this behavioral limit in object recognition reflects limitations in the temporal processing capacity of early visual areas tuned to basic features or high-level areas tuned to complex objects. We used functional MRI (fMRI) to measure the temporal processing capacity of multiple areas along the ventral visual pathway progressing from the primary visual cortex (V1) to high-level object-selective regions, specifically the fusiform face area (FFA) and parahippocampal place area (PPA). Subjects viewed successive images of faces or houses at presentation rates varying from 2.3 to 37.5 items/s while performing an object discrimination task. Measures of the temporal frequency response profile of each visual area revealed a systematic decline in peak tuning across the visual hierarchy. Areas V1–V3 showed peak activity at rapid presentation rates of 18–25 items/s, area V4v peaked at intermediate rates (9 items/s), and the FFA and PPA peaked at the slowest temporal rates (4–5 items/s). Our results reveal a progressive loss in the temporal processing capacity of the human visual system as information is transferred from early visual areas to higher areas. These data suggest that temporal limitations in object recognition likely result from the limited processing capacity of high-level object-selective areas rather than that of earlier stages of visual processing.

Download Full-text