Spatial and temporal context jointly modulate the sensory response within the ventral visual stream

AbstractBoth spatial and temporal context play an important role in visual perception and behavior. Humans can extract statistical regularities from both forms of context to help processing the present and to construct expectations about the future. Numerous studies have found reduced neural responses to expected stimuli compared to unexpected stimuli, for both spatial and temporal regularities. However, it is largely unclear whether and how these forms of context interact. In the current fMRI study, thirty-three human volunteers were exposed to object stimuli that could be expected or surprising in terms of their spatial and temporal context. We found a reliable independent contribution of both spatial and temporal context in modulating the neural response. Specifically, neural responses to stimuli in expected compared to unexpected contexts were suppressed throughout the ventral visual stream. Interestingly, the modulation by spatial context was stronger in magnitude and more reliable than modulations by temporal context. These results suggest that while both spatial and temporal context serve as a prior that can modulate sensory processing in a similar fashion, predictions of spatial context may be a more powerful modulator in the visual system.Significance StatementBoth temporal and spatial context can affect visual perception, however it is largely unclear if and how these different forms of context interact in modulating sensory processing. When manipulating both temporal and spatial context expectations, we found that they jointly affected sensory processing, evident as a suppression of neural responses for expected compared to unexpected stimuli. Interestingly, the modulation by spatial context was stronger than that by temporal context. Together, our results suggest that spatial context may be a stronger modulator of neural responses than temporal context within the visual system. Thereby, the present study provides new evidence how different types of predictions jointly modulate perceptual processing.

Download Full-text

Spatial and Temporal Context Jointly Modulate the Sensory Response within the Ventral Visual Stream

Journal of Cognitive Neuroscience ◽

10.1162/jocn_a_01792 ◽

2021 ◽

pp. 1-16

Author(s):

Tao He ◽

David Richter ◽

Zhiguo Wang ◽

Floris P. de Lange

Keyword(s):

Sensory Response ◽

Temporal Context ◽

Neural Responses ◽

Visual Stream ◽

Fmri Study ◽

Human Volunteers ◽

Different Types ◽

Statistical Regularities ◽

Ventral Visual Stream ◽

And Behavior

Download Full-text

Orbito-frontal Cortex is Necessary for Temporal Context Memory

Journal of Cognitive Neuroscience ◽

10.1162/jocn.2009.21316 ◽

2010 ◽

Vol 22 (8) ◽

pp. 1819-1831 ◽

Cited By ~ 48

Author(s):

Audrey Duarte ◽

Richard N. Henson ◽

Robert T. Knight ◽

Tina Emery ◽

Kim S. Graham

Keyword(s):

Frontal Cortex ◽

Critical Role ◽

Spatial Context ◽

Temporal Context ◽

Imaging Results ◽

Temporal Aspects ◽

Neuropsychological Investigation ◽

Temporal And Spatial ◽

Context Memory ◽

Encoding And Retrieval

Lesion and neuroimaging studies suggest that orbito-frontal cortex (OFC) supports temporal aspects of episodic memory. However, it is unclear whether OFC contributes to the encoding and/or retrieval of temporal context and whether it is selective for temporal relative to nontemporal (spatial) context memory. We addressed this issue with two complimentary studies: functional magnetic resonance imaging to measure OFC activity associated with successful temporal and spatial context memory during encoding and retrieval in healthy young participants, and a neuropsychological investigation to measure changes in spatial and temporal context memory in OFC lesion patients. Imaging results revealed that OFC contributed to encoding and retrieval of associations between objects and their temporal but not their spatial contexts. Consistent with this, OFC patients exhibited impairments in temporal but not spatial source memory accuracy. These results suggest that OFC plays a critical role in the formation and subsequent retrieval of temporal context.

Download Full-text

Unsupervised Models of Mouse Visual Cortex

10.1101/2021.06.16.448730 ◽

2021 ◽

Author(s):

Aran Nayebi ◽

Nathan C. L. Kong ◽

Chengxu Zhuang ◽

Justin L. Gardner ◽

Anthony M. Norcia ◽

...

Keyword(s):

Visual Cortex ◽

Visual System ◽

Network Architectures ◽

Deep Convolutional Neural Networks ◽

Visual Stream ◽

Computational Theory ◽

Visual Coding ◽

Ventral Visual Stream ◽

Mouse Visual Cortex ◽

Image Transformations

Task-optimized deep convolutional neural networks are the most quantitatively accurate models of the primate ventral visual stream. However, such networks are implausible as a model of the mouse visual system because mouse visual cortex has a known shallower hierarchy and the supervised objectives these networks are typically trained with are likely neither ethologically relevant in content nor in quantity. Here we develop shallow network architectures that are more consistent with anatomical and physiological studies of mouse visual cortex than current models. We demonstrate that hierarchically shallow architectures trained using contrastive objective functions applied to visual-acuity-adapted images achieve neural prediction performance that exceed those of the same architectures trained in a supervised manner and result in the most quantitatively accurate models of the mouse visual system. Moreover, these models' neural predictivity significantly surpasses those of supervised, deep architectures that are known to correspond well to the primate ventral visual stream. Finally, we derive a novel measure of inter-animal consistency, and show that the best models closely match this quantity across visual areas. Taken together, our results suggest that contrastive objectives operating on shallow architectures with ethologically-motivated image transformations may be a biologically-plausible computational theory of visual coding in mice.

Download Full-text

Wiring Up Vision: Minimizing Supervised Synaptic Updates Needed to Produce a Primate Ventral Stream

10.1101/2020.06.08.140111 ◽

2020 ◽

Author(s):

Franziska Geiger ◽

Martin Schrimpf ◽

Tiago Marques ◽

James J. DiCarlo

Keyword(s):

Visual System ◽

Visual Processing ◽

Ventral Stream ◽

Visual Object ◽

Visual Object Recognition ◽

Visual Stream ◽

Ongoing Research ◽

Ventral Visual Stream ◽

And Behavior ◽

The Brain

AbstractAfter training on large datasets, certain deep neural networks are surprisingly good models of the neural mechanisms of adult primate visual object recognition. Nevertheless, these models are poor models of the development of the visual system because they posit millions of sequential, precisely coordinated synaptic updates, each based on a labeled image. While ongoing research is pursuing the use of unsupervised proxies for labels, we here explore a complementary strategy of reducing the required number of supervised synaptic updates to produce an adult-like ventral visual stream (as judged by the match to V1, V2, V4, IT, and behavior). Such models might require less precise machinery and energy expenditure to coordinate these updates and would thus move us closer to viable neuroscientific hypotheses about how the visual system wires itself up. Relative to the current leading model of the adult ventral stream, we here demonstrate that the total number of supervised weight updates can be substantially reduced using three complementary strategies: First, we find that only 2% of supervised updates (epochs and images) are needed to achieve ~80% of the match to adult ventral stream. Second, by improving the random distribution of synaptic connectivity, we find that 54% of the brain match can already be achieved “at birth” (i.e. no training at all). Third, we find that, by training only ~5% of model synapses, we can still achieve nearly 80% of the match to the ventral stream. When these three strategies are applied in combination, we find that these new models achieve ~80% of a fully trained model’s match to the brain, while using two orders of magnitude fewer supervised synaptic updates. These results reflect first steps in modeling not just primate adult visual processing during inference, but also how the ventral visual stream might be “wired up” by evolution (a model’s “birth” state) and by developmental learning (a model’s updates based on visual experience).

Download Full-text

Suppressed sensory response to predictable object stimuli throughout the ventral visual stream

10.1101/228890 ◽

2017 ◽

Author(s):

David Richter ◽

Matthias Ekman ◽

Floris P. de Lange

Keyword(s):

Visual Cortex ◽

Primary Visual Cortex ◽

Object Perception ◽

Sensory Response ◽

Neural Basis ◽

Visual Stream ◽

Lateral Occipital Complex ◽

Sensory Representation ◽

Statistical Regularities ◽

Ventral Visual Stream

AbstractPrediction plays a crucial role in perception, as prominently suggested by predictive coding theories. However, the exact form and mechanism of predictive modulations of sensory processing remain unclear, with some studies reporting a downregulation of the sensory response for predictable input, while others observed an enhanced response. In a similar vein, downregulation of the sensory response for predictable input has been linked to either sharpening or dampening of the sensory representation, which are opposite in nature. In the present study we set out to investigate the neural consequences of perceptual expectation of object stimuli throughout the visual hierarchy, using fMRI in human volunteers. Participants (n=24) were exposed to pairs of sequentially presented object images in a statistical learning paradigm, in which the first object predicted the identity of the second object. Image transitions were not task relevant; thus all learning of statistical regularities was incidental. We found strong suppression of neural responses to expected compared to unexpected stimuli throughout the ventral visual stream, including primary visual cortex (V1), lateral occipital complex (LOC), and anterior ventral visual areas. Expectation suppression in LOC, but not V1, scaled positively with image preference, lending support to the dampening account of expectation suppression in object perception.Significance StatementStatistical regularities permeate our world and help us to perceive and understand our surroundings. It has been suggested that the brain fundamentally relies on predictions and constructs models of the world in order to make sense of sensory information. Previous research on the neural basis of prediction has documented expectation suppression, i.e. suppressed responses to expected compared to unexpected stimuli. In the present study we queried the presence and characteristics of expectation suppression throughout the ventral visual stream. We demonstrate robust expectation suppression in the entire ventral visual pathway, and underlying this suppression a dampening of the sensory representation in object-selective visual cortex, but not in primary visual cortex. Taken together, our results provide novel evidence in support of theories conceptualizing perception as an active inference process, which selectively dampens cortical representations of predictable objects. This dampening may support our ability to automatically filter out irrelevant, predictable objects.

Download Full-text

Statistical learning attenuates visual activity only for attended stimuli

10.1101/653782 ◽

2019 ◽

Author(s):

David Richter ◽

Floris P. de Lange

Keyword(s):

Statistical Learning ◽

Sensory Information ◽

Neural Responses ◽

Visual Stream ◽

Fmri Study ◽

Human Volunteers ◽

Statistical Regularities ◽

Ventral Visual Stream ◽

Sensory Attenuation ◽

And Behavior

AbstractPerception and behavior can be guided by predictions, which are often based on learned statistical regularities. Neural responses to expected stimuli are frequently found to be attenuated after statistical learning. However, whether this sensory attenuation following statistical learning occurs automatically or depends on attention remains unknown. In the present fMRI study, we exposed human volunteers to sequentially presented object stimuli, in which the first object predicted the identity of the second object. We observed a strong attenuation of neural activity for expected compared to unexpected stimuli in the ventral visual stream. Crucially, this sensory attenuation was only apparent when stimuli were attended, and vanished when attention was directed away from the predictable objects. These results put important constraints on neurocomputational theories that cast perception as a process of probabilistic integration of prior knowledge and sensory information.

Download Full-text

Image content is more important than Bouma’s Law for scene metamers

10.1101/378521 ◽

2018 ◽

Author(s):

Thomas S. A. Wallis ◽

Christina M. Funke ◽

Alexander S. Ecker ◽

Leon A. Gatys ◽

Felix A. Wichmann ◽

...

Keyword(s):

Perceptual Experience ◽

Receptive Fields ◽

Retinal Eccentricity ◽

High Fidelity ◽

Neural Responses ◽

Visual Stream ◽

Scale Factors ◽

Area V2 ◽

Ventral Visual Stream ◽

Human Participants

AbstractWe subjectively perceive our visual field with high fidelity, yet large peripheral distortions can go unnoticed and peripheral objects can be difficult to identify (crowding). A recent paper proposed a model of the mid-level ventral visual stream in which neural responses were averaged over an area of space that increased as a function of eccentricity (scaling). Human participants could not discriminate synthesised model images from each other (they were metamers) when scaling was about half the retinal eccentricity. This result implicated ventral visual area V2 and approximated “Bouma’s Law” of crowding. It has subsequently been interpreted as a link between crowding zones, receptive field scaling, and our rich perceptual experience. However, participants in this experiment never saw the original images. We find that participants can easily discriminate real and model-generated images at V2 scaling. Lower scale factors than even V1 receptive fields may be required to generate metamers. Efficiently explaining why scenes look as they do may require incorporating segmentation processes and global organisational constraints in addition to local pooling.

Download Full-text

Familiarity increases processing speed in the visual system

10.1101/670489 ◽

2019 ◽

Author(s):

Mariya E. Manahova ◽

Eelke Spaak ◽

Floris P. de Lange

Keyword(s):

Visual System ◽

Brain Activity ◽

Spatial Dynamics ◽

Neural Response ◽

Visual Response ◽

Neural Responses ◽

Visual Stream ◽

Perceptual Performance ◽

Evoked Activity ◽

Human Participants

AbstractFamiliarity with a stimulus leads to an attenuated neural response to the stimulus. Alongside this attenuation, recent studies have also observed a truncation of stimulus-evoked activity for familiar visual input. One proposed function of this truncation is to rapidly put neurons in a state of readiness to respond to new input. Here, we examined this hypothesis by presenting human participants with target stimuli that were embedded in rapid streams of familiar or novel distractor stimuli at different speeds of presentation, while recording brain activity using magnetoencephalography (MEG) and measuring behavioral performance. We investigated the temporal and spatial dynamics of signal truncation and whether this phenomenon bears relationship to participants’ ability to categorize target items within a visual stream. Behaviorally, target categorization performance was markedly better when the target was embedded within familiar distractors, and this benefit became more pronounced with increasing speed of presentation. Familiar distractors showed a truncation of neural activity in the visual system, and this truncation was strongest for the fastest presentation speeds. Moreover, neural processing of the target was stronger when it was preceded by familiar distractors. Taken together, these findings suggest that truncation of neural responses for familiar items may result in stronger processing of relevant target information, resulting in superior perceptual performance.Significance statementThe visual response to familiar input is attenuated more rapidly than for novel input. Here we find that this truncation of the neural response for familiar input is strongest for very fast image presentations. We also find a tentative function for this truncation: the neural response to a target image that is embedded within distractors is much greater when the distractors are familiar than when they are novel. Similarly, target categorization performance is much better when the target is embedded within familiar distractors, and this advantage is most obvious for very fast image presentations. This suggests that neural truncation helps to rapidly put neurons in a state of readiness to respond to new input.

Download Full-text

Familiarity Increases Processing Speed in the Visual System

Journal of Cognitive Neuroscience ◽

10.1162/jocn_a_01507 ◽

2020 ◽

Vol 32 (4) ◽

pp. 722-733 ◽

Cited By ~ 1

Author(s):

Mariya E. Manahova ◽

Eelke Spaak ◽

Floris P. de Lange

Keyword(s):

Visual System ◽

Brain Activity ◽

Spatial Dynamics ◽

Neural Response ◽

Visual Stream ◽

Perceptual Performance ◽

Temporal And Spatial ◽

Evoked Activity ◽

Human Participants ◽

Cortical Regions

Familiarity with a stimulus leads to an attenuated neural response to the stimulus. Alongside this attenuation, recent studies have also observed a truncation of stimulus-evoked activity for familiar visual input. One proposed function of this truncation is to rapidly put neurons in a state of readiness to respond to new input. Here, we examined this hypothesis by presenting human participants with target stimuli that were embedded in rapid streams of familiar or novel distractor stimuli at different speeds of presentation, while recording brain activity using magnetoencephalography and measuring behavioral performance. We investigated the temporal and spatial dynamics of signal truncation and whether this phenomenon bears relationship to participants' ability to categorize target items within a visual stream. Behaviorally, target categorization performance was markedly better when the target was embedded within familiar distractors, and this benefit became more pronounced with increasing speed of presentation. Familiar distractors showed a truncation of neural activity in the visual system. This truncation was strongest for the fastest presentation speeds and peaked in progressively more anterior cortical regions as presentation speeds became slower. Moreover, the neural response evoked by the target was stronger when this target was preceded by familiar distractors. Taken together, these findings demonstrate that item familiarity results in a truncated neural response, is associated with stronger processing of relevant target information, and leads to superior perceptual performance.

Download Full-text

Statistical learning attenuates visual activity only for attended stimuli

eLife ◽

10.7554/elife.47869 ◽

2019 ◽

Vol 8 ◽

Cited By ~ 14

Author(s):

David Richter ◽

Floris P de Lange

Keyword(s):

Statistical Learning ◽

Sensory Information ◽

Neural Responses ◽

Visual Stream ◽

Fmri Study ◽

Human Volunteers ◽

Statistical Regularities ◽

Ventral Visual Stream ◽

Sensory Attenuation ◽

And Behavior

Perception and behavior can be guided by predictions, which are often based on learned statistical regularities. Neural responses to expected stimuli are frequently found to be attenuated after statistical learning. However, whether this sensory attenuation following statistical learning occurs automatically or depends on attention remains unknown. In the present fMRI study, we exposed human volunteers to sequentially presented object stimuli, in which the first object predicted the identity of the second object. We observed a reliable attenuation of neural activity for expected compared to unexpected stimuli in the ventral visual stream. Crucially, this sensory attenuation was only apparent when stimuli were attended, and vanished when attention was directed away from the predictable objects. These results put important constraints on neurocomputational theories that cast perception as a process of probabilistic integration of prior knowledge and sensory information.

Download Full-text