Image-Computable Ideal Observers for Tasks with Natural Stimuli

2020 ◽  
Vol 6 (1) ◽  
pp. 491-517 ◽  
Author(s):  
Johannes Burge
Keyword(s):  
Visual Processing ◽  
Visual Information ◽  
Neural Coding ◽  
Ideal Observer ◽  
Perceptual Task ◽  
Vision Science ◽  
Natural Stimuli ◽  
Ideal Observers ◽  
Perceptual Performance ◽  
And Performance

An ideal observer is a theoretical model observer that performs a specific sensory-perceptual task optimally, making the best possible use of the available information given physical and biological constraints. An image-computable ideal observer (pixels in, estimates out) is a particularly powerful type of ideal observer that explicitly models the flow of visual information from the stimulus-encoding process to the eventual decoding of a sensory-perceptual estimate. Image-computable ideal observer analyses underlie some of the most important results in vision science. However, most of what we know from ideal observers about visual processing and performance derives from relatively simple tasks and relatively simple stimuli. This review describes recent efforts to develop image-computable ideal observers for a range of tasks with natural stimuli and shows how these observers can be used to predict and understand perceptual and neurophysiological performance. The reviewed results establish principled links among models of neural coding, computational methods for dimensionality reduction, and sensory-perceptual performance in tasks with natural stimuli.

scholarly journals Bayesian natural selection and the evolution of perceptual systems

2002 ◽  
Vol 357 (1420) ◽  
pp. 419-448 ◽  
Author(s):  
Wilson S. Geisler ◽  
Randy L. Diehl
Keyword(s):  
Natural Selection ◽  
Decision Theory ◽  
Ideal Observer ◽  
Statistical Decision Theory ◽  
Statistical Decision ◽  
Natural Stimuli ◽  
Ideal Observers ◽  
Formal Framework ◽  
Bayesian Statistical Decision Theory ◽  
Perceptual Systems

In recent years, there has been much interest in characterizing statistical properties of natural stimuli in order to better understand the design of perceptual systems. A fruitful approach has been to compare the processing of natural stimuli in real perceptual systems with that of ideal observers derived within the framework of Bayesian statistical decision theory. While this form of optimization theory has provided a deeper understanding of the information contained in natural stimuli as well as of the computational principles employed in perceptual systems, it does not directly consider the process of natural selection, which is ultimately responsible for design. Here we propose a formal framework for analysing how the statistics of natural stimuli and the process of natural selection interact to determine the design of perceptual systems. The framework consists of two complementary components. The first is a maximum fitness ideal observer, a standard Bayesian ideal observer with a utility function appropriate for natural selection. The second component is a formal version of natural selection based upon Bayesian statistical decision theory. Maximum fitness ideal observers and Bayesian natural selection are demonstrated in several examples. We suggest that the Bayesian approach is appropriate not only for the study of perceptual systems but also for the study of many other systems in biology.

scholarly journals Temporal stability of stimulus representation increases along rodent visual cortical hierarchies

2021 ◽  
Vol 12 (1) ◽  
Author(s):  
Eugenio Piasini ◽  
Liviu Soltuzu ◽  
Paolo Muratore ◽  
Riccardo Caramellino ◽  
Kasper Vinken ◽  
...  
Keyword(s):  
Visual Processing ◽  
Visual Information ◽  
Neural Coding ◽  
Temporal Stability ◽  
Receptive Fields ◽  
Ventral Stream ◽  
Dynamic Stimuli ◽  
The Stability ◽  
Visual Cortical ◽  
The Impact

AbstractCortical representations of brief, static stimuli become more invariant to identity-preserving transformations along the ventral stream. Likewise, increased invariance along the visual hierarchy should imply greater temporal persistence of temporally structured dynamic stimuli, possibly complemented by temporal broadening of neuronal receptive fields. However, such stimuli could engage adaptive and predictive processes, whose impact on neural coding dynamics is unknown. By probing the rat analog of the ventral stream with movies, we uncovered a hierarchy of temporal scales, with deeper areas encoding visual information more persistently. Furthermore, the impact of intrinsic dynamics on the stability of stimulus representations grew gradually along the hierarchy. A database of recordings from mouse showed similar trends, additionally revealing dependencies on the behavioral state. Overall, these findings show that visual representations become progressively more stable along rodent visual processing hierarchies, with an important contribution provided by intrinsic processing.

scholarly journals A Bayesian and efficient observer model explains concurrent attractive and repulsive history biases in visual perception

eLife ◽  
2020 ◽  
Vol 9 ◽  
Author(s):  
Matthias Fritsche ◽  
Eelke Spaak ◽  
Floris P de Lange
Keyword(s):  
Visual Processing ◽  
Visual Information ◽  
Temporal Pattern ◽  
Visual Experience ◽  
Ideal Observer ◽  
Serial Dependence ◽  
Short Term ◽  
The Past ◽  
Observer Model

Human perceptual decisions can be repelled away from (repulsive adaptation) or attracted towards recent visual experience (attractive serial dependence). It is currently unclear whether and how these repulsive and attractive biases interact during visual processing and what computational principles underlie these history dependencies. Here we disentangle repulsive and attractive biases by exploring their respective timescales. We find that perceptual decisions are concurrently attracted towards the short-term perceptual history and repelled from stimuli experienced up to minutes into the past. The temporal pattern of short-term attraction and long-term repulsion cannot be captured by an ideal Bayesian observer model alone. Instead, it is well captured by an ideal observer model with efficient encoding and Bayesian decoding of visual information in a slowly changing environment. Concurrent attractive and repulsive history biases in perceptual decisions may thus be the consequence of the need for visual processing to simultaneously satisfy constraints of efficiency and stability.

scholarly journals Intrinsic dynamics enhance temporal stability of stimulus representation along rodent visual cortical hierarchies

2019 ◽  
Author(s):  
Eugenio Piasini ◽  
Liviu Soltuzu ◽  
Paolo Muratore ◽  
Riccardo Caramellino ◽  
Kasper Vinken ◽  
...  
Keyword(s):  
Visual Processing ◽  
Visual Information ◽  
Neural Coding ◽  
Temporal Stability ◽  
Ventral Stream ◽  
Dynamic Stimuli ◽  
The Stability ◽  
Visual Cortical ◽  
Predictive Processes ◽  
The Impact

SummaryAlong the ventral stream, cortical representations of brief, static stimuli become gradually more invariant to identity-preserving transformations. In the presence of long, temporally structured dynamic stimuli, higher invariance should imply temporally persistent representations at the top of this functional hierarchy. However, such stimuli could engage adaptive and predictive processes, whose impact on neural coding dynamics is unknown. By probing the rat analogue of the ventral stream with movies, we uncovered a hierarchy of temporal scales, with deeper areas encoding visual information more persistently. Furthermore, the impact of intrinsic dynamics on the stability of stimulus representations gradually grew along the hierarchy. Analysis of a large dataset of recordings from the mouse visual hierarchy yielded similar trends, revealing also their dependence on the behavioral state of the animal. Overall, these findings show that visual representations become progressively more stable along rodent visual processing hierarchies, with an important contribution provided by intrinsic processing.

scholarly journals A Bayesian and efficient observer model explains concurrent attractive and repulsive history biases in visual perception

2020 ◽  
Author(s):  
Matthias Fritsche ◽  
Eelke Spaak ◽  
Floris P. de Lange
Keyword(s):  
Visual Processing ◽  
Visual Information ◽  
Temporal Pattern ◽  
Visual Experience ◽  
Ideal Observer ◽  
Serial Dependence ◽  
Short Term ◽  
The Past ◽  
Observer Model

AbstractPerceptual decisions can be repelled away from (repulsive adaptation) or attracted towards recent visual experience (attractive serial dependence). It is currently unclear whether and how these repulsive and attractive biases interact during visual processing and what computational principles may underlie these history dependencies. In the current study, we disentangle repulsive and attractive biases by exploring the respective timescales over which current visual processing is influenced by previous experience. Across four experiments, we find that perceptual decisions about stimulus orientation are concurrently attracted towards the short-term perceptual history and repelled from stimuli experienced up to minutes into the past. We show that the temporal pattern of short-term attraction and long-term repulsion cannot be captured by an ideal Bayesian observer model alone. Instead, it is well captured by an ideal observer model with efficient encoding and Bayesian decoding of visual information in a slowly changing environment. Concurrent attractive and repulsive history biases in perceptual decisions may thus be the consequence of the need for visual processing to simultaneously satisfy constraints of both efficiency and stability.

Exploring The Effect of Visual Information Degradation on Human Perception and Performance In A Human-Telerobot System

2020 ◽  
Vol 64 (1) ◽  
pp. 1302-1307
Author(s):  
Richard Stone ◽  
Minglu Wang ◽  
Thomas Schnieders ◽  
Esraa Abdelall
Keyword(s):  
Visual Information ◽  
Human Perception ◽  
Low Level ◽  
Hit Rate ◽  
Word Level ◽  
Level Of Automation ◽  
Significant Difference ◽  
Human Operators ◽  
And Performance ◽  
High Level

Human-robotic interaction system are increasingly becoming integrated into industrial, commercial and emergency service agencies. It is critical that human operators understand and trust automation when these systems support and even make important decisions. The following study focused on human-in-loop telerobotic system performing a reconnaissance operation. Twenty-four subjects were divided into groups based on level of automation (Low-Level Automation (LLA), and High-Level Automation (HLA)). Results indicated a significant difference between low and high word level of control in hit rate when permanent error occurred. In the LLA group, the type of error had a significant effect on the hit rate. In general, the high level of automation was better than the low level of automation, especially if it was more reliable, suggesting that subjects in the HLA group could rely on the automatic implementation to perform the task more effectively and more accurately.

Effects of Symbol Structure on Visual Information Processing

1983 ◽  
Vol 27 (5) ◽  
pp. 354-354
Author(s):  
Bruce W. Hamill ◽  
Robert A. Virzi
Keyword(s):  
Information Processing ◽  
Visual Processing ◽  
Visual Information ◽  
Context Effects ◽  
Response Times ◽  
Search Time ◽  
Visual Information Processing ◽  
Array Size ◽  
Serial Search ◽  
Feature Structure

This investigation addresses the problem of attention in the processing of symbolic information from visual displays. Its scope includes the nature of attentive processes, the structural properties of stimuli that influence visual information processing mechanisms, and the manner in which these factors interact in perception. Our purpose is to determine the effects of configural feature structure on visual information processing. It is known that for stimuli comprising separable features, one can distinguish between conditions in which only one relevant feature differs among stimuli in the array being searched and conditions in which conjunctions of two (or more) features differ: Since the visual process of conjoining separable features is additive, this fact is reflected in search time as a function of array size, with feature conditions yielding flat curves associated with parallel search (no increase in search time across array sizes) and conjunction conditions yielding linearly increasing curves associated with serial search. We studied configural-feature stimuli within this framework to determine the nature of visual processing for such stimuli as a function of their feature structure. Response times of subjects searching for particular targets among structured arrays of distractors were measured in a speeded visual search task. Two different sets of stimulus materials were studied in array sizes of up to 32 stimuli, using both tachistoscope and microcomputer-based CRT presentation for each. Our results with configural stimuli indicate serial search in all of the conditions, with the slope of the response-time-by-array-size function being steeper for conjunction conditions than for feature conditions. However, for each of the two sets of stimuli we studied, there was one configuration that stood apart from the others in its set in that it yielded significantly faster response times, and in that conjunction conditions involving these particular stimuli tended to cluster with the feature conditions rather than with the other conjunction conditions. In addition to these major effects of particular targets, context effects also appeared in our results as effects of the various distractor sets used; certain of these context effects appear to be reversible. The effects of distractor sets on target search were studied in considerable detail. We have found interesting differences in visual processing between stimuli comprising separable features and those comprising configural features. We have also been able to characterize the effects we have found with configural-feature stimuli as being related to the specific feature structure of the target stimulus in the context of the specific feature structure of distractor stimuli. These findings have strong implications for the design of symbology that can enhance visual performance in the use of automated displays.

scholarly journals The Neurophysiology of Backward Visual Masking: Information Analysis

1999 ◽  
Vol 11 (3) ◽  
pp. 300-311 ◽  
Author(s):  
Edmund T. Rolls ◽  
Martin J. Tovée ◽  
Stefano Panzeri
Keyword(s):  
Visual Processing ◽  
Visual Information ◽  
Computational Models ◽  
Direct Evidence ◽  
Visual Masking ◽  
Neuronal Response ◽  
Short Soas ◽  
Stimulus Onset ◽  
Backward Masking ◽  
Neuronal Responses

Backward masking can potentially provide evidence of the time needed for visual processing, a fundamental constraint that must be incorporated into computational models of vision. Although backward masking has been extensively used psychophysically, there is little direct evidence for the effects of visual masking on neuronal responses. To investigate the effects of a backward masking paradigm on the responses of neurons in the temporal visual cortex, we have shown that the response of the neurons is interrupted by the mask. Under conditions when humans can just identify the stimulus, with stimulus onset asynchronies (SOA) of 20 msec, neurons in macaques respond to their best stimulus for approximately 30 msec. We now quantify the information that is available from the responses of single neurons under backward masking conditions when two to six faces were shown. We show that the information available is greatly decreased as the mask is brought closer to the stimulus. The decrease is more marked than the decrease in firing rate because it is the selective part of the firing that is especially attenuated by the mask, not the spontaneous firing, and also because the neuronal response is more variable at short SOAs. However, even at the shortest SOA of 20 msec, the information available is on average 0.1 bits. This compares to 0.3 bits with only the 16-msec target stimulus shown and a typical value for such neurons of 0.4 to 0.5 bits with a 500-msec stimulus. The results thus show that considerable information is available from neuronal responses even under backward masking conditions that allow the neurons to have their main response in 30 msec. This provides evidence for how rapid the processing of visual information is in a cortical area and provides a fundamental constraint for understanding how cortical information processing operates.

scholarly journals Informative neural representations of unseen objects during higher-order processing in human brains and deep artificial networks

2021 ◽  
Author(s):  
Ning Mei ◽  
Roberto Santana ◽  
David Soto
Keyword(s):  
Visual Processing ◽  
Visual Information ◽  
Model Simulation ◽  
Conscious Experience ◽  
Neural Representation ◽  
Visual Stream ◽  
Order Processing ◽  
High Level ◽  
Unseen Objects ◽  
Human Brains

AbstractDespite advances in the neuroscience of visual consciousness over the last decades, we still lack a framework for understanding the scope of unconscious processing and how it relates to conscious experience. Previous research observed brain signatures of unconscious contents in visual cortex, but these have not been identified in a reliable manner, with low trial numbers and signal detection theoretic constraints not allowing to decisively discard conscious perception. Critically, the extent to which unconscious content is represented in high-level processing stages along the ventral visual stream and linked prefrontal areas remains unknown. Using a within-subject, high-precision, highly-sampled fMRI approach, we show that unconscious contents, even those associated with null sensitivity, can be reliably decoded from multivoxel patterns that are highly distributed along the ventral visual pathway and also involving prefrontal substrates. Notably, the neural representation in these areas generalised across conscious and unconscious visual processing states, placing constraints on prior findings that fronto-parietal substrates support the representation of conscious contents and suggesting revisions to models of consciousness such as the neuronal global workspace. We then provide a computational model simulation of visual information processing/representation in the absence of perceptual sensitivity by using feedforward convolutional neural networks trained to perform a similar visual task to the human observers. The work provides a novel framework for pinpointing the neural representation of unconscious knowledge across different task domains.

scholarly journals The Binding Ring Illusion: assimilation affects the perceived size of a circular array

F1000Research ◽  
2013 ◽  
Vol 2 ◽  
pp. 58 ◽  
Author(s):  
J Daniel McCarthy ◽  
Colin Kupitz ◽  
Gideon P Caplovitz
Keyword(s):  
Visual Processing ◽  
Visual Information ◽  
Circular Array ◽  
Multiple Sources ◽  
Spatial Frequencies ◽  
Constructive Process ◽  
Novel Variant ◽  
Assimilation Process ◽  
Circular Contour ◽  
High Level

Our perception of an object’s size arises from the integration of multiple sources of visual information including retinal size, perceived distance and its size relative to other objects in the visual field. This constructive process is revealed through a number of classic size illusions such as the Delboeuf Illusion, the Ebbinghaus Illusion and others illustrating size constancy. Here we present a novel variant of the Delbouef and Ebbinghaus size illusions that we have named the Binding Ring Illusion. The illusion is such that the perceived size of a circular array of elements is underestimated when superimposed by a circular contour – a binding ring – and overestimated when the binding ring slightly exceeds the overall size of the array. Here we characterize the stimulus conditions that lead to the illusion, and the perceptual principles that underlie it. Our findings indicate that the perceived size of an array is susceptible to the assimilation of an explicitly defined superimposed contour. Our results also indicate that the assimilation process takes place at a relatively high level in the visual processing stream, after different spatial frequencies have been integrated and global shape has been constructed. We hypothesize that the Binding Ring Illusion arises due to the fact that the size of an array of elements is not explicitly defined and therefore can be influenced (through a process of assimilation) by the presence of a superimposed object that does have an explicit size.

Sign in / Sign up

Export Citation Format

Share Document