An Image Reconstruction Framework for Characterizing Early Vision

We developed an image-computable observer model of the early visual system that operates on fully naturalistic input, based on a framework of Bayesian image reconstruction from retinal cone mosaic excitations. Our model extends previous work on ideal observer analysis and the evaluation of performance beyond psychophysical discrimination tasks, takes into account the statistical regularities of our visual environment, and provides a unifying framework for answering a wide range of questions regarding early vision. Using the error in the reconstruction as a metric, we analyzed the variations of the number of different photoreceptor types on human retina as an optimal design problem. In addition, the reconstructions allow both visualization and quantification of information loss due to physiological optics and cone mosaic sampling, and how these vary with eccentricity. Furthermore, in simulations of color deficiencies and interferometric experiments, we found that the reconstructed images provide a reasonable proxy for directly modeling subjects' percepts. Lastly, we used the reconstruction-based observer for the analysis of psychophysical threshold, and found notable interactions between spatial frequency and chromatic direction in the resulting spatial contrast sensitivity function. Our method should be widely applicable to many experiments and practical applications in which early vision plays an important role.

Predicting the partition of behavioral variability in speed perception with naturalistic stimuli

A core goal of visual neuroscience is to predict human perceptual performance from natural signals. Performance in any natural task can be impacted by at least three sources of uncertainty: stimulus variability, internal noise, and sub-optimal computations. Determining the relative importance of these factors has been a focus of interest for decades, but most successes have been achieved with simple tasks and simple stimuli. Drawing quantitative links directly from natural signals to perceptual performance has proven a substantial challenge. Here, we develop an image-computable (pixels in, estimates out) Bayesian ideal observer that makes optimal use of the statistics relating image movies to speed. The optimal computations bear striking resemblance to descriptive models proposed to account for neural activity in area MT. We develop a model based on the ideal, stimulate it with naturalistic signals, predict the behavioral signatures of each performance-limiting factor, and test the predictions in an interlocking series of speed discrimination experiments. The critical experiment collects human responses to repeated presentations of each unique image movie. The model, highly constrained by the earlier experiments, tightly predicts human response consistency without free parameters. This result implies that human observers use near-optimal computations to estimate speed, and that human performance is near-exclusively limited by natural stimulus variability and internal noise. The results demonstrate that human performance can be predicted from a task-specific statistical analysis of naturalistic stimuli, show that image-computable ideal observer analysis can be generalized from simple to natural stimuli, and encourage similar analyses in other domains.

Do people ask good questions?

People ask questions in order to efficiently learn about the world. But do people ask good questions? In this work, we designed an intuitive, game-based task that allowed people to ask natural language questions to resolve their uncertainty. Question quality was measured through Bayesian ideal-observer models that considered large spaces of possible game states. During free-form question generation, participants asked a creative variety of useful and goal-directed questions, yet they rarely asked the best questions as identified by the Bayesian ideal-observers (Experiment 1). In subsequent experiments, participants strongly preferred the best questions when evaluating questions that they did not generate themselves (Experiments 2 & 3). On the one hand, our results show that people can accurately evaluate question quality, even when the set of questions is diverse and an ideal-observer analysis has large computational requirements. On the other hand, people have a limited ability to synthesize maximally-informative questions from scratch, suggesting a bottleneck in the question asking process.

Adaptive shaping of cortical response selectivity in the vibrissa pathway

One embodiment of context-dependent sensory processing is bottom-up adaptation, where persistent stimuli decrease neuronal firing rate over hundreds of milliseconds. Adaptation is not, however, simply the fatigue of the sensory pathway, but shapes the information flow and selectivity to stimulus features. Adaptation enhances spatial discriminability (distinguishing stimulus location) while degrading detectability (reporting presence of the stimulus), for both the ideal observer of the cortex and awake, behaving animals. However, how the dynamics of the adaptation shape the cortical response and this detection and discrimination tradeoff is unknown, as is to what degree this phenomenon occurs on a continuum as opposed to a switching of processing modes. Using voltage-sensitive dye imaging in anesthetized rats to capture the temporal and spatial characteristics of the cortical response to tactile inputs, we showed that the suppression of the cortical response, in both magnitude and spatial spread, is continuously modulated by the increasing amount of energy in the adapting stimulus, which is nonuniquely determined by its frequency and velocity. Single-trial ideal observer analysis demonstrated a tradeoff between detectability and spatial discriminability up to a moderate amount of adaptation, which corresponds to the frequency range in natural whisking. This was accompanied by a decrease in both detectability and discriminability with high-energy adaptation, which indicates a more complex coupling between detection and discrimination than a simple switching of modes. Taken together, the results suggest that adaptation operates on a continuum and modulates the tradeoff between detectability and discriminability that has implications for information processing in ethological contexts.

Cross inhibition from ON to OFF pathway improves the efficiency of contrast encoding in the mammalian retina

The retina is divided into parallel and mostly independent ON and OFF pathways, but the ON pathway “cross” inhibits the OFF pathway. Cross inhibition was thought to improve signal processing by the OFF pathway, but its effect on contrast encoding had not been tested experimentally. To quantify the effect of cross inhibition on the encoding of contrast, we presented a dark flash to an in vitro preparation of the mammalian retina. We then recorded excitatory currents, inhibitory currents, membrane voltages, and spikes from OFF α-ganglion cells. The recordings were subjected to an ideal observer analysis that used Bayesian methods to determine how accurately the recordings detected the dark flash. We found that cross inhibition increases the detection accuracy of currents and membrane voltages. Yet these improvements in encoding do not fully reach the spike train, because cross inhibition also hyperpolarizes the OFF α-cell below spike threshold, preventing small signals in the membrane voltages at low contrast from reaching the spike train. The ultimate effect of cross inhibition is to increase the accuracy with which the spike train detects moderate contrast, but reduce the accuracy with which it detects low contrast. In apparent compensation for the loss of accuracy at low contrast, cross inhibition, by hyperpolarizing the OFF α-cell, reduces the number of spikes required to detect the dark flash and thereby increases encoding efficiency.