Optimized but not maximized cue integration for 3D visual perception

Reconstructing three-dimensional (3D) scenes from two-dimensional (2D) retinal images is an ill-posed problem. Despite this, our 3D perception of the world based on 2D retinal images is seemingly accurate and precise. The integration of distinct visual cues is essential for robust 3D perception in humans, but it is unclear if this mechanism is conserved in non-human primates, and how the underlying neural architecture constrains 3D perception. Here we assess 3D perception in macaque monkeys using a surface orientation discrimination task. We find that perception is generally accurate, but precision depends on the spatial pose of the surface and available cues. The results indicate that robust perception is achieved by dynamically reweighting the integration of stereoscopic and perspective cues according to their pose-dependent reliabilities. They further suggest that 3D perception is influenced by a prior for the 3D orientation statistics of natural scenes. We compare the data to simulations based on the responses of 3D orientation selective neurons. The results are explained by a model in which two independent neuronal populations representing stereoscopic and perspective cues (with perspective signals from the two eyes combined using nonlinear canonical computations) are optimally integrated through linear summation. Perception of combined-cue stimuli is optimal given this architecture. However, an alternative architecture in which stereoscopic cues and perspective cues detected by each eye are represented by three independent populations yields two times greater precision than observed. This implies that, due to canonical computations, cue integration for 3D perception is optimized but not maximized.Author summaryOur eyes only sense two-dimensional projections of the world (like a movie on a screen), yet we perceive the world in three dimensions. To create reliable 3D percepts, the human visual system integrates distinct visual signals according to their reliabilities, which depend on conditions such as how far away an object is located and how it is oriented. Here we find that non-human primates similarly integrate different 3D visual signals, and that their perception is influenced by the 3D orientation statistics of natural scenes. Cue integration is thus a conserved mechanism for creating robust 3D percepts by the primate brain. Using simulations of neural population activity, based on neuronal recordings from the same animals, we show that some computations which occur widely in the brain facilitate 3D perception, while others hinder perception. This work addresses key questions about how neural systems solve the difficult problem of generating 3D percepts, identifies a plausible neural architecture for implementing robust 3D vision, and reveals how neural computation can simultaneously optimize and curb perception.