Frequency transfer of the ventriloquism aftereffect

The ventriloquism aftereffect (VAE) describes the persistent shift of perceived sound location after having been adapted to a ventriloquism condition, in which the sound was repeatedly paired with a displaced visual stimulus. In the latter case, participants consistently mislocalize the sound in the direction of the visual stimulus (ventriloquism effect, VE). Previous studies provide conflicting reports regarding the strength of the VAE, ranging from 0 to nearly 100%. Moreover, there is controversy about its generalization to different sounds than the one inducing the VE, ranging from no transfer at all, to full transfer across different sound spectra. Here, we imposed the VE for three different sounds: a low-frequency and a high-frequency narrow-band noise, and a broadband Gaussian white noise (GWN). In the adaptation phase, listeners generated fast goal-directed head movements to localize the sound, presented across a 70 deg range in the horizontal plane, while ignoring a visual distracter that was consistently displaced 10 deg to the right of the sound. In the post-adaptation phase, participants localized narrow-band sounds with center frequencies from 0.5 to 8 kHz, as well as GWN, without the visual distracter. Our results show that the VAE amounted to approximately 40% of the VE and generalized well across the entire frequency domain. We also found that the strength of the VAE correlated with the pre-adaptation sound-localization performance. We compare our results with previous reports and discuss different hypotheses regarding optimal audio-visual cue integration.

The context of experienced sensory discrepancies shapes multisensory integration and recalibration differently

Whether two sensory cues interact during perceptual judgments depends on their immediate properties, but as suggested by Bayesian models, also on the observer's a priori belief that these originate from a common source. While in many experiments this a priori belief is considered fixed, in real life it must adapt to the momentary context or environment. To understand the adaptive nature of human multisensory perception we investigated the context-sensitivity of spatial judgements in a ventriloquism paradigm. We exposed observers to audio-visual stimuli whose discrepancy either varied over a wider (±46°) or a narrower range (±26°) and hypothesized that exposure to a wider range of discrepancies would facilitate multisensory binding by increasing participants a priori belief about a common source for a given discrepancy. Our data support this hypothesis by revealing an enhanced integration (ventriloquism) bias in the wider context, which was echoed in Bayesian causal inference models fit to participants' data, which assigned a stronger a priori integration tendency during the wider context. Interestingly, the immediate ventriloquism aftereffect, a multisensory response bias obtained following a multisensory test trial, was not affected by the contextual manipulation, although participants' confidence in their spatial judgments differed between contexts for both integration and recalibration trials. These results highlight the context-sensitivity of multisensory binding and suggest that the immediate ventriloquism aftereffect is not a purely sensory-level consequence of the multisensory integration process.

Multiple spatial reference frames underpin perceptual recalibration to audio-visual discrepancies

In dynamic multisensory environments, the perceptual system corrects for discrepancies arising between modalities. For instance, in the ventriloquism aftereffect (VAE), spatial disparities introduced between visual and auditory stimuli lead to a perceptual recalibration of auditory space. Previous research has shown that the VAE is underpinned by multiple recalibration mechanisms tuned to different timescales, however it remains unclear whether these mechanisms use common or distinct spatial reference frames. Here we asked whether the VAE operates in eye- or head-centred reference frames across a range of adaptation timescales, from a few seconds to a few minutes. We developed a novel paradigm for selectively manipulating the contribution of eye- versus head-centred visual signals to the VAE by manipulating auditory locations relative to either the head orientation or the point of fixation. Consistent with previous research, we found both eye- and head-centred frames contributed to the VAE across all timescales. However, we found no evidence for an interaction between spatial reference frames and adaptation duration. Our results indicate that the VAE is underpinned by multiple spatial reference frames that are similarly leveraged by the underlying time-sensitive mechanisms.

A model of the reference frame of the ventriloquism aftereffect

Background: Ventriloquism aftereffect (VAE), observed as a shift in the perceived locations of sounds after audiovisual stimulation, requires reference frame (RF) alignment since hearing and vision encode space in different RFs (head-centered, HC, vs. eye-centered, EC). Experimental studies examining the RF of VAE found inconsistent results: a mixture of HC and EC RFs was observed for VAE induced in the central region, while a predominantly HC RF was observed in the periphery. Here, a computational model examines these inconsistencies, as well as a newly observed EC adaptation induced by AV-aligned audiovisual stimuli. Methods: The model has two versions, each containing two additively combined components: a saccade-related component characterizing the adaptation in auditory-saccade responses, and auditory space representation adapted by ventriloquism signals either in the HC RF (HC version) or in a combination of HC and EC RFs (HEC version). Results: The HEC model performed better than the HC model in the main simulation considering all the data, while the HC model was more appropriate when only the AV-aligned adaptation data were simulated. Conclusion: Visual signals in a uniform mixed HC+EC RF are likely used to calibrate the auditory spatial representation, even after the EC-referenced auditory-saccade adaptation is accounted for.

Robust spatial ventriloquism effect and trial-by-trial aftereffect under memory interference

Our brain adapts to discrepancies in the sensory inputs. One example is provided by the ventriloquism effect, experienced when the sight and sound of an object are displaced. Here the discrepant multisensory stimuli not only result in a biased localization of the sound, but also recalibrate the perception of subsequent unisensory acoustic information in the so-called ventriloquism aftereffect. This aftereffect has been linked to memory-related processes based on its parallels to general sequential effects in perceptual decision making experiments and insights obtained in neuroimaging studies. For example, we have recently implied memory-related medial parietal regions in the trial-by-trial ventriloquism aftereffect. Here, we tested the hypothesis that the trial-by-trial (or immediate) ventriloquism aftereffect is indeed susceptible to manipulations interfering with working memory. Across three experiments we systematically manipulated the temporal delays between stimuli and response for either the ventriloquism or the aftereffect trials, or added a sensory-motor masking trial in between. Our data reveal no significant impact of either of these manipulations on the aftereffect, suggesting that the recalibration reflected by the trial-by-trial ventriloquism aftereffect is surprisingly resilient to manipulations interfering with memory-related processes.

Robust spatial ventriloquism effect and aftereffect under memory interference

ABSTRACTOur brain adapts to discrepancies in the sensory inputs. One example is provided by the ventriloquism effect, experienced when the sight and sound of an object are displaced. Here the discrepant multisensory stimuli not only result in a biased localization of the sound, but also recalibrate the perception of subsequent unisensory acoustic information in the so-called ventriloquism aftereffect. This aftereffect has been linked to memory-related processes based on its parallels to general sequential effects in perceptual decision making experiments and insights obtained in neuroimaging studies. For example, we have recently implied memory-related medial parietal regions in the trial-by-trial ventriloquism aftereffect. Here, we tested the hypothesis that the ventriloquism aftereffect is indeed susceptible to manipulations interfering with working memory. Across three experiments we systematically manipulated the temporal delays between stimuli and response for either the ventriloquism or the aftereffect trials, or added a sensory-motor masking trial in between. Our data reveal no significant impact of either of these manipulations on the aftereffect, suggesting that the recalibration reflected by the ventriloquism aftereffect is surprisingly resilient to manipulations interfering with memory-related processes.

Sensory- and memory-related drivers for altered ventriloquism effects and aftereffects in older adults

ABSTRACTThe manner in which humans exploit multisensory information for subsequent decisions changes with age. Multiple causes for such age-effects are being discussed, including a reduced precision in peripheral sensory representations, changes in cognitive inference about causal relations between sensory cues, and a decline in memory contributing to altered sequential patterns of multisensory behaviour. To dissociate these putative contributions, we investigated how healthy young and older adults integrate audio-visual spatial information within trials (the ventriloquism effect) and between trials (the ventriloquism aftereffect). With both a model-free and (Bayesian) model-based analyses we found that both biases differed between groups. Our results attribute the age-change in the ventriloquism bias to a decline in spatial hearing rather than a change in cognitive processes. This decline in peripheral function, combined with a more prominent influence from preceding responses rather than preceding stimuli in the elderly, can also explain the observed age-effect in the ventriloquism aftereffect. Our results suggest a transition from a sensory- to a behavior-driven influence of past multisensory experience on perceptual decisions with age, due to reduced sensory precision and change in memory capacity.

Hemisphere-Specific Properties of the Ventriloquism Aftereffect in Humans and Monkeys

ABSTRACTVisual calibration of auditory space requires re-alignment of representations differing in 1) format (auditory hemispheric channels vs. visual maps) and 2) reference frames (head-centered vs. eye-centered). Here, a ventriloquism paradigm from Kopčo et al. (J Neurosci, 29, 13809-13814) was used to examine these processes in humans and monkeys for ventriloquism induced within one spatial hemifield. Results show that 1) the auditory representation is adapted even by aligned audio-visual stimuli, and 2) the spatial reference frame is primarily head-centered in humans but mixed in monkeys. These results support the view that the ventriloquism aftereffect is driven by multiple spatially non-uniform processes.PACS numbers: 43.66.Pn, 43.66.Qp, 43.66.Mk