Substituting facial movements in singers changes the sounds of musical intervals

Cross-modal integration is ubiquitous within perception and, in humans, the McGurk effect demonstrates that seeing a person articulating speech can change what we hear into a new auditory percept. It remains unclear whether cross-modal integration of sight and sound generalizes to other visible vocal articulations like those made by singers. We surmise that perceptual integrative effects should involve music deeply, since there is ample indeterminacy and variability in its auditory signals. We show that switching videos of sung musical intervals changes systematically the estimated distance between two notes of a musical interval so that pairing the video of a smaller sung interval to a relatively larger auditory led to compression effects on rated intervals, whereas the reverse led to a stretching effect. In addition, after seeing a visually switched video of an equally-tempered sung interval and then hearing the same interval played on the piano, the two intervals were judged often different though they differed only in instrument. These findings reveal spontaneous, cross-modal, integration of vocal sounds and clearly indicate that strong integration of sound and sight can occur beyond the articulations of natural speech.

The unity hypothesis revisited: can the gender incongruent McGurk effect be disrupted by priming?

The “unity assumption hypothesis” contends that higher-level factors, such as a perceiver’s belief and prior experience, modulate multisensory integration. The McGurk illusion exemplifies such integration. When a visual velar /ga/ is dubbed with an auditory bilabial /ba/, listeners unify the discrepant signals with knowledge that open lips cannot produce /ba/ and a fusion percept /da/ is perceived. Previous research claimed to have falsified this theory by demonstrating the McGurk effect occurs even when a face is dubbed with a gender incongruent voice. But perhaps stronger evidence than just an apparent incongruence between unfamiliar faces and voices is needed to prevent perceptual unity. Here we investigated whether the McGurk illusion with gender incongruent stimuli can be disrupted by priming with appropriate pairing of face and voice. In an online experiment, 89 participants aged 18-62, were randomly allocated to experience experimental trials containing either a male or female face with incongruent gender voice. The number of times participants experienced a McGurk illusion was measured before and after a training block which familiarized them with the true pairings of face and voice. After training and priming, the susceptibility to the McGurk effects decreased significantly on average. The findings support the notion that unity assumptions modulate intersensory bias, and confirm and extend previous studies using gender incongruous McGurk stimuli.

Own-race faces promote integrated audiovisual speech information

The other-race effect indicates a perceptual advantage when processing own-race faces. This effect has been demonstrated in individuals’ recognition of facial identity and emotional expressions. However, it remains unclear whether the other-race effect also exists in multisensory domains. We conducted two experiments to provide evidence for the other-race effect in facial speech recognition, using the McGurk effect. Experiment 1 tested this issue among East Asian adults, examining the magnitude of the McGurk effect during stimuli using speakers from two different races (own-race vs. other-race). We found that own-race faces induced a stronger McGurk effect than other-race faces. Experiment 2 indicated that the other-race effect was not simply due to different levels of attention being paid to the mouths of own- and other-race speakers. Our findings demonstrated that own-race faces enhance the weight of visual input during audiovisual speech perception, and they provide evidence of the own-race effect in the audiovisual interaction for speech perception in adults.

Manual Gestures Modulate Early Neural Responses in Loudness Perception

How different sensory modalities interact to shape perception is a fundamental question in cognitive neuroscience. Previous studies in audiovisual interaction have focused on abstract levels such as categorical representation (e.g., McGurk effect). It is unclear whether the cross-modal modulation can extend to low-level perceptual attributes. This study used motional manual gestures to test whether and how the loudness perception can be modulated by visual-motion information. Specifically, we implemented a novel paradigm in which participants compared the loudness of two consecutive sounds whose intensity changes around the just noticeable difference (JND), with manual gestures concurrently presented with the second sound. In two behavioral experiments and two EEG experiments, we investigated our hypothesis that the visual-motor information in gestures would modulate loudness perception. Behavioral results showed that the gestural information biased the judgment of loudness. More importantly, the EEG results demonstrated that early auditory responses around 100 ms after sound onset (N100) were modulated by the gestures. These consistent results in four behavioral and EEG experiments suggest that visual-motor processing can integrate with auditory processing at an early perceptual stage to shape the perception of a low-level perceptual attribute such as loudness, at least under challenging listening conditions.

The other-race effect on the McGurk effect in infancy

This study investigated the difference in the McGurk effect between own-race-face and other-race-face stimuli among Japanese infants from 5 to 9 months of age. The McGurk effect results from infants using information from a speaker’s face in audiovisual speech integration. We hypothesized that the McGurk effect varies with the speaker’s race because of the other-race effect, which indicates an advantage for own-race faces in our face processing system. Experiment 1 demonstrated the other-race effect on audiovisual speech integration such that the infants ages 5–6 months and 8–9 months are likely to perceive the McGurk effect when observing an own-race-face speaker, but not when observing an other-race-face speaker. Experiment 2 found the other-race effect on audiovisual speech integration regardless of irrelevant speech identity cues. Experiment 3 confirmed the infants’ ability to differentiate two auditory syllables. These results showed that infants are likely to integrate voice with an own-race-face, but not with an other-race-face. This implies the role of experiences with own-race-faces in the development of audiovisual speech integration. Our findings also contribute to the discussion of whether perceptual narrowing is a modality-general, pan-sensory process.

The McGurk Effect In Relation to Musicians' Abilities

The original study of the McGurk Effect, a perceptual phenomenon caused by contradictory audiovisual stimuli fusing together to create the illusion of a third sound, was carried out by psychologists McGurk and MacDonald in 1976. The results of early experiments displayed that observers used both auditory and visual signals while being spoken to, auditory signals being the sound waves entering their ears, and visual signals being how the speaker moved his face while pronouncing a word. When conflicting signals are given, a third sound is perceived, as the brain is disoriented from the different signals. The idea that musicians have superior audiovisual cortexes have led some to speculate if musicians are as susceptible to the McGurk Effect as non-musicians. To research the susceptibility of musicians to the McGurk Effect, the experiment conducted included a total of 40 subjects, 20 musicians and 20 non-musicians. The subjects were played a control video of a speaker saying “ga” and were then presented with four audiovisually incongruent videos, all containing a speaker mouthing the word “ga” with the audio recording of the speaker saying “ba” dubbed on. Two main 2x2 Chi Square tests and fifteen secondary 2x2 Chi Squares tests were run in total. The two main tests, which compared the amount of McGurk interpretations to either audio or visual interpretations, both produced a p-value of <.0005. Upon further research, 25.7% of musicians reported a McGurk interpretation, as opposed to 52.2% of non-musicians, which implied that musicians are less susceptible to the McGurk effect.

High-Level Perception and Multimodal Perception

What is the correct procedure for determining the contents of perception? Philosophers tackling this question increasingly rely on empirically oriented procedures. This chapter argues that this strategy constitutes an improvement over the armchair methodology of phenomenal contrast arguments, but that there is a respect in which current empirical procedures remain limited: they are unimodal in nature, wrongly treating the senses as isolatable. The chapter thus has two aims: first, to motivate a reorientation of the admissible contents debate into a multimodal framework. The second is to explore whether experimental studies of multimodal perception support a so-called Liberal account of perception’s admissible contents. The chapter concludes that the McGurk effect and the ventriloquist effect are both explicable without the postulation of high-level content, but that at least one multimodal experimental paradigm may necessitate such content: the rubber hand illusion. One upshot is that Conservatives who claim that the Liberal view intolerably broadens the scope of perceptual illusions, particularly from the perspective of perceptual psychology, should pursue other arguments against that view.