Electrophysiological Indices of Audiovisual Speech Perception: Beyond the McGurk Effect and Speech in Noise

Visual information on a talker’s face can influence what a listener hears. Commonly used approaches to study this include mismatched audiovisual stimuli (e.g., McGurk type stimuli) or visual speech in auditory noise. In this paper we discuss potential limitations of these approaches and introduce a novel visual phonemic restoration method. This method always presents the same visual stimulus (e.g., /ba/) dubbed with a matched auditory stimulus (/ba/) or one that has weakened consonantal information and sounds more /a/-like). When this reduced auditory stimulus (or /a/) is dubbed with the visual /ba/, a visual influence will result in effectively ‘restoring’ the weakened auditory cues so that the stimulus is perceived as a /ba/. An oddball design in which participants are asked to detect the /a/ among a stream of more frequently occurring /ba/s while either a speaking face or face with no visual speech was used. In addition, the same paradigm was presented for a second contrast in which participants detected /pa/ among /ba/s, a contrast which should be unaltered by the presence of visual speech. Behavioral and some ERP findings reflect the expected phonemic restoration for the /ba/ vs. /a/ contrast; specifically, we observed reduced accuracy and P300 response in the presence of visual speech. Further, we report an unexpected finding of reduced accuracy and P300 response for both speech contrasts in the presence of visual speech, suggesting overall modulation of the auditory signal in the presence of visual speech. Consistent with this, we observed a mismatch negativity (MMN) effect for the /ba/ vs. /pa/ contrast only that was larger in absence of visual speech. We discuss the potential utility for this paradigm for listeners who cannot respond actively, such as infants and individuals with developmental disabilities.

Download Full-text

Reducing Playback Rate of Audiovisual Speech Leads to a Surprising Decrease in the McGurk Effect

Multisensory Research ◽

10.1163/22134808-00002586 ◽

2018 ◽

Vol 31 (1-2) ◽

pp. 19-38 ◽

Cited By ~ 5

Author(s):

John F. Magnotti ◽

Debshila Basu Mallick ◽

Michael S. Beauchamp

Keyword(s):

Visual Information ◽

Mcgurk Effect ◽

Visual Speech ◽

Large Individual ◽

Audiovisual Speech ◽

Unexpected Finding ◽

Video Playback ◽

Natural Rate ◽

Audiovisual Speech Perception ◽

Bayesian Integration

We report the unexpected finding that slowing video playback decreases perception of the McGurk effect. This reduction is counter-intuitive because the illusion depends on visual speech influencing the perception of auditory speech, and slowing speech should increase the amount of visual information available to observers. We recorded perceptual data from 110 subjects viewing audiovisual syllables (either McGurk or congruent control stimuli) played back at one of three rates: the rate used by the talker during recording (the natural rate), a slow rate (50% of natural), or a fast rate (200% of natural). We replicated previous studies showing dramatic variability in McGurk susceptibility at the natural rate, ranging from 0–100% across subjects and from 26–76% across the eight McGurk stimuli tested. Relative to the natural rate, slowed playback reduced the frequency of McGurk responses by 11% (79% of subjects showed a reduction) and reduced congruent accuracy by 3% (25% of subjects showed a reduction). Fast playback rate had little effect on McGurk responses or congruent accuracy. To determine whether our results are consistent with Bayesian integration, we constructed a Bayes-optimal model that incorporated two assumptions: individuals combine auditory and visual information according to their reliability, and changing playback rate affects sensory reliability. The model reproduced both our findings of large individual differences and the playback rate effect. This work illustrates that surprises remain in the McGurk effect and that Bayesian integration provides a useful framework for understanding audiovisual speech perception.

Download Full-text

Assessing audiovisual saliency and visual-information content in the articulation of consonants and vowels on audiovisual temporal perception

Seeing and Perceiving ◽

10.1163/187847612x646514 ◽

2012 ◽

Vol 25 (0) ◽

pp. 29

Author(s):

Argiro Vatakis ◽

Charles Spence

Keyword(s):

Visual Information ◽

Temporal Order ◽

Temporal Integration ◽

Visual Speech ◽

Auditory Signal ◽

Audiovisual Speech Perception ◽

Speech Stream ◽

Point Of Subjective Simultaneity ◽

Speech Token ◽

Manner Of Articulation

Research has revealed different temporal integration windows between and within different speech-tokens. The limited speech-tokens tested to date has not allowed for the proper evaluation of whether such differences are task or stimulus driven? We conducted a series of experiments to investigate how the physical differences associated with speech articulation affect the temporal aspects of audiovisual speech perception. Videos of consonants and vowels uttered by three speakers were presented. Participants made temporal order judgments (TOJs) regarding which speech-stream had been presented first. The sensitivity of participants’ TOJs and the point of subjective simultaneity (PSS) were analyzed as a function of the place, manner of articulation, and voicing for consonants, and the height/backness of the tongue and lip-roundedness for vowels. The results demonstrated that for the case of place of articulation/roundedness, participants were more sensitive to the temporal order of highly-salient speech-signals with smaller visual-leads at the PSS. This was not the case when the manner of articulation/height was evaluated. These findings suggest that the visual-speech signal provides substantial cues to the auditory-signal that modulate the relative processing times required for the perception of the speech-stream. A subsequent experiment explored how the presentation of different sources of visual-information modulated such findings. Videos of three consonants were presented under natural and point-light (PL) viewing conditions revealing parts, or the whole, face. Preliminary analysis revealed no differences in TOJ accuracy under different viewing conditions. However, the PSS data revealed significant differences in viewing conditions depending on the speech token uttered (e.g., larger visual-leads for PL-lip/teeth/tongue-only views).

Download Full-text

Weaker McGurk Effect for Rubin’s Vase-Type Speech in People With High Autistic Traits

Multisensory Research ◽

10.1163/22134808-bja10047 ◽

2021 ◽

pp. 1-17

Author(s):

Yuta Ujiie ◽

Kohske Takahashi

Keyword(s):

Speech Recognition ◽

Speech Perception ◽

Visual Information ◽

Autistic Traits ◽

Autism Spectrum ◽

Mcgurk Effect ◽

Visual Speech ◽

Audiovisual Speech ◽

Autism Spectrum Quotient ◽

Audiovisual Speech Perception

Abstract While visual information from facial speech modulates auditory speech perception, it is less influential on audiovisual speech perception among autistic individuals than among typically developed individuals. In this study, we investigated the relationship between autistic traits (Autism-Spectrum Quotient; AQ) and the influence of visual speech on the recognition of Rubin’s vase-type speech stimuli with degraded facial speech information. Participants were 31 university students (13 males and 18 females; mean age: 19.2, SD: 1.13 years) who reported normal (or corrected-to-normal) hearing and vision. All participants completed three speech recognition tasks (visual, auditory, and audiovisual stimuli) and the AQ–Japanese version. The results showed that accuracies of speech recognition for visual (i.e., lip-reading) and auditory stimuli were not significantly related to participants’ AQ. In contrast, audiovisual speech perception was less susceptible to facial speech perception among individuals with high rather than low autistic traits. The weaker influence of visual information on audiovisual speech perception in autism spectrum disorder (ASD) was robust regardless of the clarity of the visual information, suggesting a difficulty in the process of audiovisual integration rather than in the visual processing of facial speech.

Download Full-text

Increased connectivity among sensory and motor regions during visual and audiovisual speech perception

10.1101/2020.12.15.422726 ◽

2020 ◽

Author(s):

Jonathan E Peelle ◽

Brent Spehar ◽

Michael S Jones ◽

Sarah McConkey ◽

Joel Myerson ◽

...

Keyword(s):

Visual Cortex ◽

Speech Perception ◽

Brain Activity ◽

Premotor Cortex ◽

Temporal Cortex ◽

Primary Auditory Cortex ◽

Visual Speech ◽

Auditory Signal ◽

Audiovisual Speech ◽

Audiovisual Speech Perception

In everyday conversation, we usually process the talker's face as well as the sound of their voice. Access to visual speech information is particularly useful when the auditory signal is degraded. Here we used fMRI to monitor brain activity while adults (n = 60) were presented with visual-only, auditory-only, and audiovisual words. As expected, audiovisual speech perception recruited both auditory and visual cortex, with a trend towards increased recruitment of premotor cortex in more difficult conditions (for example, in substantial background noise). We then investigated neural connectivity using psychophysiological interaction (PPI) analysis with seed regions in both primary auditory cortex and primary visual cortex. Connectivity between auditory and visual cortices was stronger in audiovisual conditions than in unimodal conditions, including a wide network of regions in posterior temporal cortex and prefrontal cortex. Taken together, our results suggest a prominent role for cross-region synchronization in understanding both visual-only and audiovisual speech.

Download Full-text

Sound Location Can Influence Audiovisual Speech Perception When Spatial Attention Is Manipulated

Seeing and Perceiving ◽

10.1163/187847511x557308 ◽

2011 ◽

Vol 24 (1) ◽

pp. 67-90 ◽

Cited By ~ 11

Author(s):

Riikka Möttönen ◽

Kaisa Tiippana ◽

Mikko Sams ◽

Hanna Puharinen

Keyword(s):

Speech Perception ◽

Spatial Attention ◽

Reaction Times ◽

Mcgurk Effect ◽

Visual Speech ◽

Audiovisual Speech ◽

Sound Location ◽

Audiovisual Speech Perception ◽

The Right ◽

Talking Face

AbstractAudiovisual speech perception has been considered to operate independent of sound location, since the McGurk effect (altered auditory speech perception caused by conflicting visual speech) has been shown to be unaffected by whether speech sounds are presented in the same or different location as a talking face. Here we show that sound location effects arise with manipulation of spatial attention. Sounds were presented from loudspeakers in five locations: the centre (location of the talking face) and 45°/90° to the left/right. Auditory spatial attention was focused on a location by presenting the majority (90%) of sounds from this location. In Experiment 1, the majority of sounds emanated from the centre, and the McGurk effect was enhanced there. In Experiment 2, the major location was 90° to the left, causing the McGurk effect to be stronger on the left and centre than on the right. Under control conditions, when sounds were presented with equal probability from all locations, the McGurk effect tended to be stronger for sounds emanating from the centre, but this tendency was not reliable. Additionally, reaction times were the shortest for a congruent audiovisual stimulus, and this was the case independent of location. Our main finding is that sound location can modulate audiovisual speech perception, and that spatial attention plays a role in this modulation.

Download Full-text

Hearing Lips and Seeing Voices: the Origins and Development of the ‘McGurk Effect’ and Reflections on Audio–Visual Speech Perception Over the Last 40 Years

Multisensory Research ◽

10.1163/22134808-00002548 ◽

2018 ◽

Vol 31 (1-2) ◽

pp. 7-18 ◽

Cited By ~ 3

Author(s):

John MacDonald

Keyword(s):

Speech Perception ◽

Visual Illusion ◽

Simultaneous Presentation ◽

Mcgurk Effect ◽

Visual Speech ◽

Audiovisual Speech ◽

Audiovisual Speech Perception ◽

Profound Impact ◽

Visual Speech Perception

In 1976 Harry McGurk and I published a paper in Nature, entitled ‘Hearing Lips and Seeing Voices’. The paper described a new audio–visual illusion we had discovered that showed the perception of auditorily presented speech could be influenced by the simultaneous presentation of incongruent visual speech. This hitherto unknown effect has since had a profound impact on audiovisual speech perception research. The phenomenon has come to be known as the ‘McGurk effect’, and the original paper has been cited in excess of 4800 times. In this paper I describe the background to the discovery of the effect, the rationale for the generation of the initial stimuli, the construction of the exemplars used and the serendipitous nature of the finding. The paper will also cover the reaction (and non-reaction) to the Nature publication, the growth of research on, and utilizing the ‘McGurk effect’ and end with some reflections on the significance of the finding.

Download Full-text

Audio-visual integration in noise: Influence of auditory and visual stimulus degradation on eye movements and perception of the McGurk effect

Attention Perception & Psychophysics ◽

10.3758/s13414-020-02042-x ◽

2020 ◽

Vol 82 (7) ◽

pp. 3544-3557 ◽

Cited By ~ 2

Author(s):

Jemaine E. Stacey ◽

Christina J. Howard ◽

Suvobrata Mitra ◽

Paula C. Stacey

Keyword(s):

Brain Research ◽

Mcgurk Effect ◽

Visual Signal ◽

Auditory Signal ◽

Clear Speech ◽

Hearing Research ◽

The Face ◽

Visual Degradation ◽

Auditory Noise ◽

Modality Preference

AbstractSeeing a talker’s face can aid audiovisual (AV) integration when speech is presented in noise. However, few studies have simultaneously manipulated auditory and visual degradation. We aimed to establish how degrading the auditory and visual signal affected AV integration. Where people look on the face in this context is also of interest; Buchan, Paré and Munhall (Brain Research, 1242, 162–171, 2008) found fixations on the mouth increased in the presence of auditory noise whilst Wilson, Alsius, Paré and Munhall (Journal of Speech, Language, and Hearing Research, 59(4), 601–615, 2016) found mouth fixations decreased with decreasing visual resolution. In Condition 1, participants listened to clear speech, and in Condition 2, participants listened to vocoded speech designed to simulate the information provided by a cochlear implant. Speech was presented in three levels of auditory noise and three levels of visual blurring. Adding noise to the auditory signal increased McGurk responses, while blurring the visual signal decreased McGurk responses. Participants fixated the mouth more on trials when the McGurk effect was perceived. Adding auditory noise led to people fixating the mouth more, while visual degradation led to people fixating the mouth less. Combined, the results suggest that modality preference and where people look during AV integration of incongruent syllables varies according to the quality of information available.

Download Full-text

Audiovisual Speech Perception

Perception ◽

10.1068/v970029 ◽

1997 ◽

Vol 26 (1_suppl) ◽

pp. 347-347

Author(s):

M Sams

Keyword(s):

Speech Perception ◽

Visual Information ◽

Word Meaning ◽

Source Area ◽

Mcgurk Effect ◽

Audiovisual Speech ◽

Audiovisual Speech Perception ◽

Processing Level ◽

Early Processing ◽

Audiovisual Fusion

Persons with hearing loss use visual information from articulation to improve their speech perception. Even persons with normal hearing utilise visual information, especially when the stimulus-to-noise ratio is poor. A dramatic demonstration of the role of vision in speech perception is the audiovisual fusion called the ‘McGurk effect’. When the auditory syllable /pa/ is presented in synchrony with the face articulating the syllable /ka/, the subject usually perceives /ta/ or /ka/. The illusory perception is clearly auditory in nature. We recently studied the audiovisual fusion (acoustical /p/, visual /k/) for Finnish (1) syllables, and (2) words. Only 3% of the subjects perceived the syllables according to the acoustical input, ie in 97% of the subjects the perception was influenced by the visual information. For words the percentage of acoustical identifications was 10%. The results demonstrate a very strong influence of visual information of articulation in face-to-face speech perception. Word meaning and sentence context have a negligible influence on the fusion. We have also recorded neuromagnetic responses of the human cortex when the subjects both heard and saw speech. Some subjects showed a distinct response to a ‘McGurk’ stimulus. The response was rather late, emerging about 200 ms from the onset of the auditory stimulus. We suggest that the perisylvian cortex, close to the source area for the auditory 100 ms response (M100), may be activated by the discordant stimuli. The behavioural and neuromagnetic results suggest a precognitive audiovisual speech integration occurring at a relatively early processing level.

Download Full-text

Neural Correlates of Modality-Sensitive Deviance Detection in the Audiovisual Oddball Paradigm

Brain Sciences ◽

10.3390/brainsci10060328 ◽

2020 ◽

Vol 10 (6) ◽

pp. 328

Author(s):

Melissa Randazzo ◽

Ryan Priefer ◽

Paul J. Smith ◽

Amanda Nagler ◽

Trey Avery ◽

...

Keyword(s):

Speech Perception ◽

Visual Information ◽

Time Window ◽

Mcgurk Effect ◽

Predictive Processing ◽

Visual Signal ◽

Oddball Paradigm ◽

Audiovisual Speech ◽

Late Time ◽

Audiovisual Speech Perception

The McGurk effect, an incongruent pairing of visual /ga/–acoustic /ba/, creates a fusion illusion /da/ and is the cornerstone of research in audiovisual speech perception. Combination illusions occur given reversal of the input modalities—auditory /ga/-visual /ba/, and percept /bga/. A robust literature shows that fusion illusions in an oddball paradigm evoke a mismatch negativity (MMN) in the auditory cortex, in absence of changes to acoustic stimuli. We compared fusion and combination illusions in a passive oddball paradigm to further examine the influence of visual and auditory aspects of incongruent speech stimuli on the audiovisual MMN. Participants viewed videos under two audiovisual illusion conditions: fusion with visual aspect of the stimulus changing, and combination with auditory aspect of the stimulus changing, as well as two unimodal auditory- and visual-only conditions. Fusion and combination deviants exerted similar influence in generating congruency predictions with significant differences between standards and deviants in the N100 time window. Presence of the MMN in early and late time windows differentiated fusion from combination deviants. When the visual signal changes, a new percept is created, but when the visual is held constant and the auditory changes, the response is suppressed, evoking a later MMN. In alignment with models of predictive processing in audiovisual speech perception, we interpreted our results to indicate that visual information can both predict and suppress auditory speech perception.

Download Full-text

Early and late beta-band power reflect audiovisual perception in the McGurk illusion

Journal of Neurophysiology ◽

10.1152/jn.00783.2014 ◽

2015 ◽

Vol 113 (7) ◽

pp. 2342-2350 ◽

Cited By ~ 21

Author(s):

Yadira Roa Romero ◽

Daniel Senkowski ◽

Julian Keil

Keyword(s):

Auditory Perception ◽

Audiovisual Integration ◽

Visual Speech ◽

Audiovisual Speech ◽

Beta Band ◽

Audiovisual Speech Perception ◽

Subsequent Formation ◽

Band Power ◽

Mcgurk Illusion ◽

Audiovisual Stimuli

The McGurk illusion is a prominent example of audiovisual speech perception and the influence that visual stimuli can have on auditory perception. In this illusion, a visual speech stimulus influences the perception of an incongruent auditory stimulus, resulting in a fused novel percept. In this high-density electroencephalography (EEG) study, we were interested in the neural signatures of the subjective percept of the McGurk illusion as a phenomenon of speech-specific multisensory integration. Therefore, we examined the role of cortical oscillations and event-related responses in the perception of congruent and incongruent audiovisual speech. We compared the cortical activity elicited by objectively congruent syllables with incongruent audiovisual stimuli. Importantly, the latter elicited a subjectively congruent percept: the McGurk illusion. We found that early event-related responses (N1) to audiovisual stimuli were reduced during the perception of the McGurk illusion compared with congruent stimuli. Most interestingly, our study showed a stronger poststimulus suppression of beta-band power (13–30 Hz) at short (0–500 ms) and long (500–800 ms) latencies during the perception of the McGurk illusion compared with congruent stimuli. Our study demonstrates that auditory perception is influenced by visual context and that the subsequent formation of a McGurk illusion requires stronger audiovisual integration even at early processing stages. Our results provide evidence that beta-band suppression at early stages reflects stronger stimulus processing in the McGurk illusion. Moreover, stronger late beta-band suppression in McGurk illusion indicates the resolution of incongruent physical audiovisual input and the formation of a coherent, illusory multisensory percept.

Download Full-text