scholarly journals Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect

2015 ◽  
Vol 137 (1) ◽  
pp. 362-377 ◽  
Author(s):  
Olha Nahorna ◽  
Frédéric Berthommier ◽  
Jean-Luc Schwartz
Keyword(s):  
Mcgurk Effect ◽  
Visual Speech ◽  
Scene Analysis

scholarly journals Visually-biased perception in cochlear implant users: a study of the McGurk and sound-induced flash illusions

2021 ◽  
Author(s):  
Iliza M Butera ◽  
Ryan A Stevenson ◽  
René H Gifford ◽  
Mark T Wallace
Keyword(s):  
Cochlear Implant ◽  
Outcome Measures ◽  
Audiovisual Integration ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Speech Comprehension ◽  
Speech Stimuli ◽  
Speech Cues ◽  
Biased Perception

The reduction in spectral resolution by cochlear implants oftentimes requires complementary visual speech cues to aid in understanding. Despite substantial clinical characterization of auditory-only speech outcome measures, relatively little is known about the audiovisual integrative abilities that most cochlear implant (CI) users rely on for daily speech comprehension. In this study, we tested audiovisual integration in 63 CI users and 69 normal-hearing (NH) controls using the McGurk and sound-induced flash illusions. This study is the largest to-date measuring the McGurk effect in this population and the first to test the sound-induced flash illusion. When presented with conflicting audiovisual speech stimuli (i.e., the phoneme "ba" dubbed onto the viseme "ga"), we found that 55 CI users (87%) reported a fused percept of "da" or "tha" on at least one trial. However, overall, we found that CI users experienced the McGurk effect less often than controls--a result that was concordant with results with the sound-induced flash illusion where the pairing of a single circle flashing on the screen with multiple beeps resulted in fewer illusory flashes for CI users. While illusion perception in these two tasks appears to be uncorrelated among CI users, we identified a negative correlation in the NH group. Because neither illusion appears to provide further explanation of variability in CI outcome measures, further research is needed to determine how these findings relate to CI users' speech understanding, particularly in ecological listening conditions that are naturally multisensory.

The McGurk Effect: Auditory Visual Speech Perception’s Piltdown Man

2017 ◽  
Author(s):  
Dominic Massaro
Keyword(s):  
Mcgurk Effect ◽  
Visual Speech

scholarly journals Animated virtual characters to explore audio-visual speech in controlled and naturalistic environments

2020 ◽  
Vol 10 (1) ◽  
Author(s):  
Raphaël Thézé ◽  
Mehdi Ali Gadiri ◽  
Louis Albert ◽  
Antoine Provost ◽  
Anne-Lise Giraud ◽  
...  
Keyword(s):  
Speech Processing ◽  
Background Noise ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Natural Speech ◽  
Virtual Characters ◽  
Speech Stimuli ◽  
Stimulus Timing ◽  
Phonetic Features ◽  
Set Up

Abstract Natural speech is processed in the brain as a mixture of auditory and visual features. An example of the importance of visual speech is the McGurk effect and related perceptual illusions that result from mismatching auditory and visual syllables. Although the McGurk effect has widely been applied to the exploration of audio-visual speech processing, it relies on isolated syllables, which severely limits the conclusions that can be drawn from the paradigm. In addition, the extreme variability and the quality of the stimuli usually employed prevents comparability across studies. To overcome these limitations, we present an innovative methodology using 3D virtual characters with realistic lip movements synchronized on computer-synthesized speech. We used commercially accessible and affordable tools to facilitate reproducibility and comparability, and the set-up was validated on 24 participants performing a perception task. Within complete and meaningful French sentences, we paired a labiodental fricative viseme (i.e. /v/) with a bilabial occlusive phoneme (i.e. /b/). This audiovisual mismatch is known to induce the illusion of hearing /v/ in a proportion of trials. We tested the rate of the illusion while varying the magnitude of background noise and audiovisual lag. Overall, the effect was observed in 40% of trials. The proportion rose to about 50% with added background noise and up to 66% when controlling for phonetic features. Our results conclusively demonstrate that computer-generated speech stimuli are judicious, and that they can supplement natural speech with higher control over stimulus timing and content.

scholarly journals Auditory–Visual Speech Integration in Bipolar Disorder: A Preliminary Study

Languages ◽  
2018 ◽  
Vol 3 (4) ◽  
pp. 38 ◽  
Author(s):  
Arzu Yordamlı ◽  
Doğu Erdener
Keyword(s):  
Bipolar Disorder ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Physiological Data ◽  
Striking Difference ◽  
Control Group ◽  
Speech Stimuli ◽  
Visual Speech Information ◽  
Speech Information ◽  
And Control

This study aimed to investigate how individuals with bipolar disorder integrate auditory and visual speech information compared to healthy individuals. Furthermore, we wanted to see whether there were any differences between manic and depressive episode bipolar disorder patients with respect to auditory and visual speech integration. It was hypothesized that the bipolar group’s auditory–visual speech integration would be weaker than that of the control group. Further, it was predicted that those in the manic phase of bipolar disorder would integrate visual speech information more robustly than their depressive phase counterparts. To examine these predictions, a McGurk effect paradigm with an identification task was used with typical auditory–visual (AV) speech stimuli. Additionally, auditory-only (AO) and visual-only (VO, lip-reading) speech perceptions were also tested. The dependent variable for the AV stimuli was the amount of visual speech influence. The dependent variables for AO and VO stimuli were accurate modality-based responses. Results showed that the disordered and control groups did not differ in AV speech integration and AO speech perception. However, there was a striking difference in favour of the healthy group with respect to the VO stimuli. The results suggest the need for further research whereby both behavioural and physiological data are collected simultaneously. This will help us understand the full dynamics of how auditory and visual speech information are integrated in people with bipolar disorder.

Auditory-visual speech integration by prelinguistic infants: Perception of an emergent consonant in the McGurk effect

2004 ◽  
Vol 45 (4) ◽  
pp. 204-220 ◽  
Author(s):  
Denis Burnham ◽  
Barbara Dodd
Keyword(s):  
Mcgurk Effect ◽  
Visual Speech

Sound Location Can Influence Audiovisual Speech Perception When Spatial Attention Is Manipulated

2011 ◽  
Vol 24 (1) ◽  
pp. 67-90 ◽  
Author(s):  
Riikka Möttönen ◽  
Kaisa Tiippana ◽  
Mikko Sams ◽  
Hanna Puharinen
Keyword(s):  
Speech Perception ◽  
Spatial Attention ◽  
Reaction Times ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Audiovisual Speech ◽  
Sound Location ◽  
Audiovisual Speech Perception ◽  
The Right ◽  
Talking Face

AbstractAudiovisual speech perception has been considered to operate independent of sound location, since the McGurk effect (altered auditory speech perception caused by conflicting visual speech) has been shown to be unaffected by whether speech sounds are presented in the same or different location as a talking face. Here we show that sound location effects arise with manipulation of spatial attention. Sounds were presented from loudspeakers in five locations: the centre (location of the talking face) and 45°/90° to the left/right. Auditory spatial attention was focused on a location by presenting the majority (90%) of sounds from this location. In Experiment 1, the majority of sounds emanated from the centre, and the McGurk effect was enhanced there. In Experiment 2, the major location was 90° to the left, causing the McGurk effect to be stronger on the left and centre than on the right. Under control conditions, when sounds were presented with equal probability from all locations, the McGurk effect tended to be stronger for sounds emanating from the centre, but this tendency was not reliable. Additionally, reaction times were the shortest for a congruent audiovisual stimulus, and this was the case independent of location. Our main finding is that sound location can modulate audiovisual speech perception, and that spatial attention plays a role in this modulation.

Reducing Playback Rate of Audiovisual Speech Leads to a Surprising Decrease in the McGurk Effect

2018 ◽  
Vol 31 (1-2) ◽  
pp. 19-38 ◽  
Author(s):  
John F. Magnotti ◽  
Debshila Basu Mallick ◽  
Michael S. Beauchamp
Keyword(s):  
Visual Information ◽  
Mcgurk Effect ◽  
Visual Speech ◽  
Large Individual ◽  
Audiovisual Speech ◽  
Unexpected Finding ◽  
Video Playback ◽  
Natural Rate ◽  
Audiovisual Speech Perception ◽  
Bayesian Integration

We report the unexpected finding that slowing video playback decreases perception of the McGurk effect. This reduction is counter-intuitive because the illusion depends on visual speech influencing the perception of auditory speech, and slowing speech should increase the amount of visual information available to observers. We recorded perceptual data from 110 subjects viewing audiovisual syllables (either McGurk or congruent control stimuli) played back at one of three rates: the rate used by the talker during recording (the natural rate), a slow rate (50% of natural), or a fast rate (200% of natural). We replicated previous studies showing dramatic variability in McGurk susceptibility at the natural rate, ranging from 0–100% across subjects and from 26–76% across the eight McGurk stimuli tested. Relative to the natural rate, slowed playback reduced the frequency of McGurk responses by 11% (79% of subjects showed a reduction) and reduced congruent accuracy by 3% (25% of subjects showed a reduction). Fast playback rate had little effect on McGurk responses or congruent accuracy. To determine whether our results are consistent with Bayesian integration, we constructed a Bayes-optimal model that incorporated two assumptions: individuals combine auditory and visual information according to their reliability, and changing playback rate affects sensory reliability. The model reproduced both our findings of large individual differences and the playback rate effect. This work illustrates that surprises remain in the McGurk effect and that Bayesian integration provides a useful framework for understanding audiovisual speech perception.

Sign in / Sign up

Export Citation Format

Share Document