Visual speech improves the intelligibility of time-expanded auditory speech

Neuroreport ◽  
2009 ◽  
Vol 20 (5) ◽  
pp. 473-477 ◽  
Author(s):  
Akihiro Tanaka ◽  
Shuichi Sakamoto ◽  
Komi Tsumura ◽  
Yôiti Suzuki
Keyword(s):  
Visual Speech ◽  
Auditory Speech

scholarly journals Children perceive speech onsets by ear and eye

2016 ◽  
Vol 44 (1) ◽  
pp. 185-215 ◽  
Author(s):  
SUSAN JERGER ◽  
MARKUS F. DAMIAN ◽  
NANCY TYE-MURRAY ◽  
HERVÉ ABDI
Keyword(s):  
Speech Perception ◽  
Growth Period ◽  
Visual Speech ◽  
Cognitive Resources ◽  
Audiovisual Speech ◽  
Auditory Input ◽  
Phonological Priming ◽  
Complex Tasks ◽  
Auditory Speech ◽  
Developmental Theories

AbstractAdults use vision to perceive low-fidelity speech; yet how children acquire this ability is not well understood. The literature indicates that children show reduced sensitivity to visual speech from kindergarten to adolescence. We hypothesized that this pattern reflects the effects of complex tasks and a growth period with harder-to-utilize cognitive resources, not lack of sensitivity. We investigated sensitivity to visual speech in children via the phonological priming produced by low-fidelity (non-intact onset) auditory speech presented audiovisually (see dynamic face articulate consonant/rhyme b/ag; hear non-intact onset/rhyme: –b/ag) vs. auditorily (see still face; hear exactly same auditory input). Audiovisual speech produced greater priming from four to fourteen years, indicating that visual speech filled in the non-intact auditory onsets. The influence of visual speech depended uniquely on phonology and speechreading. Children – like adults – perceive speech onsets multimodally. Findings are critical for incorporating visual speech into developmental theories of speech perception.

Visual speech recalibrates auditory speech identification

2002 ◽  
Vol 112 (5) ◽  
pp. 2245-2245
Author(s):  
Paul Bertelson ◽  
Jean Vroomen ◽  
Beatrice de Gelder
Keyword(s):  
Visual Speech ◽  
Auditory Speech ◽  
Speech Identification

scholarly journals Linking Activity in Human Superior Temporal Cortex to Perception of Noisy Audiovisual Speech

2020 ◽  
Author(s):  
Johannes Rennig ◽  
Michael S Beauchamp
Keyword(s):  
Univariate Analysis ◽  
Temporal Cortex ◽  
Visual Speech ◽  
Button Press ◽  
Auditory Information ◽  
Bold Response ◽  
Auditory Speech ◽  
Auditory Component ◽  
Visual Speech Information ◽  
Speech Information

AbstractRegions of the human posterior superior temporal gyrus and sulcus (pSTG/S) respond to the visual mouth movements that constitute visual speech and the auditory vocalizations that constitute auditory speech. We hypothesized that these multisensory responses in pSTG/S underlie the observation that comprehension of noisy auditory speech is improved when it is accompanied by visual speech. To test this idea, we presented audiovisual sentences that contained either a clear auditory component or a noisy auditory component while measuring brain activity using BOLD fMRI. Participants reported the intelligibility of the speech on each trial with a button press. Perceptually, adding visual speech to noisy auditory sentences rendered them much more intelligible. Post-hoc trial sorting was used to examine brain activations during noisy sentences that were more or less intelligible, focusing on multisensory speech regions in the pSTG/S identified with an independent visual speech localizer. Univariate analysis showed that less intelligible noisy audiovisual sentences evoked a weaker BOLD response, while more intelligible sentences evoked a stronger BOLD response that was indistinguishable from clear sentences. To better understand these differences, we conducted a multivariate representational similarity analysis. The pattern of response for intelligible noisy audiovisual sentences was more similar to the pattern for clear sentences, while the response pattern for unintelligible noisy sentences was less similar. These results show that for both univariate and multivariate analyses, successful integration of visual and noisy auditory speech normalizes responses in pSTG/S, providing evidence that multisensory subregions of pSTG/S are responsible for the perceptual benefit of visual speech.Significance StatementEnabling social interactions, including the production and perception of speech, is a key function of the human brain. Speech perception is a complex computational problem that the brain solves using both visual information from the talker’s facial movements and auditory information from the talker’s voice. Visual speech information is particularly important under noisy listening conditions when auditory speech is difficult or impossible to understand alone Regions of the human cortex in posterior superior temporal lobe respond to the visual mouth movements that constitute visual speech and the auditory vocalizations that constitute auditory speech. We show that the pattern of activity in cortex reflects the successful multisensory integration of auditory and visual speech information in the service of perception.

scholarly journals Lip movements entrain the observers’ low-frequency brain oscillations to facilitate speech intelligibility

eLife ◽  
2016 ◽  
Vol 5 ◽  
Author(s):  
Hyojin Park ◽  
Christoph Kayser ◽  
Gregor Thut ◽  
Joachim Gross
Keyword(s):  
Visual Cortex ◽  
Speech Processing ◽  
Speech Intelligibility ◽  
Brain Activity ◽  
Low Frequency ◽  
Visual Speech ◽  
Visual Signals ◽  
Partial Coherence ◽  
Auditory Speech ◽  
Oscillatory Brain Activity

During continuous speech, lip movements provide visual temporal signals that facilitate speech processing. Here, using MEG we directly investigated how these visual signals interact with rhythmic brain activity in participants listening to and seeing the speaker. First, we investigated coherence between oscillatory brain activity and speaker’s lip movements and demonstrated significant entrainment in visual cortex. We then used partial coherence to remove contributions of the coherent auditory speech signal from the lip-brain coherence. Comparing this synchronization between different attention conditions revealed that attending visual speech enhances the coherence between activity in visual cortex and the speaker’s lips. Further, we identified a significant partial coherence between left motor cortex and lip movements and this partial coherence directly predicted comprehension accuracy. Our results emphasize the importance of visually entrained and attention-modulated rhythmic brain activity for the enhancement of audiovisual speech processing.

scholarly journals Vision perceptually restores auditory spectral dynamics in speech

2020 ◽  
Vol 117 (29) ◽  
pp. 16920-16927 ◽  
Author(s):  
John Plass ◽  
David Brang ◽  
Satoru Suzuki ◽  
Marcia Grabowecky
Keyword(s):  
Visual Cues ◽  
Perceptual System ◽  
Visual Speech ◽  
Auditory Information ◽  
Speech Comprehension ◽  
Facilitation Effect ◽  
Time Frequency ◽  
Auditory Speech ◽  
High Level ◽  
Frequency Modulations

Visual speech facilitates auditory speech perception, but the visual cues responsible for these benefits and the information they provide remain unclear. Low-level models emphasize basic temporal cues provided by mouth movements, but these impoverished signals may not fully account for the richness of auditory information provided by visual speech. High-level models posit interactions among abstract categorical (i.e., phonemes/visemes) or amodal (e.g., articulatory) speech representations, but require lossy remapping of speech signals onto abstracted representations. Because visible articulators shape the spectral content of speech, we hypothesized that the perceptual system might exploit natural correlations between midlevel visual (oral deformations) and auditory speech features (frequency modulations) to extract detailed spectrotemporal information from visual speech without employing high-level abstractions. Consistent with this hypothesis, we found that the time–frequency dynamics of oral resonances (formants) could be predicted with unexpectedly high precision from the changing shape of the mouth during speech. When isolated from other speech cues, speech-based shape deformations improved perceptual sensitivity for corresponding frequency modulations, suggesting that listeners could exploit this cross-modal correspondence to facilitate perception. To test whether this type of correspondence could improve speech comprehension, we selectively degraded the spectral or temporal dimensions of auditory sentence spectrograms to assess how well visual speech facilitated comprehension under each degradation condition. Visual speech produced drastically larger enhancements during spectral degradation, suggesting a condition-specific facilitation effect driven by cross-modal recovery of auditory speech spectra. The perceptual system may therefore use audiovisual correlations rooted in oral acoustics to extract detailed spectrotemporal information from visual speech.

scholarly journals Visual speech fills in both discrimination and identification of non-intact auditory speech in children

2017 ◽  
Vol 45 (2) ◽  
pp. 392-414 ◽  
Author(s):  
SUSAN JERGER ◽  
MARKUS F. DAMIAN ◽  
RACHEL P. MCALPINE ◽  
HERVÉ ABDI
Keyword(s):  
Receptive Vocabulary ◽  
Visual Speech ◽  
Speech Sounds ◽  
Auditory Speech ◽  
Vocabulary Skills ◽  
Auditory Mode ◽  
Phoneme Discrimination

AbstractTo communicate, children must discriminate and identify speech sounds. Because visual speech plays an important role in this process, we explored how visual speech influences phoneme discrimination and identification by children. Critical items had intact visual speech (e.g.bæz) coupled to non-intact (excised onsets) auditory speech (signified by /–b/æz). Children discriminated syllable pairs that differed in intactness (i.e. bæz:/–b/æz) and identified non-intact nonwords (/–b/æz). We predicted that visual speech would cause children to perceive the non-intact onsets as intact, resulting in moresameresponses for discrimination and more intact (i.e.bæz) responses for identification in the audiovisual than auditory mode. Visual speech for the easy-to-speechread /b/ but not for the difficult-to-speechread /g/ boosted discrimination and identification (about 35–45%) in children from four to fourteen years. The influence of visual speech on discrimination was uniquely associated with the influence of visual speech on identification and receptive vocabulary skills.

scholarly journals Author response: The visual speech head start improves perception and reduces superior temporal cortex responses to auditory speech

2019 ◽  
Author(s):  
Patrick J Karas ◽  
John F Magnotti ◽  
Brian A Metzger ◽  
Lin L Zhu ◽  
Kristen B Smith ◽  
...  
Keyword(s):  
Head Start ◽  
Temporal Cortex ◽  
Visual Speech ◽  
Author Response ◽  
Auditory Speech ◽  
Superior Temporal Cortex

scholarly journals Children use visual speech to compensate for non-intact auditory speech

2014 ◽  
Vol 126 ◽  
pp. 295-312 ◽  
Author(s):  
Susan Jerger ◽  
Markus F. Damian ◽  
Nancy Tye-Murray ◽  
Hervé Abdi
Keyword(s):  
Visual Speech ◽  
Auditory Speech
Sign in / Sign up

Export Citation Format

Share Document