scholarly journals Audio Visual Integration with Competing Sources in the Framework of Audio Visual Speech Scene Analysis

2016 ◽  
pp. 399-408 ◽  
Author(s):  
Attigodu Chandrashekara Ganesh ◽  
Frédéric Berthommier ◽  
Jean-Luc Schwartz
Keyword(s):  
Visual Speech ◽  
Scene Analysis ◽  
Visual Integration

Psychologically-Inspired Audio-Visual Speech Recognition Using Coarse Speech Recognition and Missing Feature Theory

2017 ◽  
Vol 29 (1) ◽  
pp. 105-113 ◽  
Author(s):  
Kazuhiro Nakadai ◽  
◽  
Tomoaki Koiwa ◽  
Keyword(s):  
Speech Recognition ◽  
Real World ◽  
Visual Speech ◽  
Noise Robustness ◽  
Visual Integration ◽  
The Real ◽  
Visual Speech Recognition ◽  
Missing Feature Theory ◽  
Missing Feature ◽  
Feature Theory

[abstFig src='/00290001/10.jpg' width='300' text='System architecture of AVSR based on missing feature theory and P-V grouping' ] Audio-visual speech recognition (AVSR) is a promising approach to improving the noise robustness of speech recognition in the real world. For AVSR, the auditory and visual units are the phoneme and viseme, respectively. However, these are often misclassified in the real world because of noisy input. To solve this problem, we propose two psychologically-inspired approaches. One is audio-visual integration based on missing feature theory (MFT) to cope with missing or unreliable audio and visual features for recognition. The other is phoneme and viseme grouping based on coarse-to-fine recognition. Preliminary experiments show that these two approaches are effective for audio-visual speech recognition. Integration based on MFT with an appropriate weight improves the recognition performance by −5 dB. This is the case even in a noisy environment, in which most speech recognition systems do not work properly. Phoneme and viseme grouping further improved the AVSR performance, particularly at a low signal-to-noise ratio.**This work is an extension of our publication “Tomoaki Koiwa et al.: Coarse speech recognition by audio-visual integration based on missing feature theory, IROS 2007, pp.1751-1756, 2007.”

scholarly journals Visual Speech Benefit in Clear and Degraded Speech Depends on the Auditory Intelligibility of the Talker and the Number of Background Talkers

2019 ◽  
Vol 23 ◽  
pp. 233121651983786 ◽  
Author(s):  
Catherine L. Blackburn ◽  
Pádraig T. Kitterick ◽  
Gary Jones ◽  
Christian J. Sumner ◽  
Paula C. Stacey
Keyword(s):  
Speech Intelligibility ◽  
Noise Signal ◽  
Sine Wave ◽  
Theory Model ◽  
Visual Speech ◽  
Clear Speech ◽  
Visual Integration ◽  
Degraded Speech ◽  
The Face ◽  
Vocoded Speech

Perceiving speech in background noise presents a significant challenge to listeners. Intelligibility can be improved by seeing the face of a talker. This is of particular value to hearing impaired people and users of cochlear implants. It is well known that auditory-only speech understanding depends on factors beyond audibility. How these factors impact on the audio-visual integration of speech is poorly understood. We investigated audio-visual integration when either the interfering background speech (Experiment 1) or intelligibility of the target talkers (Experiment 2) was manipulated. Clear speech was also contrasted with sine-wave vocoded speech to mimic the loss of temporal fine structure with a cochlear implant. Experiment 1 showed that for clear speech, the visual speech benefit was unaffected by the number of background talkers. For vocoded speech, a larger benefit was found when there was only one background talker. Experiment 2 showed that visual speech benefit depended upon the audio intelligibility of the talker and increased as intelligibility decreased. Degrading the speech by vocoding resulted in even greater benefit from visual speech information. A single “independent noise” signal detection theory model predicted the overall visual speech benefit in some conditions but could not predict the different levels of benefit across variations in the background or target talkers. This suggests that, similar to audio-only speech intelligibility, the integration of audio-visual speech cues may be functionally dependent on factors other than audibility and task difficulty, and that clinicians and researchers should carefully consider the characteristics of their stimuli when assessing audio-visual integration.

scholarly journals Audio-visual speech scene analysis: Characterization of the dynamics of unbinding and rebinding the McGurk effect

2015 ◽  
Vol 137 (1) ◽  
pp. 362-377 ◽  
Author(s):  
Olha Nahorna ◽  
Frédéric Berthommier ◽  
Jean-Luc Schwartz
Keyword(s):  
Mcgurk Effect ◽  
Visual Speech ◽  
Scene Analysis

Lipreading and audio-visual speech perception

1992 ◽  
Vol 335 (1273) ◽  
pp. 71-78 ◽  
Keyword(s):  
Speech Perception ◽  
Visual Cues ◽  
Visual Speech ◽  
Speech Signals ◽  
Visual Integration ◽  
Computer Animations ◽  
Visual Speech Perception

This paper reviews progress in understanding the psychology of lipreading and audio-visual speech perception. It considers four questions. What distinguishes better from poorer lipreaders? What are the effects of introducing a delay between the acoustical and optical speech signals? What have attempts to produce computer animations of talking faces contributed to our understanding of the visual cues that distinguish consonants and vowels? Finally, how should the process of audio-visual integration in speech perception be described; that is, how are the sights and sounds of talking faces represented at their conflux?

Task Complexity Alters Proprioceptive-Visual Integration

2013 ◽  
Author(s):  
Christine Sutter ◽  
Jochen Musseler
Keyword(s):  
Task Complexity ◽  
Visual Integration

Analysis of Roofing Materials for Rural Buildings Projects: Costs and Visual Integration

2016 ◽  
Vol 2 (09) ◽  
pp. 587-592
Author(s):  
Antonio M. Reyes-Rodríguez ◽  
Justo García-Sanz-Calcedo ◽  
Lorenzo García-Moruno ◽  
Julio Hernández-Blanco
Keyword(s):  
Visual Integration ◽  
Rural Buildings ◽  
Roofing Materials

Scene Analysis from Viewing Orientations in a Shooting Environment of Multiple Mobile Phones

2014 ◽  
Vol E97.A (11) ◽  
pp. 2178-2180
Author(s):  
Shogo TOKAI ◽  
Takayoshi MORIOKA ◽  
Hiroyuki HASE
Keyword(s):  
Mobile Phones ◽  
Scene Analysis
Sign in / Sign up

Export Citation Format

Share Document