Method and apparatus for audio-visual speech detection and recognition

Purpose The purpose of this article is to examine explanations for pure-tone average–spondee threshold differences in functional hearing loss. Method Loudness magnitude estimation functions were obtained from 24 participants for pure tones (0.5 and 1.0 kHz), vowels, spondees, and speech-shaped noise as a function of level (20–90 dB SPL). Participants listened monaurally through earphones. Loudness predictions were obtained for the same stimuli by using a computational, dynamic loudness model. Results When evaluated at the same SPL, speech-shaped noise was judged louder than vowels/spondees, which were judged louder than tones. Equal-loudness levels were inferred from fitted loudness functions for the group. For the clinical application, the 2.1-dB difference between spondees and tones at equal loudness became a 12.1-dB difference when the stimuli were converted from SPL to HL. Conclusions Nearly all of the pure-tone average–spondee threshold differences in functional hearing loss are attributable to references for calibration for 0 dB HL for tones and speech, which are based on detection and recognition, respectively. The recognition threshold for spondees is roughly 9 dB higher than the speech detection threshold; persons feigning a loss, who base loss magnitude on loudness, do not consider this difference. Furthermore, the dynamic loudness model was more accurate than the static model.

Download Full-text

Neural oscillations in the temporal pole for a temporally congruent audio-visual speech detection task

Scientific Reports ◽

10.1038/srep37973 ◽

2016 ◽

Vol 6 (1) ◽

Cited By ~ 10

Author(s):

Takefumi Ohki ◽

Atsuko Gunji ◽

Yuichi Takei ◽

Hidetoshi Takahashi ◽

Yuu Kaneko ◽

...

Keyword(s):

Detection Task ◽

Visual Speech ◽

Neural Oscillations ◽

Speech Detection ◽

Temporal Pole

Download Full-text

Integration of Audio video Speech Recognition using LSTM and Feed Forward Convolutional Neural Network

10.21203/rs.3.rs-173380/v1 ◽

2021 ◽

Author(s):

Shashidhar R ◽

Sudarshan Patil Kulkarni

Keyword(s):

Neural Network ◽

Speech Recognition ◽

Good Accuracy ◽

Visual Speech ◽

Feed Forward Neural Network ◽

Final Model ◽

Feed Forward ◽

Speech Detection ◽

Visual Speech Recognition ◽

Lip Reading

Abstract In the current scenario, audio visual speech recognition is one of the emerging fields of research, but there is still deficiency of appropriate visual features for recognition of visual speech. Human lip-readers are increasingly being presented as useful in the gathering of forensic evidence but, like all human, suffer from unreliability in analyzing the lip movement. Here we used a custom dataset and design the system in such a way that it predicts the output for the lip reading. The problem of speaker independent lip-reading is very demanding due to unpredictable variations between people. Also due to recent developments and advances in the fields of signal processing and computer vision. The task of automating the lip reading is becoming a field of great interest. Here we use MFCC techniques for audio processing and LSTM method for visual speech recognition and finally integrate the audio and video using feed forward neural network (FFNN) and also got good accuracy. That is why the AVSR technique attract a great attention as a reliable solution for the speech detection problem. The final model was capable of taking more appropriate decision while predicting the spoken word. We were able to get a good accuracy of about 92.38% for the final model.

Download Full-text

Investigating the audio–visual speech detection advantage

Speech Communication ◽

10.1016/j.specom.2004.09.008 ◽

2004 ◽

Vol 44 (1-4) ◽

pp. 19-30 ◽

Cited By ~ 36

Author(s):

Jeesun Kim ◽

Chris Davis

Keyword(s):

Visual Speech ◽

Speech Detection

Download Full-text

Infants and Adults Use Visual Cues to Improve Detection and Discrimination of Speech in Noise

Journal of Speech Language and Hearing Research ◽

10.1044/2019_jslhr-h-19-0106 ◽

2019 ◽

Vol 62 (10) ◽

pp. 3860-3875 ◽

Cited By ~ 3

Author(s):

Kaylah Lalonde ◽

Lynne A. Werner

Keyword(s):

Visual Cues ◽

Visual Speech ◽

Visual Signal ◽

Auditory Signal ◽

Speech Discrimination ◽

Linear Modeling ◽

Speech Detection ◽

Speech In Noise ◽

Speech Cues ◽

Visual Onset

Purpose This study assessed the extent to which 6- to 8.5-month-old infants and 18- to 30-year-old adults detect and discriminate auditory syllables in noise better in the presence of visual speech than in auditory-only conditions. In addition, we examined whether visual cues to the onset and offset of the auditory signal account for this benefit. Method Sixty infants and 24 adults were randomly assigned to speech detection or discrimination tasks and were tested using a modified observer-based psychoacoustic procedure. Each participant completed 1–3 conditions: auditory-only, with visual speech, and with a visual signal that only cued the onset and offset of the auditory syllable. Results Mixed linear modeling indicated that infants and adults benefited from visual speech on both tasks. Adults relied on the onset–offset cue for detection, but the same cue did not improve their discrimination. The onset–offset cue benefited infants for both detection and discrimination. Whereas the onset–offset cue improved detection similarly for infants and adults, the full visual speech signal benefited infants to a lesser extent than adults on the discrimination task. Conclusions These results suggest that infants' use of visual onset–offset cues is mature, but their ability to use more complex visual speech cues is still developing. Additional research is needed to explore differences in audiovisual enhancement (a) of speech discrimination across speech targets and (b) with increasingly complex tasks and stimuli.

Download Full-text