scholarly journals Linguistic constraints modulate speech timing in an oscillating neural network

2020 ◽  
Author(s):  
Sanne Ten Oever ◽  
Andrea E. Martin
Keyword(s):  
Sensory Processing ◽  
Speech Processing ◽  
Language Models ◽  
Speech Input ◽  
Speech Timing ◽  
Temporal Phase ◽  
Word Onset ◽  
Knowledge Of Language ◽  
Linguistic Constraints ◽  
Onset Delays

AbstractNeuronal oscillations putatively track speech in order to optimize sensory processing. However, it is unclear how isochronous brain oscillations can track pseudo-rhythmic speech input. Here we investigate how top-down predictions flowing from internal language models interact with oscillations during speech processing. We show that word-to-word onset delays are shorter when words are spoken in predictable contexts. A computational model including oscillations, feedback, and inhibition is able to track the natural pseudo-rhythmic word-to-word onset differences. As the model processes, it generates temporal phase codes, which are a candidate mechanism for carrying information forward in time in the system. Intriguingly, the model’s response is more rhythmic for non-isochronous compared to isochronous speech when onset times are proportional to predictions from the internal model. These results show that oscillatory tracking of temporal speech dynamics relies not only on the input acoustics, but also on the linguistic constraints flowing from knowledge of language.

scholarly journals An oscillating computational model can track pseudo-rhythmic speech by using linguistic predictions

eLife ◽  
2021 ◽  
Vol 10 ◽  
Author(s):  
Sanne ten Oever ◽  
Andrea E Martin
Keyword(s):  
Computational Model ◽  
Sensory Processing ◽  
Empirical Data ◽  
Temporal Dynamics ◽  
Language Models ◽  
Brain Oscillations ◽  
Neuronal Oscillations ◽  
Speech Input ◽  
Temporal Phase ◽  
Speech Time

Neuronal oscillations putatively track speech in order to optimize sensory processing. However, it is unclear how isochronous brain oscillations can track pseudo-rhythmic speech input. Here we propose that oscillations can track pseudo-rhythmic speech when considering that speech time is dependent on content-based predictions flowing from internal language models. We show that temporal dynamics of speech are dependent on the predictability of words in a sentence. A computational model including oscillations, feedback, and inhibition is able to track pseudo-rhythmic speech input. As the model processes, it generates temporal phase codes, which are a candidate mechanism for carrying information forward in time. The model is optimally sensitive to the natural temporal speech dynamics and can explain empirical data on temporal speech illusions. Our results suggest that speech tracking does not have to rely only on the acoustics but could also exploit ongoing interactions between oscillations and constraints flowing from internal language models.

scholarly journals Automatic Speech Recognition Predicts Speech Intelligibility and Comprehension for Listeners With Simulated Age-Related Hearing Loss

2017 ◽  
Vol 60 (9) ◽  
pp. 2394-2405 ◽  
Author(s):  
Lionel Fontan ◽  
Isabelle Ferrané ◽  
Jérôme Farinas ◽  
Julien Pinquier ◽  
Julien Tardieu ◽  
...  
Keyword(s):  
Hearing Loss ◽  
Speech Recognition ◽  
Automatic Speech Recognition ◽  
Hearing Aids ◽  
Speech Processing ◽  
Fine Tuning ◽  
Language Models ◽  
Age Related ◽  
Age Related Hearing Loss ◽  
Asr System

Purpose The purpose of this article is to assess speech processing for listeners with simulated age-related hearing loss (ARHL) and to investigate whether the observed performance can be replicated using an automatic speech recognition (ASR) system. The long-term goal of this research is to develop a system that will assist audiologists/hearing-aid dispensers in the fine-tuning of hearing aids. Method Sixty young participants with normal hearing listened to speech materials mimicking the perceptual consequences of ARHL at different levels of severity. Two intelligibility tests (repetition of words and sentences) and 1 comprehension test (responding to oral commands by moving virtual objects) were administered. Several language models were developed and used by the ASR system in order to fit human performances. Results Strong significant positive correlations were observed between human and ASR scores, with coefficients up to .99. However, the spectral smearing used to simulate losses in frequency selectivity caused larger declines in ASR performance than in human performance. Conclusion Both intelligibility and comprehension scores for listeners with simulated ARHL are highly correlated with the performances of an ASR-based system. In the future, it needs to be determined if the ASR system is similarly successful in predicting speech processing in noise and by older people with ARHL.

scholarly journals Adaptive Plasticity Under Adverse Listening Conditions is Disrupted in Developmental Dyslexia

2020 ◽  
pp. 1-11
Author(s):  
Yafit Gabay ◽  
Lori L. Holt
Keyword(s):  
Word Recognition ◽  
Developmental Dyslexia ◽  
Speech Processing ◽  
Acoustic Noise ◽  
Spoken Language ◽  
Adaptive Plasticity ◽  
Signal Distortion ◽  
Language Recognition ◽  
Speech Input ◽  
Accented Speech

Abstract Objective: Acoustic distortions to the speech signal impair spoken language recognition, but healthy listeners exhibit adaptive plasticity consistent with rapid adjustments in how the distorted speech input maps to speech representations, perhaps through engagement of supervised error-driven learning. This puts adaptive plasticity in speech perception in an interesting position with regard to developmental dyslexia inasmuch as dyslexia impacts speech processing and may involve dysfunction in neurobiological systems hypothesized to be involved in adaptive plasticity. Method: Here, we examined typical young adult listeners (N = 17), and those with dyslexia (N = 16), as they reported the identity of native-language monosyllabic spoken words to which signal processing had been applied to create a systematic acoustic distortion. During training, all participants experienced incremental signal distortion increases to mildly distorted speech along with orthographic and auditory feedback indicating word identity following response across a brief, 250-trial training block. During pretest and posttest phases, no feedback was provided to participants. Results: Word recognition across severely distorted speech was poor at pretest and equivalent across groups. Training led to improved word recognition for the most severely distorted speech at posttest, with evidence that adaptive plasticity generalized to support recognition of new tokens not previously experienced under distortion. However, training-related recognition gains for listeners with dyslexia were significantly less robust than for control listeners. Conclusions: Less efficient adaptive plasticity to speech distortions may impact the ability of individuals with dyslexia to deal with variability arising from sources like acoustic noise and foreign-accented speech.

scholarly journals Speech processing and production in two-year-old children acquiring isiXhosa: A tale of two children

2016 ◽  
Vol 63 (2) ◽  
Author(s):  
Michelle Pascoe ◽  
Kate Rossouw ◽  
Laura Fish ◽  
Charne Jansen ◽  
Natalie Manley ◽  
...  
Keyword(s):  
Speech Processing ◽  
Auditory Discrimination ◽  
Relative Strength ◽  
Auditory Task ◽  
Typical Development ◽  
Limited Data ◽  
Speech Input ◽  
Speech Output ◽  
Phonological Knowledge ◽  
Input Processing

We investigated the speech processing and production of 2-year-old children acquiring isiXhosa in South Africa. Two children (2 years, 5 months; 2 years, 8 months) are presented as single cases. Speech input processing, stored phonological knowledge and speech output are described, based on data from auditory discrimination, naming, and repetition tasks. Both children were approximating adult levels of accuracy in their speech output, although naming was constrained by vocabulary. Performance across tasks was variable: One child showed a relative strength with repetition, and experienced most difficulties with auditory discrimination. The other performed equally well in naming and repetition, and obtained 100% for her auditory task. There is limited data regarding typical development of isiXhosa, and the focus has mainly been on speech production. This exploratory study describes typical development of isiXhosa using a variety of tasks understood within a psycholinguistic framework. We describe some ways in which speech and language therapists can devise and carry out assessment with children in situations where few formal assessments exist, and also detail the challenges of such work.Keywords: psycholinguistic assessment; phonology; psycholinguistic framework; naming; repetition; auditory discrimination

scholarly journals Individual Differences in the Use of Acoustic-Phonetic Versus Lexical Cues for Speech Perception

2021 ◽  
Vol 6 ◽  
Author(s):  
Nikole Giovannone ◽  
Rachel M. Theodore
Keyword(s):  
Speech Perception ◽  
Speech Processing ◽  
Receptive Language ◽  
Language Ability ◽  
Natural Speech ◽  
Stimulus Input ◽  
Lexical Information ◽  
Speech Input ◽  
High Conflict ◽  
Lexical Cues

Previous research suggests that individuals with weaker receptive language show increased reliance on lexical information for speech perception relative to individuals with stronger receptive language, which may reflect a difference in how acoustic-phonetic and lexical cues are weighted for speech processing. Here we examined whether this relationship is the consequence of conflict between acoustic-phonetic and lexical cues in speech input, which has been found to mediate lexical reliance in sentential contexts. Two groups of participants completed standardized measures of language ability and a phonetic identification task to assess lexical recruitment (i.e., a Ganong task). In the high conflict group, the stimulus input distribution removed natural correlations between acoustic-phonetic and lexical cues, thus placing the two cues in high competition with each other; in the low conflict group, these correlations were present and thus competition was reduced as in natural speech. The results showed that 1) the Ganong effect was larger in the low compared to the high conflict condition in single-word contexts, suggesting that cue conflict dynamically influences online speech perception, 2) the Ganong effect was larger for those with weaker compared to stronger receptive language, and 3) the relationship between the Ganong effect and receptive language was not mediated by the degree to which acoustic-phonetic and lexical cues conflicted in the input. These results suggest that listeners with weaker language ability down-weight acoustic-phonetic cues and rely more heavily on lexical knowledge, even when stimulus input distributions reflect characteristics of natural speech input.

scholarly journals A technology prototype system for rating therapist empathy from audio recordings in addiction counseling

2016 ◽  
Vol 2 ◽  
pp. e59 ◽  
Author(s):  
Bo Xiao ◽  
Chewei Huang ◽  
Zac E. Imel ◽  
David C. Atkins ◽  
Panayiotis Georgiou ◽  
...  
Keyword(s):  
Speech Processing ◽  
Processing System ◽  
Language Models ◽  
Automated System ◽  
Prototype System ◽  
Observational Assessment ◽  
Speech Technology ◽  
Addiction Counseling ◽  
Audio Recordings ◽  
Therapist Empathy

Scaling up psychotherapy services such as for addiction counseling is a critical societal need. One challenge is ensuring quality of therapy, due to the heavy cost of manual observational assessment. This work proposes a speech technology-based system to automate the assessment of therapist empathy—a key therapy quality index—from audio recordings of the psychotherapy interactions. We designed a speech processing system that includes voice activity detection and diarization modules, and an automatic speech recognizer plus a speaker role matching module to extract the therapist’s language cues. We employed Maximum Entropy models, Maximum Likelihood language models, and a Lattice Rescoring method to characterize highvs.low empathic language. We estimated therapy-session level empathy codes using utterance level evidence obtained from these models. Our experiments showed that the fully automated system achieved a correlation of 0.643 between expert annotated empathy codes and machine-derived estimations, and an accuracy of 81% in classifying highvs.low empathy, in comparison to a 0.721 correlation and 86% accuracy in the oracle setting using manual transcripts. The results show that the system provides useful information that can contribute to automatic quality insurance and therapist training.

scholarly journals Wait for it! Stronger influence of context on categorical perception in Danish than Norwegian

2019 ◽  
Author(s):  
Byurakn Ishkhanyan ◽  
Anders Højen ◽  
Riccardo Fusaroli ◽  
Christer Johansson ◽  
Kristian Tylén ◽  
...  
Keyword(s):  
Language Processing ◽  
Speech Processing ◽  
Context Effect ◽  
Categorical Perception ◽  
Time Window ◽  
Relative Weight ◽  
Linguistic Differences ◽  
Speech Input ◽  
Phonetic Information ◽  
Processing Strategies

Speech input is often noisy and ambiguous. Yet listenersusually do not have difficulties understanding it. A keyhypothesis is that in speech processing acoustic-phoneticbottom-up processing is complemented by top-downcontextual information. This context effect is larger when theambiguous word is only separated from a disambiguating word by a few syllables compared to many syllables, suggesting that there is a limited time window for processing acoustic-phonetic information with the help of context. Here, we argue that the relative weight of bottom-up and top-down processes may be different for languages that have different phonological properties. We report an experiment comparing two closely related languages, Danish and Norwegian. We show that Danish speakers do indeed rely on context more than Norwegian speakers do. These results highlight the importance of investigating cross-linguistic differences in speech processing, suggesting that speakers of different languages may develop different language processing strategies.

scholarly journals Combining predictive coding with neural oscillations optimizes on-line speech processing

2018 ◽  
Author(s):  
Sevada Hovsepyan ◽  
Itsaso Olasagasti ◽  
Anne-Lise Giraud
Keyword(s):  
Sensory Processing ◽  
Speech Processing ◽  
Predictive Coding ◽  
Theta Oscillations ◽  
Neural Oscillations ◽  
Speech Comprehension ◽  
Contextual Cues ◽  
Continuous Speech ◽  
Neural Representations ◽  
On Line

ABSTRACTSpeech comprehension requires segmenting continuous speech to connect it on-line with discrete linguistic neural representations. This process relies on theta-gamma oscillation coupling, which tracks syllables and encodes them in decipherable neural activity. Speech comprehension also strongly depends on contextual cues predicting speech structure and content. To explore the effects of theta-gamma coupling on bottom-up/top-down dynamics during on-line speech perception, we designed a generative model that can recognize syllable sequences in continuous speech. The model uses theta oscillations to detect syllable onsets and align both gamma-rate encoding activity with syllable boundaries and predictions with speech input. We observed that the model performed best when theta oscillations were used to align gamma units with input syllables, i.e. when bidirectional information flows were coordinated, and internal timing knowledge was exploited. This work demonstrates that notions of predictive coding and neural oscillations can usefully be brought together to account for dynamic on-line sensory processing.

scholarly journals BLiMP: The Benchmark of Linguistic Minimal Pairs for English

2020 ◽  
Vol 8 ◽  
pp. 377-392
Author(s):  
Alex Warstadt ◽  
Alicia Parrish ◽  
Haokun Liu ◽  
Anhad Mohananey ◽  
Wei Peng ◽  
...  
Keyword(s):  
State Of The Art ◽  
Negative Polarity ◽  
Minimal Pair ◽  
Language Models ◽  
Linguistic Knowledge ◽  
Negative Polarity Items ◽  
Polarity Items ◽  
Minimal Pairs ◽  
Knowledge Of Language ◽  
N Gram

We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP), 1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands.

Sign in / Sign up

Export Citation Format

Share Document