Natural Infant-Directed Speech Facilitates Neural Tracking of Prosody

Infants prefer to be addressed with infant-directed speech (IDS). IDS benefits language acquisition through amplified low-frequency amplitude modulations. It has been reported that this amplification increases electrophysiological tracking of IDS compared to adult-directed speech (ADS). It is still unknown which particular frequency band triggers this effect. Here, we compare tracking at the rates of syllables and prosodic stress, which are both critical to word segmentation and recognition. In mother-infant dyads (n=30), mothers described novel objects to their 9-month-olds while infants' EEG was recorded. For IDS, mothers were instructed to speak to their children as they typically do, while for ADS, mothers described the objects as if speaking with an adult. Phonetic analyses confirmed that pitch features were more prototypically infant-directed in the IDS-condition compared to the ADS-condition. Neural tracking of speech was assessed by speech-brain coherence, which measures the synchronization between speech envelope and EEG. Results revealed significant speech-brain coherence at both syllabic and prosodic stress rates, indicating that infants track speech in IDS and ADS at both rates. We found significantly higher speech-brain coherence for IDS compared to ADS in the prosodic stress rate but not the syllabic rate. This indicates that the IDS benefit arises primarily from enhanced prosodic stress. Thus, neural tracking is sensitive to parents’ speech adaptations during natural interactions, possibly facilitating higher-level inferential processes such as word segmentation from continuous speech.

Two Stages of Speech Envelope Tracking in Human Auditory Cortex Modulated by Speech Intelligibility

The envelope is essential for speech perception. Recent studies have shown that cortical activity can track the acoustic envelope. However, whether the tracking strength reflects the extent of speech intelligibility processing remains controversial. Here, using stereo-electroencephalogram (sEEG) technology, we directly recorded the activity in human auditory cortex while subjects listened to either natural or noise-vocoded speech. These two stimuli have approximately identical envelopes, but the noise-vocoded speech does not have speech intelligibility. We found two stages of envelope tracking in auditory cortex: an early high-γ (60-140 Hz) power stage (delay ≈ 49 ms) that preferred the noise-vocoded speech, and a late θ (4-8 Hz) phase stage (delay ≈ 178 ms) that preferred the natural speech. Furthermore, the decoding performance of high-γ power was better in primary auditory cortex than in non-primary auditory cortex, consistent with its short tracking delay. We also found distinct lateralization effects: high-γ power envelope tracking dominated left auditory cortex, while θ phase showed better decoding performance in right auditory cortex. In sum, we suggested a functional dissociation between high-γ power and θ phase: the former reflects fast and automatic processing of brief acoustic features, while the latter correlates to slow build-up processing facilitated by speech intelligibility.

Decoding speech information from EEG data with 4, 7 and 11 month-old infants: Contrasting convolutional neural network, mutual information-based and backward linear models

Computational models that successfully translate neural activity into speech are multiplying in the adult literature, with non-linear convolutional neural network (CNN) approaches joining the more frequently-employed linear and mutual information (MI) models. Despite the promise of these methods for uncovering the neural basis of language acquisition by the human brain, similar studies with infants are rare. Existing infant studies rely on simpler cross-correlation and other linear techniques and aim only to establish neural tracking of the broadband speech envelope. Here, three novel computational models were applied to measure whether low-frequency speech envelope information was encoded in infant neural activity. Backward linear and CNN models were applied to estimate speech information from neural activity using linear versus nonlinear approaches, and a MI model measured how well the acoustic stimuli were encoded in infant neural responses. Fifty infants provided EEG recordings when aged 4, 7, and 11 months, while listening passively to natural speech (sung nursery rhymes) presented by video with a female singer. Each model computed speech information for these nursery rhymes in two different frequency bands, delta (1 – 4 Hz) and theta (4 – 8 Hz), thought to provide different types of linguistic information. All three models demonstrated significant levels of performance for delta-band and theta-band neural activity from 4 months of age. All models also demonstrated higher accuracy for the delta-band neural response in the infant brain. However, only the linear and MI models showed developmental (age-related) effects, and these developmental effects differed by model. Accordingly, the choice of algorithm used to decode speech envelope information from neural activity in the infant brain may determine the developmental conclusions that can be drawn. Better understanding of the strengths and weaknesses of each modelling approach will be fundamental to improving our understanding of how the human brain builds a language system.

Listening to speech in noisy scenes: Antithetical contribution of primary and non-primary auditory cortex

Invasive and non-invasive electrophysiological measurements during cocktail-party-like listening indicate that neural activity in human auditory cortex (AC) tracks the envelope of relevant speech. Due to the measurement s limited coverage and/or spatial resolution, however, the distinct contribution of primary and non-primary auditory areas remains unclear. Using 7-Tesla fMRI, here we measured brain responses of participants attending to one speaker, without and with another concurrent speaker. Using voxel-wise modeling, we observed significant speech envelope tracking in bilateral Heschl s gyrus (HG) and middle superior temporal sulcus (mSTS), despite the sluggish fMRI responses and slow temporal sampling. Neural activity was either positively (HG) or negatively (mSTS) correlated to the speech envelope. Spatial pattern analyses indicated that whereas tracking in HG reflected both relevant and (to a lesser extent) non-relevant speech, right mSTS selectively represented the relevant speech signal. These results indicate that primary and non-primary AC antithetically process ongoing speech suggesting a push-pull of acoustic and linguistic information.

A kinematic-acoustic analysis of gesture-speech coupling in persons with aphasia

Aphasia is a profound speech pathology hampering speech production and/or comprehension. People with Aphasia (PWA) use more manual gestures than neurotypicals. This intuitively invokes the idea of some kind of compensatory mechanism, but there is variable evidence of this gesture-boosting effect on speech processes. The status quo in gesture research with PWA is an emphasis on categorical analysis of gesture types, how often they are recruited, and whether they aid communication or speaking. However, there are increasingly louder calls for the investigation of gesture and speech as continuous entangled modes of expression. In neurotypical adults, expressive moments of gesture and speech are synchronized. It is unknown how this multimodal prosody is instantiated in PWA. In the current study we perform the first acoustic-kinematic gesture-speech analysis in persons of aphasia (Wernicke, Broca, Anomic) relative to age-matched controls, where we apply several multimodal signal processing methods (e.g., motion tracking). Specifically, we related the speech peaks (smoothed amplitude envelope change) with that of the nearest peaks in the gesture acceleration profile. We obtained that the magnitude of gesture vs. speech peaks are positively related across the groups, though more variably for PWA, and such coupling was related to less severe Aphasia-related symptoms in PWA. When comparing the temporal ordering of speech envelope versus acceleration peaks, no statistically reliable differences between controls and PWA were observed. We did however obtain when gesture kinematic peaks tended to anticipate speech peaks more, this was related to less severe aphasia-related symptoms. Finally, we show that both gesture and speech have slower quasi-rhythmic structure, indicating that next to speech, gesture is slowed down too. We conclude that future aphasia research will benefit from the study of gesture-speech interconnections.

Envelope reconstruction of speech and music highlights stronger tracking of speech at low frequencies

The human brain tracks amplitude fluctuations of both speech and music, which reflects acoustic processing in addition to the encoding of higher-order features and one’s cognitive state. Comparing neural tracking of speech and music envelopes can elucidate stimulus-general mechanisms, but direct comparisons are confounded by differences in their envelope spectra. Here, we use a novel method of frequency-constrained reconstruction of stimulus envelopes using EEG recorded during passive listening. We expected to see music reconstruction match speech in a narrow range of frequencies, but instead we found that speech was reconstructed better than music for all frequencies we examined. Additionally, models trained on all stimulus types performed as well or better than the stimulus-specific models at higher modulation frequencies, suggesting a common neural mechanism for tracking speech and music. However, speech envelope tracking at low frequencies, below 1 Hz, was associated with increased weighting over parietal channels, which was not present for the other stimuli. Our results highlight the importance of low-frequency speech tracking and suggest an origin from speech-specific processing in the brain.

Comprehension under noise in L1 and L2: Acoustic entrainment of speech envelope

Listeners regularly comprehend continuous speech despite noisy conditions. Previous studies show that cortical entrainment to speech degrades under noise, predicts comprehension, and increases for non-native listeners. We test the hypothesis that listeners similarly increase cortical entrainment for both L2 and noisy L1 speech, after controlling for comprehension. Twenty-four Chinese-English bilinguals underwent EEG while listening to one hour of an audiobook, mixed with three levels of noise, in Mandarin and English and answered comprehension questions. We estimated cortical entrainment for one-minute tracks using the multivariate temporal response function (mTRF). Contrary to our prediction, entrainment of the L2 was significantly lower than L1, while L1 entrainment significantly increased when speech was masked by noise without reducing comprehension. However, greater L2 proficiency was positively associated with greater entrainment. We discuss how studies of entrainment relating to noise and bilingualism might be reconciled with an approach focused on exerted rather than demanded effort.