Encoding of speech in convolutional layers and the brain stem based on language experience

Comparing artificial neural networks (ANNs) with outputs of brain imaging techniques has recently seen substantial advances in (computer) vision and text-based language models. Here, we propose a framework to compare biological and artificial neural computations of spoken language representations and propose several new challenges to this paradigm. Using a technique proposed by Begus and Zhou (2021b), we can analyze encoding of any acoustic property in intermediate convolutional layers of an artificial neural network. This allows us to test similarities in speech encoding between the brain and artificial neural networks in a way that is more interpretable than the majority of existing proposals that focus on correlations and supervised models. We introduce fully unsupervised deep generative models (the Generative Adversarial Network architecture) trained on raw speech to the brain-and-ANN-comparison paradigm, which enable testing of both the production and perception principles in human speech. We present a framework that parallels electrophysiological experiments measuring complex Auditory Brainstem Response (cABR) in human brain with intermediate layers in deep convolutional networks. We compared peak latency in cABR relative to the stimulus in the brain stem experiment, and in intermediate convolutional layers relative to the input/output in deep convolutional networks. We also examined and compared the effect of prior language exposure on the peak latency in cABR, and in intermediate convolutional layers of a phonetic property. Specifically, the phonetic property (i.e., VOT =10 ms) is perceived differently by English vs. Spanish speakers as voiced (e.g. [ba]) vs voiceless (e.g. [pa]). Critically, the cABR peak latency to the VOT phonetic property is different between English and Spanish speakers, and peak latency in intermediate convolutional layers is different between English-trained and Spanish-trained computational models. Substantial similarities in peak latency encoding between the human brain and intermediate convolutional networks emerge based on results from eight trained networks (including a replication experiment). The proposed technique can be used to compare encoding between the human brain and intermediate convolutional layers for any acoustic property.

Cortico-Cerebellar Hyper-Connections and Reduced Purkinje Cells Behind Abnormal Eyeblink Conditioning in a Computational Model of Autism Spectrum Disorder

Empirical evidence suggests that children with autism spectrum disorder (ASD) show abnormal behavior during delay eyeblink conditioning. They show a higher conditioned response learning rate and earlier peak latency of the conditioned response signal. The neuronal mechanisms underlying this autistic behavioral phenotype are still unclear. Here, we use a physiologically constrained spiking neuron model of the cerebellar-cortical system to investigate which features are critical to explaining atypical learning in ASD. Significantly, the computer simulations run with the model suggest that the higher conditioned responses learning rate mainly depends on the reduced number of Purkinje cells. In contrast, the earlier peak latency mainly depends on the hyper-connections of the cerebellum with sensory and motor cortex. Notably, the model has been validated by reproducing the behavioral data collected from studies with real children. Overall, this article is a starting point to understanding the link between the behavioral and neurobiological basis in ASD learning. At the end of the paper, we discuss how this knowledge could be critical for devising new treatments.

Listening Effort in Cochlear Implant Users: The Effect of Speech Intelligibility, Noise Reduction Processing, and Working Memory Capacity on the Pupil Dilation Response

Purpose: This study aimed to evaluate the effect of speech recognition performance, working memory capacity (WMC), and a noise reduction algorithm (NRA) on listening effort as measured with pupillometry in cochlear implant (CI) users while listening to speech in noise. Method: Speech recognition and pupil responses (peak dilation, peak latency, and release of dilation) were measured during a speech recognition task at three speech-to-noise ratios (SNRs) with an NRA in both on and off conditions. WMC was measured with a reading span task. Twenty experienced CI users participated in this study. Results: With increasing SNR and speech recognition performance, (a) the peak pupil dilation decreased by only a small amount, (b) the peak latency decreased, and (c) the release of dilation after the sentences increased. The NRA had no effect on speech recognition in noise or on the peak or latency values of the pupil response but caused less release of dilation after the end of the sentences. A lower reading span score was associated with higher peak pupil dilation but was not associated with peak latency, release of dilation, or speech recognition in noise. Conclusions: In CI users, speech perception is effortful, even at higher speech recognition scores and high SNRs, indicating that CI users are in a chronic state of increased effort in communication situations. The application of a clinically used NRA did not improve speech perception, nor did it reduce listening effort. Participants with a relatively low WMC exerted relatively more listening effort but did not have better speech reception thresholds in noise.

Sympathetic Response to Postural Perturbation in Stance

The purpose of the present study was to elucidate whether the sympathetic response to perturbation in stance represents multiple mental responses, whether perturbation-induced fear of fall is one of the mental responses, and whether the sympathetic response is task specific. While healthy humans maintained stance, the support surface of the feet translated in the forward or backward direction. The phasic electrodermal response (EDR), representing the sympathetic response, appeared 1–1.5 s after the support surface translation. Mostly, perturbation-induced EDRs comprised one peak, but some EDRs were comprised of two peaks. The onset latency of the two-peak EDR was much shorter than that of the one-peak EDR. The second peak latency of the two-peak EDR was similar to the peak latency of the one-peak EDR, indicating that the first peak of the two-peak EDR was an additional component preceding the one-peak EDR. This finding supports a view that perturbation-induced EDR in stance sometimes represents multiple mental responses. The amplitude of the EDR had a positive and significant correlation with fear, indicating that perturbation-induced EDR in stance partially represents perturbation-induced fear of fall. The EDR amplitude was dependent on the translation amplitude and direction, indicating that perturbation-induced EDR in stance is a task specific response. The EDR appeared earlier when the participants prepared to answer a question or when the perturbation was self-triggered, indicating that adding cognitive load induces earlier perturbation-induced mental responses.

Modality differences in ERP components between somatosensory and auditory Go/No-go paradigms in prepubescent children

We investigated modality differences in the N2 and P3 components of event-related potentials (ERPs) between somatosensory and auditory Go/No-go paradigms in eighteen healthy prepubescent children (mean age: 125.9±4.2 months). We also evaluated the relationship between behavioral responses (reaction time, reaction time variability, and omission and commission error rates) and amplitudes and latencies of N2 and P3 during somatosensory and auditory Go/No-go paradigms. The peak latency of No-go-N2 was significantly shorter than that of Go-N2 during somatosensory paradigms, but not during auditory paradigms. The peak amplitude of P3 was significantly larger during somatosensory than auditory paradigms, and the peak latency of P3 was significantly shorter during somatosensory than auditory paradigms. Correlations between behavioral responses and the P3 component were not found during somatosensory paradigms. On the other hand, in auditory paradigms, correlations were detected between the reaction time and peak amplitude of No-go-P3, and between the reaction time variability and peak latency of No-go-P3. A correlation was noted between commission error and the peak latency of No-go-N2 during somatosensory paradigms. Compared with previous adult studies using both somatosensory and auditory Go/No-go paradigms, the relationships between behavioral responses and ERP components would be weak in prepubescent children. Our data provide findings to advance understanding of the neural development of motor execution and inhibition processing, that is dependent on or independent of the stimulus modality.

Similar CNV Neurodynamic Patterns between Sub- and Supra-Second Time Perception

In the field of time psychology, the functional significance of the contingent negative variation (CNV) component in time perception and whether the processing mechanisms of sub- and supra-second are similar or different still remain unclear. In the present study, event-related potential (ERP) technology and classical temporal discrimination tasks were used to explore the neurodynamic patterns of sub- and supra-second time perception. In Experiment 1, the standard interval (SI) was fixed at 500 ms, and the comparison interval (CI) ranged from 200 ms to 800 ms. In Experiment 2, the SI was fixed at 2000 ms, and the CI ranged from 1400 ms to 2600 ms. Participants were required to judge whether the CI was longer or shorter than the SI. The ERP results showed similar CNV activity patterns in the two experiments. Specifically, CNV amplitude would be more negative when the CI was longer or closer to the memorized SI. CNV peak latency increased significantly until the CI reached the memorized SI. We propose that CNV amplitude might reflect the process of temporal comparison, and CNV peak latency might represent the process of temporal decision-making. To our knowledge, it is the first ERP task explicitly testing the two temporal scales, sub- and supra-second timing, in one study. Taken together, the present study reveals a similar functional significance of CNV between sub- and supra-second time perception.

Hyperexcitability in Older Adults with Elevated Beta-Amyloid

Background: Growing evidence links beta-amyloid (Aβ) and neuronal hyperexcitability in preclinical mouse models of Alzheimer’s disease (AD). The aim of this study was to compare neuronal excitability between cognitively normal amyloid positive (CNAβ+) and those without elevated amyloid (CNAβ-) older adults. We hypothesized CNAβ+ participants would show hyperexcitability, indexed by greater peak P3 event-related potential peak amplitude, shorter peak latency, and changes in event-related power, compared to CNAβ-.Methods: CNAβ+ participants (n = 17, age: 73 ± 5, 11 women, MOCA scores 26 ± 2) and 17 CNAβ- participants group-matched for age, sex, and MOCA completed the a working memory task (n-back with n = 0, 1, 2) test while wearing a 256-channel EEG net. P3 peak amplitude and latency of the nontarget, target and task difference (nontarget – target), and event-related power, extracted from Fz (main outcome), Cz, and Pz were compared between groups using linear mixed models. Mean Aβ standard uptake value ratios (SUVR) were correlated with P3 amplitude and latency using Pearson r.Results: P3 peak amplitude of the task difference (p = 0.048) and P3 peak latency of non-targets trials (p = 0.006) at Fz differed between groups. Similarly, power was lower in the delta band (p = 0.04) for nontargets at Fz in CNAβ+ participants. CNAβ+ participants also demonstrated higher theta and alpha power in channels at Cz and Pz, but no changes in P3 ERP. Strong correlations were found between mean Aβ SUVR and latency of the 1-back (r = -0.69; p = 0.003) and 2-back (r = -0.69; p = 0.004) of the task difference at channel Fz in the CNAβ+ group.Conclusions: Our pilot data suggests that elevated amyloid in cognitive normal older adults is associated with hyperexcitability in P3 ERP. Further research is warranted to determine the validity of ERP in predicting clinical, neurobiological, and functional manifestations of AD.

Prediction of Second Language Proficiency Based on Electroencephalographic Signals Measured While Listening to Natural Speech

This study had two goals: to clarify the relationship between electroencephalographic (EEG) features estimated while non-native speakers listened to a second language (L2) and their proficiency in L2 determined by a conventional paper test and to provide a predictive model for L2 proficiency based on EEG features. We measured EEG signals from 205 native Japanese speakers, who varied widely in English proficiency while they listened to natural speech in English. Following the EEG measurement, they completed a conventional English listening test for Japanese speakers. We estimated multivariate temporal response functions separately for word class, speech rate, word position, and parts of speech. We found significant negative correlations between listening score and 17 EEG features, which included peak latency of early components (corresponding to N1 and P2) for both open and closed class words and peak latency and amplitude of a late component (corresponding to N400) for open class words. On the basis of the EEG features, we generated a predictive model for Japanese speakers’ English listening proficiency. The correlation coefficient between the true and predicted listening scores was 0.51. Our results suggest that L2 or foreign language ability can be assessed using neural signatures measured while listening to natural speech, without the need of a conventional paper test.

Decreased P2 Waveform Reflects Impaired Brain Executive Function Induced by 12 h of Low Homeostatic Sleep Pressure: Evidence From an Event-Related Potential Study

Homeostatic sleep pressure can cause cognitive impairment, in which executive function is the most affected. Previous studies have mainly focused on high homeostatic sleep pressure (long-term sleep deprivation); thus, there is still little related neuro-psycho-physiological evidence based on low homeostatic sleep pressure (12 h of continuous wakefulness) that affects executive function. This study aimed to investigate the impact of lower homeostatic sleep pressure on executive function. Our study included 14 healthy young male participants tested using the Go/NoGo task in normal resting wakefulness (10:00 am) and after low homeostatic sleep pressure (10:00 pm). Behavioral data (response time and accuracy) were collected, and electroencephalogram (EEG) data were recorded simultaneously, using repeated measures analysis of variance for data analysis. Compared with resting wakefulness, the participants’ response time to the Go stimulus was shortened after low homeostatic sleep pressure, and the correct response rate was reduced. Furthermore, the peak amplitude of Go–P2 decreased significantly, and the peak latency did not change significantly. For NoGo stimulation, the peak amplitude of NoGo–P2 decreased significantly (p < 0.05), and the peak latency was significantly extended (p < 0.05). Thus, the P2 wave is likely related to the attention and visual processing and reflects the early judgment of the perceptual process. Therefore, the peak amplitude of Go–P2 and NoGo–P2 decreased, whereas the peak latency of NoGo–P2 increased, indicating that executive function is impaired after low homeostatic sleep pressure. This study has shown that the P2 wave is a sensitive indicator that reflects the effects of low homeostatic sleep pressure on executive function, and that it is also an important window to observe the effect of homeostatic sleep pressure and circadian rhythm on cognitive function.

Physiological and Medico-Social Research Trends of the Wave P300 and More Late Components of Visual Event-Related Potentials

To extend the application of the late waves of the event-related potentials (ERPs) to multiple modalities, devices and software the underlying physiological mechanisms and responses of the brain for a particular sensory system and mental function must be carefully examined. The objective of this study was aimed to study the sensory processes of the “human-computer interaction” model when classifying visual images with an incomplete set of signs based on the analysis of early, middle, late and slow ERPs components. 26 healthy subjects (men) aged 20–26 years were investigated. ERPs in 19 monopolar sites according to the 10/20 system were recorded. Discriminant and factor analyzes (BMDP Statistical Software) were applied. The component N450 is the most specialized indicator of the perception of unrecognizable (oddball) visual images. The amplitude of the ultra-late components N750 and N900 is also higher under conditions of presentation of the oddball image, regardless of the location of the registration points. In brain pathology along with the pronounced asymmetry of the wave distribution, reduction of the N150 wave and lengthening of its peak latency, a line of regularities were noted. These include–a pronounced reduction in peak latency P250 and N350, an increased amplitude of N350 in the frontal and central points of registration, a decrease in the amplitude of N450 in the left frontal cortex and its increase in the occipital registration points, activation of the occipital cortex at a time interval of 400–500 ms, as well as fusion later waves. We called such phenomena of the development of cognitive ERP in brain pathology “the incongruence of ERP components”. The results of the research are discussed in the light of the paradigm of the P300 wave application in brain-computer interface systems, as well as with the peculiarities in brain pathology.