Face masks impair reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

AbstractFace masks have become a prevalent measure during the Covid-19 pandemic to counteract the transmission of SARS-CoV 2. An unintended “side-effect” of face masks is their adverse influence on speech perception especially in challenging listening situations. So far, behavioural studies have not pinpointed exactly which feature(s) of speech processing face masks affect in such listening situations. We conducted an audiovisual (AV) multi-speaker experiment using naturalistic speech (i.e. an audiobook). In half of the trials, the target speaker wore a (surgical) face mask, while we measured the brain activity of normal hearing participants via magnetoencephalography (MEG). A decoding model on the clear AV speech (i.e. no additional speaker and target speaker not wearing a face mask) was trained and used to reconstruct crucial speech features in each condition. We found significant main effects of face masks on the reconstruction of acoustic features, such as the speech envelope and spectral speech features (i.e. pitch and formant frequencies), while reconstruction of higher level features of speech segmentation (phoneme and word onsets) were especially impaired through masks in difficult listening situations, i.e. when a distracting speaker was also presented. Our findings demonstrate the detrimental impact face masks have on listening and speech perception, thus extending previous behavioural results. Supporting the idea of visual facilitation of speech is the fact that we used surgical face masks in our study, which only show mild effects on speech acoustics. This idea is in line with recent research, also by our group, showing that visual cortical regions track spectral modulations. Since hearing impairment usually affects higher frequencies, the detrimental effect of face masks might pose a particular challenge for individuals who likely need the visual information about higher frequencies (e.g. formants) to compensate.

Download Full-text

Speech Perception through Face Masks by Children and Adults

10.33774/coe-2021-l88qk ◽

2021 ◽

Author(s):

Julia Schwarz ◽

Katrina (Kechun) Li ◽

Jasper Hong Sim ◽

Yixin Zhang ◽

Elizabeth Buchanan-Worster ◽

...

Keyword(s):

Speech Perception ◽

Acoustic Signal ◽

Speech Processing ◽

Face Mask ◽

Semantic Context ◽

Final Position ◽

Semantic Cues ◽

Visual Obstruction ◽

First Results ◽

Processing Language

Face masks can cause speech processing difficulties. However, it is unclear to what extent these difficulties are caused by the visual obstruction of the speaker’s mouth or by changes of the acoustic signal, and whether the effects can be found regardless of semantic context. In the present study, children and adults performed a cued shadowing task online, repeating the last word of English sentences. Target words were embedded in sentence-final position and manipulated visually, acoustically, and by semantic context (cloze probability). First results from 16 children and 16 adults suggest that processing language through face masks leads to slower responses in both groups, but visual, acoustic, and semantic cues all significantly reduce the mask effect. Although children were less proficient in predictive speech processing overall, they were still able to use semantic cues to compensate for face mask effects in a similar fashion to adults.

Download Full-text

Time Course of Attentional Modulation in the Frontal Eye Field During Curve Tracing

Journal of Neurophysiology ◽

10.1152/jn.91050.2008 ◽

2009 ◽

Vol 101 (4) ◽

pp. 1813-1822 ◽

Cited By ~ 16

Author(s):

P. S. Khayat ◽

A. Pooresmaeili ◽

P. R. Roelfsema

Keyword(s):

Visual Information ◽

Time Course ◽

Stimulus Presentation ◽

Frontal Eye Field ◽

Stimulus Selection ◽

Visual Response ◽

Visual Responses ◽

Cortical Regions ◽

Visual Cortical ◽

Curve Tracing

Neurons in the frontal eye fields (FEFs) register incoming visual information and select visual stimuli that are relevant for behavior. Here we investigated the timing of the visual response and the timing of selection by recording from single FEF neurons in a curve-tracing task that requires shifts of attention followed by an oculomotor response. We found that the behavioral selection signal in area FEF had a latency of 147 ms and that it was delayed substantially relative to the visual response, which occurred 50 ms after stimulus presentation. We compared the FEF responses to activity previously recorded in the primary visual cortex (area V1) during the same task. Visual responses in area V1 preceded the FEF responses, but the latencies of selection signals in areas V1 and FEF were similar. The similarity of timing of selection signals in structures at opposite ends of the visual cortical processing hierarchy supports the view that stimulus selection occurs in an interaction between widely separated cortical regions.

Download Full-text

Learning to Integrate Auditory and Visual Information in Speech Perception

PsycEXTRA Dataset ◽

10.1037/e537052012-627 ◽

2004 ◽

Author(s):

Joseph D. W. Stephens ◽

Lori L. Holt

Keyword(s):

Speech Perception ◽

Visual Information

Download Full-text

The Role of Primate Prefrontal Cortex in Bias and Shift Between Visual Dimensions

Cerebral Cortex ◽

10.1093/cercor/bhz072 ◽

2019 ◽

Vol 30 (1) ◽

pp. 85-99 ◽

Cited By ~ 5

Author(s):

Farshad A Mansouri ◽

Mark J Buckley ◽

Daniel J Fehring ◽

Keiji Tanaka

Keyword(s):

Prefrontal Cortex ◽

Anterior Cingulate ◽

Top Down ◽

Attentional Processes ◽

Prefrontal Cortical ◽

Frontopolar Cortex ◽

Dorsolateral Prefrontal ◽

Cortical Regions ◽

Visual Cortical ◽

Bilateral Lesions

Abstract Imaging and neural activity recording studies have shown activation in the primate prefrontal cortex when shifting attention between visual dimensions is necessary to achieve goals. A fundamental unanswered question is whether representations of these dimensions emerge from top-down attentional processes mediated by prefrontal regions or from bottom-up processes within visual cortical regions. We hypothesized a causative link between prefrontal cortical regions and dimension-based behavior. In large cohorts of humans and macaque monkeys, performing the same attention shifting task, we found that both species successfully shifted between visual dimensions, but both species also showed a significant behavioral advantage/bias to a particular dimension; however, these biases were in opposite directions in humans (bias to color) versus monkeys (bias to shape). Monkeys’ bias remained after selective bilateral lesions within the anterior cingulate cortex (ACC), frontopolar cortex, dorsolateral prefrontal cortex (DLPFC), orbitofrontal cortex (OFC), or superior, lateral prefrontal cortex. However, lesions within certain regions (ACC, DLPFC, or OFC) impaired monkeys’ ability to shift between these dimensions. We conclude that goal-directed processing of a particular dimension for the executive control of behavior depends on the integrity of prefrontal cortex; however, representation of competing dimensions and bias toward them does not depend on top-down prefrontal-mediated processes.

Download Full-text

Auditory cortical micro-networks show differential connectivity during voice and speech processing in humans

Communications Biology ◽

10.1038/s42003-021-02328-2 ◽

2021 ◽

Vol 4 (1) ◽

Author(s):

Florence Steiner ◽

Marine Bobin ◽

Sascha Frühholz

Keyword(s):

Speech Processing ◽

Neural Circuits ◽

Brain Activity ◽

Inhibitory Pathways ◽

Ac Networks ◽

Directional Connectivity ◽

The Right ◽

Voice Processing ◽

Differential Connectivity ◽

Voice And Speech

AbstractThe temporal voice areas (TVAs) in bilateral auditory cortex (AC) appear specialized for voice processing. Previous research assumed a uniform functional profile for the TVAs which are broadly spread along the bilateral AC. Alternatively, the TVAs might comprise separate AC nodes controlling differential neural functions for voice and speech decoding, organized as local micro-circuits. To investigate micro-circuits, we modeled the directional connectivity between TVA nodes during voice processing in humans while acquiring brain activity using neuroimaging. Results show several bilateral AC nodes for general voice decoding (speech and non-speech voices) and for speech decoding in particular. Furthermore, non-hierarchical and differential bilateral AC networks manifest distinct excitatory and inhibitory pathways for voice and speech processing. Finally, while voice and speech processing seem to have distinctive but integrated neural circuits in the left AC, the right AC reveals disintegrated neural circuits for both sounds. Altogether, we demonstrate a functional heterogeneity in the TVAs for voice decoding based on local micro-circuits.

Download Full-text

Phase synchronization of oscillatory brain activity and the matching of top-down and bottom up visual information

International Journal of Psychophysiology ◽

10.1016/j.ijpsycho.2008.05.414 ◽

2008 ◽

Vol 69 (3) ◽

pp. 162

Author(s):

P. Sauseng

Keyword(s):

Visual Information ◽

Phase Synchronization ◽

Brain Activity ◽

Top Down ◽

Bottom Up ◽

Oscillatory Brain Activity

Download Full-text

Abnormal speech perception in schizophrenia with auditory hallucinations

Acta Neuropsychiatrica ◽

10.1111/j.0924-2708.2004.00071.x ◽

2004 ◽

Vol 16 (3) ◽

pp. 154-159 ◽

Cited By ~ 13

Author(s):

Seung-Hwan Lee ◽

Young-Cho Chung ◽

Jong-Chul Yang ◽

Yong-Ku Kim ◽

Kwang-Yoon Suh

Keyword(s):

Speech Perception ◽

Speech Processing ◽

Auditory Hallucination ◽

Verbal Working Memory ◽

Continuous Performance Task ◽

Perception System ◽

Significant Difference ◽

Significant Impairment ◽

Schizophrenic Patients ◽

Cpt I

Background:The neurobiological mechanism of auditory hallucination (AH) in schizophrenia remains elusive, but AH can be caused by the abnormality in the speech perception system based on the speech perception neural network model.Objectives:The purpose of this study was to investigate whether schizophrenic patients with AH have the speech processing impairment as compared with schizophrenic patients without AH, and whether the speech perception ability could be improved after AH had subsided.Methods:Twenty-four schizophrenic patients with AH were compared with 25 schizophrenic patients without AH. Narrative speech perception was assessed using a masked speech tracking (MST) task with three levels of superimposed phonetic noise. Sentence repetition task (SRT) and auditory continuous performance task (CPT) were used to assess grammar-dependent verbal working memory and non-language attention, respectively. These tests were measured before and after treatment in both groups.Results:Before treatment, schizophrenic patients with AH showed significant impairments in MST compared with those without AH. There were no significant differences in SRT and CPT correct (CPT-C) rates between both groups, but CPT incorrect (CPT-I) rate showed a significant difference. The low-score CPI-I group showed a significant difference in MST performance between the two groups, while the high-score CPI-I group did not. After treatment (after AH subsided), the hallucinating schizophrenic patients still had significant impairment in MST performance compared with non-hallucinating schizophrenic patients.Conclusions:Our results support the claim that schizophrenic patients with AH are likely to have a disturbance of the speech perception system. Moreover, our data suggest that non-language attention might be a key factor influencing speech perception ability and that speech perception dysfunction might be a trait marker in schizophrenia with AH.

Download Full-text

Refixation control in free viewing: a specialized mechanism divulged by eye-movement-related brain activity

Journal of Neurophysiology ◽

10.1152/jn.00121.2018 ◽

2018 ◽

Vol 120 (5) ◽

pp. 2311-2324 ◽

Cited By ~ 4

Author(s):

Andrey R. Nikolaev ◽

Radha Nila Meghanathan ◽

Cees van Leeuwen

Keyword(s):

Eye Movement ◽

Visual Information ◽

Direct Evidence ◽

Search Task ◽

Visual Search Task ◽

Brain Activity ◽

Movement Control ◽

Planning Stage ◽

Eye Fixations ◽

Free Viewing

In free viewing, the eyes return to previously visited locations rather frequently, even though the attentional and memory-related processes controlling eye-movement show a strong antirefixation bias. To overcome this bias, a special refixation triggering mechanism may have to be recruited. We probed the neural evidence for such a mechanism by combining eye tracking with EEG recording. A distinctive signal associated with refixation planning was observed in the EEG during the presaccadic interval: the presaccadic potential was reduced in amplitude before a refixation compared with normal fixations. The result offers direct evidence for a special refixation mechanism that operates in the saccade planning stage of eye movement control. Once the eyes have landed on the revisited location, acquisition of visual information proceeds indistinguishably from ordinary fixations. NEW & NOTEWORTHY A substantial proportion of eye fixations in human natural viewing behavior are revisits of recently visited locations, i.e., refixations. Our recently developed methods enabled us to study refixations in a free viewing visual search task, using combined eye movement and EEG recording. We identified in the EEG a distinctive refixation-related signal, signifying a control mechanism specific to refixations as opposed to ordinary eye fixations.

Download Full-text

Neural correlates of motion processing through echolocation, source hearing, and vision in blind echolocation experts and sighted echolocation novices

Journal of Neurophysiology ◽

10.1152/jn.00501.2013 ◽

2014 ◽

Vol 111 (1) ◽

pp. 112-127 ◽

Cited By ~ 28

Author(s):

L. Thaler ◽

J. L. Milne ◽

S. R. Arnott ◽

D. Kish ◽

M. A. Goodale

Keyword(s):

Visual Motion ◽

Brain Activity ◽

Region Of Interest ◽

Occipital Cortex ◽

Neural Substrates ◽

Motion Processing ◽

Planum Temporale ◽

Show Evidence ◽

Visual Cortical ◽

Source Motion

We have shown in previous research (Thaler L, Arnott SR, Goodale MA. PLoS One 6: e20162, 2011) that motion processing through echolocation activates temporal-occipital cortex in blind echolocation experts. Here we investigated how neural substrates of echo-motion are related to neural substrates of auditory source-motion and visual-motion. Three blind echolocation experts and twelve sighted echolocation novices underwent functional MRI scanning while they listened to binaural recordings of moving or stationary echolocation or auditory source sounds located either in left or right space. Sighted participants' brain activity was also measured while they viewed moving or stationary visual stimuli. For each of the three modalities separately (echo, source, vision), we then identified motion-sensitive areas in temporal-occipital cortex and in the planum temporale. We then used a region of interest (ROI) analysis to investigate cross-modal responses, as well as laterality effects. In both sighted novices and blind experts, we found that temporal-occipital source-motion ROIs did not respond to echo-motion, and echo-motion ROIs did not respond to source-motion. This double-dissociation was absent in planum temporale ROIs. Furthermore, temporal-occipital echo-motion ROIs in blind, but not sighted, participants showed evidence for contralateral motion preference. Temporal-occipital source-motion ROIs did not show evidence for contralateral preference in either blind or sighted participants. Our data suggest a functional segregation of processing of auditory source-motion and echo-motion in human temporal-occipital cortex. Furthermore, the data suggest that the echo-motion response in blind experts may represent a reorganization rather than exaggeration of response observed in sighted novices. There is the possibility that this reorganization involves the recruitment of “visual” cortical areas.

Download Full-text