Lip-Reading Enables the Brain to Synthesize Auditory Features of Unknown Silent Speech

Mathieu Bourguignon; Martijn Baart; Efthymia C. Kapnoula; Nicola Molinaro

doi:10.1523/jneurosci.1101-19.2019

Searching through functional space reveals distributed visual, auditory, and semantic coding in the human brain

10.1101/2020.04.20.052175 ◽

2020 ◽

Author(s):

Sreejan Kumar ◽

Cameron T. Ellis ◽

Thomas O’Connell ◽

Marvin M Chun ◽

Nicholas B. Turk-Browne

Keyword(s):

Human Brain ◽

Language Processing ◽

Brain Function ◽

Computational Models ◽

Brain Activity ◽

Functional Space ◽

Semantic Features ◽

Brain Functions ◽

Auditory Features ◽

The Brain

AbstractThe extent to which brain functions are localized or distributed is a foundational question in neuroscience. In the human brain, common fMRI methods such as cluster correction, atlas parcellation, and anatomical searchlight are biased by design toward finding localized representations. Here we introduce the functional searchlight approach as an alternative to anatomical searchlight analysis, the most commonly used exploratory multivariate fMRI technique. Functional searchlight removes any anatomical bias by grouping voxels based only on functional similarity and ignoring anatomical proximity. We report evidence that visual and auditory features from deep neural networks and semantic features from a natural language processing model are more widely distributed across the brain than previously acknowledged. This approach provides a new way to evaluate and constrain computational models with brain activity and pushes our understanding of human brain function further along the spectrum from strict modularity toward distributed representation.

Download Full-text

The purity of auditory memory

Philosophical Transactions of the Royal Society of London Series B Biological Sciences ◽

10.1098/rstb.1983.0053 ◽

1983 ◽

Vol 302 (1110) ◽

pp. 251-265 ◽

Cited By ~ 39

Keyword(s):

Speech Perception ◽

Visual Information ◽

Auditory Memory ◽

Immediate Memory ◽

System A ◽

Auditory Sensory Memory ◽

Auditory Experience ◽

Auditory Features ◽

Silent Speech ◽

Selection Of

Recent evidence from experiments on immediate memory indicates unambiguously that silent speech perception can produce typically ‘auditory’ effects while there is either active or passive mouthing of the relevant articulatory gestures. This result falsifies previous theories of auditory sensory memory (pre-categorical acoustic store) that insisted on external auditory stimulation as indispensible for access to the system. A resolution is proposed that leaves the properties of pre-categorical acoustic store much as they were assumed to be before but adds the possibility that visual information can affect the selection of auditory features in a pre-categorical stage of speech perception. In common terms, a speaker’s facial gestures (or one’s own) can influence auditory experience independently of determining what it was that was said. Some results in word perception that encourage this view are discussed.

Download Full-text

Glutamate Receptors: Finally Fingered in Inherited Epilepsy?

Epilepsy Currents ◽

10.1111/j.1535-7511.2007.00191.x ◽

2007 ◽

Vol 7 (4) ◽

pp. 110-111

Author(s):

Nicholas P. Poolos

Keyword(s):

Synaptic Transmission ◽

Autosomal Dominant ◽

Transmembrane Protein ◽

Hippocampal Slices ◽

Partial Epilepsy ◽

Receptor Complex ◽

Neuronal Protein ◽

Human Epilepsy ◽

Auditory Features ◽

The Brain

Epilepsy-Related Ligand/Receptor Complex LGI1 and ADAM22 Regulate Synaptic Transmission. Fukata Y, Adesnik H, Iwanaga T, Bredt DS, Nicoll RA, Fukata M. Science 2006;313(5794):1792–1795. Abnormally synchronized synaptic transmission in the brain causes epilepsy. Most inherited forms of epilepsy result from mutations in ion channels. However, one form of epilepsy, autosomal dominant partial epilepsy with auditory features (ADPEAF), is characterized by mutations in a secreted neuronal protein, LGI1. We show that ADAM22, a transmembrane protein that when mutated itself causes seizure, serves as a receptor for LGI1. LGI1 enhances AMPA receptor-mediated synaptic transmission in hippocampal slices. The mutated form of LGI1 fails to bind to ADAM22. ADAM22 is anchored to the postsynaptic density by cytoskeletal scaffolds containing stargazin. These studies in rat brain indicate possible avenues for understanding human epilepsy.

Download Full-text

Hearing through lip-reading: the brain synthesizes features of absent speech

10.1101/395483 ◽

2018 ◽

Cited By ~ 5

Author(s):

Mathieu Bourguignon ◽

Martijn Baart ◽

Efthymia C. Kapnoula ◽

Nicola Molinaro

Keyword(s):

Biological Motion ◽

Real Life ◽

Low Frequency ◽

Angular Gyrus ◽

Synthesis Process ◽

Lip Reading ◽

Auditory Cortices ◽

The Right ◽

High Level ◽

The Brain

AbstractLip-reading is crucial to understand speech in challenging conditions. Neuroimaging investigations have revealed that lip-reading activates auditory cortices in individuals covertly repeating absent—but known—speech. However, in real-life, one usually has no detailed information about the content of upcoming speech. Here we show that during silent lip-reading of unknown speech, activity in auditory cortices entrains more to absent speech than to seen lip movements at frequencies below 1 Hz. This entrainment to absent speech was characterized by a speech-to-brain delay of 50–100 ms as when actually listening to speech. We also observed entrainment to lip movements at the same low frequency in the right angular gyrus, an area involved in processing biological motion. These findings demonstrate that the brain can synthesize high-level features of absent unknown speech sounds from lip-reading that can facilitate the processing of the auditory input. Such a synthesis process may help explain well-documented bottom-up perceptual effects.

Download Full-text

Speech Perception as a Multimodal Phenomenon

Current Directions in Psychological Science ◽

10.1111/j.1467-8721.2008.00615.x ◽

2008 ◽

Vol 17 (6) ◽

pp. 405-409 ◽

Cited By ~ 79

Author(s):

Lawrence D. Rosenblum

Keyword(s):

Speech Perception ◽

Semantic Context ◽

Visual Speech ◽

Audiovisual Speech ◽

Lexical Status ◽

Auditory Speech ◽

Lip Reading ◽

Imaging Research ◽

Speech Information ◽

The Brain

Speech perception is inherently multimodal. Visual speech (lip-reading) information is used by all perceivers and readily integrates with auditory speech. Imaging research suggests that the brain treats auditory and visual speech similarly. These findings have led some researchers to consider that speech perception works by extracting amodal information that takes the same form across modalities. From this perspective, speech integration is a property of the input information itself. Amodal speech information could explain the reported automaticity, immediacy, and completeness of audiovisual speech integration. However, recent findings suggest that speech integration can be influenced by higher cognitive properties such as lexical status and semantic context. Proponents of amodal accounts will need to explain these results.

Download Full-text