Cortical Hub for Flavor Sensation in Rodents

The experience of eating is inherently multimodal, combining intraoral gustatory, olfactory, and somatosensory signals into a single percept called flavor. As foods and beverages enter the mouth, movements associated with chewing and swallowing activate somatosensory receptors in the oral cavity, dissolve tastants in the saliva to activate taste receptors, and release volatile odorant molecules to retronasally activate olfactory receptors in the nasal epithelium. Human studies indicate that sensory cortical areas are important for intraoral multimodal processing, yet their circuit-level mechanisms remain unclear. Animal models allow for detailed analyses of neural circuits due to the large number of molecular tools available for tracing and neuronal manipulations. In this review, we concentrate on the anatomical and neurophysiological evidence from rodent models toward a better understanding of the circuit-level mechanisms underlying the cortical processing of flavor. While more work is needed, the emerging view pertaining to the multimodal processing of food and beverages is that the piriform, gustatory, and somatosensory cortical regions do not function solely as independent areas. Rather they act as an intraoral cortical hub, simultaneously receiving and processing multimodal sensory information from the mouth to produce the rich and complex flavor experience that guides consummatory behavior.

Clustering of neuropsychological traits of preschoolers

Neuropsychological tests (targeting cognitive, linguistic, motor, and executive abilities) are grouped in neuropsychological domains that are thought to be stable through adulthood. However, this assumption does not always hold true, particularly during young children’s early developmental phase. Here, we explored how the neuropsychological profile of typical Spanish-speaking preschoolers varied and consolidated with age. We recruited 643 monolingual Latin-American children from Mexico, Colombia, and Guatemala, with ages spanning from 30 to 71 months of age, and applied a novel neuropsychological examination which combined a total of 52 tests covering five classical neuropsychological domains: receptive, expressive, attention/memory, processing, and executive functions. These tests’ scores uncovered a correlational structure across neuropsychological functions that could not be explained by chance. Notably, these correlations’ overall strength, but not their interdependence across domains, dramatically increased with age. Moreover, by applying conventional clustering techniques to classify the experimental data, we found a stable representation of two clusters of children with distinctive traits, with cultural factors contributing to this classification scheme. We also found that the tasks were well organized in a network of abilities, where nodes with highest highest interconnectedness were those that required multimodal processing. These results contribute to our understanding of children’s ‘normal’ development and could help identify how failure in particular functions forecasts the emergence of neurodevelopmental disorders. Our analytic methods might become useful to characterize individual differences and improve educational practices and interventions.

Attention-Sharing Initiative of Multimodal Processing in Simultaneous Interpreting

This study sets out to describe simultaneous interpreters' attention-sharing initiatives when exposed under input from both videotaped speech recording and real-time transcriptions. Separation of mental energy in acquiring visual input accords with the human brain's statistic optimization principle where the same property of an object is presented through diverse fashions. In examining professional interpreters' initiatives, the authors invited five professional English-Chinese conference interpreters to simultaneously interpret a videotaped speech with real-time captions generated by speech recognition engine while meanwhile monitoring their eye movements. The results indicate the professional interpreters' preferences in referring to visually presented captions along with the speaker's facial expressions, where low-frequency words, proper names, and numbers gained greater attention than words with higher frequency. This phenomenon might be explained by the working memory theory in which the central executive enables redundancy gains retrieved from dual-channel information.

Multimodal processing in simultaneous interpreting with text

The present study focuses on (in)congruence of input between the visual and the auditory modality in simultaneous interpreting with text. We asked twenty-four professional conference interpreters to simultaneously interpret an aurally and visually presented text with controlled incongruences in three categories (numbers, names and control words), while measuring interpreting accuracy and eye movements. The results provide evidence for the dominance of the visual modality, which goes against the professional standard of following the auditory modality in the case of incongruence. Numbers enjoyed the greatest accuracy across conditions possibly due to simple cross-language semantic mappings. We found no evidence for a facilitation effect for congruent items, and identified an impeding effect of the presence of the visual text for incongruent items. These results might be interpreted either as evidence for the Colavita effect (in which visual stimuli take precedence over auditory ones) or as strategic behaviour applied by professional interpreters to avoid risk.

Multimodal frequency representations are embedded in modality-defined cortical sensory systems

Sensory information is represented and elaborated in hierarchical cortical systems that are thought to be dedicated to individual sensory modalities. This traditional view of sensory cortex organization has been challenged by recent evidence of multimodal responses in primary and association sensory areas. Although it is indisputable that sensory areas respond to multiple modalities, it remains unclear whether these multimodal responses reflect selective information processing for particular stimulus features. Here, we used fMRI adaptation to identify brain regions that are sensitive to the temporal frequency information contained in auditory, tactile, and audiotactile stimulus sequences. A number of brain regions distributed over the parietal and temporal lobes exhibited frequency-selective temporal response modulation for both auditory and tactile stimulus events, as indexed by repetition suppression effects. A smaller set of regions responded to crossmodal adaptation sequences in a frequency-dependent manner. Despite an extensive overlap of multimodal frequency-selective responses across the parietal and temporal lobes, representational similarity analysis revealed a cortical “regional landscape” that clearly reflected distinct somatosensory and auditory processing systems that converged on modality-invariant areas. These structured relationships between brain regions were also evident in spontaneous signal fluctuation patterns measured at rest. Our results reveal that multimodal processing in human cortex can be feature-specific and that multimodal frequency representations are embedded in the intrinsically hierarchical organization of cortical sensory systems.Significance StatementA hallmark of traditional brain organization models is the segregation of signals from the different senses in modality-dedicated brain regions. Recent evidence showing multimodal activity in brain regions thought to be dedicated to a single modality have challenged the traditional sensory cortex model. Notably, few studies have explored the feature-specificity of multimodal responses found in sensory cortex. Here, we used fMRI adaptation to identify parietal and temporal cortex regions which exhibited sensitivity to both tactile and auditory frequency information. These univariate results demonstrate that multimodal processing in sensory cortex can be feature-specific. Using the same data, though, we found clear evidence of modality-based cortical organization estimated from multivariate response patterns and spontaneous BOLD signal fluctuations. Thus, our results reveal an embedding of feature-specific multimodal processing in traditionally-defined cortical systems.

Dendritic Ca2+ dynamics and multimodal processing in a cricket antennal interneuron

The integration of stimuli of different modalities is fundamental to information processing within the nervous system. A descending interneuron in the cricket brain, with prominent dendrites in the deutocerebrum, receives input from three sensory modalities: touch of the antennal flagellum, strain of the antennal base, and visual stimulation. Using calcium imaging, we demonstrate that each modality drives a Ca2+ increase in a different dendritic region. Moreover, touch of the flagellum is represented in a topographic map along the neuron’s dendrites. Using intracellular recording, we investigated the effects of Ca2+ on spike shape through the application of the Ca2+ channel antagonist Cd2+ and identified probable Ca2+-dependent K+ currents. NEW & NOTEWORTHY Different dendritic regions of the cricket brain neuron DBNi1-2 showed localized Ca2+ increases when three modalities of stimulation (touch of the flagellum, strain at antennal base, and visual input) were given. Touch stimulation induces localized Ca2+ increases according to a topographic map of the antenna. Ca2+ appears to activate K+ currents in DBNi1-2.

Deep learning: from speech recognition to language and multimodal processing

While artificial neural networks have been in existence for over half a century, it was not until year 2010 that they had made a significant impact on speech recognition with a deep form of such networks. This invited paper, based on my keynote talk given at Interspeech conference in Singapore in September 2014, will first reflect on the historical path to this transformative success, after providing brief reviews of earlier studies on (shallow) neural networks and on (deep) generative models relevant to the introduction of deep neural networks (DNN) to speech recognition several years ago. The role of well-timed academic-industrial collaboration is highlighted, so are the advances of big data, big compute, and the seamless integration between the application-domain knowledge of speech and general principles of deep learning. Then, an overview is given on sweeping achievements of deep learning in speech recognition since its initial success. Such achievements, summarized into six major areas in this article, have resulted in across-the-board, industry-wide deployment of deep learning in speech recognition systems. Next, more challenging applications of deep learning, natural language and multimodal processing, are selectively reviewed and analyzed. Examples include machine translation, knowledgebase completion, information retrieval, and automatic image captioning, where fresh ideas from deep learning, continuous-space embedding in particular, are shown to be revolutionizing these application areas albeit with less rapid pace than for speech and image recognition. Finally, a number of key issues in deep learning are discussed, and future directions are analyzed for perceptual tasks such as speech, image, and video, as well as for cognitive tasks involving natural language.