Мультимодальность – мультимедийность – мультиканальность и др. Альтернативные формы передачи информации как проблема лингвистической теории и терминологии

Статья посвящена популярной в современной лингвистике проблеме мультимодальности, т.е. взаимодействию в рамках одного и того же дискурса разных способов кодирования информации – с использованием разных семиотических кодов. Авторы исходят из предпосылки, что суще- ствование альтернативных планов выражения сообщений обусловлено разнородностью их со- держания, а также имеющейся предрасположенностью типов содержания (семантического или прагматического) к определенному способу оформления. Мультимодальный подход рассматрива- ется в контексте медиатизации и технологизации социальной коммуникации. Авторы показывают значительную вариативность существующей в данной области терминологии, объясняют трудно- сти, связанные с употреблением некоторых терминов. Отдельный раздел статьи посвящен общей типологии модульной архитектоники сообщения, учитывающей (в качестве форм репрезентации смысла) разные уровни языковой организации сообщения, разные каналы, разные коды, разные медиаустройства, разные языки и разные (параллельные) сообщения. Multimodality – multimediality – multicanality etc. Alternative forms of the transmission of information as a problem of linguistic theory and terminology Summary: The article is devoted to the problem of multimodality, popular in modern linguistics and seen as an interaction within the same discourse of different methods of encoding and decoding information. The authors proceed from the premise that the existence of alternative plans of the expression of messages is due to the heterogeneity of their content, as well as the existing predisposition of the types of content (semantic or pragmatic) to a certain way of explication. The multimodal approach is considered in the context of the mediatization and technologization of social communication. The authors show a significant variability of the terminology in this area and explain the difficulties associated with the use of some terms. A separate section of the article is devoted to the general typology of modular message architectonics, which takes into account (as forms of meaning representation) different levels of the linguistic organization of a message, different channels, different codes, different media devices, different languages and different (parallel) messages.

Exploring digital interactivity: a multimodal and social semiotic approach to intersemiotic construction in COVID China website

This paper explores intersemiotic construction and digital interactivity through a multimodal social semiotic lens examining the semiotic instantiations of interactive signs on the homepage of “Fighting COVID-19 the Chinese Way” (covid-19.chinadaily.com.cn), a website created to disseminate information on news and development of COVID prevention and control practices in China to contribute to global efforts to fight the pandemic. The study focuses on how digital interactivity is afforded by the COVID China website, where the interactive signs are ideationally and compositionally constructed for meaning representation and interpersonally for text-viewer relation construal. From the multimodal social semiotic perspective, systemic functional model-based cluster and intersemiotic analyses are applied to explore the visual, spatial, and linguistic features that contribute to the design and construction of interactive semiotic signs on the COVID China website and afford digital interactivity for viewers’ action potentials. This study extends the analytical focus to the semiotic instantiations of interactive signs and their intersemiotic construction process that stimulate the enabling of interactivity, instead of the interactivity per se, and demonstrates how different semiotic instantiations of interactive signs are featured and interact to afford digital interactivity. It argues for an integrated lens in analysis to look at the interactive signs not only as signs of action with action-enabling forms but also as signs of meanings that afford user-page interactivity.

Idiosyncratic Tower of Babel: Individual Differences in Word-Meaning Representation Increase as Word Abstractness Increases

Humans primarily rely on language to communicate, on the basis of a shared understanding of the basic building blocks of communication: words. Do we mean the same things when we use the same words? Although cognitive neural research on semantics has revealed the common principles of word-meaning representation, the factors underlying the potential individual variations in word meanings are unknown. Here, we empirically characterized the intersubject consistency of 90 words across 20 adult subjects (10 female) using both behavioral measures (rating-based semantic-relationship patterns) and neuroimaging measures (word-evoked brain activity patterns). Across both the behavioral and neuroimaging experiments, we showed that the magnitude of individual disagreements on word meanings could be modeled on the basis of how much language or sensory experience is associated with a word and that this variation increases with word abstractness. Uncovering the cognitive and neural origins of word-meaning disagreements across individuals has implications for potential mechanisms to modulate such disagreements.

Neural oscillations reflect meaning identification for novel words in context

During language processing, people make rapid use of contextual information to promote comprehension of upcoming words. When new words are learned implicitly, information contained in the surrounding context can provide constraints on their possible meaning. In the current study, EEG was recorded as participants listened to a series of three sentences, each containing an identical target pseudoword, with the aim of using contextual information in the surrounding language to identify a meaning representation for the novel word. In half of trials, sentences were semantically coherent so that participants could develop a single representation for the novel word that fit all contexts. Other trials contained unrelated sentence contexts so that meaning associations were not possible. We observed greater theta band enhancement over the left-hemisphere across central and posterior electrodes in response to pseudowords processed across semantically related compared to unrelated contexts. Additionally, relative alpha and beta band suppression was increased prior to pseudoword onset in trials where contextual information more readily promoted pseudoword-meaning associations. Under the hypothesis that theta enhancement indexes processing demands during lexical access, the current study provides evidence for selective online memory retrieval to novel words learned implicitly in a spoken context.

Improving Neural Machine Translation with AMR Semantic Graphs

The Seq2Seq model and its variants (ConvSeq2Seq and Transformer) emerge as a promising novel solution to the machine translation problem. However, these models only focus on exploiting knowledge from bilingual sentences without paying much attention to utilizing external linguistic knowledge sources such as semantic representations. Not only do semantic representations can help preserve meaning but they also minimize the data sparsity problem. However, to date, semantic information remains rarely integrated into machine translation models. In this study, we examine the effect of abstract meaning representation (AMR) semantic graphs in different machine translation models. Experimental results on the IWSLT15 English-Vietnamese dataset have proven the efficiency of the proposed model, expanding the use of external language knowledge sources to significantly improve the performance of machine translation models, especially in the application of low-resource language pairs.

Neural bases of audio-visual integration of socially meaningful information in macaques

Social interactions rely on the interpretation of semantic and emotional information, often from multiple sensory modalities. In primates, both audition and vision serve the interpretation of communicative signals. Autistic individuals present deficits in both social communication and audio-visual integration. At present, the neural mechanisms subserving the interpretation of complex audio-visual social events are unknown. Based on heart rate estimates and functional neuroimaging, we show that macaque monkeys associate affiliative facial expressions or social scenes with corresponding affiliative vocalizations, aggressive expressions or scenes with corresponding aggressive vocalizations and escape visual scenes with scream vocalizations, while suppressing vocalizations that are incongruent with the visual context. This process is subserved by two distinct functional networks, homologous to the human emotional and attentional networks activated during the processing of visual social information. These networks are thus critical for the construction of social meaning representation, and provide grounds for the audio-visual deficits observed in autism.One-sentence summary Macaques extract social meaning from visual and auditory input recruiting face and voice patches and a broader emotional and attentional network.