The face is central to primate multicomponent signals

A wealth of experimental and observational evidence suggests that faces have become increasingly important in the communication system of primates over evolutionary time and that both the static and moveable aspects of faces convey considerable information. Therefore, whenever there is a visual component to any multicomponent signal the face is potentially relevant. However, the role of the face is not always considered in primate multicomponent communication research. We review the literature and make a case for greater focus on the face going forward. We propose that the face can be overlooked for two main reasons: first, due to methodological difficulty. Examination of multicomponent signals in primates is difficult, so scientists tend to examine a limited number of signals in combination. Detailed examination of the subtle and dynamic components of facial signals is particularly hard to achieve in studies of primates. Second, due to a common assumption that the face contains “emotional” content. A priori categorisation of facial behavior as “emotional” ignores the potentially communicative and predictive information present in the face that might contribute to signals. In short, we argue that the face is central to multicomponent signals (and also many multimodal signals) and suggest future directions for investigating this phenomenon.

Parasite defensive limb movements enhance signal attraction in male little torrent frogs: insight into the evolution of multimodal signals

Many animals rely on complex sexual signals that target multiple senses to attract mates and repel rivals. These multimodal mating displays can however also attract unintended receivers, which can be an important driver of signal complexity. Despite being taxonomically widespread, we often lack insight into how multimodal signals evolve from unimodal signals and in particular what roles unintended eavesdroppers play. Here we assess whether the physical movements of parasite defense behavior increase the complexity and attractiveness of an acoustic sexual signal in the little torrent frog (Amolops torrentis). Calling males of this species often display limb movements in order to defend against blood sucking parasites such as frog biting midges that eavesdrop on their acoustic signal. Through mate choice tests we show that some of these midge evoked movements influence female preference for acoustic signals. Our data suggest that midge induced movements may be incorporated into a sexual display, targeting both hearing and vision in the intended receiver. Females may play an important role in incorporating these multiple components because they prefer signals which combine multiple programs. Our results thus help to understand the relationship between ecological and sexual selection pressure operating on signalers and how in turn this may influence multimodal signal evolution.

Active Inference Through Energy Minimization in Multimodal Affective Human–Robot Interaction

During communication, humans express their emotional states using various modalities (e.g., facial expressions and gestures), and they estimate the emotional states of others by paying attention to multimodal signals. To ensure that a communication robot with limited resources can pay attention to such multimodal signals, the main challenge involves selecting the most effective modalities among those expressed. In this study, we propose an active perception method that involves selecting the most informative modalities using a criterion based on energy minimization. This energy-based model can learn the probability of the network state using energy values, whereby a lower energy value represents a higher probability of the state. A multimodal deep belief network, which is an energy-based model, was employed to represent the relationships between the emotional states and multimodal sensory signals. Compared to other active perception methods, the proposed approach demonstrated improved accuracy using limited information in several contexts associated with affective human–robot interaction. We present the differences and advantages of our method compared to other methods through mathematical formulations using, for example, information gain as a criterion. Further, we evaluate performance of our method, as pertains to active inference, which is based on the free energy principle. Consequently, we establish that our method demonstrated superior performance in tasks associated with mutually correlated multimodal information.

Decoupled Acoustic and Visual Components in the Multimodal Signals of the Common Cuckoo (Cuculus canorus)

Because of its parasitic habits, reproduction costs of the common cuckoo (Cuculus canorus) are mostly spent in pre-laying activities. Female costs are limited to searching host nests and laying eggs, whereas, males spend time in performing intense vocal displays, possibly with territorial purpose. This last aspect, together with a sexual plumage dimorphism, points to both intra- and inter-sexual selections operating within this species. One element triggering sexual selection is a differential fitness accrued by different phenotypes. Before analyzing possible sexual selection mechanisms operating in cuckoos, it is therefore necessary to verify whether there is a variability among male secondary characters by describing and quantifying them. Here we aimed to quantitatively characterize the main two potential candidates of sexual selection traits, i.e., calls and displays, shown by males at perches. During the 2019 breeding season, in a site within the Po Plain, we both audio and video recorded cuckoo males at five different perches. We analyzed acoustic variables as well as display sequences searching for potential correlations. We found a significant variation among calls that could be clustered into four vocal types. We also found that no visual displays were associated with vocal displays; cuckoo males were either vocal and motionless or soundless and active. We discuss our results under the perspective of the potential value of sexual selection in brood parasites and its role in its parasitic habit.

Audiovisual incongruence differentially impacts left and right hemisphere sensorimotor oscillations: Potential applications to production

Speech production gives rise to distinct auditory and somatosensory feedback signals which are dynamically integrated to enable online monitoring and error correction, though it remains unclear how the sensorimotor system supports the integration of these multimodal signals. Capitalizing on the parity of sensorimotor processes supporting perception and production, the current study employed the McGurk paradigm to induce multimodal sensory congruence/incongruence. EEG data from a cohort of 39 typical speakers were decomposed with independent component analysis to identify bilateral mu rhythms; indices of sensorimotor activity. Subsequent time-frequency analyses revealed bilateral patterns of event related desynchronization (ERD) across alpha and beta frequency ranges over the time course of perceptual events. Right mu activity was characterized by reduced ERD during all cases of audiovisual incongruence, while left mu activity was attenuated and protracted in McGurk trials eliciting sensory fusion. Results were interpreted to suggest distinct hemispheric contributions, with right hemisphere mu activity supporting a coarse incongruence detection process and left hemisphere mu activity reflecting a more granular level of analysis including phonological identification and incongruence resolution. Findings are also considered in regard to incongruence detection and resolution processes during production.

Musth and sexual selection in elephants: a review of signalling properties and potential fitness consequences

Sexual selection mediated by multimodal signals is common among polygynous species, including seasonally breeding mammals. Indirect benefit models provide plausible explanations for how and why mate selection can occur in the absence of direct benefits. Musth — an asynchronous reproductive state in male elephants — facilitates both inter- and intrasexual selection via indirect benefits, and it is further communicated through a multimodal signal. In this review, we synthesise existing evidence that supports the hypothesis that musth is a multimodal signal subject to sexual selection and that male elephants increase their direct fitness by propagating this signal while females accrue indirect benefits. Musth is characterised by a suite of physiological and behavioural changes, serving to facilitate copulation between the sexes, and via multisensory modalities musth conveys honest information about the condition of a male. Female elephants mate preferentially with musth males, increasing their own fitness in the absence of direct benefits. In addition, musth resolves dynamic dominance hierarchies among male elephants and often eliminates the need for costly physical combat. Future work in this field should investigate potential postcopulatory selection mechanisms in elephants, including sperm competition and cryptic female choice. These topics join other fundamental questions related to sexual selection, signalling, and indirect benefits that are still unanswered in elephants.

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis

Multimodal sentiment analysis and emotion recognition represent a major research direction in natural language processing (NLP). With the rapid development of online media, people often express their emotions on a topic in the form of video, and the signals it transmits are multimodal, including language, visual, and audio. Therefore, the traditional unimodal sentiment analysis method is no longer applicable, which requires the establishment of a fusion model of multimodal information to obtain sentiment understanding. In previous studies, scholars used the feature vector cascade method when fusing multimodal data at each time step in the middle layer. This method puts each modal information in the same position and does not distinguish between strong modal information and weak modal information among multiple modalities. At the same time, this method does not pay attention to the embedding characteristics of multimodal signals across the time dimension. In response to the above problems, this paper proposes a new method and model for processing multimodal signals, which takes into account the delay and hysteresis characteristics of multimodal signals across the time dimension. The purpose is to obtain a multimodal fusion feature emotion analysis representation. We evaluate our method on the multimodal sentiment analysis benchmark dataset CMU Multimodal Opinion Sentiment and Emotion Intensity Corpus (CMU-MOSEI). We compare our proposed method with the state-of-the-art model and show excellent results.

Rhythm in dyadic interactions

This review paper discusses rhythmic interactions and distinguishes them from non-rhythmic interactions. We report on communicative behaviours in social and sexual contexts, as found in dyads of humans, non-human primates, non-primate mammals, birds, anurans and insects. We discuss observed instances of rhythm in dyadic interactions, identify knowledge gaps and propose suggestions for future research. We find that most studies on rhythmicity in interactive signals mainly focus on one modality (acoustic or visual) and we suggest more work should be performed on multimodal signals. Although the social functions of interactive rhythms have been fairly well described, developmental research on rhythms used to regulate social interactions is still lacking. Future work should also focus on identifying the exact timing mechanisms involved. Rhythmic signalling behaviours are widespread and critical in regulating social interactions across taxa, but many questions remain unexplored. A multidisciplinary, comparative cross-species approach may help provide answers. This article is part of the theme issue ‘Synchrony and rhythm interaction: from the brain to behavioural ecology’.

The interaction of fundamental torsional guided waves from axial and oblique defects in pipes

A quantitative study of the interaction of the T(0,1) torsional mode with axial and oblique defects in a pipe is presented in this paper. A mode decomposition technique employing the chirplet transform is used to separate the multimodal signals reflected from the defects. Reflection signals are obtained from experiments on a carbon steel pipe. The influence of the crack length and inclination angle on the reflection is investigated. The reflection from an axial defect is found to consist of a series of wave pulses with gradually decaying amplitude. The results show that the reflection coefficient of an axial crack initially increases with the crack length but finally reaches an oscillating regime. Furthermore, for an oblique crack, it is revealed that the reflection coefficient is linearly dependent on the equivalent circumferential extent of the defect and is independent of the axial length.