scholarly journals Multimodal Affect Models: An Investigation of Relative Salience of Audio and Visual Cues for Emotion Prediction

2021 ◽  
Vol 3 ◽  
Author(s):  
Jingyao Wu ◽  
Ting Dang ◽  
Vidhyasaharan Sethu ◽  
Eliathamby Ambikairajah
Keyword(s):  
Emotion Recognition ◽  
Visual Cues ◽  
Emotion Perception ◽  
Arousal Level ◽  
Multiple Cues ◽  
Model Experiments ◽  
Multimodal Systems ◽  
Recognition Systems ◽  
Change Prediction ◽  
Relative Salience

People perceive emotions via multiple cues, predominantly speech and visual cues, and a number of emotion recognition systems utilize both audio and visual cues. Moreover, the perception of static aspects of emotion (speaker's arousal level is high/low) and the dynamic aspects of emotion (speaker is becoming more aroused) might be perceived via different expressive cues and these two aspects are integrated to provide a unified sense of emotion state. However, existing multimodal systems only focus on single aspect of emotion perception and the contributions of different modalities toward modeling static and dynamic emotion aspects are not well explored. In this paper, we investigate the relative salience of audio and video modalities to emotion state prediction and emotion change prediction using a Multimodal Markovian affect model. Experiments conducted in the RECOLA database showed that audio modality is better at modeling the emotion state of arousal and video for emotion state of valence, whereas audio shows superior advantages over video in modeling emotion changes for both arousal and valence.

Male salamanders remember individuals based on chemical or visual cues

Behaviour ◽  
2009 ◽  
Vol 146 (11) ◽  
pp. 1485-1498 ◽  
Author(s):  
Nancy Kohn ◽  
Robert Jaeger
Keyword(s):  
Filter Paper ◽  
Forest Floor ◽  
Visual Cues ◽  
Chemical Cues ◽  
Plethodon Cinereus ◽  
Multiple Cues

AbstractThe use of multiple cues can enhance the detection, recognition, discrimination, and memorability of individuals by receivers. We conducted two experiments, using only males, to test whether territorial red-backed salamanders, Plethodon cinereus, could use only chemical or only visual cues to remember familiar conspecifics. In both experiments, focal males spent significantly more time threatening unfamiliar than familiar male intruders. They also chemoinvestigated the filter paper containing chemical cues of unfamiliar intruders more often than that of familiar intruders. These results suggest that red-backed salamanders can use both chemical and visual cues to recognize familiar individuals, allowing them to distinguish between less threatening neighbours and more threatening intruders in the heterogeneous forest floor habitat, where visual cues alone would not always be available.

No effect of cognitive performance on post-intervention improvement in emotion recognition

2017 ◽  
Vol 41 (S1) ◽  
pp. S190-S190
Author(s):  
V.P. Bozikas ◽  
S. Tsotsi ◽  
A. Dardagani ◽  
E. Dandi ◽  
E.I. Nazlidou ◽  
...  
Keyword(s):  
Emotion Recognition ◽  
Cognitive Performance ◽  
Neuropsychological Test ◽  
Emotion Perception ◽  
First Episode ◽  
Psychotic Episode ◽  
Post Intervention ◽  
Basic Emotions ◽  
The Difference ◽  
Competing Interest

Deficits in emotion perception in patients with first episode of psychosis have been reported by many researchers. Till now, training programs have focused mainly in patients with schizophrenia and not in first psychotic episode (FEP) patients. We used a new intervention for facial affect recognition in a group of 35 FEP patients (26 male). The emotion recognition intervention included coloured pictures of individuals expressing six basic emotions (happiness, sadness, anger, disgust, surprise, fear) and a neutral emotion. The patients were trained to detect changes in facial features, according to the emotion displayed. A comprehensive battery of neuropsychological tests was also administered, measuring attention, memory, working memory, visuospatial ability and executive function by using specific tests of the Cambridge Neuropsychological Test Automated Battery (CANTAB). We tried to explore whether cognitive performance can explain the difference noted between the original assessment of emotion recognition and the post-intervention assessment. According to our data, overall cognitive performance did not correlate with post-intervention change in emotion recognition. Specific cognitive domains did not correlate with this change, either. According the above mentioned results, no significant correlation between neuropsychological performance and post-intervention improvement in emotion recognition was noted. This finding may suggest that interventions for emotion recognition may target specific processes that underlie emotion perception and their effect can be independent of general cognitive function.Disclosure of interestThe authors have not supplied their declaration of competing interest.

Multimodal emotion perception in young and elderly patients with multiple sclerosis

2021 ◽  
Author(s):  
Maxime Montembeault ◽  
Estefania Brando ◽  
Kim Charest ◽  
Alexandra Tremblay ◽  
Élaine Roger ◽  
...  
Keyword(s):  
Emotion Recognition ◽  
Real Life ◽  
Emotion Perception ◽  
Recognition Task ◽  
Self Report ◽  
Self Monitoring ◽  
Facial Emotions ◽  
Multimodal Emotion Recognition ◽  
The Impact ◽  
Better Than

Background. Studies suggest that emotion recognition and empathy are impaired in patients with MS (pwMS). Nonetheless, most studies of emotion recognition have used facial stimuli, are restricted to young samples, and rely self-report assessments of empathy. The aims of this study are to determine the impact of MS and age on multimodal emotion recognition (facial emotions and vocal emotional bursts) and on socioemotional sensitivity (as reported by the participants and their informants). We also aim to investigate the associations between emotion recognition, socioemotional sensitivity, and cognitive measures. Methods. We recruited 13 young healthy controls (HC), 14 young pwMS, 14 elderly HC and 15 elderly pwMS. They underwent a short neuropsychological battery, an experimental emotion recognition task including facial emotions and vocal emotional bursts. Both participants and their study informants completed the Revised-Self Monitoring Scale (RSMS) to assess the participant’s socioemotional sensitivity. Results. There was a significant effect of age and group on recognition of both facial emotions and emotional vocal bursts, HC performing significantly better than pwMS, and young participants performing better than elderly participants (no interaction effect). The same effects were observed on self-reported socioemotional sensitivity. However, lower socioemotional sensitivity in pwMS was not reported by the informants. Finally, multimodal emotion recognition did not correlate with socioemotional sensitivity, but it correlated with global cognitive severity. Conclusion. PwMS present with multimodal emotion perception deficits. Our results extend previous findings of decreased emotion perception and empathy to a group of elderly pwMS, in which advancing age does not accentuate these deficits. However, the decreased socioemotional sensitivity reported by pwMS does not appear to be observed by their relatives, nor to correlate with their emotion perception impairments. Future studies should investigate the real-life impacts of emotion perception deficits in pwMS.

The relationship of emotion recognition with neuropsychological performance in patients with first episode psychosis

2017 ◽  
Vol 41 (S1) ◽  
pp. S190-S190
Author(s):  
A. Dardagani ◽  
E. Dandi ◽  
S. Tsotsi ◽  
M. Nazou ◽  
A. Lagoudis ◽  
...  
Keyword(s):  
Emotion Recognition ◽  
Visual Memory ◽  
Emotion Perception ◽  
First Episode ◽  
Psychotic Episode ◽  
Visuospatial Ability ◽  
Cognitive Domains ◽  
Neuropsychological Dysfunction ◽  
First Psychotic Episode ◽  
The Relationship

The relationship between neuropsychological dysfunction and emotion perception has been frequently noted in various studies. Attention, for example, has been found to play an important role in emotion processing and recognition. Not many studies though, have examined this relationship in first psychotic episode patients. The aim of the present study was to explore the nature of the relation between performance in cognitive tests and a test that measures emotion perception. In a sample of 46 first psychotic episode patients (22 male), we administered a comprehensive battery of neuropsychological non-verbal tests and an emotion recognition test. The cognitive domains of attention, memory, working memory, visuospatial ability and executive function were examined, by using specific tests of the Cambridge Neuropsychological Test Automated Battery (CANTAB). The emotion recognition assessment comprised a new test that includes 35 coloured pictures of individuals expressing six basic emotions (happiness, sadness, anger, disgust, surprise, fear) and a neutral emotion. We used partial correlation–controlling for the effect of age–and we found a statistically significant relationship between emotion recognition and overall cognitive performance. More specifically, attention, visual memory and visuospatial ability positively correlated with emotion recognition. In regard to specific cognitive domains, attention positively correlated with anger and fear, whereas visual memory correlated with happiness and fear. In conclusion, it seems that the role of underlying visual processes in emotion perception has to be further examined and evaluated in this group of patients.Disclosure of interestThe authors have not supplied their declaration of competing interest.

Facial emotion recognition in borderline personality disorder

2012 ◽  
Vol 43 (9) ◽  
pp. 1953-1963 ◽  
Author(s):  
A. R. Daros ◽  
K. K. Zakzanis ◽  
A. C. Ruocco
Keyword(s):  
Borderline Personality Disorder ◽  
Personality Disorder ◽  
Emotion Recognition ◽  
Emotion Dysregulation ◽  
Emotion Perception ◽  
Borderline Personality ◽  
Facial Emotion Recognition ◽  
Facial Emotion ◽  
Clinical Feature ◽  
Speed Accuracy

BackgroundEmotion dysregulation represents a core symptom of borderline personality disorder (BPD). Deficits in emotion perception are thought to underlie this clinical feature, although studies examining emotion recognition abilities in BPD have yielded inconsistent findings.MethodThe results of 10 studies contrasting facial emotion recognition in patients with BPD (n = 266) and non-psychiatric controls (n = 255) were quantitatively synthesized using meta-analytic techniques.ResultsPatients with BPD were less accurate than controls in recognizing facial displays of anger and disgust, although their most pronounced deficit was in correctly identifying neutral (no emotion) facial expressions. These results could not be accounted for by speed/accuracy in the test-taking approach of BPD patients.ConclusionsPatients with BPD have difficulties recognizing specific negative emotions in faces and may misattribute emotions to faces depicting neutral expressions. The contribution of state-related emotion perception biases to these findings requires further clarification.

Sign in / Sign up

Export Citation Format

Share Document