Emotion Recognition from Skeletal Movements

Automatic emotion recognition has become an important trend in many artificial intelligence (AI) based applications and has been widely explored in recent years. Most research in the area of automated emotion recognition is based on facial expressions or speech signals. Although the influence of the emotional state on body movements is undeniable, this source of expression is still underestimated in automatic analysis. In this paper, we propose a novel method to recognise seven basic emotional states—namely, happy, sad, surprise, fear, anger, disgust and neutral—utilising body movement. We analyse motion capture data under seven basic emotional states recorded by professional actor/actresses using Microsoft Kinect v2 sensor. We propose a new representation of affective movements, based on sequences of body joints. The proposed algorithm creates a sequential model of affective movement based on low level features inferred from the spacial location and the orientation of joints within the tracked skeleton. In the experimental results, different deep neural networks were employed and compared to recognise the emotional state of the acquired motion sequences. The experimental results conducted in this work show the feasibility of automatic emotion recognition from sequences of body gestures, which can serve as an additional source of information in multimodal emotion recognition.

Download Full-text

Emotion recognition using Kinect motion capture data of human gaits

PeerJ ◽

10.7717/peerj.2364 ◽

2016 ◽

Vol 4 ◽

pp. e2364 ◽

Cited By ~ 18

Author(s):

Shun Li ◽

Liqing Cui ◽

Changye Zhu ◽

Baobin Li ◽

Nan Zhao ◽

...

Keyword(s):

Emotion Recognition ◽

Emotional State ◽

Low Cost ◽

Information Source ◽

Principal Component ◽

Microsoft Kinect ◽

Main Body ◽

Emotional States ◽

System Transformation ◽

Neutral State

Automatic emotion recognition is of great value in many applications, however, to fully display the application value of emotion recognition, more portable, non-intrusive, inexpensive technologies need to be developed. Human gaits could reflect the walker’s emotional state, and could be an information source for emotion recognition. This paper proposed a novel method to recognize emotional state through human gaits by using Microsoft Kinect, a low-cost, portable, camera-based sensor. Fifty-nine participants’ gaits under neutral state, induced anger and induced happiness were recorded by two Kinect cameras, and the original data were processed through joint selection, coordinate system transformation, sliding window gauss filtering, differential operation, and data segmentation. Features of gait patterns were extracted from 3-dimentional coordinates of 14 main body joints by Fourier transformation and Principal Component Analysis (PCA). The classifiers NaiveBayes, RandomForests, LibSVM and SMO (Sequential Minimal Optimization) were trained and evaluated, and the accuracy of recognizing anger and happiness from neutral state achieved 80.5% and 75.4%. Although the results of distinguishing angry and happiness states were not ideal in current study, it showed the feasibility of automatically recognizing emotional states from gaits, with the characteristics meeting the application requirements.

Download Full-text

Speech-based human emotion recognition

10.32920/ryerson.14651964 ◽

2021 ◽

Author(s):

Talieh Seyed Tabtabae

Keyword(s):

Emotion Recognition ◽

Emotional State ◽

Speaker Identification ◽

Recognition System ◽

Research Area ◽

Speech Signals ◽

Spectral Features ◽

Emotional States ◽

Human Communication ◽

Set Up

Automatic Emotion Recognition (AER) is an emerging research area in the Human-Computer Interaction (HCI) field. As Computers are becoming more and more popular every day, the study of interaction between humans (users) and computers is catching more attention. In order to have a more natural and friendly interface between humans and computers, it would be beneficial to give computers the ability to recognize situations the same way a human does. Equipped with an emotion recognition system, computers will be able to recognize their users' emotional state and show the appropriate reaction to that. In today's HCI systems, machines can recognize the speaker and also content of the speech, using speech recognition and speaker identification techniques. If machines are equipped with emotion recognition techniques, they can also know "how it is said" to react more appropriately, and make the interaction more natural. One of the most important human communication channels is the auditory channel which carries speech and vocal intonation. In fact people can perceive each other's emotional state by the way they talk. Therefore in this work the speech signals are analyzed in order to set up an automatic system which recognizes the human emotional state. Six discrete emotional states have been considered and categorized in this research: anger, happiness, fear, surprise, sadness, and disgust. A set of novel spectral features are proposed in this contribution. Two approaches are applied and the results are compared. In the first approach, all the acoustic features are extracted from consequent frames along the speech signals. The statistical values of features are considered to constitute the features vectors. Suport Vector Machine (SVM), which is a relatively new approach in the field of machine learning is used to classify the emotional states. In the second approach, spectral features are extracted from non-overlapping logarithmically-spaced frequency sub-bands. In order to make use of all the extracted information, sequence discriminant SVMs are adopted. The empirical results show that the employed techniques are very promising.

Download Full-text

Automatic Emotion Recognition Based on Non-Contact Gaits Information

Advanced Methodologies and Technologies in Artificial Intelligence, Computer Simulation, and Human-Computer Interaction - Advances in Computer and Electrical Engineering ◽

10.4018/978-1-5225-7368-5.ch005 ◽

2019 ◽

pp. 54-67

Author(s):

Jingying Wang ◽

Baobin Li ◽

Changye Zhu ◽

Shun Li ◽

Tingshao Zhu

Keyword(s):

Machine Learning ◽

Emotion Recognition ◽

Emotional State ◽

Experimental Results ◽

Learning Models ◽

Face Expression ◽

Gait Features ◽

Emotion Labels ◽

Machine Learning Models

Automatic emotion recognition was of great value in many applications; however, to fully display the application value of emotion recognition, more portable, non-intrusive, inexpensive technologies need to be developed. Except face expression and voices, human gaits could reflect the walker's emotional state too. By utilizing 59 participants' gaits data with emotion labels, the authors train machine learning models that are able to “sense” individual emotion. Experimental results show these models work very well and prove that gait features are effective in characterizing and recognizing emotions.

Download Full-text

EEG-based emotion recognition with deep convolutional neural networks

Biomedical Engineering / Biomedizinische Technik ◽

10.1515/bmt-2019-0306 ◽

2020 ◽

Vol 0 (0) ◽

Author(s):

Mehmet Akif Ozdemir ◽

Murside Degirmenci ◽

Elif Izci ◽

Aydin Akan

Keyword(s):

Neural Networks ◽

Emotion Recognition ◽

Convolutional Neural Networks ◽

Spatial Information ◽

Emotional State ◽

Human Interaction ◽

Emotional States ◽

Eeg Signals ◽

Deep Convolutional Neural Networks ◽

Spectral Topology

AbstractThe emotional state of people plays a key role in physiological and behavioral human interaction. Emotional state analysis entails many fields such as neuroscience, cognitive sciences, and biomedical engineering because the parameters of interest contain the complex neuronal activities of the brain. Electroencephalogram (EEG) signals are processed to communicate brain signals with external systems and make predictions over emotional states. This paper proposes a novel method for emotion recognition based on deep convolutional neural networks (CNNs) that are used to classify Valence, Arousal, Dominance, and Liking emotional states. Hence, a novel approach is proposed for emotion recognition with time series of multi-channel EEG signals from a Database for Emotion Analysis and Using Physiological Signals (DEAP). We propose a new approach to emotional state estimation utilizing CNN-based classification of multi-spectral topology images obtained from EEG signals. In contrast to most of the EEG-based approaches that eliminate spatial information of EEG signals, converting EEG signals into a sequence of multi-spectral topology images, temporal, spectral, and spatial information of EEG signals are preserved. The deep recurrent convolutional network is trained to learn important representations from a sequence of three-channel topographical images. We have achieved test accuracy of 90.62% for negative and positive Valence, 86.13% for high and low Arousal, 88.48% for high and low Dominance, and finally 86.23% for like–unlike. The evaluations of this method on emotion recognition problem revealed significant improvements in the classification accuracy when compared with other studies using deep neural networks (DNNs) and one-dimensional CNNs.

Download Full-text

Human Emotional State Assessment Based on a Video Portrayal

Experimental Psychology (Russia) ◽

10.17759/exppsy.2020130401 ◽

2020 ◽

Vol 13 (4) ◽

pp. 4-24 ◽

Cited By ~ 1

Author(s):

V.A. Barabanschikov ◽

E.V. Suvorova

Keyword(s):

Emotion Recognition ◽

Recognition Test ◽

Emotional State ◽

State Assessment ◽

Identification Accuracy ◽

Emotional Expressions ◽

Emotional States ◽

Wide Range ◽

Similarities And Differences ◽

Valence And Arousal

The article is devoted to the results of approbation of the Geneva Emotion Recognition Test (GERT), a Swiss method for assessing dynamic emotional states, on Russian sample. Identification accuracy and the categorical fields’ structure of emotional expressions of a “living” face are analysed. Similarities and differences in the perception of affective groups of dynamic emotions in the Russian and Swiss samples are considered. A number of patterns of recognition of multi-modal expressions with changes in valence and arousal of emotions are described. Differences in the perception of dynamics and statics of emotional expressions are revealed. GERT method confirmed it’s high potential for solving a wide range of academic and applied problems.

Download Full-text

Multimodal Emotion Recognition from Art Using Sequential Co-Attention

Journal of Imaging ◽

10.3390/jimaging7080157 ◽

2021 ◽

Vol 7 (8) ◽

pp. 157

Author(s):

Tsegaye Misikir Tashu ◽

Sakina Hajiyeva ◽

Tomas Horvath

Keyword(s):

Feature Extraction ◽

Emotion Recognition ◽

Experimental Results ◽

Multimodal Emotion Recognition

In this study, we present a multimodal emotion recognition architecture that uses both feature-level attention (sequential co-attention) and modality attention (weighted modality fusion) to classify emotion in art. The proposed architecture helps the model to focus on learning informative and refined representations for both feature extraction and modality fusion. The resulting system can be used to categorize artworks according to the emotions they evoke; recommend paintings that accentuate or balance a particular mood; search for paintings of a particular style or genre that represents custom content in a custom state of impact. Experimental results on the WikiArt emotion dataset showed the efficiency of the approach proposed and the usefulness of three modalities in emotion recognition.

Download Full-text

Emotion Recognition Using Smart Watch Sensor Data: A Mixed-Design Study (Preprint)

10.2196/preprints.10153 ◽

2018 ◽

Cited By ~ 1

Author(s):

Juan Carlos Quiroz ◽

Elena Geangu ◽

Min Hooi Yong

Keyword(s):

Heart Rate ◽

Emotion Recognition ◽

Emotional State ◽

Sensor Data ◽

Emotional States ◽

Mixed Design ◽

Design Study ◽

Heart Rate Monitor ◽

Smart Watch

BACKGROUND Research in psychology has shown that the way a person walks reflects that person's current mood (or emotional state). Recent studies have started using smartphones to detect emotional states from movement data. OBJECTIVE This study investigates the use of movement sensor data from a smart watch to infer an individual's emotional state. We present our findings on a user study with 50 participants. METHODS The experimental design is a mixed-design study; within-subjects (emotions; happy, sad, neutral) and between-subjects (stimulus type: audio visual "movie clips", audio "music clips"). Each participant experienced both emotions in a single stimulus type. All participants walked 250m while wearing a smart watch on one wrist and a heart rate monitor strap on their chest. They also had to answer a short questionnaire (20 items; PANAS) before and after experiencing each emotion. The heart rate monitor serves as a supplementary information to our data. We performed time-series analysis on the data from the smart watch and a t-test on the questionnaire items to measure the change in emotional state. The heart rate data was analyzed using one-way ANOVA. We extracted features from the time-series using sliding windows and used the features to train and validate classifiers that determine an individual's emotion. RESULTS We had 50 young adults participate in our study, with 49 included for the affective PANAS questionnaire and all for the feature extraction. Participants reported feeling less negative affect after watching sad videos or after listening to the sad music, P < .006. For the task of emotion recognition using classifiers, our results show that the personal models outperformed personal baselines, and achieve median accuracies higher than 78% for all conditions of the design study for the binary classification of happiness vs sadness. CONCLUSIONS Our findings show that we are able to detect the changes in emotional state with data obtained from the smartwatch as well as behavioral responses. Together with the high accuracies achieved across all users for the classification of happy vs sad emotional states, this is further evidence for the hypothesis that movement sensor data can be used for emotion recognition.

Download Full-text

Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces

Brain Sciences ◽

10.3390/brainsci10100687 ◽

2020 ◽

Vol 10 (10) ◽

pp. 687 ◽

Cited By ~ 1

Author(s):

Zhipeng He ◽

Zina Li ◽

Fuzhou Yang ◽

Lei Wang ◽

Jingcong Li ◽

...

Keyword(s):

Emotion Recognition ◽

Affective Computing ◽

Experimental Results ◽

Brain Computer Interfaces ◽

Continuous Development ◽

Research Directions ◽

Sensory Stimuli ◽

Computer Interfaces ◽

Brain Signals ◽

Multimodal Emotion Recognition

With the continuous development of portable noninvasive human sensor technologies such as brain–computer interfaces (BCI), multimodal emotion recognition has attracted increasing attention in the area of affective computing. This paper primarily discusses the progress of research into multimodal emotion recognition based on BCI and reviews three types of multimodal affective BCI (aBCI): aBCI based on a combination of behavior and brain signals, aBCI based on various hybrid neurophysiology modalities and aBCI based on heterogeneous sensory stimuli. For each type of aBCI, we further review several representative multimodal aBCI systems, including their design principles, paradigms, algorithms, experimental results and corresponding advantages. Finally, we identify several important issues and research directions for multimodal emotion recognition based on BCI.

Download Full-text

Deep time-delay Markov network for prediction and modeling the stress and emotions state transition

Scientific Reports ◽

10.1038/s41598-020-75155-w ◽

2020 ◽

Vol 10 (1) ◽

Author(s):

Barlian Henryranu Prasetio ◽

Hiroki Tamura ◽

Koichi Tanno

Keyword(s):

Time Delay ◽

State Transition ◽

Emotional State ◽

Emotional States ◽

Deep Time ◽

Training Time ◽

Markov Network ◽

Prediction Result ◽

Novel Method ◽

Ablation Experiment

Abstract To recognize stress and emotion, most of the existing methods only observe and analyze speech patterns from present-time features. However, an emotion (especially for stress) can change because it was triggered by an event while speaking. To address this issue, we propose a novel method for predicting stress and emotions by analyzing prior emotional states. We named this method the deep time-delay Markov network (DTMN). Structurally, the proposed DTMN contains a hidden Markov model (HMM) and a time-delay neural network (TDNN). We evaluated the effectiveness of the proposed DTMN by comparing it with several state transition methods in predicting an emotional state from time-series (sequences) speech data of the SUSAS dataset. The experimental results show that the proposed DTMN can accurately predict present emotional states by outperforming the baseline systems in terms of the prediction error rate (PER). We then modeled the emotional state transition using a finite Markov chain based on the prediction result. We also conducted an ablation experiment to observe the effect of different HMM values and TDNN parameters on the prediction result and the computational training time of the proposed DTMN.

Download Full-text

Emotion Recognition Using Smart Watch Sensor Data: Mixed-Design Study

JMIR Mental Health ◽

10.2196/10153 ◽

2018 ◽

Vol 5 (3) ◽

pp. e10153 ◽

Cited By ~ 10

Author(s):

Juan Carlos Quiroz ◽

Elena Geangu ◽

Min Hooi Yong

Keyword(s):

Heart Rate ◽

Emotion Recognition ◽

Emotional State ◽

Sensor Data ◽

Emotional States ◽

Mixed Design ◽

Design Study ◽

Heart Rate Monitor ◽

Smart Watch

Background Research in psychology has shown that the way a person walks reflects that person’s current mood (or emotional state). Recent studies have used mobile phones to detect emotional states from movement data. Objective The objective of our study was to investigate the use of movement sensor data from a smart watch to infer an individual’s emotional state. We present our findings of a user study with 50 participants. Methods The experimental design is a mixed-design study: within-subjects (emotions: happy, sad, and neutral) and between-subjects (stimulus type: audiovisual “movie clips” and audio “music clips”). Each participant experienced both emotions in a single stimulus type. All participants walked 250 m while wearing a smart watch on one wrist and a heart rate monitor strap on the chest. They also had to answer a short questionnaire (20 items; Positive Affect and Negative Affect Schedule, PANAS) before and after experiencing each emotion. The data obtained from the heart rate monitor served as supplementary information to our data. We performed time series analysis on data from the smart watch and a t test on questionnaire items to measure the change in emotional state. Heart rate data was analyzed using one-way analysis of variance. We extracted features from the time series using sliding windows and used features to train and validate classifiers that determined an individual’s emotion. Results Overall, 50 young adults participated in our study; of them, 49 were included for the affective PANAS questionnaire and 44 for the feature extraction and building of personal models. Participants reported feeling less negative affect after watching sad videos or after listening to sad music, P<.006. For the task of emotion recognition using classifiers, our results showed that personal models outperformed personal baselines and achieved median accuracies higher than 78% for all conditions of the design study for binary classification of happiness versus sadness. Conclusions Our findings show that we are able to detect changes in the emotional state as well as in behavioral responses with data obtained from the smartwatch. Together with high accuracies achieved across all users for classification of happy versus sad emotional states, this is further evidence for the hypothesis that movement sensor data can be used for emotion recognition.

Download Full-text