scholarly journals Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition

2019 ◽  
Vol 11 (5) ◽  
pp. 105 ◽  
Author(s):  
Yongrui Huang ◽  
Jianhao Yang ◽  
Siyu Liu ◽  
Jiahui Pan
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Transfer Learning ◽  
Facial Expressions ◽  
Support Vector ◽  
Fusion Method ◽  
Online Experiment ◽  
Valence Space ◽  
Valence And Arousal ◽  
Boosting Technique

Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.

scholarly journals Personalized models for facial emotion recognition through transfer learning

2020 ◽  
Vol 79 (47-48) ◽  
pp. 35811-35828
Author(s):  
Martina Rescigno ◽  
Matteo Spezialetti ◽  
Silvia Rossi
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Transfer Learning ◽  
Affective Computing ◽  
Human Life ◽  
Personal Data ◽  
Single Subject ◽  
Human Machine Interaction ◽  
Subjective Data ◽  
Valence And Arousal

AbstractEmotions represent a key aspect of human life and behavior. In recent years, automatic recognition of emotions has become an important component in the fields of affective computing and human-machine interaction. Among many physiological and kinematic signals that could be used to recognize emotions, acquiring facial expression images is one of the most natural and inexpensive approaches. The creation of a generalized, inter-subject, model for emotion recognition from facial expression is still a challenge, due to anatomical, cultural and environmental differences. On the other hand, using traditional machine learning approaches to create a subject-customized, personal, model would require a large dataset of labelled samples. For these reasons, in this work, we propose the use of transfer learning to produce subject-specific models for extracting the emotional content of facial images in the valence/arousal dimensions. Transfer learning allows us to reuse the knowledge assimilated from a large multi-subject dataset by a deep-convolutional neural network and employ the feature extraction capability in the single subject scenario. In this way, it is possible to reduce the amount of labelled data necessary to train a personalized model, with respect to relying just on subjective data. Our results suggest that generalized transferred knowledge, in conjunction with a small amount of personal data, is sufficient to obtain high recognition performances and improvement with respect to both a generalized model and personal models. For both valence and arousal dimensions, quite good performances were obtained (RMSE = 0.09 and RMSE = 0.1 for valence and arousal, respectively). Overall results suggested that both the transferred knowledge and the personal data helped in achieving this improvement, even though they alternated in providing the main contribution. Moreover, in this task, we observed that the benefits of transferring knowledge are so remarkable that no specific active or passive sampling techniques are needed for selecting images to be labelled.

scholarly journals Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition

2017 ◽  
Vol 2017 ◽  
pp. 1-8 ◽  
Author(s):  
Yongrui Huang ◽  
Jianhao Yang ◽  
Pengkai Liao ◽  
Jiahui Pan
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expressions ◽  
Multimodal Fusion ◽  
Support Vector ◽  
Basic Emotion ◽  
Input Signals ◽  
Vector Machines ◽  
Fusion Methods ◽  
Multimodal Emotion Recognition

This paper proposes two multimodal fusion methods between brain and peripheral signals for emotion recognition. The input signals are electroencephalogram and facial expression. The stimuli are based on a subset of movie clips that correspond to four specific areas of valance-arousal emotional space (happiness, neutral, sadness, and fear). For facial expression detection, four basic emotion states (happiness, neutral, sadness, and fear) are detected by a neural network classifier. For EEG detection, four basic emotion states and three emotion intensity levels (strong, ordinary, and weak) are detected by two support vector machines (SVM) classifiers, respectively. Emotion recognition is based on two decision-level fusion methods of both EEG and facial expression detections by using a sum rule or a production rule. Twenty healthy subjects attended two experiments. The results show that the accuracies of two multimodal fusion detections are 81.25% and 82.75%, respectively, which are both higher than that of facial expression (74.38%) or EEG detection (66.88%). The combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources.

scholarly journals Review on Emotion Recognition Using Facial Expressions

2021 ◽  
Vol 5 (3) ◽  
pp. 1-4
Author(s):  
Abozar Atya Mohamed Atya ◽  
Khalid Hamid Bilal
Keyword(s):  
Neural Network ◽  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expressions ◽  
Support Vector ◽  
Average Accuracy ◽  
The Face ◽  
Artificial Intelligence Technology ◽  
Artificial Neural Network Ann ◽  
Remarkable Advantage

The advent of artificial intelligence technology has reduced the gap between humans and machines as equips man to create more near-perfect humanoids. Facial expression is an important tool to communicate one’s emotions as a non-verbally overview of emotion recognition using facial expressions. A remarkable advantage of such a technique recently improved public security through tracking and recognizing, thus led to the high attention to keep up the scientific research in the field. The approaches used for facial expression include classifiers like Support Vector Machine (SVM), Artificial Neural Network (ANN), Convolution Neural Network (CNN), Active Appearance and Machine learning which all used to classify emotions based on certain parts of interest on the face like lips, lower jaw, eyebrows, cheeks and many more. By comparison, the reviews have shown that the average accuracy of the basic emotion ranged from 51% up to 100%, whereas carrying through 7% to 13% in the compound emotions, hence indicated that the indispensable emotion is much comfortable to recognize.

scholarly journals Fostering Emotion Recognition in Children with Autism Spectrum Disorder

2021 ◽  
Vol 5 (10) ◽  
pp. 57
Author(s):  
Vinícius Silva ◽  
Filomena Soares ◽  
João Sena Esteves ◽  
Cristina P. Santos ◽  
Ana Paula Pereira
Keyword(s):  
Autism Spectrum Disorder ◽  
Social Interaction ◽  
Emotion Recognition ◽  
Facial Expressions ◽  
Social Communication ◽  
Autism Spectrum ◽  
Spectrum Disorder ◽  
Support Vector ◽  
Robotic Platform ◽  
Children With Asd

Facial expressions are of utmost importance in social interactions, allowing communicative prompts for a speaking turn and feedback. Nevertheless, not all have the ability to express themselves socially and emotionally in verbal and non-verbal communication. In particular, individuals with Autism Spectrum Disorder (ASD) are characterized by impairments in social communication, repetitive patterns of behaviour, and restricted activities or interests. In the literature, the use of robotic tools is reported to promote social interaction with children with ASD. The main goal of this work is to develop a system capable of automatic detecting emotions through facial expressions and interfacing them with a robotic platform (Zeno R50 Robokind® robotic platform, named ZECA) in order to allow social interaction with children with ASD. ZECA was used as a mediator in social communication activities. The experimental setup and methodology for a real-time facial expression (happiness, sadness, anger, surprise, fear, and neutral) recognition system was based on the Intel® RealSense™ 3D sensor and on facial features extraction and multiclass Support Vector Machine classifier. The results obtained allowed to infer that the proposed system is adequate in support sessions with children with ASD, giving a strong indication that it may be used in fostering emotion recognition and imitation skills.

scholarly journals 3D Facial Expression Recognition for Defining Users’ Inner Requirements—An Emotional Design Case Study

2019 ◽  
Vol 9 (11) ◽  
pp. 2218 ◽  
Author(s):  
Maria Grazia Violante ◽  
Federica Marcolin ◽  
Enrico Vezzetti ◽  
Luca Ulrich ◽  
Gianluca Billia ◽  
...  
Keyword(s):  
Facial Expression ◽  
Facial Expressions ◽  
Facial Expression Recognition ◽  
Quality Function Deployment ◽  
Three Dimensional ◽  
Support Vector ◽  
Expression Recognition ◽  
Emotional Design ◽  
3D Data

This study proposes a novel quality function deployment (QFD) design methodology based on customers’ emotions conveyed by facial expressions. The current advances in pattern recognition related to face recognition techniques have fostered the cross-fertilization and pollination between this context and other fields, such as product design and human-computer interaction. In particular, the current technologies for monitoring human emotions have supported the birth of advanced emotional design techniques, whose main focus is to convey users’ emotional feedback into the design of novel products. As quality functional deployment aims at transforming the voice of customers into engineering features of a product, it appears to be an appropriate and promising nest in which to embed users’ emotional feedback with new emotional design methodologies, such as facial expression recognition. This way, the present methodology consists in interviewing the user and acquiring his/her face with a depth camera (allowing three-dimensional (3D) data), clustering the face information into different emotions with a support vector machine classificator, and assigning customers’ needs weights relying on the detected facial expressions. The proposed method has been applied to a case study in the context of agriculture and validated by a consortium. The approach appears sound and capable of collecting the unconscious feedback of the interviewee.

scholarly journals Student Prize-Winning Abstracts 2010

2011 ◽  
Vol 12 (1) ◽  
pp. 77-77
Author(s):  
Sharpley Hsieh ◽  
Olivier Piguet ◽  
John R. Hodges
Keyword(s):  
Emotion Recognition ◽  
Facial Expressions ◽  
Negative Emotions ◽  
Emotion Detection ◽  
Recognition Of Facial Expressions ◽  
Patient Groups ◽  
The Neural Networks ◽  
Emotional Dimensions ◽  
Recognition Of Emotions ◽  
Valence And Arousal

AbstractIntroduction: Frontotemporal dementia (FTD) is a progressive neurode-generative brain disease characterised clinically by abnormalities in behaviour, cognition and language. Two subgroups, behavioural-variant FTD (bvFTD) and semantic dementia (SD), also show impaired emotion recognition particularly for negative emotions. This deficit has been demonstrated using visual stimuli such as facial expressions. Whether recognition of emotions conveyed through other modalities — for example, music — is also impaired has not been investigated. Methods: Patients with bvFTD, SD and Alzheimer's disease (AD), as well as healthy age-matched controls, labeled tunes according to the emotion conveyed (happy, sad, peaceful or scary). In addition, each tune was also rated along two orthogonal emotional dimensions: valence (pleasant/unpleasant) and arousal (stimulating/relaxing). Participants also undertook a facial emotion recognition test and other cognitive tests. Integrity of basic music detection (tone, tempo) was also examined. Results: Patient groups were matched for disease severity. Overall, patients did not differ from controls with regard to basic music processing or for the recognition of facial expressions. Ratings of valence and arousal were similar across groups. In contrast, SD patients were selectively impaired at recognising music conveying negative emotions (sad and scary). Patients with bvFTD did not differ from controls. Conclusion: Recognition of emotions in music appears to be selectively affected in some FTD subgroups more than others, a disturbance of emotion detection which appears to be modality specific. This finding suggests dissociation in the neural networks necessary for the processing of emotions depending on modality.

scholarly journals Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

2021 ◽  
Vol 11 (24) ◽  
pp. 11738
Author(s):  
Thomas Teixeira ◽  
Éric Granger ◽  
Alessandro Lameiras Koerich
Keyword(s):  
Neural Networks ◽  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expressions ◽  
Convolutional Neural Networks ◽  
Affective Computing ◽  
Spatial Information ◽  
Short Term Memory ◽  
State Of The Art ◽  
Fine Tuning

Facial expressions are one of the most powerful ways to depict specific patterns in human behavior and describe the human emotional state. However, despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot correctly handle variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task, even for humans. This paper investigates the suitability of state-of-the-art deep learning architectures based on convolutional neural networks (CNNs) to deal with long video sequences captured in the wild for continuous emotion recognition. For such an aim, several 2D CNN models that were designed to model spatial information are extended to allow spatiotemporal representation learning from videos, considering a complex and multi-dimensional emotion space, where continuous values of valence and arousal must be predicted. We have developed and evaluated convolutional recurrent neural networks, combining 2D CNNs and long short term-memory units and inflated 3D CNN models, which are built by inflating the weights of a pre-trained 2D CNN model during fine-tuning, using application-specific videos. Experimental results on the challenging SEWA-DB dataset have shown that these architectures can effectively be fine-tuned to encode spatiotemporal information from successive raw pixel images and achieve state-of-the-art results on such a dataset.

Brain Computer Interface Using Motor Imagery and Facial Expressions to Control a Mobile Robot

2018 ◽  
Author(s):  
James Kuffuor ◽  
Biswanath Samanta
Keyword(s):  
Facial Expression ◽  
Mobile Robot ◽  
Facial Expressions ◽  
Motor Imagery ◽  
Brain Computer Interface ◽  
Hybrid Approach ◽  
Computer Interface ◽  
Support Vector ◽  
Correct Identification ◽  
Characteristic Features

A study is presented on brain computer interface (BCI) using motor imagery (MI) and facial expressions to control a mobile robot. Traditionally, only MI signals are used in BCI applications. In this paper a hybrid approach of using both MI and facial expression stimulations for BCI is proposed. Electroencephalography (EEG) signals were acquired using a sensor system and processed for several MI and facial expressions to extract characteristic features. The features were used to train support vector machine (SVM) based classifiers and the trained classifiers were used to recognize test signals for correct identification of MI and facial expressions. A system was developed to implement the BCI using MI and facial expressions to control a mobile robot. Results of training using MI and facial expressions, individually and together are presented for comparison. The combined features from MI and facial expression stimulations were found to give performance similar to facial expressions but better than MI only.

scholarly journals Simultaneous evolutionary neural network based automated video based facial expression analysis

2017 ◽  
Vol 7 (1.1) ◽  
pp. 125
Author(s):  
S. Jeyalaksshmi ◽  
S. Prasanna
Keyword(s):  
Neural Network ◽  
Facial Expression ◽  
Facial Expressions ◽  
Real Life ◽  
Support Vector ◽  
Detection Accuracy ◽  
Physical Fatigue ◽  
Adaboost Algorithm ◽  
Evolutionary Neural Network ◽  
Fatigue Detection

In real life scenario, facial expressions and emotions are nothing but responses to the external and internal events of human being. In Human Computer Interaction (HCI), recognition of end user’s expressions and emotions from the video streaming plays very important role. In such systems it is required to track the dynamic changes in human face movements quickly in order to deliver the required response system. In real time applications, this Facial Expression Recognition (FER) is very helpful like physical fatigue detection based on facial detection and expressions such as driver fatigue detection in order to prevent the accidents on road. Face expression based physical fatigue analysis or detection is out of scope of this work, but this work proposed a Simultaneous Evolutionary Neural Network (SENN) classification scheme is proposed for recognising human emotion or expression. In this work, at first, automatically detects and tracks facial landmarks in videos, and face is detected by using enhanced adaboost algorithm with haar features. Then, in order to describe facial expression modifications, geometric features are taken out and the Local Binary Pattern (LBP) is extracted to improve the detection accuracy and it has a much lower-dimensional size. With the aim of examining the temporal facial expression modifications, we apply SENN probabilistic classifiers, which examine the facial expressions in individual frames, and after that promulgate the likelihoods during the course of the video to take the temporal features of facial expressions such as glad, sad, anger, and fear feelings. The experimental results show that the performance of proposed SENN scheme is attained better results compared than existing recognition schemes like Time-Delay Neural Network with Support Vector Regression (TDNN-SVR) and SVR. 

Sign in / Sign up

Export Citation Format

Share Document