Emotion Recognition by Correlating Facial Expressions and EEG Analysis

Emotion recognition is a fundamental task that any affective computing system must perform to adapt to the user’s current mood. The analysis of electroencephalography signals has gained notoriety in studying human emotions because of its non-invasive nature. This paper presents a two-stage deep learning model to recognize emotional states by correlating facial expressions and brain signals. Most of the works related to the analysis of emotional states are based on analyzing large segments of signals, generally as long as the evoked potential lasts, which could cause many other phenomena to be involved in the recognition process. Unlike with other phenomena, such as epilepsy, there is no clearly defined marker of when an event begins or ends. The novelty of the proposed model resides in the use of facial expressions as markers to improve the recognition process. This work uses a facial emotion recognition technique (FER) to create identifiers each time an emotional response is detected and uses them to extract segments of electroencephalography (EEG) records that a priori will be considered relevant for the analysis. The proposed model was tested on the DEAP dataset.

Download Full-text

Machine-Learning-Based Emotion Recognition System Using EEG Signals

Computers ◽

10.3390/computers9040095 ◽

2020 ◽

Vol 9 (4) ◽

pp. 95

Author(s):

Rania Alhalaseh ◽

Suzan Alasasfeh

Keyword(s):

Signal Processing ◽

Feature Extraction ◽

Emotion Recognition ◽

Emotional States ◽

Eeg Signals ◽

Processing Stage ◽

Brain Signals ◽

Specificity And Sensitivity ◽

Mode Decomposition ◽

Proposed Model

Many scientific studies have been concerned with building an automatic system to recognize emotions, and building such systems usually relies on brain signals. These studies have shown that brain signals can be used to classify many emotional states. This process is considered difficult, especially since the brain’s signals are not stable. Human emotions are generated as a result of reactions to different emotional states, which affect brain signals. Thus, the performance of emotion recognition systems by brain signals depends on the efficiency of the algorithms used to extract features, the feature selection algorithm, and the classification process. Recently, the study of electroencephalography (EEG) signaling has received much attention due to the availability of several standard databases, especially since brain signal recording devices have become available in the market, including wireless ones, at reasonable prices. This work aims to present an automated model for identifying emotions based on EEG signals. The proposed model focuses on creating an effective method that combines the basic stages of EEG signal handling and feature extraction. Different from previous studies, the main contribution of this work relies in using empirical mode decomposition/intrinsic mode functions (EMD/IMF) and variational mode decomposition (VMD) for signal processing purposes. Despite the fact that EMD/IMFs and VMD methods are widely used in biomedical and disease-related studies, they are not commonly utilized in emotion recognition. In other words, the methods used in the signal processing stage in this work are different from the methods used in literature. After the signal processing stage, namely in the feature extraction stage, two well-known technologies were used: entropy and Higuchi’s fractal dimension (HFD). Finally, in the classification stage, four classification methods were used—naïve Bayes, k-nearest neighbor (k-NN), convolutional neural network (CNN), and decision tree (DT)—for classifying emotional states. To evaluate the performance of our proposed model, experiments were applied to a common database called DEAP based on many evaluation models, including accuracy, specificity, and sensitivity. The experiments showed the efficiency of the proposed method; a 95.20% accuracy was achieved using the CNN-based method.

Download Full-text

A Machine Learning Approach to EEG-based Prediction of Human Affective States Using Recursive Feature Elimination Method

MATEC Web of Conferences ◽

10.1051/matecconf/202133504001 ◽

2021 ◽

Vol 335 ◽

pp. 04001

Author(s):

Didar Dadebayev ◽

Goh Wei Wei ◽

Tan Ee Xion

Keyword(s):

Feature Selection ◽

Fractal Dimension ◽

Emotion Recognition ◽

Affective Computing ◽

Feature Selection Method ◽

Recursive Feature Elimination ◽

Support Vector ◽

Emotional States ◽

Affective States ◽

Emotion Classification

Emotion recognition, as a branch of affective computing, has attracted great attention in the last decades as it can enable more natural brain-computer interface systems. Electroencephalography (EEG) has proven to be an effective modality for emotion recognition, with which user affective states can be tracked and recorded, especially for primitive emotional events such as arousal and valence. Although brain signals have been shown to correlate with emotional states, the effectiveness of proposed models is somewhat limited. The challenge is improving accuracy, while appropriate extraction of valuable features might be a key to success. This study proposes a framework based on incorporating fractal dimension features and recursive feature elimination approach to enhance the accuracy of EEG-based emotion recognition. The fractal dimension and spectrum-based features to be extracted and used for more accurate emotional state recognition. Recursive Feature Elimination will be used as a feature selection method, whereas the classification of emotions will be performed by the Support Vector Machine (SVM) algorithm. The proposed framework will be tested with a widely used public database, and results are expected to demonstrate higher accuracy and robustness compared to other studies. The contributions of this study are primarily about the improvement of the EEG-based emotion classification accuracy. There is a potential restriction of how generic the results can be as different EEG dataset might yield different results for the same framework. Therefore, experimenting with different EEG dataset and testing alternative feature selection schemes can be very interesting for future work.

Download Full-text

The Use of Portable EEG Devices in Development of Immersive Virtual Reality Environments for Converting Emotional States into Specific Commands

Proceedings ◽

10.3390/proceedings2020054043 ◽

2020 ◽

Vol 54 (1) ◽

pp. 43

Author(s):

Catarina Sá ◽

Paulo Veloso Gomes ◽

António Marques ◽

António Correia

Keyword(s):

Virtual Reality ◽

Portable Devices ◽

Immersive Virtual Reality ◽

Emotional States ◽

Social Functions ◽

Interactive Environments ◽

Oculus Rift ◽

Non Invasive ◽

Brain Signals

The application of electroencephalography electrodes in Virtual Reality (VR) glasses allows users to relate cognitive, emotional, and social functions with the exposure to certain stimuli. The development of non-invasive portable devices, coupled with VR, allows for the collection of electroencephalographic data. One of the devices that embraced this new trend is Looxid LinkTM, a system that adds electroencephalography to HTC VIVETM, VIVE ProTM, VIVE Pro EyeTM, or Oculus Rift STM glasses to create interactive environments using brain signals. This work analyzes the possibility of using the Looxid LinkTM device to perceive, evaluate and monitor the emotions of users exposed to VR.

Download Full-text

Analyzing Emotional Oscillatory Brain Network for Valence and Arousal-Based Emotion Recognition Using EEG Data

International Journal of Information Technology & Decision Making ◽

10.1142/s0219622019500238 ◽

2019 ◽

Vol 18 (04) ◽

pp. 1359-1378

Author(s):

Jianzhuo Yan ◽

Hongzhi Kuai ◽

Jianhui Chen ◽

Ning Zhong

Keyword(s):

Emotion Recognition ◽

Affective Computing ◽

Affective State ◽

Feature Selection Method ◽

Emotional Valence ◽

Brain Network ◽

Previous Method ◽

Emotional States ◽

Eeg Data ◽

Valence And Arousal

Emotion recognition is a highly noteworthy and challenging work in both cognitive science and affective computing. Currently, neurobiology studies have revealed the partially synchronous oscillating phenomenon within brain, which needs to be analyzed from oscillatory synchronization. This combination of oscillations and synchronism is worthy of further exploration to achieve inspiring learning of the emotion recognition models. In this paper, we propose a novel approach of valence and arousal-based emotion recognition using EEG data. First, we construct the emotional oscillatory brain network (EOBN) inspired by the partially synchronous oscillating phenomenon for emotional valence and arousal. And then, a coefficient of variation and Welch’s [Formula: see text]-test based feature selection method is used to identify the core pattern (cEOBN) within EOBN for different emotional dimensions. Finally, an emotional recognition model (ERM) is built by combining cEOBN-inspired information obtained in the above process and different classifiers. The proposed approach can combine oscillation and synchronization characteristics of multi-channel EEG signals for recognizing different emotional states under the valence and arousal dimensions. The cEOBN-based inspired information can effectively reduce the dimensionality of the data. The experimental results show that the previous method can be used to detect affective state at a reasonable level of accuracy.

Download Full-text

Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

Applied Sciences ◽

10.3390/app112411738 ◽

2021 ◽

Vol 11 (24) ◽

pp. 11738

Author(s):

Thomas Teixeira ◽

Éric Granger ◽

Alessandro Lameiras Koerich

Keyword(s):

Neural Networks ◽

Facial Expression ◽

Emotion Recognition ◽

Facial Expressions ◽

Convolutional Neural Networks ◽

Affective Computing ◽

Spatial Information ◽

Short Term Memory ◽

State Of The Art ◽

Fine Tuning

Facial expressions are one of the most powerful ways to depict specific patterns in human behavior and describe the human emotional state. However, despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot correctly handle variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task, even for humans. This paper investigates the suitability of state-of-the-art deep learning architectures based on convolutional neural networks (CNNs) to deal with long video sequences captured in the wild for continuous emotion recognition. For such an aim, several 2D CNN models that were designed to model spatial information are extended to allow spatiotemporal representation learning from videos, considering a complex and multi-dimensional emotion space, where continuous values of valence and arousal must be predicted. We have developed and evaluated convolutional recurrent neural networks, combining 2D CNNs and long short term-memory units and inflated 3D CNN models, which are built by inflating the weights of a pre-trained 2D CNN model during fine-tuning, using application-specific videos. Experimental results on the challenging SEWA-DB dataset have shown that these architectures can effectively be fine-tuned to encode spatiotemporal information from successive raw pixel images and achieve state-of-the-art results on such a dataset.

Download Full-text

A Review of Emotion Recognition Based on EEG using DEAP Dataset

International Journal of Scientific Research in Science Engineering and Technology ◽

10.32628/ijsrset218352 ◽

2021 ◽

pp. 298-303

Author(s):

Rama Chaudhary ◽

Ram Avtar Jaswal

Keyword(s):

Facial Expression ◽

Emotion Recognition ◽

Facial Expressions ◽

Physiological Signals ◽

Emotional States ◽

Modern Time ◽

Human Machine Interaction ◽

Electrocardiogram Ecg ◽

Electroencephalogram Eeg ◽

Machine Interaction

In modern time, the human-machine interaction technology has been developed so much for recognizing human emotional states depending on physiological signals. The emotional states of human can be recognized by using facial expressions, but sometimes it doesn’t give accurate results. For example, if we detect the accuracy of facial expression of sad person, then it will not give fully satisfied result because sad expression also include frustration, irritation, anger, etc. therefore, it will not be possible to determine the particular expression. Therefore, emotion recognition using Electroencephalogram (EEG), Electrocardiogram (ECG) has gained so much attraction because these are based on brain and heart signals respectively. So, after analyzing all the factors, it is decided to recognize emotional states based on EEG using DEAP Dataset. So that, the better accuracy can be achieved.

Download Full-text

Generalisation and robustness investigation for facial and speech emotion recognition using bio-inspired spiking neural networks

Soft Computing ◽

10.1007/s00500-020-05501-7 ◽

2021 ◽

Vol 25 (3) ◽

pp. 1717-1730

Author(s):

Esma Mansouri-Benssassi ◽

Juan Ye

Keyword(s):

Facial Expression ◽

Emotion Recognition ◽

Affective Computing ◽

Third Party ◽

Speech Emotion Recognition ◽

Expression Recognition ◽

Emotional States ◽

Learning Approaches ◽

Feature Representations ◽

The Face

AbstractEmotion recognition through facial expression and non-verbal speech represents an important area in affective computing. They have been extensively studied from classical feature extraction techniques to more recent deep learning approaches. However, most of these approaches face two major challenges: (1) robustness—in the face of degradation such as noise, can a model still make correct predictions? and (2) cross-dataset generalisation—when a model is trained on one dataset, can it be used to make inference on another dataset?. To directly address these challenges, we first propose the application of a spiking neural network (SNN) in predicting emotional states based on facial expression and speech data, then investigate, and compare their accuracy when facing data degradation or unseen new input. We evaluate our approach on third-party, publicly available datasets and compare to the state-of-the-art techniques. Our approach demonstrates robustness to noise, where it achieves an accuracy of 56.2% for facial expression recognition (FER) compared to 22.64% and 14.10% for CNN and SVM, respectively, when input images are degraded with the noise intensity of 0.5, and the highest accuracy of 74.3% for speech emotion recognition (SER) compared to 21.95% of CNN and 14.75% for SVM when audio white noise is applied. For generalisation, our approach achieves consistently high accuracy of 89% for FER and 70% for SER in cross-dataset evaluation and suggests that it can learn more effective feature representations, which lead to good generalisation of facial features and vocal characteristics across subjects.

Download Full-text

Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing

Sensors ◽

10.3390/s20185163 ◽

2020 ◽

Vol 20 (18) ◽

pp. 5163

Author(s):

Javier Marín-Morales ◽

Carmen Llinares ◽

Jaime Guixeres ◽

Mariano Alcañiz

Keyword(s):

Virtual Reality ◽

Emotion Recognition ◽

Affective Computing ◽

Critical Role ◽

Machine Learning Techniques ◽

Future Research ◽

Immersive Virtual Reality ◽

Emotional States ◽

Daily Lives ◽

Research Areas

Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which allows researchers to simulate environments in controlled laboratory conditions with high levels of sense of presence and interactivity, is becoming more popular in emotion research. Moreover, its synergy with implicit measurements and machine-learning techniques has the potential to impact transversely in many research areas, opening new opportunities for the scientific community. This paper presents a systematic review of the emotion recognition research undertaken with physiological and behavioural measures using head-mounted displays as elicitation devices. The results highlight the evolution of the field, give a clear perspective using aggregated analysis, reveal the current open issues and provide guidelines for future research.

Download Full-text

Advances in Multimodal Emotion Recognition Based on Brain–Computer Interfaces

Brain Sciences ◽

10.3390/brainsci10100687 ◽

2020 ◽

Vol 10 (10) ◽

pp. 687 ◽

Cited By ~ 1

Author(s):

Zhipeng He ◽

Zina Li ◽

Fuzhou Yang ◽

Lei Wang ◽

Jingcong Li ◽

...

Keyword(s):

Emotion Recognition ◽

Affective Computing ◽

Experimental Results ◽

Brain Computer Interfaces ◽

Continuous Development ◽

Research Directions ◽

Sensory Stimuli ◽

Computer Interfaces ◽

Brain Signals ◽

Multimodal Emotion Recognition

With the continuous development of portable noninvasive human sensor technologies such as brain–computer interfaces (BCI), multimodal emotion recognition has attracted increasing attention in the area of affective computing. This paper primarily discusses the progress of research into multimodal emotion recognition based on BCI and reviews three types of multimodal affective BCI (aBCI): aBCI based on a combination of behavior and brain signals, aBCI based on various hybrid neurophysiology modalities and aBCI based on heterogeneous sensory stimuli. For each type of aBCI, we further review several representative multimodal aBCI systems, including their design principles, paradigms, algorithms, experimental results and corresponding advantages. Finally, we identify several important issues and research directions for multimodal emotion recognition based on BCI.

Download Full-text

An EEG Database and Its Initial Benchmark Emotion Classification Performance

Computational and Mathematical Methods in Medicine ◽

10.1155/2020/8303465 ◽

2020 ◽

Vol 2020 ◽

pp. 1-14

Author(s):

Ayan Seal ◽

Puthi Prem Nivesh Reddy ◽

Pingali Chaithanya ◽

Arramada Meghana ◽

Kamireddy Jahnavi ◽

...

Keyword(s):

Emotion Recognition ◽

Affective Computing ◽

Industrial Applications ◽

Classification Performance ◽

Discrete Wavelet ◽

Emotional States ◽

Emotion Classification ◽

Eeg Signals ◽

Learning Machine ◽

Facial Images

Human emotion recognition has been a major field of research in the last decades owing to its noteworthy academic and industrial applications. However, most of the state-of-the-art methods identified emotions after analyzing facial images. Emotion recognition using electroencephalogram (EEG) signals has got less attention. However, the advantage of using EEG signals is that it can capture real emotion. However, very few EEG signals databases are publicly available for affective computing. In this work, we present a database consisting of EEG signals of 44 volunteers. Twenty-three out of forty-four are females. A 32 channels CLARITY EEG traveler sensor is used to record four emotional states namely, happy, fear, sad, and neutral of subjects by showing 12 videos. So, 3 video files are devoted to each emotion. Participants are mapped with the emotion that they had felt after watching each video. The recorded EEG signals are considered further to classify four types of emotions based on discrete wavelet transform and extreme learning machine (ELM) for reporting the initial benchmark classification performance. The ELM algorithm is used for channel selection followed by subband selection. The proposed method performs the best when features are captured from the gamma subband of the FP1-F7 channel with 94.72% accuracy. The presented database would be available to the researchers for affective recognition applications.

Download Full-text