Protected multimodal emotion recognition

In this thesis, we propose Protected Multimodal Emotion recognition (PMM-ER), an emotion recognition approach that includes security features against the growing rate of cyber-attacks on various databases, including emotion databases. The analysis on the frequently used encryption algorithms has led to the modified encryption algorithm proposed in this work. The system is able to recognize 7 different emotions, i.e. happiness, sadness, surprise, fear, disgust and anger, as well as a neutral emotion state, based on 2D video frames, 3D vertices, and audio wave information. Several well-known features are employed, including the HSV colour feature, iterative closest point (ICP) and Mel-frequency cepstral coefficients (MFCCs). We also propose a novel approach to feature fusion including both decision- and feature-level fusion, and some well-known classification and feature extraction algorithms such as principle component analysis (PCA), linear discernment analysis (LDA) and canonical correlation analysis (CCA) are compared in this study.

Deep spatio-temporal feature fusion with compact bilinear pooling for multimodal emotion recognition

Computer Vision and Image Understanding ◽

10.1016/j.cviu.2018.06.005 ◽

2018 ◽

Vol 174 ◽

pp. 33-42 ◽

Cited By ~ 6

Author(s):

Dung Nguyen ◽

Kien Nguyen ◽

Sridha Sridharan ◽

David Dean ◽

Clinton Fookes

Keyword(s):

Emotion Recognition ◽

Feature Fusion ◽

Spatio Temporal ◽

Temporal Feature

Decision-Level Fusion Method for Emotion Recognition using Multimodal Emotion Recognition Information

2018 15th International Conference on Ubiquitous Robots (UR) ◽

10.1109/urai.2018.8441795 ◽

2018 ◽

Cited By ~ 3

Author(s):

Kyu-Seob Song ◽

Young-Hoon Nho ◽

Ju-Hwan Seo ◽

Dong-soo Kwon

Keyword(s):

Emotion Recognition ◽

Fusion Method ◽

Decision Level ◽

Decision Level Fusion ◽

Feature Fusion In Multimodal Emotion Recognition System For Enhancement Of Human-Machine Interaction

IOP Conference Series Materials Science and Engineering ◽

10.1088/1757-899x/1084/1/012004 ◽

2021 ◽

Vol 1084 (1) ◽

pp. 012004

Author(s):

S Veni ◽

R Anand ◽

DIVYA MOHAN ◽

ELDHO PAUL

Keyword(s):

Emotion Recognition ◽

Feature Fusion ◽

Recognition System ◽

Human Machine Interaction ◽

Machine Interaction

Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism

Computational and Mathematical Methods in Medicine ◽

10.1155/2021/9967592 ◽

2021 ◽

Vol 2021 ◽

pp. 1-12

Author(s):

Yifeng Zhao ◽

Deyun Chen

Keyword(s):

Emotion Recognition ◽

Feature Fusion ◽

Image Features ◽

Attention Mechanism ◽

Matlab Simulation ◽

Eeg Signals ◽

Recognition Method ◽

Character Extraction ◽

Bidirectional Lstm ◽

Multimodal Emotion Recognition

Due to the complexity of human emotions, there are some similarities between different emotion features. The existing emotion recognition method has the problems of difficulty of character extraction and low accuracy, so the bidirectional LSTM and attention mechanism based on the expression EEG multimodal emotion recognition method are proposed. Firstly, facial expression features are extracted based on the bilinear convolution network (BCN), and EEG signals are transformed into three groups of frequency band image sequences, and BCN is used to fuse the image features to obtain the multimodal emotion features of expression EEG. Then, through the LSTM with the attention mechanism, important data is extracted in the process of timing modeling, which effectively avoids the randomness or blindness of sampling methods. Finally, a feature fusion network with a three-layer bidirectional LSTM structure is designed to fuse the expression and EEG features, which is helpful to improve the accuracy of emotion recognition. On the MAHNOB-HCI and DEAP datasets, the proposed method is tested based on the MATLAB simulation platform. Experimental results show that the attention mechanism can enhance the visual effect of the image, and compared with other methods, the proposed method can extract emotion features from expressions and EEG signals more effectively, and the accuracy of emotion recognition is higher.

Automatic Emotion Perception Using Eye Movement Information for E-Healthcare Systems

Sensors ◽

10.3390/s18092826 ◽

2018 ◽

Vol 18 (9) ◽

pp. 2826 ◽

Cited By ~ 7

Author(s):

Yang Wang ◽

Zhao Lv ◽

Yongjun Zheng

Keyword(s):

Eye Movement ◽

Healthcare System ◽

Emotional State ◽

Feature Fusion ◽

Emotion Perception ◽

Emotional States ◽

Time Frequency ◽

Promising Tool ◽

Novel Approach ◽

Facing the adolescents and detecting their emotional state is vital for promoting rehabilitation therapy within an E-Healthcare system. Focusing on a novel approach for a sensor-based E-Healthcare system, we propose an eye movement information-based emotion perception algorithm by collecting and analyzing electrooculography (EOG) signals and eye movement video synchronously. Specifically, we extract the time-frequency eye movement features by firstly applying the short-time Fourier transform (STFT) to raw multi-channel EOG signals. Subsequently, in order to integrate time domain eye movement features (i.e., saccade duration, fixation duration, and pupil diameter), we investigate two feature fusion strategies: feature level fusion (FLF) and decision level fusion (DLF). Recognition experiments have been also performed according to three emotional states: positive, neutral, and negative. The average accuracies are 88.64% (the FLF method) and 88.35% (the DLF with maximal rule method), respectively. Experimental results reveal that eye movement information can effectively reflect the emotional state of the adolescences, which provides a promising tool to improve the performance of the E-Healthcare system.

An Investigation of a Feature-Level Fusion for Noisy Speech Emotion Recognition

Computers ◽

10.3390/computers8040091 ◽

2019 ◽

Vol 8 (4) ◽

pp. 91 ◽

Cited By ~ 1

Author(s):

Sara Sekkate ◽

Mohammed Khalil ◽

Abdellah Adib ◽

Sofia Ben Jebara

Keyword(s):

Emotion Recognition ◽

Feature Representation ◽

Speech Emotion Recognition ◽

Discrete Wavelet ◽

Large Set ◽

Mel Frequency Cepstral Coefficients ◽

Feature Extraction Method ◽

Feature Level Fusion ◽

Low Dimensional ◽

Because one of the key issues in improving the performance of Speech Emotion Recognition (SER) systems is the choice of an effective feature representation, most of the research has focused on developing a feature level fusion using a large set of features. In our study, we propose a relatively low-dimensional feature set that combines three features: baseline Mel Frequency Cepstral Coefficients (MFCCs), MFCCs derived from Discrete Wavelet Transform (DWT) sub-band coefficients that are denoted as DMFCC, and pitch based features. Moreover, the performance of the proposed feature extraction method is evaluated in clean conditions and in the presence of several real-world noises. Furthermore, conventional Machine Learning (ML) and Deep Learning (DL) classifiers are employed for comparison. The proposal is tested using speech utterances of both of the Berlin German Emotional Database (EMO-DB) and Interactive Emotional Dyadic Motion Capture (IEMOCAP) speech databases through speaker independent experiments. Experimental results show improvement in speech emotion detection over baselines.