Metric Learning Based Feature Representation with Gated Fusion Model for Speech Emotion Recognition

2021 ◽  
Author(s):  
Yuan Gao ◽  
Jiaxing Liu ◽  
Longbiao Wang ◽  
Jianwu Dang
Keyword(s):  
Emotion Recognition ◽  
Metric Learning ◽  
Feature Representation ◽  
Speech Emotion Recognition ◽  
Fusion Model

scholarly journals Cross-Individual Affective Detection Using EEG Signals with Audio-Visual Embedding

2021 ◽  
Author(s):  
Zhen Liang ◽  
Xihao Zhang ◽  
Rushuang Zhou ◽  
Li Zhang ◽  
Linling Li ◽  
...  
Keyword(s):  
Emotion Recognition ◽  
Temporal Dynamics ◽  
Recognition Performance ◽  
Critical Issue ◽  
Feature Representation ◽  
Fusion Model ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Brain Data ◽  
Eeg Features

How to effectively and efficiently extract valid and reliable features from high-dimensional electroencephalography (EEG), particularly how to fuse the spatial and temporal dynamic brain information into a better feature representation, is a critical issue in brain data analysis. Most current EEG studies work in a task driven manner and explore the valid EEG features with a supervised model, which would be limited by the given labels to a great extent. In this paper, we propose a practical hybrid unsupervised deep convolutional recurrent generative adversarial network based EEG feature characterization and fusion model, which is termed as EEGFuseNet. EEGFuseNet is trained in an unsupervised manner, and deep EEG features covering both spatial and temporal dynamics are automatically characterized. Comparing to the existing features, the characterized deep EEG features could be considered to be more generic and independent of any specific EEG task. The performance of the extracted deep and low-dimensional features by EEGFuseNet is carefully evaluated in an unsupervised emotion recognition application based on three public emotion databases. The results demonstrate the proposed EEGFuseNet is a robust and reliable model, which is easy to train and performs efficiently in the representation and fusion of dynamic EEG features. In particular, EEGFuseNet is established as an optimal unsupervised fusion model with promising cross-subject emotion recognition performance. It proves EEGFuseNet is capable of characterizing and fusing deep features that imply comparative cortical dynamic significance corresponding to the changing of different emotion states, and also demonstrates the possibility of realizing EEG based cross-subject emotion recognition in a pure unsupervised manner.

scholarly journals f-Similarity Preservation Loss for Soft Labels: A Demonstration on Cross-Corpus Speech Emotion Recognition

2019 ◽  
Vol 33 ◽  
pp. 5725-5732
Author(s):  
Biqiao Zhang ◽  
Yuqing Kong ◽  
Georg Essl ◽  
Emily Mower Provost
Keyword(s):  
Neural Networks ◽  
Emotion Recognition ◽  
Loss Function ◽  
Deep Neural Networks ◽  
Metric Learning ◽  
Loss Functions ◽  
Speech Emotion Recognition ◽  
Subjective Data ◽  
Dual Form ◽  
Deep Metric Learning

In this paper, we propose a Deep Metric Learning (DML) approach that supports soft labels. DML seeks to learn representations that encode the similarity between examples through deep neural networks. DML generally presupposes that data can be divided into discrete classes using hard labels. However, some tasks, such as our exemplary domain of speech emotion recognition (SER), work with inherently subjective data, data for which it may not be possible to identify a single hard label. We propose a family of loss functions, fSimilarity Preservation Loss (f-SPL), based on the dual form of f-divergence for DML with soft labels. We show that the minimizer of f-SPL preserves the pairwise label similarities in the learned feature embeddings. We demonstrate the efficacy of the proposed loss function on the task of cross-corpus SER with soft labels. Our approach, which combines f-SPL and classification loss, significantly outperforms a baseline SER system with the same structure but trained with only classification loss in most experiments. We show that the presented techniques are more robust to over-training and can learn an embedding space in which the similarity between examples is meaningful.

scholarly journals Deep Multimodal Emotion Recognition on Human Speech: A Review

2021 ◽  
Vol 11 (17) ◽  
pp. 7962
Author(s):  
Panagiotis Koromilas ◽  
Theodoros Giannakopoulos
Keyword(s):  
Emotion Recognition ◽  
Visual Information ◽  
Multimodal Interaction ◽  
Representation Learning ◽  
Basic Feature ◽  
Feature Representation ◽  
Speech Emotion Recognition ◽  
Temporal Dimension ◽  
Multimodal Interactions ◽  
Depth Analysis

This work reviews the state of the art in multimodal speech emotion recognition methodologies, focusing on audio, text and visual information. We provide a new, descriptive categorization of methods, based on the way they handle the inter-modality and intra-modality dynamics in the temporal dimension: (i) non-temporal architectures (NTA), which do not significantly model the temporal dimension in both unimodal and multimodal interaction; (ii) pseudo-temporal architectures (PTA), which also assume an oversimplification of the temporal dimension, although in one of the unimodal or multimodal interactions; and (iii) temporal architectures (TA), which try to capture both unimodal and cross-modal temporal dependencies. In addition, we review the basic feature representation methods for each modality, and we present aggregated evaluation results on the reported methodologies. Finally, we conclude this work with an in-depth analysis of the future challenges related to validation procedures, representation learning and method robustness.

scholarly journals A Novel DBN Feature Fusion Model for Cross-Corpus Speech Emotion Recognition

2016 ◽  
Vol 2016 ◽  
pp. 1-11 ◽  
Author(s):  
Zou Cairong ◽  
Zhang Xinran ◽  
Zha Cheng ◽  
Zhao Li
Keyword(s):  
Emotion Recognition ◽  
Feature Fusion ◽  
Image Features ◽  
Speech Emotion Recognition ◽  
Feature Subset ◽  
Fusion Model ◽  
Emotional Information ◽  
Separate Source ◽  
Speech Spectrum ◽  
New Feature

The feature fusion from separate source is the current technical difficulties of cross-corpus speech emotion recognition. The purpose of this paper is to, based on Deep Belief Nets (DBN) in Deep Learning, use the emotional information hiding in speech spectrum diagram (spectrogram) as image features and then implement feature fusion with the traditional emotion features. First, based on the spectrogram analysis by STB/Itti model, the new spectrogram features are extracted from the color, the brightness, and the orientation, respectively; then using two alternative DBN models they fuse the traditional and the spectrogram features, which increase the scale of the feature subset and the characterization ability of emotion. Through the experiment on ABC database and Chinese corpora, the new feature subset compared with traditional speech emotion features, the recognition result on cross-corpus, distinctly advances by 8.8%. The method proposed provides a new idea for feature fusion of emotion recognition.

scholarly journals An Investigation of a Feature-Level Fusion for Noisy Speech Emotion Recognition

Computers ◽  
2019 ◽  
Vol 8 (4) ◽  
pp. 91 ◽  
Author(s):  
Sara Sekkate ◽  
Mohammed Khalil ◽  
Abdellah Adib ◽  
Sofia Ben Jebara
Keyword(s):  
Emotion Recognition ◽  
Feature Representation ◽  
Speech Emotion Recognition ◽  
Discrete Wavelet ◽  
Large Set ◽  
Mel Frequency Cepstral Coefficients ◽  
Feature Extraction Method ◽  
Feature Level Fusion ◽  
Low Dimensional ◽  
Level Fusion

Because one of the key issues in improving the performance of Speech Emotion Recognition (SER) systems is the choice of an effective feature representation, most of the research has focused on developing a feature level fusion using a large set of features. In our study, we propose a relatively low-dimensional feature set that combines three features: baseline Mel Frequency Cepstral Coefficients (MFCCs), MFCCs derived from Discrete Wavelet Transform (DWT) sub-band coefficients that are denoted as DMFCC, and pitch based features. Moreover, the performance of the proposed feature extraction method is evaluated in clean conditions and in the presence of several real-world noises. Furthermore, conventional Machine Learning (ML) and Deep Learning (DL) classifiers are employed for comparison. The proposal is tested using speech utterances of both of the Berlin German Emotional Database (EMO-DB) and Interactive Emotional Dyadic Motion Capture (IEMOCAP) speech databases through speaker independent experiments. Experimental results show improvement in speech emotion detection over baselines.

Sign in / Sign up

Export Citation Format

Share Document