scholarly journals An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos

2020 ◽  
Vol 34 (01) ◽  
pp. 303-311 ◽  
Author(s):  
Sicheng Zhao ◽  
Yunsheng Ma ◽  
Yang Gu ◽  
Jufeng Yang ◽  
Tengfei Xing ◽  
...  
Keyword(s):  
Neural Networks ◽  
Emotion Recognition ◽  
State Of The Art ◽  
Source Code ◽  
Cross Entropy ◽  
Attention Network ◽  
Audio Features ◽  
End To End ◽  
3D Cnn ◽  
And Training

Emotion recognition in user-generated videos plays an important role in human-centered computing. Existing methods mainly employ traditional two-stage shallow pipeline, i.e. extracting visual and/or audio features and training classifiers. In this paper, we propose to recognize video emotions in an end-to-end manner based on convolutional neural networks (CNNs). Specifically, we develop a deep Visual-Audio Attention Network (VAANet), a novel architecture that integrates spatial, channel-wise, and temporal attentions into a visual 3D CNN and temporal attentions into an audio 2D CNN. Further, we design a special classification loss, i.e. polarity-consistent cross-entropy loss, based on the polarity-emotion hierarchy constraint to guide the attention generation. Extensive experiments conducted on the challenging VideoEmotion-8 and Ekman-6 datasets demonstrate that the proposed VAANet outperforms the state-of-the-art approaches for video emotion recognition. Our source code is released at: https://github.com/maysonma/VAANet.

scholarly journals Utterance Level Feature Aggregation with Deep Metric Learning for Speech Emotion Recognition

Sensors ◽  
2021 ◽  
Vol 21 (12) ◽  
pp. 4233
Author(s):  
Bogdan Mocanu ◽  
Ruxandra Tapu ◽  
Titus Zaharia
Keyword(s):  
Emotion Recognition ◽  
Loss Function ◽  
State Of The Art ◽  
Disease Diagnosis ◽  
Data Representation ◽  
Speech Emotion Recognition ◽  
Audio Features ◽  
Global Accuracy ◽  
Space Data ◽  
Art Techniques

Emotion is a form of high-level paralinguistic information that is intrinsically conveyed by human speech. Automatic speech emotion recognition is an essential challenge for various applications; including mental disease diagnosis; audio surveillance; human behavior understanding; e-learning and human–machine/robot interaction. In this paper, we introduce a novel speech emotion recognition method, based on the Squeeze and Excitation ResNet (SE-ResNet) model and fed with spectrogram inputs. In order to overcome the limitations of the state-of-the-art techniques, which fail in providing a robust feature representation at the utterance level, the CNN architecture is extended with a trainable discriminative GhostVLAD clustering layer that aggregates the audio features into compact, single-utterance vector representation. In addition, an end-to-end neural embedding approach is introduced, based on an emotionally constrained triplet loss function. The loss function integrates the relations between the various emotional patterns and thus improves the latent space data representation. The proposed methodology achieves 83.35% and 64.92% global accuracy rates on the RAVDESS and CREMA-D publicly available datasets, respectively. When compared with the results provided by human observers, the gains in global accuracy scores are superior to 24%. Finally, the objective comparative evaluation with state-of-the-art techniques demonstrates accuracy gains of more than 3%.

scholarly journals Empirical Investigation of Optimization Algorithms in Neural Machine Translation

2017 ◽  
Vol 108 (1) ◽  
pp. 13-25 ◽  
Author(s):  
Parnia Bahar ◽  
Tamer Alkhouli ◽  
Jan-Thorsten Peter ◽  
Christopher Jan-Steffen Brix ◽  
Hermann Ney
Keyword(s):  
Neural Networks ◽  
Machine Translation ◽  
Optimization Problem ◽  
Empirical Investigation ◽  
State Of The Art ◽  
Optimization Techniques ◽  
Neural Machine Translation ◽  
Translation Quality ◽  
And Training ◽  
Dimensional Optimization

AbstractTraining neural networks is a non-convex and a high-dimensional optimization problem. In this paper, we provide a comparative study of the most popular stochastic optimization techniques used to train neural networks. We evaluate the methods in terms of convergence speed, translation quality, and training stability. In addition, we investigate combinations that seek to improve optimization in terms of these aspects. We train state-of-the-art attention-based models and apply them to perform neural machine translation. We demonstrate our results on two tasks: WMT 2016 En→Ro and WMT 2015 De→En.

scholarly journals End-to-End Multimodal Emotion Recognition Using Deep Neural Networks

2017 ◽  
Vol 11 (8) ◽  
pp. 1301-1309 ◽  
Author(s):  
Panagiotis Tzirakis ◽  
George Trigeorgis ◽  
Mihalis A. Nicolaou ◽  
Bjorn W. Schuller ◽  
Stefanos Zafeiriou
Keyword(s):  
Neural Networks ◽  
Emotion Recognition ◽  
Deep Neural Networks ◽  
End To End ◽  
Multimodal Emotion Recognition

scholarly journals Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

2021 ◽  
Vol 11 (24) ◽  
pp. 11738
Author(s):  
Thomas Teixeira ◽  
Éric Granger ◽  
Alessandro Lameiras Koerich
Keyword(s):  
Neural Networks ◽  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expressions ◽  
Convolutional Neural Networks ◽  
Affective Computing ◽  
Spatial Information ◽  
Short Term Memory ◽  
State Of The Art ◽  
Fine Tuning

Facial expressions are one of the most powerful ways to depict specific patterns in human behavior and describe the human emotional state. However, despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot correctly handle variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task, even for humans. This paper investigates the suitability of state-of-the-art deep learning architectures based on convolutional neural networks (CNNs) to deal with long video sequences captured in the wild for continuous emotion recognition. For such an aim, several 2D CNN models that were designed to model spatial information are extended to allow spatiotemporal representation learning from videos, considering a complex and multi-dimensional emotion space, where continuous values of valence and arousal must be predicted. We have developed and evaluated convolutional recurrent neural networks, combining 2D CNNs and long short term-memory units and inflated 3D CNN models, which are built by inflating the weights of a pre-trained 2D CNN model during fine-tuning, using application-specific videos. Experimental results on the challenging SEWA-DB dataset have shown that these architectures can effectively be fine-tuned to encode spatiotemporal information from successive raw pixel images and achieve state-of-the-art results on such a dataset.

scholarly journals Singular Learning of Deep Multilayer Perceptrons for EEG-Based Emotion Recognition

2021 ◽  
Vol 3 ◽  
Author(s):  
Weili Guo ◽  
Guangyu Li ◽  
Jianfeng Lu ◽  
Jian Yang
Keyword(s):  
Neural Networks ◽  
Emotion Recognition ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
High Reliability ◽  
Multilayer Perceptrons ◽  
Learning Technology ◽  
Human Computer Interactions ◽  
Training Process ◽  
Specific Influence

Human emotion recognition is an important issue in human–computer interactions, and electroencephalograph (EEG) has been widely applied to emotion recognition due to its high reliability. In recent years, methods based on deep learning technology have reached the state-of-the-art performance in EEG-based emotion recognition. However, there exist singularities in the parameter space of deep neural networks, which may dramatically slow down the training process. It is very worthy to investigate the specific influence of singularities when applying deep neural networks to EEG-based emotion recognition. In this paper, we mainly focus on this problem, and analyze the singular learning dynamics of deep multilayer perceptrons theoretically and numerically. The results can help us to design better algorithms to overcome the serious influence of singularities in deep neural networks for EEG-based emotion recognition.

Deep Learning Based Human Emotional State Recognition in a Video

2020 ◽  
Vol 12 (1) ◽  
pp. 51-59
Author(s):  
A. A. Moskvin ◽  
A.G. Shishkin
Keyword(s):  
Neural Networks ◽  
Emotion Recognition ◽  
Visual Information ◽  
Network Architecture ◽  
Neural Network Architecture ◽  
State Recognition ◽  
Audio Features ◽  
E Learning ◽  
Audio Information ◽  
Selection Of

Human emotions play significant role in everyday life. There are a lot of applications of automatic emotion recognition in medicine, e-learning, monitoring, marketing etc. In this paper the method and neural network architecture for real-time human emotion recognition by audio-visual data are proposed. To classify one of seven emotions, deep neural networks, namely, convolutional and recurrent neural networks are used. Visual information is represented by a sequence of 16 frames of 96 × 96 pixels, and audio information - by 140 features for each of a sequence of 37 temporal windows. To reduce the number of audio features autoencoder was used. Audio information in conjunction with visual one is shown to increase recognition accuracy up to 12%. The developed system being not demanding to be computing resources is dynamic in terms of selection of parameters, reducing or increasing the number of emotion classes, as well as the ability to easily add, accumulate and use information from other external devices for further improvement of classification accuracy. 

scholarly journals End-to-end speech emotion recognition using a novel context-stacking dilated convolution neural network

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Duowei Tang ◽  
Peter Kuppens ◽  
Luc Geurts ◽  
Toon van Waterschoot
Keyword(s):  
Neural Network ◽  
Emotion Recognition ◽  
Speech Signal ◽  
State Of The Art ◽  
Speech Emotion Recognition ◽  
Model Parameters ◽  
Proposed Model ◽  
End To End ◽  
The Impact ◽  
Temporal Dependencies

AbstractAmongst the various characteristics of a speech signal, the expression of emotion is one of the characteristics that exhibits the slowest temporal dynamics. Hence, a performant speech emotion recognition (SER) system requires a predictive model that is capable of learning sufficiently long temporal dependencies in the analysed speech signal. Therefore, in this work, we propose a novel end-to-end neural network architecture based on the concept of dilated causal convolution with context stacking. Firstly, the proposed model consists only of parallelisable layers and is hence suitable for parallel processing, while avoiding the inherent lack of parallelisability occurring with recurrent neural network (RNN) layers. Secondly, the design of a dedicated dilated causal convolution block allows the model to have a receptive field as large as the input sequence length, while maintaining a reasonably low computational cost. Thirdly, by introducing a context stacking structure, the proposed model is capable of exploiting long-term temporal dependencies hence providing an alternative to the use of RNN layers. We evaluate the proposed model in SER regression and classification tasks and provide a comparison with a state-of-the-art end-to-end SER model. Experimental results indicate that the proposed model requires only 1/3 of the number of model parameters used in the state-of-the-art model, while also significantly improving SER performance. Further experiments are reported to understand the impact of using various types of input representations (i.e. raw audio samples vs log mel-spectrograms) and to illustrate the benefits of an end-to-end approach over the use of hand-crafted audio features. Moreover, we show that the proposed model can efficiently learn intermediate embeddings preserving speech emotion information.

scholarly journals AAR-CNNs: Auto Adaptive Regularized Convolutional Neural Networks

2018 ◽  
Author(s):  
Yao Lu ◽  
Guangming Lu ◽  
Yuanrong Xu ◽  
Bob Zhang
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
State Of The Art ◽  
Experimental Results ◽  
Training Phase ◽  
Low Resolution ◽  
Adaptive Regularization ◽  
End To End ◽  
Overfitting Problem

In order to address the overfitting problem caused by the small or simple training datasets and the large model’s size in Convolutional Neural Networks (CNNs), a novel Auto Adaptive Regularization (AAR) method is proposed in this paper. The relevant networks can be called AAR-CNNs. AAR is the first method using the “abstraction extent” (predicted by AE net) and a tiny learnable module (SE net) to auto adaptively predict more accurate and individualized regularization information. The AAR module can be directly inserted into every stage of any popular networks and trained end to end to improve the networks’ flexibility. This method can not only regularize the network at both the forward and the backward processes in the training phase, but also regularize the network on a more refined level (channel or pixel level) depending on the abstraction extent’s form. Comparative experiments are performed on low resolution ImageNet, CIFAR and SVHN datasets. Experimental results show that the AAR-CNNs can achieve state-of-the-art performances on these datasets.

scholarly journals HalluciNet-ing Spatiotemporal Representations Using a 2D-CNN

Signals ◽  
2021 ◽  
Vol 2 (3) ◽  
pp. 604-618
Author(s):  
Paritosh Parmar ◽  
Brendan Morris
Keyword(s):  
Neural Networks ◽  
State Of The Art ◽  
Scene Recognition ◽  
Multitask Learning ◽  
Training Phase ◽  
Dynamic Scene ◽  
Improve Performance ◽  
Computing Power ◽  
Practical Standpoint ◽  
3D Cnn

Spatiotemporal representations learned using 3D convolutional neural networks (CNN) are currently used in state-of-the-art approaches for action-related tasks. However, 3D-CNN are notorious for being memory and compute resource intensive as compared with more simple 2D-CNN architectures. We propose to hallucinate spatiotemporal representations from a 3D-CNN teacher with a 2D-CNN student. By requiring the 2D-CNN to predict the future and intuit upcoming activity, it is encouraged to gain a deeper understanding of actions and how they evolve. The hallucination task is treated as an auxiliary task, which can be used with any other action-related task in a multitask learning setting. Thorough experimental evaluation, it is shown that the hallucination task indeed helps improve performance on action recognition, action quality assessment, and dynamic scene recognition tasks. From a practical standpoint, being able to hallucinate spatiotemporal representations without an actual 3D-CNN can enable deployment in resource-constrained scenarios, such as with limited computing power and/or lower bandwidth. We also observed that our hallucination task has utility not only during the training phase, but also during the pre-training phase.

Sign in / Sign up

Export Citation Format

Share Document