Facial Expression Recognition Using Visual Saliency and Deep Learning

Spatial Transformer Networks are considered a powerful algorithm to learn the main areas of an image, but still, they could be more efficient by receiving images with embedded expert knowledge. This paper aims to improve the performance of conventional Spatial Transformers when applied to Facial Expression Recognition. Based on the Spatial Transformers’ capacity of spatial manipulation within networks, we propose different extensions to these models where effective attentional regions are captured employing facial landmarks or facial visual saliency maps. This specific attentional information is then hardcoded to guide the Spatial Transformers to learn the spatial transformations that best fit the proposed regions for better recognition results. For this study, we use two datasets: AffectNet and FER-2013. For AffectNet, we achieve a 0.35% point absolute improvement relative to the traditional Spatial Transformer, whereas for FER-2013, our solution gets an increase of 1.49% when models are fine-tuned with the Affectnet pre-trained weights.

Download Full-text

A survey of facial expression recognition based on deep learning

2020 15th IEEE Conference on Industrial Electronics and Applications (ICIEA) ◽

10.1109/iciea48937.2020.9248180 ◽

2020 ◽

Author(s):

Heng Wei ◽

Zhi Zhang

Keyword(s):

Deep Learning ◽

Facial Expression ◽

Facial Expression Recognition ◽

Expression Recognition

Download Full-text

Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network

Sensors ◽

10.3390/s21093046 ◽

2021 ◽

Vol 21 (9) ◽

pp. 3046

Author(s):

Shervin Minaee ◽

Mehdi Minaei ◽

Amirali Abdolrashidi

Keyword(s):

Deep Learning ◽

Facial Expression ◽

Facial Expression Recognition ◽

Expression Recognition ◽

Visualization Technique ◽

Convolutional Network ◽

The Past ◽

Multiple Datasets ◽

The Face ◽

Traditional Approaches

Facial expression recognition has been an active area of research over the past few decades, and it is still challenging due to the high intra-class variation. Traditional approaches for this problem rely on hand-crafted features such as SIFT, HOG, and LBP, followed by a classifier trained on a database of images or videos. Most of these works perform reasonably well on datasets of images captured in a controlled condition but fail to perform as well on more challenging datasets with more image variation and partial faces. In recent years, several works proposed an end-to-end framework for facial expression recognition using deep learning models. Despite the better performance of these works, there are still much room for improvement. In this work, we propose a deep learning approach based on attentional convolutional network that is able to focus on important parts of the face and achieves significant improvement over previous models on multiple datasets, including FER-2013, CK+, FERG, and JAFFE. We also use a visualization technique that is able to find important facial regions to detect different emotions based on the classifier’s output. Through experimental results, we show that different emotions are sensitive to different parts of the face.

Download Full-text

Deep Learning for Real Time Facial Expression Recognition in Social Robots

Neural Information Processing - Lecture Notes in Computer Science ◽

10.1007/978-3-030-04221-9_35 ◽

2018 ◽

pp. 392-402 ◽

Cited By ~ 3

Author(s):

Ariel Ruiz-Garcia ◽

Nicola Webb ◽

Vasile Palade ◽

Mark Eastwood ◽

Mark Elshaw

Keyword(s):

Deep Learning ◽

Facial Expression ◽

Real Time ◽

Facial Expression Recognition ◽

Social Robots ◽

Expression Recognition

Download Full-text

Real Time Facial Expression Recognition System Based on Deep Learning

International Journal of Recent Technology and Engineering - 2 ◽

10.35940/ijrte.b1591.0982s1119 ◽

2019 ◽

Vol 8 (2S11) ◽

pp. 4047-4051

Keyword(s):

Deep Learning ◽

Facial Expression ◽

Real Time ◽

Facial Expression Recognition ◽

Recognition System ◽

Expression Recognition ◽

Or Education ◽

Face Expression ◽

Active Research ◽

Main Components

The automatic detection of facial expressions is an active research topic, since its wide fields of applications in human-computer interaction, games, security or education. However, the latest studies have been made in controlled laboratory environments, which is not according to real world scenarios. For that reason, a real time Facial Expression Recognition System (FERS) is proposed in this paper, in which a deep learning approach is applied to enhance the detection of six basic emotions: happiness, sadness, anger, disgust, fear and surprise in a real-time video streaming. This system is composed of three main components: face detection, face preparation and face expression classification. The results of proposed FERS achieve a 65% of accuracy, trained over 35558 face images..

Download Full-text