scholarly journals On-Road Driver Emotion Recognition Using Facial Expression

2022 ◽  
Vol 12 (2) ◽  
pp. 807
Author(s):  
Huafei Xiao ◽  
Wenbo Li ◽  
Guanzhong Zeng ◽  
Yingzhang Wu ◽  
Jiyong Xue ◽  
...  
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Data Augmentation ◽  
Recognition Task ◽  
Superior Performance ◽  
Expression Recognition ◽  
Emotion Detection ◽  
Backbone Networks ◽  
Driving Task ◽  
Machine Systems

With the development of intelligent automotive human-machine systems, driver emotion detection and recognition has become an emerging research topic. Facial expression-based emotion recognition approaches have achieved outstanding results on laboratory-controlled data. However, these studies cannot represent the environment of real driving situations. In order to address this, this paper proposes a facial expression-based on-road driver emotion recognition network called FERDERnet. This method divides the on-road driver facial expression recognition task into three modules: a face detection module that detects the driver’s face, an augmentation-based resampling module that performs data augmentation and resampling, and an emotion recognition module that adopts a deep convolutional neural network pre-trained on FER and CK+ datasets and then fine-tuned as a backbone for driver emotion recognition. This method adopts five different backbone networks as well as an ensemble method. Furthermore, to evaluate the proposed method, this paper collected an on-road driver facial expression dataset, which contains various road scenarios and the corresponding driver’s facial expression during the driving task. Experiments were performed on the on-road driver facial expression dataset that this paper collected. Based on efficiency and accuracy, the proposed FERDERnet with Xception backbone was effective in identifying on-road driver facial expressions and obtained superior performance compared to the baseline networks and some state-of-the-art networks.

scholarly journals eXnet: An Efficient Approach for Emotion Recognition in the Wild

Sensors ◽  
2020 ◽  
Vol 20 (4) ◽  
pp. 1087
Author(s):  
Muhammad Naveed Riaz ◽  
Yao Shen ◽  
Muhammad Sohail ◽  
Minyi Guo
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expression Recognition ◽  
Data Augmentation ◽  
Raspberry Pi ◽  
Expression Recognition ◽  
Extensive Evaluation ◽  
Benchmark Datasets ◽  
Augmentation Techniques ◽  
In The Wild

Facial expression recognition has been well studied for its great importance in the areas of human–computer interaction and social sciences. With the evolution of deep learning, there have been significant advances in this area that also surpass human-level accuracy. Although these methods have achieved good accuracy, they are still suffering from two constraints (high computational power and memory), which are incredibly critical for small hardware-constrained devices. To alleviate this issue, we propose a new Convolutional Neural Network (CNN) architecture eXnet (Expression Net) based on parallel feature extraction which surpasses current methods in accuracy and contains a much smaller number of parameters (eXnet: 4.57 million, VGG19: 14.72 million), making it more efficient and lightweight for real-time systems. Several modern data augmentation techniques are applied for generalization of eXnet; these techniques improve the accuracy of the network by overcoming the problem of overfitting while containing the same size. We provide an extensive evaluation of our network against key methods on Facial Expression Recognition 2013 (FER-2013), Extended Cohn-Kanade Dataset (CK+), and Real-world Affective Faces Database (RAF-DB) benchmark datasets. We also perform ablation evaluation to show the importance of different components of our architecture. To evaluate the efficiency of eXnet on embedded systems, we deploy it on Raspberry Pi 4B. All these evaluations show the superiority of eXnet for emotion recognition in the wild in terms of accuracy, the number of parameters, and size on disk.

scholarly journals Toward Continuous Social Phenotyping: Analyzing Gaze Patterns in an Emotion Recognition Task for Children With Autism Through Wearable Smart Glasses

10.2196/13810 ◽  
2020 ◽  
Vol 22 (4) ◽  
pp. e13810 ◽  
Author(s):  
Anish Nag ◽  
Nick Haber ◽  
Catalin Voss ◽  
Serena Tamura ◽  
Jena Daniels ◽  
...  
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expression Recognition ◽  
Eye Gaze ◽  
Recognition Task ◽  
Children With Autism ◽  
Expression Recognition ◽  
Eye Tracker ◽  
Gaze Tracking ◽  
Smart Glasses

Background Several studies have shown that facial attention differs in children with autism. Measuring eye gaze and emotion recognition in children with autism is challenging, as standard clinical assessments must be delivered in clinical settings by a trained clinician. Wearable technologies may be able to bring eye gaze and emotion recognition into natural social interactions and settings. Objective This study aimed to test: (1) the feasibility of tracking gaze using wearable smart glasses during a facial expression recognition task and (2) the ability of these gaze-tracking data, together with facial expression recognition responses, to distinguish children with autism from neurotypical controls (NCs). Methods We compared the eye gaze and emotion recognition patterns of 16 children with autism spectrum disorder (ASD) and 17 children without ASD via wearable smart glasses fitted with a custom eye tracker. Children identified static facial expressions of images presented on a computer screen along with nonsocial distractors while wearing Google Glass and the eye tracker. Faces were presented in three trials, during one of which children received feedback in the form of the correct classification. We employed hybrid human-labeling and computer vision–enabled methods for pupil tracking and world–gaze translation calibration. We analyzed the impact of gaze and emotion recognition features in a prediction task aiming to distinguish children with ASD from NC participants. Results Gaze and emotion recognition patterns enabled the training of a classifier that distinguished ASD and NC groups. However, it was unable to significantly outperform other classifiers that used only age and gender features, suggesting that further work is necessary to disentangle these effects. Conclusions Although wearable smart glasses show promise in identifying subtle differences in gaze tracking and emotion recognition patterns in children with and without ASD, the present form factor and data do not allow for these differences to be reliably exploited by machine learning systems. Resolving these challenges will be an important step toward continuous tracking of the ASD phenotype.

scholarly journals Toward Continuous Social Phenotyping: Analyzing Gaze Patterns in an Emotion Recognition Task for Children With Autism Through Wearable Smart Glasses (Preprint)

2019 ◽  
Author(s):  
Anish Nag ◽  
Nick Haber ◽  
Catalin Voss ◽  
Serena Tamura ◽  
Jena Daniels ◽  
...  
Keyword(s):  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expression Recognition ◽  
Eye Gaze ◽  
Recognition Task ◽  
Children With Autism ◽  
Expression Recognition ◽  
Eye Tracker ◽  
Gaze Tracking ◽  
Smart Glasses

BACKGROUND Several studies have shown that facial attention differs in children with autism. Measuring eye gaze and emotion recognition in children with autism is challenging, as standard clinical assessments must be delivered in clinical settings by a trained clinician. Wearable technologies may be able to bring eye gaze and emotion recognition into natural social interactions and settings. OBJECTIVE This study aimed to test: (1) the feasibility of tracking gaze using wearable smart glasses during a facial expression recognition task and (2) the ability of these gaze-tracking data, together with facial expression recognition responses, to distinguish children with autism from neurotypical controls (NCs). METHODS We compared the eye gaze and emotion recognition patterns of 16 children with autism spectrum disorder (ASD) and 17 children without ASD via wearable smart glasses fitted with a custom eye tracker. Children identified static facial expressions of images presented on a computer screen along with nonsocial distractors while wearing Google Glass and the eye tracker. Faces were presented in three trials, during one of which children received feedback in the form of the correct classification. We employed hybrid human-labeling and computer vision–enabled methods for pupil tracking and world–gaze translation calibration. We analyzed the impact of gaze and emotion recognition features in a prediction task aiming to distinguish children with ASD from NC participants. RESULTS Gaze and emotion recognition patterns enabled the training of a classifier that distinguished ASD and NC groups. However, it was unable to significantly outperform other classifiers that used only age and gender features, suggesting that further work is necessary to disentangle these effects. CONCLUSIONS Although wearable smart glasses show promise in identifying subtle differences in gaze tracking and emotion recognition patterns in children with and without ASD, the present form factor and data do not allow for these differences to be reliably exploited by machine learning systems. Resolving these challenges will be an important step toward continuous tracking of the ASD phenotype.

Emotional Facial Expression Recognition Task

2010 ◽  
Author(s):  
M. Fischer-Shofty ◽  
S. G. Shamay-Tsoorya ◽  
H. Harari ◽  
Y. Levkovitz
Keyword(s):  
Facial Expression ◽  
Facial Expression Recognition ◽  
Recognition Task ◽  
Expression Recognition ◽  
Emotional Facial Expression

scholarly journals Are affective factors related to individual differences in facial expression recognition?

2020 ◽  
Vol 7 (9) ◽  
pp. 190699
Author(s):  
Sarah A. H. Alharbi ◽  
Katherine Button ◽  
Lingshan Zhang ◽  
Kieran J. O'Shea ◽  
Vanessa Fasolt ◽  
...  
Keyword(s):  
Individual Differences ◽  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expression Recognition ◽  
Recognition Performance ◽  
Expression Recognition ◽  
General Anxiety ◽  
Affective Factors ◽  
Multimodal Emotion Recognition ◽  
Anxiety Subscale

Evidence that affective factors (e.g. anxiety, depression, affect) are significantly related to individual differences in emotion recognition is mixed. Palermo et al . (Palermo et al . 2018 J. Exp. Psychol. Hum. Percept. Perform. 44 , 503–517) reported that individuals who scored lower in anxiety performed significantly better on two measures of facial-expression recognition (emotion-matching and emotion-labelling tasks), but not a third measure (the multimodal emotion recognition test). By contrast, facial-expression recognition was not significantly correlated with measures of depression, positive or negative affect, empathy, or autistic-like traits. Because the range of affective factors considered in this study and its use of multiple expression-recognition tasks mean that it is a relatively comprehensive investigation of the role of affective factors in facial expression recognition, we carried out a direct replication. In common with Palermo et al . (Palermo et al . 2018 J. Exp. Psychol. Hum. Percept. Perform. 44 , 503–517), scores on the DASS anxiety subscale negatively predicted performance on the emotion recognition tasks across multiple analyses, although these correlations were only consistently significant for performance on the emotion-labelling task. However, and by contrast with Palermo et al . (Palermo et al . 2018 J. Exp. Psychol. Hum. Percept. Perform. 44 , 503–517), other affective factors (e.g. those related to empathy) often also significantly predicted emotion-recognition performance. Collectively, these results support the proposal that affective factors predict individual differences in emotion recognition, but that these correlations are not necessarily specific to measures of general anxiety, such as the DASS anxiety subscale.

Real-time facial expression recognition using smoothed deep neural network ensemble

2020 ◽  
Vol 28 (1) ◽  
pp. 97-111
Author(s):  
Nadir Kamel Benamara ◽  
Mikel Val-Calvo ◽  
Jose Ramón Álvarez-Sánchez ◽  
Alejandro Díaz-Morcillo ◽  
Jose Manuel Ferrández-Vicente ◽  
...  
Keyword(s):  
Neural Network ◽  
Facial Expression ◽  
Emotion Recognition ◽  
Facial Expression Recognition ◽  
Facial Emotion Recognition ◽  
Training Data ◽  
Facial Emotion ◽  
Processing Unit ◽  
Expression Recognition ◽  
Neural Network Ensemble

Facial emotion recognition (FER) has been extensively researched over the past two decades due to its direct impact in the computer vision and affective robotics fields. However, the available datasets to train these models include often miss-labelled data due to the labellers bias that drives the model to learn incorrect features. In this paper, a facial emotion recognition system is proposed, addressing automatic face detection and facial expression recognition separately, the latter is performed by a set of only four deep convolutional neural network respect to an ensembling approach, while a label smoothing technique is applied to deal with the miss-labelled training data. The proposed system takes only 13.48 ms using a dedicated graphics processing unit (GPU) and 141.97 ms using a CPU to recognize facial emotions and reaches the current state-of-the-art performances regarding the challenging databases, FER2013, SFEW 2.0, and ExpW, giving recognition accuracies of 72.72%, 51.97%, and 71.82% respectively.

Facial Expression and Sex Recognition in Schizophrenia and Depression

2005 ◽  
Vol 50 (9) ◽  
pp. 525-533 ◽  
Author(s):  
Benoit Bediou ◽  
Pierre Krolak-Salmon ◽  
Mohamed Saoud ◽  
Marie-Anne Henaff ◽  
Michael Burt ◽  
...  
Keyword(s):  
Facial Expression ◽  
Facial Expression Recognition ◽  
Recognition Task ◽  
Expression Recognition ◽  
Sex Recognition ◽  
Control Subjects ◽  
Healthy Control ◽  
And Control ◽  
Expression Processing ◽  
Selective Impairment

Background: Impaired facial expression recognition in schizophrenia patients contributes to abnormal social functioning and may predict functional outcome in these patients. Facial expression processing involves individual neural networks that have been shown to malfunction in schizophrenia. Whether these patients have a selective deficit in facial expression recognition or a more global impairment in face processing remains controversial. Objective: To investigate whether patients with schizophrenia exhibit a selective impairment in facial emotional expression recognition, compared with patients with major depression and healthy control subjects. Methods: We studied performance in facial expression recognition and facial sex recognition paradigms, using original morphed faces, in a population with schizophrenia ( n = 29) and compared their scores with those of depression patients ( n = 20) and control subjects ( n = 20). Results: Schizophrenia patients achieved lower scores than both other groups in the expression recognition task, particularly in fear and disgust recognition. Sex recognition was unimpaired. Conclusion: Facial expression recognition is impaired in schizophrenia, whereas sex recognition is preserved, which highly suggests an abnormal processing of changeable facial features in this disease. A dysfunction of the top-down retrograde modulation coming from limbic and paralimbic structures on visual areas is hypothesized.

scholarly journals Evaluation of Data Augmentation Techniques for Facial Expression Recognition Systems

Electronics ◽  
2020 ◽  
Vol 9 (11) ◽  
pp. 1892
Author(s):  
Simone Porcu ◽  
Alessandro Floris ◽  
Luigi Atzori
Keyword(s):  
Facial Expression ◽  
Facial Expression Recognition ◽  
Data Augmentation ◽  
Generative Adversarial Networks ◽  
Geometric Transformation ◽  
Expression Recognition ◽  
Learning Approaches ◽  
Geometric Transformations ◽  
Augmentation Techniques ◽  
The Impact

Most Facial Expression Recognition (FER) systems rely on machine learning approaches that require large databases for an effective training. As these are not easily available, a good solution is to augment the databases with appropriate data augmentation (DA) techniques, which are typically based on either geometric transformation or oversampling augmentations (e.g., generative adversarial networks (GANs)). However, it is not always easy to understand which DA technique may be more convenient for FER systems because most state-of-the-art experiments use different settings which makes the impact of DA techniques not comparable. To advance in this respect, in this paper, we evaluate and compare the impact of using well-established DA techniques on the emotion recognition accuracy of a FER system based on the well-known VGG16 convolutional neural network (CNN). In particular, we consider both geometric transformations and GAN to increase the amount of training images. We performed cross-database evaluations: training with the "augmented" KDEF database and testing with two different databases (CK+ and ExpW). The best results were obtained combining horizontal reflection, translation and GAN, bringing an accuracy increase of approximately 30%. This outperforms alternative approaches, except for the one technique that could however rely on a quite bigger database.

scholarly journals CNN-Based Facial Expression Recognition from Annotated RGB-D Images for Human–Robot Interaction

2019 ◽  
Vol 16 (04) ◽  
pp. 1941002 ◽  
Author(s):  
Jing Li ◽  
Yang Mi ◽  
Gongfa Li ◽  
Zhaojie Ju
Keyword(s):  
Facial Expression ◽  
Facial Expression Recognition ◽  
Recognition Task ◽  
Recognition System ◽  
Human Robot Interaction ◽  
Microsoft Kinect ◽  
Depth Information ◽  
Expression Recognition ◽  
Stream Network ◽  
Depth Images

Facial expression recognition has been widely used in human computer interaction (HCI) systems. Over the years, researchers have proposed different feature descriptors, implemented different classification methods, and carried out a number of experiments on various datasets for automatic facial expression recognition. However, most of them used 2D static images or 2D video sequences for the recognition task. The main limitations of 2D-based analysis are problems associated with variations in pose and illumination, which reduce the recognition accuracy. Therefore, an alternative way is to incorporate depth information acquired by 3D sensor, because it is invariant in both pose and illumination. In this paper, we present a two-stream convolutional neural network (CNN)-based facial expression recognition system and test it on our own RGB-D facial expression dataset collected by Microsoft Kinect for XBOX in unspontaneous scenarios since Kinect is an inexpensive and portable device to capture both RGB and depth information. Our fully annotated dataset includes seven expressions (i.e., neutral, sadness, disgust, fear, happiness, anger, and surprise) for 15 subjects (9 males and 6 females) aged from 20 to 25. The two individual CNNs are identical in architecture but do not share parameters. To combine the detection results produced by these two CNNs, we propose the late fusion approach. The experimental results demonstrate that the proposed two-stream network using RGB-D images is superior to that of using only RGB images or depth images.

Sign in / Sign up

Export Citation Format

Share Document