Fast Video Facial Expression Recognition by a Deeply Tensor-Compressed LSTM Neural Network for Mobile Devices

2021 ◽  
Vol 2 (4) ◽  
pp. 1-26
Author(s):  
Peining Zhen ◽  
Hai-Bao Chen ◽  
Yuan Cheng ◽  
Zhigang Ji ◽  
Bin Liu ◽  
...  
Keyword(s):  
Neural Network ◽  
Facial Expression ◽  
Mobile Devices ◽  
Mobile Device ◽  
Facial Expression Recognition ◽  
Short Term Memory ◽  
Expression Recognition ◽  
Feature Maps ◽  
Temporal Model ◽  
Spatio Temporal

Mobile devices usually suffer from limited computation and storage resources, which seriously hinders them from deep neural network applications. In this article, we introduce a deeply tensor-compressed long short-term memory (LSTM) neural network for fast video-based facial expression recognition on mobile devices. First, a spatio-temporal facial expression recognition LSTM model is built by extracting time-series feature maps from facial clips. The LSTM-based spatio-temporal model is further deeply compressed by means of quantization and tensorization for mobile device implementation. Based on datasets of Extended Cohn-Kanade (CK+), MMI, and Acted Facial Expression in Wild 7.0, experimental results show that the proposed method achieves 97.96%, 97.33%, and 55.60% classification accuracy and significantly compresses the size of network model up to 221× with reduced training time per epoch by 60%. Our work is further implemented on the RK3399Pro mobile device with a Neural Process Engine. The latency of the feature extractor and LSTM predictor can be reduced 30.20× and 6.62× , respectively, on board with the leveraged compression methods. Furthermore, the spatio-temporal model costs only 57.19 MB of DRAM and 5.67W of power when running on the board.

scholarly journals Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

2019 ◽  
Vol 33 ◽  
pp. 3215-3223 ◽  
Author(s):  
Wissam J. Baddar ◽  
Yong Man Ro
Keyword(s):  
Facial Expression ◽  
Facial Expression Recognition ◽  
Short Term Memory ◽  
Recognition Task ◽  
Input Sequence ◽  
Expression Recognition ◽  
Cell State ◽  
Temporal Features ◽  
Additional Cell ◽  
Spatio Temporal

Spatio-temporal feature encoding is essential for encoding the dynamics in video sequences. Recurrent neural networks, particularly long short-term memory (LSTM) units, have been popular as an efficient tool for encoding spatio-temporal features in sequences. In this work, we investigate the effect of mode variations on the encoded spatio-temporal features using LSTMs. We show that the LSTM retains information related to the mode variation in the sequence, which is irrelevant to the task at hand (e.g. classification facial expressions). Actually, the LSTM forget mechanism is not robust enough to mode variations and preserves information that could negatively affect the encoded spatio-temporal features. We propose the mode variational LSTM to encode spatio-temporal features robust to unseen modes of variation. The mode variational LSTM modifies the original LSTM structure by adding an additional cell state that focuses on encoding the mode variation in the input sequence. To efficiently regulate what features should be stored in the additional cell state, additional gating functionality is also introduced. The effectiveness of the proposed mode variational LSTM is verified using the facial expression recognition task. Comparative experiments on publicly available datasets verified that the proposed mode variational LSTM outperforms existing methods. Moreover, a new dynamic facial expression dataset with different modes of variation, including various modes like pose and illumination variations, was collected to comprehensively evaluate the proposed mode variational LSTM. Experimental results verified that the proposed mode variational LSTM encodes spatio-temporal features robust to unseen modes of variation.

scholarly journals Learning Sequential Patterns Using Spatio-Temporal Attention Networks for Video Classification

2021 ◽  
Author(s):  
Shubhada Deshmukh ◽  
Manasi Patwardhan ◽  
Anjali Mahajan ◽  
Sadanand Deshpande
Keyword(s):  
Neural Network ◽  
Facial Expression ◽  
Facial Expression Recognition ◽  
Research Effort ◽  
Real Life ◽  
Vital Role ◽  
Expression Recognition ◽  
Video Classification ◽  
Attention Model ◽  
Spatio Temporal

Abstract Extensive research effort has been focused on extracting temporal patterns from videos, to improve the accuracy of video classification using a deep neural network based approaches. In this paper, we show that long term dependency patterns may not be enough to achieve sufficient improved results. We propose the Attention-based Spatio-Temporal model (AST) for video classification, which is a self-attention model that learns to attend to spatial features using Convolutional Neural Network (CNN) and temporal features using attention mechanisms. We evaluate our model on motion dependent Action recognition (UCF-101) dataset, facial expression recognition (MMI) dataset, and micro-expression recognition (CASME2) dataset and generated real-life Facial Expression Recognition (FER) dataset and improved by 10%, 4.7% and 5.6% accuracy respectively as compared to state-of-art on the three standard datasets and a synthetic dataset as well.In our research, we performed several experiments for detecting expressions and actions, the AST model plays a vital role in selecting the frames and carry the sequential context in the real-time application as well. We also experimented by extracting the features using the Active shape model (ASM) for FER and found the AST model surpasses other approaches.

scholarly journals Iranian kinect face database (IKFDB): a color-depth based face database collected by kinect v.2 sensor

2021 ◽  
Vol 3 (1) ◽  
Author(s):  
Seyed Muhammad Hossein Mousavi ◽  
S. Younes Mirinezhad
Keyword(s):  
Neural Network ◽  
Facial Expression ◽  
Facial Expression Recognition ◽  
Depth Image ◽  
Sensor Technology ◽  
Support Vector ◽  
Expression Recognition ◽  
Face Database ◽  
Depth Data ◽  
Color Depth

AbstractThis study presents a new color-depth based face database gathered from different genders and age ranges from Iranian subjects. Using suitable databases, it is possible to validate and assess available methods in different research fields. This database has application in different fields such as face recognition, age estimation and Facial Expression Recognition and Facial Micro Expressions Recognition. Image databases based on their size and resolution are mostly large. Color images usually consist of three channels namely Red, Green and Blue. But in the last decade, another aspect of image type has emerged, named “depth image”. Depth images are used in calculating range and distance between objects and the sensor. Depending on the depth sensor technology, it is possible to acquire range data differently. Kinect sensor version 2 is capable of acquiring color and depth data simultaneously. Facial expression recognition is an important field in image processing, which has multiple uses from animation to psychology. Currently, there is a few numbers of color-depth (RGB-D) facial micro expressions recognition databases existing. With adding depth data to color data, the accuracy of final recognition will be increased. Due to the shortage of color-depth based facial expression databases and some weakness in available ones, a new and almost perfect RGB-D face database is presented in this paper, covering Middle-Eastern face type. In the validation section, the database will be compared with some famous benchmark face databases. For evaluation, Histogram Oriented Gradients features are extracted, and classification algorithms such as Support Vector Machine, Multi-Layer Neural Network and a deep learning method, called Convolutional Neural Network or are employed. The results are so promising.

Conditional convolution neural network enhanced random forest for facial expression recognition

2018 ◽  
Vol 84 ◽  
pp. 251-261 ◽  
Author(s):  
Yuanyuan Liu ◽  
Xiaohui Yuan ◽  
Xi Gong ◽  
Zhong Xie ◽  
Fang Fang ◽  
...  
Keyword(s):  
Neural Network ◽  
Random Forest ◽  
Facial Expression ◽  
Facial Expression Recognition ◽  
Convolution Neural Network ◽  
Expression Recognition
Sign in / Sign up

Export Citation Format

Share Document