This dissertation presents a probabilistic state estimation framework for integrating data-driven machine learning models and a deformable facial shape model in order to estimate continuous-valued intensities of 22 different facial muscle movements, known as Action Units (AU), defined in the Facial Action Coding System (FACS). A practical approach is proposed and validated for integrating class-wise probability scores from machine learning models within a Gaussian state estimation framework. Furthermore, driven mass-spring-damper models are applied for modelling the dynamics of facial muscle movements. Both facial shape and appearance information are used for estimating AU intensities, making it a hybrid approach. Several features are designed and explored to help the probabilistic framework to deal with multiple challenges involved in automatic AU detection. The proposed AU intensity estimation method and its features are evaluated quantitatively and qualitatively using three different datasets containing either spontaneous or acted facial expressions with AU annotations. The proposed method produced temporally smoother estimates that facilitate a fine-grained analysis of facial expressions. It also performed reasonably well, even though it simultaneously estimates intensities of 22 AUs, some of which are subtle in expression or resemble each other closely. The estimated AU intensities tended to the lower range of values, and were often accompanied by a small delay in onset. This shows that the proposed method is conservative. In order to further improve performance, state-of-the-art machine learning approaches for AU detection could be integrated within the proposed probabilistic AU intensity estimation framework. In addition to AU intensity estimation, this dissertation explores the applicability of the estimated AU intensities for automatic analysis of mental states such as pain and distraction. A survey of automatic pain detection approaches, conducted as part of this dissertation, highlights the progress and deficits in this field. Several AU-based rules are designed for pain intensity estimation based on psychological evidence, and their performance is evaluated empirically. The potential of these AU-based rules to automatically generate explanations for pain detections is also illustrated. Furthermore, a preliminary analysis of the estimated AU intensities shows differences in facial actions between various distraction scenarios during simulated driving. The results of this dissertation show that more interdisciplinary research is needed to address the open challenges in the field of automatic mental state analysis, particularly to build reference datasets, to model interpersonal differences, and to generate human-comprehensible explanations of predictions.
Towards Robust and Interpretable Practical Applications of Automatic Mental State Analysis Using a Dynamic and Hybrid Facial Action Estimation Approach
