The Elements of End-to-end Deep Face Recognition: A Survey of Recent Advances

Face recognition is one of the most popular and long-standing topics in computer vision. With the recent development of deep learning techniques and large-scale datasets, deep face recognition has made remarkable progress and been widely used in many real-world applications. Given a natural image or video frame as input, an end-to-end deep face recognition system outputs the face feature for recognition. To achieve this, a typical end-to-end system is built with three key elements: face detection, face alignment, and face representation. The face detection locates faces in the image or frame. Then, the face alignment is proceeded to calibrate the faces to the canonical view and crop them with a normalized pixel size. Finally, in the stage of face representation, the discriminative features are extracted from the aligned face for recognition. Nowadays, all of the three elements are fulfilled by the technique of deep convolutional neural network. In this survey article, we present a comprehensive review about the recent advance of each element of the end-to-end deep face recognition, since the thriving deep learning techniques have greatly improved the capability of them. To start with, we present an overview of the end-to-end deep face recognition. Then, we review the advance of each element, respectively, covering many aspects such as the to-date algorithm designs, evaluation metrics, datasets, performance comparison, existing challenges, and promising directions for future research. Also, we provide a detailed discussion about the effect of each element on its subsequent elements and the holistic system. Through this survey, we wish to bring contributions in two aspects: first, readers can conveniently identify the methods which are quite strong-baseline style in the subcategory for further exploration; second, one can also employ suitable methods for establishing a state-of-the-art end-to-end face recognition system from scratch.

Granule-Based-Classifier (GbC): A Lattice Computing Scheme Applied on Tree Data Structures

Social robots keep proliferating. A critical challenge remains their sensible interaction with humans, especially in real world applications. Hence, computing with real world semantics is instrumental. Recently, the Lattice Computing (LC) paradigm has been proposed with a capacity to compute with semantics represented by partial order in a mathematical lattice data domain. In the aforementioned context, this work proposes a parametric LC classifier, namely a Granule-based-Classifier (GbC), applicable in a mathematical lattice (T,⊑) of tree data structures, each of which represents a human face. A tree data structure here emerges from 68 facial landmarks (points) computed in a data preprocessing step by the OpenFace software. The proposed (tree) representation retains human anonymity during data processing. Extensive computational experiments regarding three different pattern recognition problems, namely (1) head orientation, (2) facial expressions, and (3) human face recognition, demonstrate GbC capacities, including good classification results, and a common human face representation in different pattern recognition problems, as well as data induced granular rules in (T,⊑) that allow for (a) explainable decision-making, (b) tunable generalization enabled also by formal logic/reasoning techniques, and (c) an inherent capacity for modular data fusion extensions. The potential of the proposed techniques is discussed.

Investigating facial information content in the hand area of individuals with a congenital and acquired missing hand

Cortical remapping after hand loss in the primary somatosensory cortex (S1) is thought to be predominantly dictated by cortical proximity, with adjacent body parts remapping into the deprived area. Traditionally, this remapping has been characterised by changes in the lip representation, which is assumed to be the immediate neighbour of the hand based on electrophysiological research in non-human primates. However, the orientation of facial somatotopy in humans is debated, with contrasting work reporting both an inverted and upright topography. We aimed to fill this gap in the S1 homunculus by investigating the topographic organisation of the face. Using both univariate and multivariate approaches we examined the extent of face-to-hand remapping in individuals with a congenital and acquired missing hand (hereafter one-handers and amputees, respectively), relative to two-handed controls. Participants were asked to move different facial parts (forehead, nose, lips, tongue) during fMRI scanning. We first report evidence for an upright facial organisation in all three groups, with the upper face and not the lips bordering the hand area. We further found little evidence for remapping of all tested facial parts in amputees, with no significant relationship to the chronicity of their PLP. In contrast, we found converging evidence for a complex pattern of face remapping in congenital one-handers across all facial parts, where the location of the cortical neighbour, the forehead, is shown to shift away from the deprived hand area, which is subsequently activated by the lips and the tongue. Together, our findings demonstrate that the face representation in humans is highly plastic, but that this plasticity is restricted by the developmental stage of input deprivation, rather than cortical proximity.

View-tuned and view-invariant face encoding in IT cortex is explained by selected natural image fragments

Humans recognize individual faces regardless of variation in the facial view. The view-tuned face neurons in the inferior temporal (IT) cortex are regarded as the neural substrate for view-invariant face recognition. This study approximated visual features encoded by these neurons as combinations of local orientations and colors, originated from natural image fragments. The resultant features reproduced the preference of these neurons to particular facial views. We also found that faces of one identity were separable from the faces of other identities in a space where each axis represented one of these features. These results suggested that view-invariant face representation was established by combining view sensitive visual features. The face representation with these features suggested that, with respect to view-invariant face representation, the seemingly complex and deeply layered ventral visual pathway can be approximated via a shallow network, comprised of layers of low-level processing for local orientations and colors (V1/V2-level) and the layers which detect particular sets of low-level elements derived from natural image fragments (IT-level).

TRADITION AND INNOVATION IN THE WORKS OF O. A. KIPRENSKY: PORTRAIT OF A. K. SCHWALBE

The article considers one of the key works of the portrait genre of the 1800s — the portrait of Adam K. Schwalbe by Orest A. Kiprensky. The analysis of this work by several generations of art historians (Herman A. Nedoshivin, Natalia N. Kovalenskaya, Dmitry V. Sarabyanov, Yakov V. Brook, Irina V. Linnik) reveals its main substantial and formal features, and also clarifies issues related to the concepts of tradition and innovation. All those who have written about this work agree that the artistic image of A. K. Schwalbe's portrait is based on impressions of Rubens and Rembrandt. However, a more careful analysis of Kiprensky's work provides an opportunity to considerably expand the sources of possible borrowings not only from the masters of the past. Studying it in the context of the art of 1800s leads to the conclusion that the works of Kiprensky's contemporaries (in particular Salvatore Tonchi) contain the same motifs used in the portrait of Schwalbe, namely: attributes of "fur coat portraits", the full-face representation of the model and the tightness of space, sharp character and expressiveness of the portrait's appearance. To prove the thesis that Kiprensky was influenced by the art of his time, a large number of works (including those by unknown artists), both famous and little-known, are involved.