A comparison of deep learning models for end-to-end face-based video retrieval in unconstrained videos

Face-based video retrieval (FBVR) is the task of retrieving videos that containing the same face shown in the query image. In this article, we present the first end-to-end FBVR pipeline that is able to operate on large datasets of unconstrained, multi-shot, multi-person videos. We adapt an existing audiovisual recognition dataset to the task of FBVR and use it to evaluate our proposed pipeline. We compare a number of deep learning models for shot detection, face detection, and face feature extraction as part of our pipeline on a validation dataset made of more than 4000 videos. We obtain 97.25% mean average precision on an independent test set, composed of more than 1000 videos. The pipeline is able to extract features from videos at $$\sim $$ ∼ 7 times the real-time speed, and it is able to perform a query on thousands of videos in less than 0.5 s.

A Descriptive Survey on Face Emotion Recognition Techniques

Recognition of natural emotion from human faces has applications in Human–Computer Interaction, image and video retrieval, automated tutoring systems, smart environment as well as driver warning systems. It is also a significant indication of nonverbal communication among the individuals. The assignment of Face Emotion Recognition (FER) is predominantly complex for two reasons. The first reason is the nonexistence of a large database of training images, and the second one is about classifying the emotions, which can be complex based on the static input image. In addition, robust unbiased FER in real time remains the foremost challenge for various supervised learning-based techniques. This survey analyzes diverse techniques regarding the FER systems. It reviews a bunch of research papers and performs a significant analysis. Initially, the analysis depicts various techniques that are contributed in different research papers. In addition, this paper offers a comprehensive study regarding the chronological review and performance achievements in each contribution. The analytical review is also concerned about the measures for which the maximum performance was achieved in several contributions. Finally, the survey is extended with various research issues and gaps that can be useful for the researchers to promote improved future works on the FER models.

Intelligent Sports Video Classification Based on Deep Neural Network (DNN) Algorithm and Transfer Learning

Traditional text annotation-based video retrieval is done by manually labeling videos with text, which is inefficient and highly subjective and generally cannot accurately describe the meaning of videos. Traditional content-based video retrieval uses convolutional neural networks to extract the underlying feature information of images to build indexes and achieves similarity retrieval of video feature vectors according to certain similarity measure algorithms. In this paper, by studying the characteristics of sports videos, we propose the histogram difference method based on using transfer learning and the four-step method based on block matching for mutation detection and fading detection of video shots, respectively. By adaptive thresholding, regions with large frame difference changes are marked as candidate regions for shots, and then the shot boundaries are determined by mutation detection algorithm. Combined with the characteristics of sports video, this paper proposes a key frame extraction method based on clustering and optical flow analysis, and experimental comparison with the traditional clustering method. In addition, this paper proposes a key frame extraction algorithm based on clustering and optical flow analysis for key frame extraction of sports video. The algorithm effectively removes the redundant frames, and the extracted key frames are more representative. Through extensive experiments, the keyword fuzzy finding algorithm based on improved deep neural network and ontology semantic expansion proposed in this paper shows a more desirable retrieval performance, and it is feasible to use this method for video underlying feature extraction, annotation, and keyword finding, and one of the outstanding features of the algorithm is that it can quickly and effectively retrieve the desired video in a large number of Internet video resources, reducing the false detection rate and leakage rate while improving the fidelity, which basically meets people’s daily needs.