OPTICAL FLOW BY MULTI-SCALE ANNOTATED KEYPOINTS - A Biological Approach

Recently, video frame interpolation research developed with a convolutional neural network has shown remarkable results. However, these methods demand huge amounts of memory and run time for high-resolution videos, and are unable to process a 4K frame in a single pass. In this paper, we propose a fast 4K video frame interpolation method, based upon a multi-scale optical flow reconstruction scheme. The proposed method predicts low resolution bi-directional optical flow, and reconstructs it into high resolution. We also proposed consistency and multi-scale smoothness loss to enhance the quality of the predicted optical flow. Furthermore, we use adversarial loss to make the interpolated frame more seamless and natural. We demonstrated that the proposed method outperforms the existing state-of-the-art methods in quantitative evaluation, while it runs up to 4.39× faster than those methods for 4K videos.

Download Full-text

Deep Optical Flow Estimation Via Multi-Scale Correspondence Structure Learning

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2017/488 ◽

2017 ◽

Cited By ~ 2

Author(s):

Shanshan Zhao ◽

Xi Li ◽

Omar El Farouk Bourahla

Keyword(s):

Deep Learning ◽

Optical Flow ◽

Structure Learning ◽

Feature Space ◽

Image Correlation ◽

Flow Estimation ◽

Optical Flow Estimation ◽

Multi Scale ◽

Benchmark Datasets ◽

End To End

As an important and challenging problem in computer vision, learning based optical flow estimation aims to discover the intrinsic correspondence structure between two adjacent video frames through statistical learning. Therefore, a key issue to solve in this area is how to effectively model the multi-scale correspondence structure properties in an adaptive end-to-end learning fashion. Motivated by this observation, we propose an end-to-end multi-scale correspondence structure learning (MSCSL) approach for optical flow estimation. In principle, the proposed MSCSL approach is capable of effectively capturing the multi-scale inter-image-correlation correspondence structures within a multi-level feature space from deep learning. Moreover, the proposed MSCSL approach builds a spatial Conv-GRU neural network model to adaptively model the intrinsic dependency relationships among these multi-scale correspondence structures. Finally, the above procedures for correspondence structure learning and multi-scale dependency modeling are implemented in a unified end-to-end deep learning framework. Experimental results on several benchmark datasets demonstrate the effectiveness of the proposed approach.

Download Full-text

A Multi-Scale Phase-Based Optical Flow Method for Motion Tracking of Left Ventricle

2010 4th International Conference on Bioinformatics and Biomedical Engineering ◽

10.1109/icbbe.2010.5515478 ◽

2010 ◽

Cited By ~ 5

Author(s):

Xiaomei Yang ◽

Kenya Murase

Keyword(s):

Left Ventricle ◽

Optical Flow ◽

Motion Tracking ◽

Flow Method ◽

Multi Scale ◽

Optical Flow Method

Download Full-text

Analog-VLSI, array-processor-based, Bayesian, multi-scale optical flow estimation

International Journal of Circuit Theory and Applications ◽

10.1002/cta.342 ◽

2006 ◽

Vol 34 (1) ◽

pp. 49-75 ◽

Cited By ~ 8

Author(s):

L. Török ◽

Á. Zarándy

Keyword(s):

Optical Flow ◽

Analog Vlsi ◽

Array Processor ◽

Flow Estimation ◽

Optical Flow Estimation ◽

Multi Scale ◽

Vlsi Array

Download Full-text

Multi-scale joint feature network for micro-expression recognition

Computational Visual Media ◽

10.1007/s41095-021-0217-9 ◽

2021 ◽

Author(s):

Xinyu Li ◽

Guangshun Wei ◽

Jie Wang ◽

Yuanfeng Zhou

Keyword(s):

Optical Flow ◽

Expression Recognition ◽

Facial Motion ◽

Multi Scale ◽

Flow Image ◽

Wide Range ◽

Recognition Ability ◽

Benchmark Datasets ◽

Casme Ii ◽

Micro Expression

AbstractMicro-expression recognition is a substantive cross-study of psychology and computer science, and it has a wide range of applications (e.g., psychological and clinical diagnosis, emotional analysis, criminal investigation, etc.). However, the subtle and diverse changes in facial muscles make it difficult for existing methods to extract effective features, which limits the improvement of micro-expression recognition accuracy. Therefore, we propose a multi-scale joint feature network based on optical flow images for micro-expression recognition. First, we generate an optical flow image that reflects subtle facial motion information. The optical flow image is then fed into the multi-scale joint network for feature extraction and classification. The proposed joint feature module (JFM) integrates features from different layers, which is beneficial for the capture of micro-expression features with different amplitudes. To improve the recognition ability of the model, we also adopt a strategy for fusing the feature prediction results of the three JFMs with the backbone network. Our experimental results show that our method is superior to state-of-the-art methods on three benchmark datasets (SMIC, CASME II, and SAMM) and a combined dataset (3DB).

Download Full-text