scholarly journals CoT-AMFlow: Adaptive Modulation Network with Co-Teaching Strategy for Unsupervised Optical Flow Estimation

2020 ◽  
Author(s):  
Hengli Wang ◽  
Rui Fan ◽  
Ming Liu
Keyword(s):  
Optical Flow ◽  
Network Architecture ◽  
Teaching Strategy ◽  
Adaptive Modulation ◽  
Flow Modulation ◽  
Flow Estimation ◽  
Optical Flow Estimation ◽  
Flow Information ◽  
Partially Occluded ◽  
Unsupervised Approaches

The interpretation of ego motion and scene change is a fundamental task for mobile robots. Optical flow information can be employed to estimate motion in the surroundings. Recently, unsupervised optical flow estimation has become a research hotspot. However, unsupervised approaches are often easy to be unreliable on partially occluded or texture-less regions. To deal with this problem, we propose CoT-AMFlow in this paper, an unsupervised optical flow estimation approach. In terms of the network architecture, we develop an adaptive modulation network that employs two novel module types, flow modulation modules (FMMs) and cost volume modulation modules (CMMs), to remove outliers in challenging regions. As for the training paradigm, we adopt a co-teaching strategy, where two networks simultaneously teach each other about challenging regions to further improve accuracy. Experimental results on the MPI Sintel, KITTI Flow and Middlebury Flow benchmarks demonstrate that our CoT-AMFlow outperforms all other state-of-the-art unsupervised approaches, while still running in real time. Our project page is available at https://sites.google.com/view/cot-amflow.

scholarly journals CoT-AMFlow: Adaptive Modulation Network with Co-Teaching Strategy for Unsupervised Optical Flow Estimation

2020 ◽  
Author(s):  
Hengli Wang ◽  
Rui Fan ◽  
Ming Liu
Keyword(s):  
Optical Flow ◽  
Network Architecture ◽  
Teaching Strategy ◽  
Adaptive Modulation ◽  
Flow Modulation ◽  
Flow Estimation ◽  
Optical Flow Estimation ◽  
Flow Information ◽  
Partially Occluded ◽  
Unsupervised Approaches

The interpretation of ego motion and scene change is a fundamental task for mobile robots. Optical flow information can be employed to estimate motion in the surroundings. Recently, unsupervised optical flow estimation has become a research hotspot. However, unsupervised approaches are often easy to be unreliable on partially occluded or texture-less regions. To deal with this problem, we propose CoT-AMFlow in this paper, an unsupervised optical flow estimation approach. In terms of the network architecture, we develop an adaptive modulation network that employs two novel module types, flow modulation modules (FMMs) and cost volume modulation modules (CMMs), to remove outliers in challenging regions. As for the training paradigm, we adopt a co-teaching strategy, where two networks simultaneously teach each other about challenging regions to further improve accuracy. Experimental results on the MPI Sintel, KITTI Flow and Middlebury Flow benchmarks demonstrate that our CoT-AMFlow outperforms all other state-of-the-art unsupervised approaches, while still running in real time. Our project page is available at https://sites.google.com/view/cot-amflow.

scholarly journals CoT-AMFlow: Adaptive Modulation Network with Co-Teaching Strategy for Unsupervised Optical Flow Estimation

2020 ◽  
Author(s):  
Hengli Wang ◽  
Rui Fan ◽  
Ming Liu
Keyword(s):  
Optical Flow ◽  
Network Architecture ◽  
Teaching Strategy ◽  
Adaptive Modulation ◽  
Flow Modulation ◽  
Flow Estimation ◽  
Optical Flow Estimation ◽  
Flow Information ◽  
Partially Occluded ◽  
Unsupervised Approaches

The interpretation of ego motion and scene change is a fundamental task for mobile robots. Optical flow information can be employed to estimate motion in the surroundings. Recently, unsupervised optical flow estimation has become a research hotspot. However, unsupervised approaches are often easy to be unreliable on partially occluded or texture-less regions. To deal with this problem, we propose CoT-AMFlow in this paper, an unsupervised optical flow estimation approach. In terms of the network architecture, we develop an adaptive modulation network that employs two novel module types, flow modulation modules (FMMs) and cost volume modulation modules (CMMs), to remove outliers in challenging regions. As for the training paradigm, we adopt a co-teaching strategy, where two networks simultaneously teach each other about challenging regions to further improve accuracy. Experimental results on the MPI Sintel, KITTI Flow and Middlebury Flow benchmarks demonstrate that our CoT-AMFlow outperforms all other state-of-the-art unsupervised approaches, while still running in real time. Our project page is available at https://sites.google.com/view/cot-amflow.

scholarly journals Ghost-removal image warping for optical flow estimation

2019 ◽  
Vol 277 ◽  
pp. 02002
Author(s):  
Song Wang ◽  
Zengfu Wang
Keyword(s):  
Optical Flow ◽  
Image Warping ◽  
Estimation Methods ◽  
Flow Estimation ◽  
Optical Flow Estimation ◽  
Flow Information ◽  
Ghosting Artifacts ◽  
Compensation Technique ◽  
Image Compensation ◽  
Traditional Image

Traditional image warping methods used in optical flow estimation usually adopt simple interpolation strategies to obtain the warped images. But without considering the characteristic of occluded regions, the traditional methods may result in undesirable ghosting artifacts. To tackle this problem, in this paper we propose a novel image warping method to effectively remove ghosting artifacts. To be Specific, when given a warped image, the ghost regions are firstly discriminated using the optical flow information. Then, we use a new image compensation technique to eliminate the ghosting artifacts. The proposed method can avoid serious distortion in the warped images, therefore can prevent error propagation in the coarse-to-fine optical flow estimation schemes. Meanwhile, our approach can be easily integrated into various optical flow estimation methods. Experimental results on some popular datasets such as Flying Chairs and MPI-Sintel demonstrate that the proposed method can improve the performance of current optical flow estimation methods.

scholarly journals Implicit and Explicit Regularization for Optical Flow Estimation

Sensors ◽  
2020 ◽  
Vol 20 (14) ◽  
pp. 3855
Author(s):  
Konstantinos Karageorgos ◽  
Anastasios Dimou ◽  
Federico Alvarez ◽  
Petros Daras
Keyword(s):  
Neural Network ◽  
Optical Flow ◽  
Regularization Method ◽  
Network Architecture ◽  
Contextual Information ◽  
Semantic Segmentation ◽  
Convergence Time ◽  
Local Motion ◽  
Flow Estimation ◽  
Optical Flow Estimation

In this paper, two novel and practical regularizing methods are proposed to improve existing neural network architectures for monocular optical flow estimation. The proposed methods aim to alleviate deficiencies of current methods, such as flow leakage across objects and motion consistency within rigid objects, by exploiting contextual information. More specifically, the first regularization method utilizes semantic information during the training process to explicitly regularize the produced optical flow field. The novelty of this method lies in the use of semantic segmentation masks to teach the network to implicitly identify the semantic edges of an object and better reason on the local motion flow. A novel loss function is introduced that takes into account the objects’ boundaries as derived from the semantic segmentation mask to selectively penalize motion inconsistency within an object. The method is architecture agnostic and can be integrated into any neural network without modifying or adding complexity at inference. The second regularization method adds spatial awareness to the input data of the network in order to improve training stability and efficiency. The coordinates of each pixel are used as an additional feature, breaking the invariance properties of the neural network architecture. The additional features are shown to implicitly regularize the optical flow estimation enforcing a consistent flow, while improving both the performance and the convergence time. Finally, the combination of both regularization methods further improves the performance of existing cutting edge architectures in a complementary way, both quantitatively and qualitatively, on popular flow estimation benchmark datasets.

scholarly journals Learned optical flow for intra-operative tracking of the retinal fundus

2020 ◽  
Vol 15 (5) ◽  
pp. 827-836
Author(s):  
Claudio S. Ravasio ◽  
Theodoros Pissas ◽  
Edward Bloch ◽  
Blanca Flores ◽  
Sepehr Jalali ◽  
...  
Keyword(s):  
Optical Flow ◽  
Ground Truth ◽  
Semantic Segmentation ◽  
Robotic Systems ◽  
Sustained Delivery ◽  
Flow Estimation ◽  
Optical Flow Estimation ◽  
Points Of Interest ◽  
Flow Information ◽  
Retinal Fundus

Abstract Purpose Sustained delivery of regenerative retinal therapies by robotic systems requires intra-operative tracking of the retinal fundus. We propose a supervised deep convolutional neural network to densely predict semantic segmentation and optical flow of the retina as mutually supportive tasks, implicitly inpainting retinal flow information missing due to occlusion by surgical tools. Methods As manual annotation of optical flow is infeasible, we propose a flexible algorithm for generation of large synthetic training datasets on the basis of given intra-operative retinal images. We evaluate optical flow estimation by tracking a grid and sparsely annotated ground truth points on a benchmark of challenging real intra-operative clips obtained from an extensive internally acquired dataset encompassing representative vitreoretinal surgical cases. Results The U-Net-based network trained on the synthetic dataset is shown to generalise well to the benchmark of real surgical videos. When used to track retinal points of interest, our flow estimation outperforms variational baseline methods on clips containing tool motions which occlude the points of interest, as is routinely observed in intra-operatively recorded surgery videos. Conclusions The results indicate that complex synthetic training datasets can be used to specifically guide optical flow estimation. Our proposed algorithm therefore lays the foundation for a robust system which can assist with intra-operative tracking of moving surgical targets even when occluded.

scholarly journals Every Frame Counts: Joint Learning of Video Segmentation and Optical Flow

2020 ◽  
Vol 34 (07) ◽  
pp. 10713-10720
Author(s):  
Mingyu Ding ◽  
Zhe Wang ◽  
Bolei Zhou ◽  
Jianping Shi ◽  
Zhiwu Lu ◽  
...  
Keyword(s):  
Optical Flow ◽  
Video Segmentation ◽  
Video Clip ◽  
Semantic Segmentation ◽  
Temporal Consistency ◽  
Flow Estimation ◽  
Optical Flow Estimation ◽  
Optical Flows ◽  
Benchmark Datasets ◽  
Spatio Temporal

A major challenge for video semantic segmentation is the lack of labeled data. In most benchmark datasets, only one frame of a video clip is annotated, which makes most supervised methods fail to utilize information from the rest of the frames. To exploit the spatio-temporal information in videos, many previous works use pre-computed optical flows, which encode the temporal consistency to improve the video segmentation. However, the video segmentation and optical flow estimation are still considered as two separate tasks. In this paper, we propose a novel framework for joint video semantic segmentation and optical flow estimation. Semantic segmentation brings semantic information to handle occlusion for more robust optical flow estimation, while the non-occluded optical flow provides accurate pixel-level temporal correspondences to guarantee the temporal consistency of the segmentation. Moreover, our framework is able to utilize both labeled and unlabeled frames in the video through joint training, while no additional calculation is required in inference. Extensive experiments show that the proposed model makes the video semantic segmentation and optical flow estimation benefit from each other and outperforms existing methods under the same settings in both tasks.

Sign in / Sign up

Export Citation Format

Share Document