scholarly journals A Fast 4K Video Frame Interpolation Using a Multi-Scale Optical Flow Reconstruction Network

Symmetry ◽  
2019 ◽  
Vol 11 (10) ◽  
pp. 1251 ◽  
Author(s):  
Ahn ◽  
Jeong ◽  
Kim ◽  
Kwon ◽  
Yoo
Keyword(s):  
High Resolution ◽  
Optical Flow ◽  
State Of The Art ◽  
Interpolation Method ◽  
Video Frame ◽  
Frame Interpolation ◽  
Multi Scale ◽  
Reconstruction Scheme ◽  
Flow Reconstruction

Recently, video frame interpolation research developed with a convolutional neural network has shown remarkable results. However, these methods demand huge amounts of memory and run time for high-resolution videos, and are unable to process a 4K frame in a single pass. In this paper, we propose a fast 4K video frame interpolation method, based upon a multi-scale optical flow reconstruction scheme. The proposed method predicts low resolution bi-directional optical flow, and reconstructs it into high resolution. We also proposed consistency and multi-scale smoothness loss to enhance the quality of the predicted optical flow. Furthermore, we use adversarial loss to make the interpolated frame more seamless and natural. We demonstrated that the proposed method outperforms the existing state-of-the-art methods in quantitative evaluation, while it runs up to 4.39× faster than those methods for 4K videos.

scholarly journals A Fast 4K Video Frame Interpolation Using a Hybrid Task-Based Convolutional Neural Network

Symmetry ◽  
2019 ◽  
Vol 11 (5) ◽  
pp. 619 ◽  
Author(s):  
Ha-Eun Ahn ◽  
Jinwoo Jeong ◽  
Je Woo Kim
Keyword(s):  
Neural Network ◽  
High Resolution ◽  
Convolutional Neural Network ◽  
High Frequency ◽  
State Of The Art ◽  
Visual Quality ◽  
Video Frame ◽  
Frame Interpolation ◽  
Algorithm Efficiency ◽  
Coarse To Fine

Visual quality and algorithm efficiency are two main interests in video frame interpolation. We propose a hybrid task-based convolutional neural network for fast and accurate frame interpolation of 4K videos. The proposed method synthesizes low-resolution frames, then reconstructs high-resolution frames in a coarse-to-fine fashion. We also propose edge loss, to preserve high-frequency information and make the synthesized frames look sharper. Experimental results show that the proposed method achieves state-of-the-art performance and performs 2.69x faster than the existing methods that are operable for 4K videos, while maintaining comparable visual and quantitative quality.

scholarly journals Video Frame Interpolation via Deformable Separable Convolution

2020 ◽  
Vol 34 (07) ◽  
pp. 10607-10614 ◽  
Author(s):  
Xianhang Cheng ◽  
Zhenzhong Chen
Keyword(s):  
State Of The Art ◽  
Video Frame ◽  
Kernel Size ◽  
Frame Interpolation ◽  
Interpolation Methods ◽  
Video Frames ◽  
Convolution Process ◽  
Strong Performance ◽  
Existing Frames ◽  
Better Than

Learning to synthesize non-existing frames from the original consecutive video frames is a challenging task. Recent kernel-based interpolation methods predict pixels with a single convolution process to replace the dependency of optical flow. However, when scene motion is larger than the pre-defined kernel size, these methods yield poor results even though they take thousands of neighboring pixels into account. To solve this problem in this paper, we propose to use deformable separable convolution (DSepConv) to adaptively estimate kernels, offsets and masks to allow the network to obtain information with much fewer but more relevant pixels. In addition, we show that the kernel-based methods and conventional flow-based methods are specific instances of the proposed DSepConv. Experimental results demonstrate that our method significantly outperforms the other kernel-based interpolation methods and shows strong performance on par or even better than the state-of-the-art algorithms both qualitatively and quantitatively.

scholarly journals Validation of a new SAFRAN-based gridded precipitation product for Spain and comparisons to Spain02 and ERA-Interim

2017 ◽  
Vol 21 (4) ◽  
pp. 2187-2201 ◽  
Author(s):  
Pere Quintana-Seguí ◽  
Marco Turco ◽  
Sixto Herrera ◽  
Gonzalo Miguez-Macho
Keyword(s):  
High Resolution ◽  
Land Surface ◽  
Climate Models ◽  
Hydrological Cycle ◽  
Interpolation Method ◽  
Rain Gauge ◽  
Surface Model ◽  
Low Resolution ◽  
Precipitation Events

Abstract. Offline land surface model (LSM) simulations are useful for studying the continental hydrological cycle. Because of the nonlinearities in the models, the results are very sensitive to the quality of the meteorological forcing; thus, high-quality gridded datasets of screen-level meteorological variables are needed. Precipitation datasets are particularly difficult to produce due to the inherent spatial and temporal heterogeneity of that variable. They do, however, have a large impact on the simulations, and it is thus necessary to carefully evaluate their quality in great detail. This paper reports the quality of two high-resolution precipitation datasets for Spain at the daily time scale: the new SAFRAN-based dataset and Spain02. SAFRAN is a meteorological analysis system that was designed to force LSMs and has recently been extended to the entirety of Spain for a long period of time (1979/1980–2013/2014). Spain02 is a daily precipitation dataset for Spain and was created mainly to validate regional climate models. In addition, ERA-Interim is included in the comparison to show the differences between local high-resolution and global low-resolution products. The study compares the different precipitation analyses with rain gauge data and assesses their temporal and spatial similarities to the observations. The validation of SAFRAN with independent data shows that this is a robust product. SAFRAN and Spain02 have very similar scores, although the latter slightly surpasses the former. The scores are robust with altitude and throughout the year, save perhaps in summer when a diminished skill is observed. As expected, SAFRAN and Spain02 perform better than ERA-Interim, which has difficulty capturing the effects of the relief on precipitation due to its low resolution. However, ERA-Interim reproduces spells remarkably well in contrast to the low skill shown by the high-resolution products. The high-resolution gridded products overestimate the number of precipitation days, which is a problem that affects SAFRAN more than Spain02 and is likely caused by the interpolation method. Both SAFRAN and Spain02 underestimate high precipitation events, but SAFRAN does so more than Spain02. The overestimation of low precipitation events and the underestimation of intense episodes will probably have hydrological consequences once the data are used to force a land surface or hydrological model.

scholarly journals Collaborative Development of High Resolution Pluvial Flood Maps for Flanders

10.29007/nxqj ◽  
2018 ◽  
Author(s):  
Kris Cauwenberghs ◽  
Tom Feyaerts ◽  
Neil Hunter ◽  
Joost Dewelde ◽  
Thomas Vansteenkiste ◽  
...  
Keyword(s):  
High Resolution ◽  
State Of The Art ◽  
Hydraulic Structures ◽  
Detailed Data ◽  
Low Countries ◽  
Web Technologies ◽  
The Past ◽  
History Of ◽  
Flood Maps

As part of the low countries and with one of the highest population densities worldwide, the Flemish region has experienced a long history of flooding causing tens of millions euro damage each year. In response to this, water managers invested over the past decade in flood modelling and mapping with a fluvial origin. In recent years, pluvial flooding has also occurred numerous times in Flanders, but a region-wide map describing these processes more in detail in terms of extent, depth and probability was lacking. Following a pilot-study in 2016, the VMM undertook in 2017 the VLAGG1- project to develop a region-wide, high-resolution pluvial flood map for Flanders. Via a combination of state-of-the art methodologies and web technologies, a draft flood map was presented to a broad reviewing community across Flanders, who were then able to improve it further by adding local knowledge on known flooding and more detailed data on key hydraulic structures. In a three month period, over 7000 additions were made by 370 delegates from 165 organizations that have been incorporated into, and significantly improved the quality of the final flood maps which are due to be published in 2019.

scholarly journals Validation of a new SAFRAN-based gridded precipitation product for Spain and comparisons to Spain02 and ERA-Interim

2016 ◽  
Author(s):  
Pere Quintana-Seguí ◽  
Marco Turco ◽  
Sixto Herrera ◽  
Gonzalo Miguez-Macho
Keyword(s):  
High Resolution ◽  
Land Surface ◽  
Climate Models ◽  
Hydrological Cycle ◽  
Interpolation Method ◽  
Surface Model ◽  
Low Resolution ◽  
Skill Scores ◽  
Precipitation Events

Abstract. Offline Land-Surface Model (LSM) simulations are useful for studying the continental hydrological cycle. Because of the nonlinearities in the models, the results are very sensitive to the quality of the meteorological forcing; thus, high-quality gridded datasets of screen-level meteorological variables are needed. Precipitation datasets are particularly difficult to produce due to the inherent spatial and temporal heterogeneity of that variable. They do, however, have a large impact on the simulations, and it is thus necessary to carefully evaluate their quality in great detail. This paper reports the quality of two high-resolution precipitation datasets for Spain at the daily time scale: the new SAFRAN-based dataset and Spain02. SAFRAN is a meteorological analysis system that was designed to force LSMs and has recently been extended to the entirety of Spain for a long period of time (1979/80–2013/14). Spain02 is a daily precipitation dataset for Spain and was created mainly to validate Regional Climate Models. In addition, ERA-Interim is included in the comparison to show the differences between local high-resolution and global low-resolution products. The study compares the different precipitation analyses with rain gauge data and assesses their temporal and spatial similarities to the observations. The results show that SAFRAN and Spain02 have very similar skill scores, although the later has better scores in general. As expected, SAFRAN and Spain02 perform better than ERA-Interim, which has difficulty capturing the effects of the relief on precipitation due to its low resolution. However, ERA-Interim reproduces spells remarkably well, in contrast to the low skill shown by the high-resolution products. The high-resolution gridded products overestimate the number of precipitation days, which is a problem that affects SAFRAN more than Spain02 and is likely caused by the interpolation method. Both SAFRAN and Spain02 underestimate high precipitation events, but SAFRAN does so more than Spain02. The overestimation of low precipitation events and the underestimation of intense episodes will probably have hydrological consequences once the data are used to force a land surface or hydrological model.

scholarly journals SafeNet: Scale-normalization and Anchor-based Feature Extraction Network for Person Re-identification

2018 ◽  
Author(s):  
Kun Yuan ◽  
Qian Zhang ◽  
Chang Huang ◽  
Shiming Xiang ◽  
Chunhong Pan
Keyword(s):  
Large Scale ◽  
State Of The Art ◽  
Spatial Distributions ◽  
Body Parts ◽  
Retrieval Task ◽  
Multi Scale ◽  
Aspect Ratios ◽  
Scale Normalization ◽  
Full Body

Person Re-identification (ReID) is a challenging retrieval task that requires matching a person's image across non-overlapping camera views. The quality of fulfilling this task is largely determined on the robustness of the features that are used to describe the person. In this paper, we show the advantage of jointly utilizing multi-scale abstract information to learn powerful features over full body and parts. A scale normalization module is proposed to balance different scales through residual-based integration. To exploit the information hidden in non-rigid body parts, we propose an anchor-based method to capture the local contents by stacking convolutions of kernels with various aspect ratios, which focus on different spatial distributions. Finally, a well-defined framework is constructed for simultaneously learning the representations of both full body and parts. Extensive experiments conducted on current challenging large-scale person ReID datasets, including Market1501, CUHK03 and DukeMTMC, demonstrate that our proposed method achieves the state-of-the-art results.

scholarly journals FISR: Deep Joint Frame Interpolation and Super-Resolution with a Multi-Scale Temporal Loss

2020 ◽  
Vol 34 (07) ◽  
pp. 11278-11286 ◽  
Author(s):  
Soo Ye Kim ◽  
Jihyong Oh ◽  
Munchurl Kim
Keyword(s):  
Video Sequence ◽  
Super Resolution ◽  
Motion Artifacts ◽  
Video Frame ◽  
High Definition ◽  
Frame Interpolation ◽  
Display Devices ◽  
Multi Scale ◽  
Training Scheme ◽  
Spatio Temporal

Super-resolution (SR) has been widely used to convert low-resolution legacy videos to high-resolution (HR) ones, to suit the increasing resolution of displays (e.g. UHD TVs). However, it becomes easier for humans to notice motion artifacts (e.g. motion judder) in HR videos being rendered on larger-sized display devices. Thus, broadcasting standards support higher frame rates for UHD (Ultra High Definition) videos (4K@60 fps, 8K@120 fps), meaning that applying SR only is insufficient to produce genuine high quality videos. Hence, to up-convert legacy videos for realistic applications, not only SR but also video frame interpolation (VFI) is necessitated. In this paper, we first propose a joint VFI-SR framework for up-scaling the spatio-temporal resolution of videos from 2K 30 fps to 4K 60 fps. For this, we propose a novel training scheme with a multi-scale temporal loss that imposes temporal regularization on the input video sequence, which can be applied to any general video-related task. The proposed structure is analyzed in depth with extensive experiments.

Sign in / Sign up

Export Citation Format

Share Document