scholarly journals Flow-aware synthesis: A generic motion model for video frame interpolation

2021 ◽  
Author(s):  
Jinbo Xing ◽  
Wenbo Hu ◽  
Yuechen Zhang ◽  
Tien-Tsin Wong
Keyword(s):  
Control Variable ◽  
Frame Rate ◽  
Video Frame ◽  
Motion Model ◽  
Linear Quadratic ◽  
Frame Interpolation ◽  
Flow Prediction ◽  
Prediction Module ◽  
Non Linear ◽  
Public Datasets

AbstractA popular and challenging task in video research, frame interpolation aims to increase the frame rate of video. Most existing methods employ a fixed motion model, e.g., linear, quadratic, or cubic, to estimate the intermediate warping field. However, such fixed motion models cannot well represent the complicated non-linear motions in the real world or rendered animations. Instead, we present an adaptive flow prediction module to better approximate the complex motions in video. Furthermore, interpolating just one intermediate frame between consecutive input frames may be insufficient for complicated non-linear motions. To enable multi-frame interpolation, we introduce the time as a control variable when interpolating frames between original ones in our generic adaptive flow prediction module. Qualitative and quantitative experimental results show that our method can produce high-quality results and outperforms the existing state-of-the-art methods on popular public datasets.

scholarly journals Efficient Video Frame Interpolation Using Generative Adversarial Networks

2020 ◽  
Vol 10 (18) ◽  
pp. 6245
Author(s):  
Quang Nhat Tran ◽  
Shih-Hsuan Yang
Keyword(s):  
Video Compression ◽  
State Of The Art ◽  
Motion Blur ◽  
The State ◽  
Frame Rate ◽  
Computational Time ◽  
Generative Adversarial Networks ◽  
Video Frame ◽  
Frame Interpolation ◽  
Frame Rate Up Conversion

Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.

scholarly journals Enhanced Real-Time Intermediate Flow Estimation for Video Frame Interpolation

2021 ◽  
Author(s):  
Minseop Kim ◽  
Haechul Choi
Keyword(s):  
Optical Flow ◽  
Real Time ◽  
Estimation Method ◽  
Frame Rate ◽  
Video Frame ◽  
High Quality ◽  
Frame Interpolation ◽  
Real Time Processing ◽  
High Frame Rate ◽  
Flow Estimation

Recently, the demand for high-quality video content has rapidly been increasing, led by the development of network technology and the growth in video streaming platforms. In particular, displays with a high refresh rate, such as 120 Hz, have become popular. However, the visual quality is only enhanced if the video stream is produced at the same high frame rate. For the high quality, conventional videos with a low frame rate should be converted into a high frame rate in real time. This paper introduces a bidirectional intermediate flow estimation method for real-time video frame interpolation. A bidirectional intermediate optical flow is directly estimated to predict an accurate intermediate frame. For real-time processing, multiple frames are interpolated with a single intermediate optical flow and parts of the network are implemented in 16-bit floating-point precision. Perceptual loss is also applied to improve the cognitive performance of the interpolated frames. The experimental results showed a high prediction accuracy of 35.54 dB on the Vimeo90K triplet benchmark dataset. The interpolation speed of 84 fps was achieved for 480p resolution.

scholarly journals Analysis of Coronary Angiography Video Interpolation Methods to Reduce X-ray Exposure Frequency Based on Deep Learning

2021 ◽  
Author(s):  
Dong-xue Liang
Keyword(s):  
Deep Learning ◽  
Coronary Angiography ◽  
Coronary Arteries ◽  
Heart Surgery ◽  
Frame Rate ◽  
Video Frame ◽  
X Rays ◽  
Frame Interpolation ◽  
X Ray ◽  
Video Frame Rate

Cardiac coronary angiography is a major technique that assists physicians during interventional heart surgery. Under X-ray irradiation, the physician injects a contrast agent through a catheter and determines the coronary arteries’ state in real time. However, to obtain a more accurate state of the coronary arteries, physicians need to increase the frequency and intensity of X-ray exposure, which will inevitably increase the potential for harm to both the patient and the surgeon. In the work reported here, we use advanced deep learning algorithms to find a method of frame interpolation for coronary angiography videos that reduces the frequency of X-ray exposure by reducing the frame rate of the coronary angiography video, thereby reducing X-ray-induced damage to physicians. We established a new coronary angiography image group dataset containing 95,039 groups of images extracted from 31 videos. Each group includesthree consecutive images, which are used to train the video interpolation network model. We apply six popular frameinterpolation methods to this dataset to confirm that the video frame interpolation technology can reduce the video frame rate and reduce exposure of physicians to X-rays.

scholarly journals Video Frame Interpolation via Deformable Separable Convolution

2020 ◽  
Vol 34 (07) ◽  
pp. 10607-10614 ◽  
Author(s):  
Xianhang Cheng ◽  
Zhenzhong Chen
Keyword(s):  
State Of The Art ◽  
Video Frame ◽  
Kernel Size ◽  
Frame Interpolation ◽  
Interpolation Methods ◽  
Video Frames ◽  
Convolution Process ◽  
Strong Performance ◽  
Existing Frames ◽  
Better Than

Learning to synthesize non-existing frames from the original consecutive video frames is a challenging task. Recent kernel-based interpolation methods predict pixels with a single convolution process to replace the dependency of optical flow. However, when scene motion is larger than the pre-defined kernel size, these methods yield poor results even though they take thousands of neighboring pixels into account. To solve this problem in this paper, we propose to use deformable separable convolution (DSepConv) to adaptively estimate kernels, offsets and masks to allow the network to obtain information with much fewer but more relevant pixels. In addition, we show that the kernel-based methods and conventional flow-based methods are specific instances of the proposed DSepConv. Experimental results demonstrate that our method significantly outperforms the other kernel-based interpolation methods and shows strong performance on par or even better than the state-of-the-art algorithms both qualitatively and quantitatively.

scholarly journals A NEW APPROACH TO HIGHWAY LANE DETECTION BY USING HOUGH TRANSFORM TECHNIQUE

2017 ◽  
Author(s):  
Nur Shazwani Aminuddin ◽  
Masrullizam Mat Ibrahim ◽  
Nursabillilah Mohd Ali ◽  
Syafeeza Ahmad Radzi ◽  
Wira Hidayat Mohd Saad ◽  
...  
Keyword(s):  
Hough Transform ◽  
Average Rate ◽  
Lane Detection ◽  
Frame Rate ◽  
Video Frame ◽  
Night Time ◽  
The Road ◽  
Main Technique ◽  
On The Road ◽  
Width Measurements

This paper presents the development of a road lane detection algorithm using image processing techniques. This algorithm is developed based on dynamic videos, which are recorded using on-board cameras installed in vehicles for Malaysian highway conditions. The recorded videos are dynamic scenes of the background and the foreground, in which the detection of the objects, presence on the road area such as vehicles and road signs are more challenging caused by interference from background elements such as buildings, trees, road dividers and other related elements or objects. Thus, this algorithm aims to detect the road lanes for three significant parameter operations; vanishing point detection, road width measurements, and Region of Interest (ROI) of the road area, for detection purposes. The techniques used in the algorithm are image enhancement and edges extraction by Sobel filter, and the main technique for lane detection is a Hough Transform. The performance of the algorithm is tested and validated by using three videos of highway scenes in Malaysia with normal weather conditions, raining and a night-time scene, and an additional scene of a sunny rural road area. The video frame rate is 30fps with dimensions of 720p (1280x720) HD pixels. In the final achievement analysis, the test result shows a true positive rate, a TP lane detection  average rate of 0.925 and the capability to be used in the final application implementation.  

scholarly journals Estimating an Aggregate Economic Genotype to Facilitate Breeding and Selection of Strawberry Genotypes in Southeast Queensland

2014 ◽  
Vol 139 (3) ◽  
pp. 253-260
Author(s):  
Mark E. Herrington ◽  
Craig Hardner ◽  
Malcolm Wegener ◽  
Louella Woolcock ◽  
Mark J. Dieters
Keyword(s):  
Rank Correlation ◽  
Breeding Population ◽  
Breeding Program ◽  
Production Costs ◽  
Breeding Value ◽  
Linear Quadratic ◽  
Gross Margin ◽  
Economic Weight ◽  
Non Linear ◽  
Gross Margins

The Queensland strawberry (Fragaria ×ananassa) breeding program in subtropical Australia aims to improve sustainable profitability for the producer. Selection must account for the relative economic importance of each trait and the genetic architecture underlying these traits in the breeding population. Our study used estimates of the influence of a trait on production costs and profitability to develop a profitability index (PI) and an economic weight (i.e., change in PI for a unit change in level of trait) for each trait. The economic weights were then combined with the breeding values for 12 plant and fruit traits on over 3000 genotypes that were represented in either the current breeding population or as progenitors in the pedigree of these individuals. The resulting linear combination (i.e., sum of economic weight × breeding value for all 12 traits) estimated the overall economic worth of each genotype as H, the aggregate economic genotype. H values were validated by comparisons among commercial cultivars and were also compared with the estimated gross margins. When the H value of ‘Festival’ was set as zero, the H values of genotypes in the pedigree ranged from –0.36 to +0.28. H was highly correlated (R2 = 0.77) with the year of selection (1945–98). The gross margins were highly linearly related (R2 > 0.98) to H values when the genotype was planted on less than 50% of available area, but the relationship was non-linear [quadratic with a maximum (R2 > 0.96)] when the planted area exceeded 50%. Additionally, with H values above zero, the variation in gross margin increased with increasing H values as the percentage of area planted to a genotype increased. High correlations among some traits allowed the omission of any one of three of the 12 traits with little or no effect on ranking (Spearman’s rank correlation 0.98 or greater). Thus, these traits may be dropped from the aggregate economic genotype, leading to either cost reductions in the breeding program or increased selection intensities for the same resources. H was efficient in identifying economically superior genotypes for breeding and deployment, but because of the non-linear relationship with gross margin, calculation of a gross margin for genotypes with high H is also necessary when cultivars are deployed across more than 50% of the available area.

Sign in / Sign up

Export Citation Format

Share Document