Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure

In this paper, a new frame rate up-conversion (FRC) using Backward Multiple Reference Frame Motion (BMRF) compensated interpolation (MCI) algorithm is proposed. The BMRF algorithm has been compared with two most common MCI methods, Temporal Linear (TLIN) and Bidirectional MCI (BID) used in FRC, in the aspect of PSNR, perceptual quality, and computational complexity. The simulation results clearly showed that BMRF presented a trade-off solution between computation and interpolation quality. By using simple weighted frame interpolation between multiple frames, it reduced computation rapidly while keeping similar interpolation quality when compared to the BID method. The simplicity of the BMRF algorithm made it suitable for either software or hardware implementation essential for real-time FRC applications.

Download Full-text

Efficient Video Frame Interpolation Using Generative Adversarial Networks

Applied Sciences ◽

10.3390/app10186245 ◽

2020 ◽

Vol 10 (18) ◽

pp. 6245

Author(s):

Quang Nhat Tran ◽

Shih-Hsuan Yang

Keyword(s):

Video Compression ◽

State Of The Art ◽

Motion Blur ◽

The State ◽

Frame Rate ◽

Computational Time ◽

Generative Adversarial Networks ◽

Video Frame ◽

Frame Interpolation ◽

Frame Rate Up Conversion

Frame interpolation, which generates an intermediate frame given adjacent ones, finds various applications such as frame rate up-conversion, video compression, and video streaming. Instead of using complex network models and additional data involved in the state-of-the-art frame interpolation methods, this paper proposes an approach based on an end-to-end generative adversarial network. A combined loss function is employed, which jointly considers the adversarial loss (difference between data models), reconstruction loss, and motion blur degradation. The objective image quality metric values reach a PSNR of 29.22 dB and SSIM of 0.835 on the UCF101 dataset, similar to those of the state-of-the-art approach. The good visual quality is notably achieved by approximately one-fifth computational time, which entails possible real-time frame rate up-conversion. The interpolated output can be further improved by a GAN based refinement network that better maintains motion and color by image-to-image translation.

Download Full-text

BlockNet: A Deep Neural Network for Block-Based Motion Estimation Using Representative Matching

Symmetry ◽

10.3390/sym12050840 ◽

2020 ◽

Vol 12 (5) ◽

pp. 840

Author(s):

Junggi Lee ◽

Kyeongbo Kong ◽

Gyujin Bae ◽

Woo-Jin Song

Keyword(s):

Motion Estimation ◽

Computational Cost ◽

Frame Rate ◽

Global Motion ◽

Neural Architecture ◽

Block Motion ◽

Block Based ◽

Frame Rate Up Conversion ◽

Coarse To Fine ◽

Simple Average

Owing to the limitations of practical realizations, block-based motion is widely used as an alternative for pixel-based motion in video applications such as global motion estimation and frame rate up-conversion. We hereby present BlockNet, a compact but effective deep neural architecture for block-based motion estimation. First, BlockNet extracts rich features for a pair of input images. Then, it estimates coarse-to-fine block motion using a pyramidal structure. In each level, block-based motion is estimated using the proposed representative matching with a simple average operator. The experimental results show that BlockNet achieved a similar average end-point error with and without representative matching, whereas the proposed matching incurred 18% lower computational cost than full matching.

Download Full-text