Spatio-Temporal Learning for Video Deblurring based on Two-Stream Generative Adversarial Network

2021 ◽  
Author(s):  
Liyao Song ◽  
Quan Wang ◽  
Haiwei Li ◽  
Jiancun Fan ◽  
Bingliang Hu
Keyword(s):  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Spatio Temporal ◽  
Temporal Learning

scholarly journals TWIST-GAN: Towards Wavelet Transform and Transferred GAN for Spatio-Temporal Single Image Super Resolution

2021 ◽  
Vol 12 (6) ◽  
pp. 1-20
Author(s):  
Fayaz Ali Dharejo ◽  
Farah Deeba ◽  
Yuanchun Zhou ◽  
Bhagwan Das ◽  
Munsif Ali Jatoi ◽  
...  
Keyword(s):  
Remote Sensing ◽  
Super Resolution ◽  
Generative Adversarial Networks ◽  
Single Image ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Adversarial Networks ◽  
Image Super Resolution ◽  
Spatio Temporal ◽  
Single Image Super Resolution

Single Image Super-resolution (SISR) produces high-resolution images with fine spatial resolutions from a remotely sensed image with low spatial resolution. Recently, deep learning and generative adversarial networks (GANs) have made breakthroughs for the challenging task of single image super-resolution (SISR) . However, the generated image still suffers from undesirable artifacts such as the absence of texture-feature representation and high-frequency information. We propose a frequency domain-based spatio-temporal remote sensing single image super-resolution technique to reconstruct the HR image combined with generative adversarial networks (GANs) on various frequency bands (TWIST-GAN). We have introduced a new method incorporating Wavelet Transform (WT) characteristics and transferred generative adversarial network. The LR image has been split into various frequency bands by using the WT, whereas the transfer generative adversarial network predicts high-frequency components via a proposed architecture. Finally, the inverse transfer of wavelets produces a reconstructed image with super-resolution. The model is first trained on an external DIV2 K dataset and validated with the UC Merced Landsat remote sensing dataset and Set14 with each image size of 256 × 256. Following that, transferred GANs are used to process spatio-temporal remote sensing images in order to minimize computation cost differences and improve texture information. The findings are compared qualitatively and qualitatively with the current state-of-art approaches. In addition, we saved about 43% of the GPU memory during training and accelerated the execution of our simplified version by eliminating batch normalization layers.

scholarly journals Spatio-temporal silhouette sequence reconstruction for gait recognition against occlusion

2019 ◽  
Vol 11 (1) ◽  
Author(s):  
Md. Zasim Uddin ◽  
Daigo Muramatsu ◽  
Noriko Takemura ◽  
Md. Atiqur Rahman Ahad ◽  
Yasushi Yagi
Keyword(s):  
Gait Cycle ◽  
Gait Recognition ◽  
Real Life ◽  
Image Sequence ◽  
Body Parts ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Hinge Loss ◽  
Sequence Reconstruction ◽  
Spatio Temporal

AbstractGait-based features provide the potential for a subject to be recognized even from a low-resolution image sequence, and they can be captured at a distance without the subject’s cooperation. Person recognition using gait-based features (gait recognition) is a promising real-life application. However, several body parts of the subjects are often occluded because of beams, pillars, cars and trees, or another walking person. Therefore, gait-based features are not applicable to approaches that require an unoccluded gait image sequence. Occlusion handling is a challenging but important issue for gait recognition. In this paper, we propose silhouette sequence reconstruction from an occluded sequence (sVideo) based on a conditional deep generative adversarial network (GAN). From the reconstructed sequence, we estimate the gait cycle and extract the gait features from a one gait cycle image sequence. To regularize the training of the proposed generative network, we use adversarial loss based on triplet hinge loss incorporating Wasserstein GAN (WGAN-hinge). To the best of our knowledge, WGAN-hinge is the first adversarial loss that supervises the generator network during training by incorporating pairwise similarity ranking information. The proposed approach was evaluated on multiple challenging occlusion patterns. The experimental results demonstrate that the proposed approach outperforms the existing state-of-the-art benchmarks.

scholarly journals Spatio-temporal generative adversarial network for gait anonymization

2019 ◽  
Vol 46 ◽  
pp. 307-319 ◽  
Author(s):  
Ngoc-Dung T. Tieu ◽  
Huy H. Nguyen ◽  
Hoang-Quoc Nguyen-Son ◽  
Junichi Yamagishi ◽  
Isao Echizen
Keyword(s):  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Spatio Temporal

scholarly journals Semi-Supervised Multi-Temporal Deep Representation Fusion Network for Landslide Mapping from Aerial Orthophotos

2021 ◽  
Vol 13 (4) ◽  
pp. 548
Author(s):  
Xiaokang Zhang ◽  
Man-On Pun ◽  
Ming Liu
Keyword(s):  
Feature Space ◽  
Representation Learning ◽  
Feature Representation ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Multi Temporal ◽  
Landslide Mapping ◽  
Spatio Temporal ◽  
Reliability And Robustness ◽  
High Level

Using remote sensing techniques to monitor landslides and their resultant land cover changes is fundamentally important for risk assessment and hazard prevention. Despite enormous efforts in developing intelligent landslide mapping (LM) approaches, LM remains challenging owing to high spectral heterogeneity of very-high-resolution (VHR) images and the daunting labeling efforts. To this end, a deep learning model based on semi-supervised multi-temporal deep representation fusion network, namely SMDRF-Net, is proposed for reliable and efficient LM. In comparison with previous methods, the SMDRF-Net possesses three distinct properties. (1) Unsupervised deep representation learning at the pixel- and object-level is performed by transfer learning using the Wasserstein generative adversarial network with gradient penalty to learn discriminative deep features and retain precise outlines of landslide objects in the high-level feature space. (2) Attention-based adaptive fusion of multi-temporal and multi-level deep representations is developed to exploit the spatio-temporal dependencies of deep representations and enhance the feature representation capability of the network. (3) The network is optimized using limited samples with pseudo-labels that are automatically generated based on a comprehensive uncertainty index. Experimental results from the analysis of VHR aerial orthophotos demonstrate the reliability and robustness of the proposed approach for LM in comparison with state-of-the-art methods.

Strategies for supplementing recurrent neural network training for spatio-temporal prediction

2019 ◽  
Vol 67 (7) ◽  
pp. 545-556 ◽  
Author(s):  
Mark Schutera ◽  
Stefan Elser ◽  
Jochen Abhau ◽  
Ralf Mikut ◽  
Markus Reischl
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Data Augmentation ◽  
Autonomous Driving ◽  
Temporal Data ◽  
Generative Adversarial Network ◽  
Temporal Prediction ◽  
Adversarial Network ◽  
Augmentation Strategy ◽  
Spatio Temporal

Abstract In autonomous driving, prediction tasks address complex spatio-temporal data. This article describes the examination of Recurrent Neural Networks (RNNs) for object trajectory prediction in the image space. The proposed methods enhance the performance and spatio-temporal prediction capabilities of Recurrent Neural Networks. Two different data augmentation strategies and a hyperparameter search are implemented for this purpose. A conventional data augmentation strategy and a Generative Adversarial Network (GAN) based strategy are analyzed with respect to their ability to close the generalization gap of Recurrent Neural Networks. The results are then discussed using single-object tracklets provided by the KITTI Tracking Dataset. This work demonstrates the benefits of augmenting spatio-temporal data with GANs.

scholarly journals Wave-Tracking in the Surf Zone Using Coastal Video Imagery with Deep Neural Networks

Atmosphere ◽  
2020 ◽  
Vol 11 (3) ◽  
pp. 304 ◽  
Author(s):  
Jinah Kim ◽  
Jaeil Kim ◽  
Taekyung Kim ◽  
Dong Huh ◽  
Sofia Caires
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Surf Zone ◽  
Background Information ◽  
Wave Crest ◽  
Video Enhancement ◽  
Generative Adversarial Network ◽  
Video Imagery ◽  
Adversarial Network ◽  
Spatio Temporal

In this paper, we propose a series of procedures for coastal wave-tracking using coastal video imagery with deep neural networks. It consists of three stages: video enhancement, hydrodynamic scene separation and wave-tracking. First, a generative adversarial network, trained using paired raindrop and clean videos, is applied to remove image distortions by raindrops and to restore background information of coastal waves. Next, a hydrodynamic scene of propagated wave information is separated from surrounding environmental information in the enhanced coastal video imagery using a deep autoencoder network. Finally, propagating waves are tracked by registering consecutive images in the quality-enhanced and scene-separated coastal video imagery using a spatial transformer network. The instantaneous wave speed of each individual wave crest and breaker in the video domain is successfully estimated through learning the behavior of transformed and propagated waves in the surf zone using deep neural networks. Since it enables the acquisition of spatio-temporal information of the surf zone though the characterization of wave breakers inclusively wave run-up, we expect that the proposed framework with the deep neural networks leads to improve understanding of nearshore wave dynamics.

Sign in / Sign up

Export Citation Format

Share Document