Deep Generative Adversarial Networks for Image-to-Image Translation: A Review

Many image processing, computer graphics, and computer vision problems can be treated as image-to-image translation tasks. Such translation entails learning to map one visual representation of a given input to another representation. Image-to-image translation with generative adversarial networks (GANs) has been intensively studied and applied to various tasks, such as multimodal image-to-image translation, super-resolution translation, object transfiguration-related translation, etc. However, image-to-image translation techniques suffer from some problems, such as mode collapse, instability, and a lack of diversity. This article provides a comprehensive overview of image-to-image translation based on GAN algorithms and its variants. It also discusses and analyzes current state-of-the-art image-to-image translation techniques that are based on multimodal and multidomain representations. Finally, open issues and future research directions utilizing reinforcement learning and three-dimensional (3D) modal translation are summarized and discussed.

Download Full-text

Generative Adversarial Networks in Computer Vision

ACM Computing Surveys ◽

10.1145/3439723 ◽

2021 ◽

Vol 54 (2) ◽

pp. 1-38

Author(s):

Zhengwei Wang ◽

Qi She ◽

Tomás E. Ward

Keyword(s):

Computer Vision ◽

Generative Adversarial Networks ◽

Future Research ◽

Image Generation ◽

Adversarial Networks ◽

Image Translation ◽

The Status ◽

Loss Variant ◽

Future Research Directions ◽

Application Requirements

Generative adversarial networks (GANs) have been extensively studied in the past few years. Arguably their most significant impact has been in the area of computer vision where great advances have been made in challenges such as plausible image generation, image-to-image translation, facial attribute manipulation, and similar domains. Despite the significant successes achieved to date, applying GANs to real-world problems still poses significant challenges, three of which we focus on here. These are as follows: (1) the generation of high quality images, (2) diversity of image generation, and (3) stabilizing training. Focusing on the degree to which popular GAN technologies have made progress against these challenges, we provide a detailed review of the state-of-the-art in GAN-related research in the published scientific literature. We further structure this review through a convenient taxonomy we have adopted based on variations in GAN architectures and loss functions. While several reviews for GANs have been presented to date, none have considered the status of this field based on their progress toward addressing practical challenges relevant to computer vision. Accordingly, we review and critically discuss the most popular architecture-variant, and loss-variant GANs, for tackling these challenges. Our objective is to provide an overview as well as a critical analysis of the status of GAN research in terms of relevant progress toward critical computer vision application requirements. As we do this we also discuss the most compelling applications in computer vision in which GANs have demonstrated considerable success along with some suggestions for future research directions. Codes related to the GAN-variants studied in this work is summarized on https://github.com/sheqi/GAN_Review.

Download Full-text

Daydriex: Translating Nighttime Scenes towards Daytime Driving Experience at Night

Applied Sciences ◽

10.3390/app11052013 ◽

2021 ◽

Vol 11 (5) ◽

pp. 2013

Author(s):

Euihyeok Lee ◽

Seungwoo Kang

Keyword(s):

Input Image ◽

Generative Adversarial Networks ◽

User Perception ◽

Missing Information ◽

High Quality ◽

Driving Experience ◽

Processing Pipeline ◽

Adversarial Networks ◽

Image Translation ◽

Translation Techniques

What if the window of our cars is a magic window, which transforms dark views outside of the window at night into bright ones as we can see in the daytime? To realize such a window, one of important requirements is that the stream of transformed images displayed on the window should be of high quality so that users perceive it as real scenes in the day. Although image-to-image translation techniques based on Generative Adversarial Networks (GANs) have been widely studied, night-to-day image translation is still a challenging task. In this paper, we propose Daydriex, a processing pipeline to generate enhanced daytime translation focusing on road views. Our key idea is to supplement the missing information in dark areas of input image frames by using existing daytime images corresponding to the input images from street view services. We present a detailed processing flow and address several issues to realize our idea. Our evaluation shows that the results by Daydriex achieves lower Fréchet Inception Distance (FID) scores and higher user perception scores compared to those by CycleGAN only.

Download Full-text

Generative Adversarial Networks

ACM Computing Surveys ◽

10.1145/3459992 ◽

2021 ◽

Vol 54 (6) ◽

pp. 1-38

Author(s):

Zhipeng Cai ◽

Zuobin Xiong ◽

Honghui Xu ◽

Peng Wang ◽

Wei Li ◽

...

Keyword(s):

Language Processing ◽

Generative Adversarial Networks ◽

Future Research ◽

Data Generation ◽

Optimization Strategy ◽

Privacy And Security ◽

Adversarial Networks ◽

Initial Stage ◽

Game Theoretic ◽

Future Research Directions

Generative Adversarial Networks (GANs) have promoted a variety of applications in computer vision and natural language processing, among others, due to its generative model’s compelling ability to generate realistic examples plausibly drawn from an existing distribution of samples. GAN not only provides impressive performance on data generation-based tasks but also stimulates fertilization for privacy and security oriented research because of its game theoretic optimization strategy. Unfortunately, there are no comprehensive surveys on GAN in privacy and security, which motivates this survey to summarize systematically. The existing works are classified into proper categories based on privacy and security functions, and this survey conducts a comprehensive analysis of their advantages and drawbacks. Considering that GAN in privacy and security is still at a very initial stage and has imposed unique challenges that are yet to be well addressed, this article also sheds light on some potential privacy and security applications with GAN and elaborates on some future research directions.

Download Full-text

Text to Image Translation using Cycle GAN

International Journal of Engineering and Advanced Technology - Regular Issue ◽

10.35940/ijeat.d8703.049420 ◽

2020 ◽

Vol 9 (4) ◽

pp. 1294-1297

Keyword(s):

State Of The Art ◽

Ground Truth ◽

Image Synthesis ◽

Difficult Problem ◽

Generative Adversarial Networks ◽

Adversarial Networks ◽

Current State ◽

Image Translation ◽

Translation Methods ◽

Text Images

In the recent past, text-to-image translation was an active field of research. The ability of a network to know a sentence's context and to create a specific picture that represents the sentence demonstrates the model's ability to think more like humans. Common text--translation methods employ Generative Adversarial Networks to generate high-text-images, but the images produced do not always represent the meaning of the phrase provided to the model as input. Using a captioning network to caption generated images, we tackle this problem and exploit the gap between ground truth captions and generated captions to further enhance the network. We present detailed similarities between our system and the methods already in place. Text-to-Image synthesis is a difficult problem with plenty of space for progress despite the current state-of - the-art results. Synthesized images from current methods give the described image a rough sketch but do not capture the true essence of what the text describes. The re-penny achievement of Generative Adversarial Networks (GANs) demonstrates that they are a decent contender for the decision of design to move toward this issue.

Download Full-text

Generative adversarial networks- and ResNets-based framework for image translation with super-resolution

Journal of Electronic Imaging ◽

10.1117/1.jei.27.6.063018 ◽

2018 ◽

Vol 27 (06) ◽

pp. 1 ◽

Cited By ~ 2

Author(s):

Hui Hu

Keyword(s):

Super Resolution ◽

Generative Adversarial Networks ◽

Adversarial Networks ◽

Image Translation

Download Full-text

Three-Dimensional Thematic Map Imaging of the Yacht Port on the Example of the Polish National Sailing Centre Marina in Gdańsk

Applied Sciences ◽

10.3390/app11157016 ◽

2021 ◽

Vol 11 (15) ◽

pp. 7016

Author(s):

Pawel S. Dabrowski ◽

Cezary Specht ◽

Mariusz Specht ◽

Artur Makar

Keyword(s):

Spatial Data ◽

Convex Surface ◽

Dimensional Space ◽

Three Dimensional ◽

Future Research ◽

Two Dimensional ◽

Thematic Maps ◽

Research Directions ◽

Future Research Directions ◽

The Many

The theory of cartographic projections is a tool which can present the convex surface of the Earth on the plane. Of the many types of maps, thematic maps perform an important function due to the wide possibilities of adapting their content to current needs. The limitation of classic maps is their two-dimensional nature. In the era of rapidly growing methods of mass acquisition of spatial data, the use of flat images is often not enough to reveal the level of complexity of certain objects. In this case, it is necessary to use visualization in three-dimensional space. The motivation to conduct the study was the use of cartographic projections methods, spatial transformations, and the possibilities offered by thematic maps to create thematic three-dimensional map imaging (T3DMI). The authors presented a practical verification of the adopted methodology to create a T3DMI visualization of the marina of the National Sailing Centre of the Gdańsk University of Physical Education and Sport (Poland). The profiled characteristics of the object were used to emphasize the key elements of its function. The results confirmed the increase in the interpretative capabilities of the T3DMI method, relative to classic two-dimensional maps. Additionally, the study suggested future research directions of the presented solution.

Download Full-text

Co-DGAN: cooperating discriminator generative adversarial networks for unpaired image-to-image translation

Soft Computing ◽

10.1007/s00500-020-05475-6 ◽

2021 ◽

Author(s):

Huajun Liu ◽

Ziyan Wang ◽

Haigang Sui ◽

Qing Zhu ◽

Shubo Liu ◽

...

Keyword(s):

Generative Adversarial Networks ◽

Adversarial Networks ◽

Image Translation

Download Full-text

DPSRGAN: Dilation Patch Super-Resolution Generative Adversarial Networks

2021 6th International Conference for Convergence in Technology (I2CT) ◽

10.1109/i2ct51068.2021.9417903 ◽

2021 ◽

Author(s):

Kapil Mirchandani ◽

Kushal Chordiya

Keyword(s):

Super Resolution ◽

Generative Adversarial Networks ◽

Adversarial Networks

Download Full-text

Children’s outdoor active mobility behaviour and neighbourhood safety: a systematic review in measurement methods and future research directions

International Journal of Health Geographics ◽

10.1186/s12942-020-00254-w ◽

2021 ◽

Vol 20 (1) ◽

Author(s):

Roula Zougheibe ◽

Jianhong (Cecilia) Xia ◽

Ashraf Dewan ◽

Ori Gudes ◽

Richard Norman

Keyword(s):

Three Dimensional ◽

Measurement Methods ◽

Future Research ◽

Inclusion Criteria ◽

Eligibility Criteria ◽

Research Directions ◽

Spatial Locality ◽

Mobility Behaviour ◽

Health Studies ◽

Future Research Directions

Abstract Background Numerous studies have examined the association between safety and primary school-aged children’s forms of active mobility. However, variations in studies’ measurement methods and the elements addressed have contributed to inconsistencies in research outcomes, which may be forming a barrier to advancing researchers’ knowledge about this field. To assess where current research stands, we have synthesised the methodological measures in studies that examined the effects of neighbourhood safety exposure (perceived and measured) on children’s outdoor active mobility behaviour and used this analysis to propose future research directions. Method A systematic search of the literature in six electronic databases was conducted using pre-defined eligibility criteria and was concluded in July 2020. Two reviewers screened the literature abstracts to determine the studies’ inclusion, and two reviewers independently conducted a methodological quality assessment to rate the included studies. Results Twenty-five peer-reviewed studies met the inclusion criteria. Active mobility behaviour and health characteristics were measured objectively in 12 out of the 25 studies and were reported in another 13 studies. Twenty-one studies overlooked spatiotemporal dimensions in their analyses and outputs. Delineations of children’s neighbourhoods varied within 10 studies’ objective measures, and the 15 studies that opted for subjective measures. Safety perceptions obtained in 22 studies were mostly static and primarily collected via parents, and dissimilarities in actual safety measurement methods were present in 6 studies. The identified schematic constraints in studies’ measurement methods assisted in outlining a three-dimensional relationship between ‘what’ (determinants), ‘where’ (spatial) and ‘when’ (time) within a methodological conceptual framework. Conclusions The absence of standardised measurement methods among relevant studies may have led to the current diversity in findings regarding active mobility, spatial (locality) and temporal (time) characteristics, the neighbourhood, and the representation of safety. Ignorance of the existing gaps and heterogeneity in measures may impact the reliability of evidence and poses a limitation when synthesising findings, which could result in serious biases for policymakers. Given the increasing interest in children’s health studies, we suggested alternatives in the design and method of measures that may guide future evidence-based research for policymakers who aim to improve children’s active mobility and safety.

Download Full-text