Image Style Transfer based on Generative Adversarial Network

Robotic drawing has become increasingly popular as an entertainment and interactive tool. In this paper we present RoboCoDraw, a real-time collaborative robot-based drawing system that draws stylized human face sketches interactively in front of human users, by using the Generative Adversarial Network (GAN)-based style transfer and a Random-Key Genetic Algorithm (RKGA)-based path optimization. The proposed RoboCoDraw system takes a real human face image as input, converts it to a stylized avatar, then draws it with a robotic arm. A core component in this system is the AvatarGAN proposed by us, which generates a cartoon avatar face image from a real human face. AvatarGAN is trained with unpaired face and avatar images only and can generate avatar images of much better likeness with human face images in comparison with the vanilla CycleGAN. After the avatar image is generated, it is fed to a line extraction algorithm and converted to sketches. An RKGA-based path optimization algorithm is applied to find a time-efficient robotic drawing path to be executed by the robotic arm. We demonstrate the capability of RoboCoDraw on various face images using a lightweight, safe collaborative robot UR5.

Download Full-text

Synthesizing Depth Hand Images with GANs and Style Transfer for Hand Pose Estimation

Sensors ◽

10.3390/s19132919 ◽

2019 ◽

Vol 19 (13) ◽

pp. 2919 ◽

Cited By ~ 2

Author(s):

Wangyong He ◽

Zhongzhao Xie ◽

Yongbo Li ◽

Xinmei Wang ◽

Wendi Cai

Keyword(s):

Pose Estimation ◽

Ground Truth ◽

Training Image ◽

Training Data ◽

Generative Adversarial Network ◽

Style Transfer ◽

Visual Appearance ◽

Hand Pose Estimation ◽

Adversarial Network ◽

Hand Pose

Hand pose estimation is a critical technology of computer vision and human-computer interaction. Deep-learning methods require a considerable amount of tagged data. Accordingly, numerous labeled training data are required. This paper aims to generate depth hand images. Given a ground-truth 3D hand pose, the developed method can generate depth hand images. To be specific, a ground truth can be 3D hand poses with the hand structure contained, while the synthesized image has an identical size to that of the training image and a similar visual appearance to the training set. The developed method, inspired by the progress in the generative adversarial network (GAN) and image-style transfer, helps model the latent statistical relationship between the ground-truth hand pose and the corresponding depth hand image. The images synthesized using the developed method are demonstrated to be feasible for enhancing performance. On public hand pose datasets (NYU, MSRA, ICVL), comprehensive experiments prove that the developed method outperforms the existing works.

Download Full-text

A survey of style transfer based on generative adversarial network

10.1117/12.2607066 ◽

2021 ◽

Author(s):

Mingyu Qin ◽

Youchen Fan ◽

Baolin Liu ◽

Xu Ma

Keyword(s):

Generative Adversarial Network ◽

Style Transfer ◽

Adversarial Network

Download Full-text

Batch equalization with a generative adversarial network

Bioinformatics ◽

10.1093/bioinformatics/btaa819 ◽

2020 ◽

Vol 36 (Supplement_2) ◽

pp. i875-i883

Author(s):

Wesley Wei Qian ◽

Cassandra Xia ◽

Subhashini Venugopalan ◽

Arunachalam Narayanaswamy ◽

Michelle Dimon ◽

...

Keyword(s):

Neural Network ◽

Batch Effect ◽

Biological Information ◽

Supplementary Information ◽

Generative Adversarial Network ◽

Style Transfer ◽

Biological Features ◽

Adversarial Network ◽

Low Dimensional ◽

Equalization Method

Abstract Motivation Advances in automation and imaging have made it possible to capture a large image dataset that spans multiple experimental batches of data. However, accurate biological comparison across the batches is challenged by batch-to-batch variation (i.e. batch effect) due to uncontrollable experimental noise (e.g. varying stain intensity or cell density). Previous approaches to minimize the batch effect have commonly focused on normalizing the low-dimensional image measurements such as an embedding generated by a neural network. However, normalization of the embedding could suffer from over-correction and alter true biological features (e.g. cell size) due to our limited ability to interpret the effect of the normalization on the embedding space. Although techniques like flat-field correction can be applied to normalize the image values directly, they are limited transformations that handle only simple artifacts due to batch effect. Results We present a neural network-based batch equalization method that can transfer images from one batch to another while preserving the biological phenotype. The equalization method is trained as a generative adversarial network (GAN), using the StarGAN architecture that has shown considerable ability in style transfer. After incorporating new objectives that disentangle batch effect from biological features, we show that the equalized images have less batch information and preserve the biological information. We also demonstrate that the same model training parameters can generalize to two dramatically different types of cells, indicating this approach could be broadly applicable. Availability and implementation https://github.com/tensorflow/gan/tree/master/tensorflow_gan/examples/stargan Supplementary information Supplementary data are available at Bioinformatics online.

Download Full-text

CBNWI-50: A Deep Learning Bird Dataset for Image Translation and Resolution Improvement using Generative Adversarial Network

International Journal of Innovative Technology and Exploring Engineering - Special Issue ◽

10.35940/ijitee.i1015.0789s19 ◽

2019 ◽

Vol 8 (9S) ◽

pp. 91-102

Keyword(s):

Deep Learning ◽

Super Resolution ◽

Generative Adversarial Networks ◽

Western India ◽

Generative Adversarial Network ◽

Style Transfer ◽

Adversarial Network ◽

Image Translation ◽

Common Birds ◽

Single Image Super Resolution

Generative Adversarial Networks have gained prominence in a short span of time as they can synthesize images from latent noise by minimizing the adversarial cost function. New variants of GANs have been developed to perform specific tasks using state-of-the-art GAN models, like image translation, single image super resolution, segmentation, classification, style transfer etc. However, a combination of two GANs to perform two different applications in one model has been sparsely explored. Hence, this paper concatenates two GANs and aims to perform Image Translation using Cycle GAN model on bird images and improve their resolution using SRGAN. During the extensive survey, it is observed that most of the deep learning databases on Aves were built using the new world species (i.e. species found in North America). Hence, to bridge this gap, a new Ave database, 'Common Birds of North - Western India' (CBNWI-50), is also proposed in this work.

Download Full-text

Scribble-to-Painting Transformation with Multi-Task Generative Adversarial Networks

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2019/820 ◽

2019 ◽

Author(s):

Jinning Li ◽

Yexiang Xue

Keyword(s):

Semantic Segmentation ◽

Neural Nets ◽

Experimental Result ◽

Generative Adversarial Networks ◽

Neural Net ◽

Generative Adversarial Network ◽

Style Transfer ◽

Adversarial Network ◽

Classical Image ◽

Artistic Images

We propose the Dual Scribble-to-Painting Network (DSP-Net), which is able to produce artistic paintings based on user-generated scribbles. In scribble-to-painting transformation, a neural net has to infer additional details of the image, given relatively sparse information contained in the outlines of the scribble. Therefore, it is more challenging than classical image style transfer, in which the information content is reduced from photos to paintings. Inspired by the human cognitive process, we propose a multi-task generative adversarial network, which consists of two jointly trained neural nets -- one for generating artistic images and the other one for semantic segmentation. We demonstrate that joint training on these two tasks brings in additional benefit. Experimental result shows that DSP-Net outperforms state-of-the-art models both visually and quantitatively. In addition, we publish a large dataset for scribble-to-painting transformation.

Download Full-text