scholarly journals Infrared and visible image fusion using two-layer generative adversarial network

2021 ◽  
pp. 1-17
Author(s):  
Lei Chen ◽  
Jun Han ◽  
Feng Tian
Keyword(s):  
Ground Truth ◽  
Objective Evaluation ◽  
The Other ◽  
Data Sets ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Adversarial Network ◽  
Fused Image ◽  
Visible Images ◽  
Evaluation Parameters

Infrared (IR) images can distinguish targets from their backgrounds based on difference in thermal radiation, whereas visible images can provide texture details with high spatial resolution. The fusion of the IR and visible images has many advantages and can be applied to applications such as target detection and recognition. This paper proposes a two-layer generative adversarial network (GAN) to fuse these two types of images. In the first layer, the network generate fused images using two GANs: one uses the IR image as input and the visible image as ground truth, and the other with the visible as input and the IR as ground truth. In the second layer, the network transfer one of the two fused images generated in the first layer as input and the other as ground truth to GAN to generate the final fused image. We adopt TNO and INO data sets to verify our method, and by comparing with eight objective evaluation parameters obtained by other ten methods. It is demonstrated that our method is able to achieve better performance than state-of-arts on preserving both texture details and thermal information.

Colorization of fusion image of infrared and visible images based on parallel generative adversarial network approach

2021 ◽  
pp. 1-10
Author(s):  
Lei Chen ◽  
Jun Han ◽  
Feng Tian
Keyword(s):  
Ground Truth ◽  
Objective Evaluation ◽  
Generative Adversarial Networks ◽  
Data Sets ◽  
Fusion Image ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Low Contrast ◽  
Adversarial Network ◽  
Visible Images

Fusing the infrared (IR) and visible images has many advantages and can be applied to applications such as target detection and recognition. Colors can give more accurate and distinct features, but the low resolution and low contrast of fused images make this a challenge task. In this paper, we proposed a method based on parallel generative adversarial networks (GANs) to address the challenge. We used IR image, visible image and fusion image as ground truth of ‘L’, ‘a’ and ‘b’ of the Lab model. Through the parallel GANs, we can gain the Lab data which can be converted to RGB image. We adopt TNO and RoadScene data sets to verify our method, and compare with five objective evaluation parameters obtained by other three methods based on deep learning (DL). It is demonstrated that the proposed approach is able to achieve better performance against state-of-arts methods.

scholarly journals Learning a Generative Model for Fusing Infrared and Visible Images via Conditional Generative Adversarial Network with Dual Discriminators

2019 ◽  
Author(s):  
Han Xu ◽  
Pengwei Liang ◽  
Wei Yu ◽  
Junjun Jiang ◽  
Jiayi Ma
Keyword(s):  
Probability Distribution ◽  
State Of The Art ◽  
Infrared Image ◽  
Infrared Images ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Qualitative And Quantitative ◽  
Adversarial Network ◽  
Fused Image ◽  
Visible Images

In this paper, we propose a new end-to-end model, called dual-discriminator conditional generative adversarial network (DDcGAN), for fusing infrared and visible images of different resolutions. Unlike the pixel-level methods and existing deep learning-based methods, the fusion task is accomplished through the adversarial process between a generator and two discriminators, in addition to the specially designed content loss. The generator is trained to generate real-like fused images to fool discriminators. The two discriminators are trained to calculate the JS divergence between the probability distribution of downsampled fused images and infrared images, and the JS divergence between the probability distribution of gradients of fused images and gradients of visible images, respectively. Thus, the fused images can compensate for the features that are not constrained by the single content loss. Consequently, the prominence of thermal targets in the infrared image and the texture details in the visible image can be preserved or even enhanced in the fused image simultaneously. Moreover, by constraining and distinguishing between the downsampled fused image and the low-resolution infrared image, DDcGAN can be preferably applied to the fusion of different resolution images. Qualitative and quantitative experiments on publicly available datasets demonstrate the superiority of our method over the state-of-the-art.

scholarly journals U-GAN Model for Infrared and Visible Images Fusion

2020 ◽  
Vol 38 (4) ◽  
pp. 904-912
Author(s):  
Zhuo Chen ◽  
Ming Fang ◽  
Xu Chai ◽  
Feiran Fu ◽  
Lihong Yuan
Keyword(s):  
Image Fusion ◽  
Network Architecture ◽  
Infrared Image ◽  
Data Sets ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Adversarial Network ◽  
Convolution Structure ◽  
Fusion Images ◽  
Visible Images

Infrared and visible image fusion is an effective method to solve the lack of single sensor imaging. The purpose is that the fusion images are suitable for human eyes and conducive to the next application and processing. In order to solve the problems of incomplete feature extraction, loss of details, and less samples of common data sets, it is not conducive to training, an end-to-end network architecture for image fusion is proposed. U-net is introduced into image fusion, and the final fusion result is obtained by using the generative adversarial network. Through its special convolution structure, the important feature information is extracted to the maximum extent, and the sample does not need to be cut to avoid the problem of reducing the fusion accuracy, but also to improve the training speed. Then the U-net extracted feature is confronted with the discriminator containing infrared image, and the generator model is obtained. The experimental results show that the present algorithm can obtain the fusion image with clear outline, prominent texture and obvious target. SD, SF, SSIM, AG and other indicators are obviously improved.

scholarly journals A Generative Adversarial Network for Infrared and Visible Image Fusion Based on Semantic Segmentation

Entropy ◽  
2021 ◽  
Vol 23 (3) ◽  
pp. 376
Author(s):  
Jilei Hou ◽  
Dazhi Zhang ◽  
Wei Wu ◽  
Jiayi Ma ◽  
Huabing Zhou
Keyword(s):  
Image Fusion ◽  
Information Source ◽  
Semantic Segmentation ◽  
Input Image ◽  
Infrared Images ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Adversarial Network ◽  
Visible Images ◽  
High Level

This paper proposes a new generative adversarial network for infrared and visible image fusion based on semantic segmentation (SSGAN), which can consider not only the low-level features of infrared and visible images, but also the high-level semantic information. Source images can be divided into foregrounds and backgrounds by semantic masks. The generator with a dual-encoder-single-decoder framework is used to extract the feature of foregrounds and backgrounds by different encoder paths. Moreover, the discriminator’s input image is designed based on semantic segmentation, which is obtained by combining the foregrounds of the infrared images with the backgrounds of the visible images. Consequently, the prominence of thermal targets in the infrared images and texture details in the visible images can be preserved in the fused images simultaneously. Qualitative and quantitative experiments on publicly available datasets demonstrate that the proposed approach can significantly outperform the state-of-the-art methods.

SGAN4AbSum: A Semantic-Enhanced Generative Adversarial Network for Abstractive Text Summarization

2021 ◽  
Author(s):  
Tham Vo
Keyword(s):  
Ground Truth ◽  
Text Summarization ◽  
Generative Adversarial Network ◽  
Convolutional Network ◽  
Training Strategy ◽  
Adversarial Network ◽  
Deep Recurrent Neural Network ◽  
Benchmark Datasets ◽  
Latent Representations ◽  
Abstractive Summarization

Abstract In abstractive summarization task, most of proposed models adopt the deep recurrent neural network (RNN)-based encoder-decoder architecture to learn and generate meaningful summary for a given input document. However, most of recent RNN-based models always suffer the challenges related to the involvement of much capturing high-frequency/reparative phrases in long documents during the training process which leads to the outcome of trivial and generic summaries are generated. Moreover, the lack of thorough analysis on the sequential and long-range dependency relationships between words within different contexts while learning the textual representation also make the generated summaries unnatural and incoherent. To deal with these challenges, in this paper we proposed a novel semantic-enhanced generative adversarial network (GAN)-based approach for abstractive text summarization task, called as: SGAN4AbSum. We use an adversarial training strategy for our text summarization model in which train the generator and discriminator to simultaneously handle the summary generation and distinguishing the generated summary with the ground-truth one. The input of generator is the jointed rich-semantic and global structural latent representations of training documents which are achieved by applying a combined BERT and graph convolutional network (GCN) textual embedding mechanism. Extensive experiments in benchmark datasets demonstrate the effectiveness of our proposed SGAN4AbSum which achieve the competitive ROUGE-based scores in comparing with state-of-the-art abstractive text summarization baselines.

Reflection interference removal for infrared thermography images based on GAN

2021 ◽  
Vol 63 (9) ◽  
pp. 529-533
Author(s):  
Jiali Zhang ◽  
Yupeng Tian ◽  
LiPing Ren ◽  
Jiaheng Cheng ◽  
JinChen Shi
Keyword(s):  
Neural Networks ◽  
Infrared Thermography ◽  
Metal Surfaces ◽  
Infrared Image ◽  
Infrared Camera ◽  
Infrared Images ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Continuous Application ◽  
Visible Images

Reflection in images is common and the removal of complex noise such as image reflection is still being explored. The problem is difficult and ill-posed, not only because there is no mixing function but also because there are no constraints in the output space (the processed image). When it comes to detecting defects on metal surfaces using infrared thermography, reflection from smooth metal surfaces can easily affect the final detection results. Therefore, it is essential to remove the reflection interference in infrared images. With the continuous application and expansion of neural networks in the field of image processing, researchers have tried to apply neural networks to remove image reflection. However, they have mainly focused on reflection interference removal in visible images and it is believed that no researchers have applied neural networks to remove reflection interference in infrared images. In this paper, the authors introduce the concept of a conditional generative adversarial network (cGAN) and propose an end-to-end trained network based on this with two types of loss: perceptual loss and adversarial loss. A self-built infrared reflection image dataset from an infrared camera is used. The experimental results demonstrate the effectiveness of this GAN for removing infrared image reflection.

scholarly journals GAINESIS: Generative Artificial Intelligence NEtlists SynthesIS

Electronics ◽  
2022 ◽  
Vol 11 (2) ◽  
pp. 245
Author(s):  
Konstantinos G. Liakos ◽  
Georgios K. Georgakilas ◽  
Fotis C. Plessas ◽  
Paris Kitsos
Keyword(s):  
Artificial Intelligence ◽  
Machine Learning ◽  
Power Analysis ◽  
Public Libraries ◽  
Data Sets ◽  
Hardware Trojan ◽  
Generative Adversarial Network ◽  
Data Set ◽  
Encrypted Data ◽  
Adversarial Network

A significant problem in the field of hardware security consists of hardware trojan (HT) viruses. The insertion of HTs into a circuit can be applied for each phase of the circuit chain of production. HTs degrade the infected circuit, destroy it or leak encrypted data. Nowadays, efforts are being made to address HTs through machine learning (ML) techniques, mainly for the gate-level netlist (GLN) phase, but there are some restrictions. Specifically, the number and variety of normal and infected circuits that exist through the free public libraries, such as Trust-HUB, are based on the few samples of benchmarks that have been created from circuits large in size. Thus, it is difficult, based on these data, to develop robust ML-based models against HTs. In this paper, we propose a new deep learning (DL) tool named Generative Artificial Intelligence Netlists SynthesIS (GAINESIS). GAINESIS is based on the Wasserstein Conditional Generative Adversarial Network (WCGAN) algorithm and area–power analysis features from the GLN phase and synthesizes new normal and infected circuit samples for this phase. Based on our GAINESIS tool, we synthesized new data sets, different in size, and developed and compared seven ML classifiers. The results demonstrate that our new generated data sets significantly enhance the performance of ML classifiers compared with the initial data set of Trust-HUB.

scholarly journals Synthesizing Depth Hand Images with GANs and Style Transfer for Hand Pose Estimation

Sensors ◽  
2019 ◽  
Vol 19 (13) ◽  
pp. 2919 ◽  
Author(s):  
Wangyong He ◽  
Zhongzhao Xie ◽  
Yongbo Li ◽  
Xinmei Wang ◽  
Wendi Cai
Keyword(s):  
Pose Estimation ◽  
Ground Truth ◽  
Training Image ◽  
Training Data ◽  
Generative Adversarial Network ◽  
Style Transfer ◽  
Visual Appearance ◽  
Hand Pose Estimation ◽  
Adversarial Network ◽  
Hand Pose

Hand pose estimation is a critical technology of computer vision and human-computer interaction. Deep-learning methods require a considerable amount of tagged data. Accordingly, numerous labeled training data are required. This paper aims to generate depth hand images. Given a ground-truth 3D hand pose, the developed method can generate depth hand images. To be specific, a ground truth can be 3D hand poses with the hand structure contained, while the synthesized image has an identical size to that of the training image and a similar visual appearance to the training set. The developed method, inspired by the progress in the generative adversarial network (GAN) and image-style transfer, helps model the latent statistical relationship between the ground-truth hand pose and the corresponding depth hand image. The images synthesized using the developed method are demonstrated to be feasible for enhancing performance. On public hand pose datasets (NYU, MSRA, ICVL), comprehensive experiments prove that the developed method outperforms the existing works.

scholarly journals Joint Entity and Event Extraction with Generative Adversarial Imitation Learning

2019 ◽  
Vol 1 (2) ◽  
pp. 99-120 ◽  
Author(s):  
Tongtao Zhang ◽  
Heng Ji ◽  
Avirup Sil
Keyword(s):  
State Of The Art ◽  
Ground Truth ◽  
Event Extraction ◽  
Imitation Learning ◽  
Learning Method ◽  
Inverse Reinforcement Learning ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
The Difference ◽  
New Framework

We propose a new framework for entity and event extraction based on generative adversarial imitation learning—an inverse reinforcement learning method using a generative adversarial network (GAN). We assume that instances and labels yield to various extents of difficulty and the gains and penalties (rewards) are expected to be diverse. We utilize discriminators to estimate proper rewards according to the difference between the labels committed by the ground-truth (expert) and the extractor (agent). Our experiments demonstrate that the proposed framework outperforms state-of-the-art methods.

scholarly journals Infrared and Visible Image Fusion Based on Nonlinear Enhancement and NSST Decomposition

2020 ◽  
Author(s):  
Xiaoxue XING ◽  
Cheng LIU ◽  
Cong LUO ◽  
Tingfa XU
Keyword(s):  
Thermal Radiation ◽  
High Frequency ◽  
Geometric Analysis ◽  
Low Frequency ◽  
Fusion Rule ◽  
Fusion Algorithm ◽  
Visible Image ◽  
Fused Image ◽  
Fusion Methods ◽  
Visible Images

Abstract In Multi-scale Geometric Analysis (MGA)-based fusion methods for infrared and visible images, adopting the same representation for the two types of the images will result in the non-obvious thermal radiation target in the fused image, which can hardly be distinguished from the background. To solve the problem, a novel fusion algorithm based on nonlinear enhancement and Non-Subsampled Shearlet Transform (NSST) decomposition is proposed. Firstly, NSST is used to decompose the two source images into low- and high-frequency sub-bands. Then, the Wavelet Transform (WT) is used to decompose high-frequency sub-bands into obtain approximate sub-bands and directional detail sub-bands. The “average” fusion rule is performed for fusion for approximate sub-bands. And the “max-absolute” fusion rule is performed for fusion for directional detail sub-bands. The inverse WT is used to reconstruct the high-frequency sub-bands. To highlight the thermal radiation target, we construct a non-linear transform function to determine the fusion weight of low-frequency sub-bands, and whose parameters can be further adjusted to meet different fusion requirements. Finally, the inverse NSST is used to reconstruct the fused image. The experimental results show that the proposed method can simultaneously enhance the thermal target in infrared images and preserve the texture details in visible images, and which is competitive with or even superior to the state-of-the-art fusion methods in terms of both visual and quantitative evaluations.

Sign in / Sign up

Export Citation Format

Share Document