scholarly journals Learning a Generative Model for Fusing Infrared and Visible Images via Conditional Generative Adversarial Network with Dual Discriminators

2019 ◽  
Author(s):  
Han Xu ◽  
Pengwei Liang ◽  
Wei Yu ◽  
Junjun Jiang ◽  
Jiayi Ma
Keyword(s):  
Probability Distribution ◽  
State Of The Art ◽  
Infrared Image ◽  
Infrared Images ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Qualitative And Quantitative ◽  
Adversarial Network ◽  
Fused Image ◽  
Visible Images

In this paper, we propose a new end-to-end model, called dual-discriminator conditional generative adversarial network (DDcGAN), for fusing infrared and visible images of different resolutions. Unlike the pixel-level methods and existing deep learning-based methods, the fusion task is accomplished through the adversarial process between a generator and two discriminators, in addition to the specially designed content loss. The generator is trained to generate real-like fused images to fool discriminators. The two discriminators are trained to calculate the JS divergence between the probability distribution of downsampled fused images and infrared images, and the JS divergence between the probability distribution of gradients of fused images and gradients of visible images, respectively. Thus, the fused images can compensate for the features that are not constrained by the single content loss. Consequently, the prominence of thermal targets in the infrared image and the texture details in the visible image can be preserved or even enhanced in the fused image simultaneously. Moreover, by constraining and distinguishing between the downsampled fused image and the low-resolution infrared image, DDcGAN can be preferably applied to the fusion of different resolution images. Qualitative and quantitative experiments on publicly available datasets demonstrate the superiority of our method over the state-of-the-art.

Reflection interference removal for infrared thermography images based on GAN

2021 ◽  
Vol 63 (9) ◽  
pp. 529-533
Author(s):  
Jiali Zhang ◽  
Yupeng Tian ◽  
LiPing Ren ◽  
Jiaheng Cheng ◽  
JinChen Shi
Keyword(s):  
Neural Networks ◽  
Infrared Thermography ◽  
Metal Surfaces ◽  
Infrared Image ◽  
Infrared Camera ◽  
Infrared Images ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Continuous Application ◽  
Visible Images

Reflection in images is common and the removal of complex noise such as image reflection is still being explored. The problem is difficult and ill-posed, not only because there is no mixing function but also because there are no constraints in the output space (the processed image). When it comes to detecting defects on metal surfaces using infrared thermography, reflection from smooth metal surfaces can easily affect the final detection results. Therefore, it is essential to remove the reflection interference in infrared images. With the continuous application and expansion of neural networks in the field of image processing, researchers have tried to apply neural networks to remove image reflection. However, they have mainly focused on reflection interference removal in visible images and it is believed that no researchers have applied neural networks to remove reflection interference in infrared images. In this paper, the authors introduce the concept of a conditional generative adversarial network (cGAN) and propose an end-to-end trained network based on this with two types of loss: perceptual loss and adversarial loss. A self-built infrared reflection image dataset from an infrared camera is used. The experimental results demonstrate the effectiveness of this GAN for removing infrared image reflection.

scholarly journals Fusing Infrared and Visible Images of Different Resolutions via Total Variation Model

Sensors ◽  
2018 ◽  
Vol 18 (11) ◽  
pp. 3827 ◽  
Author(s):  
Qinglei Du ◽  
Han Xu ◽  
Yong Ma ◽  
Jun Huang ◽  
Fan Fan
Keyword(s):  
Thermal Radiation ◽  
Total Variation ◽  
Infrared Image ◽  
Infrared Images ◽  
Visible Image ◽  
Pixel Intensity ◽  
Texture Information ◽  
Fused Image ◽  
Fusion Methods ◽  
Visible Images

In infrared and visible image fusion, existing methods typically have a prerequisite that the source images share the same resolution. However, due to limitations of hardware devices and application environments, infrared images constantly suffer from markedly lower resolution compared with the corresponding visible images. In this case, current fusion methods inevitably cause texture information loss in visible images or blur thermal radiation information in infrared images. Moreover, the principle of existing fusion rules typically focuses on preserving texture details in source images, which may be inappropriate for fusing infrared thermal radiation information because it is characterized by pixel intensities, possibly neglecting the prominence of targets in fused images. Faced with such difficulties and challenges, we propose a novel method to fuse infrared and visible images of different resolutions and generate high-resolution resulting images to obtain clear and accurate fused images. Specifically, the fusion problem is formulated as a total variation (TV) minimization problem. The data fidelity term constrains the pixel intensity similarity of the downsampled fused image with respect to the infrared image, and the regularization term compels the gradient similarity of the fused image with respect to the visible image. The fast iterative shrinkage-thresholding algorithm (FISTA) framework is applied to improve the convergence rate. Our resulting fused images are similar to super-resolved infrared images, which are sharpened by the texture information from visible images. Advantages and innovations of our method are demonstrated by the qualitative and quantitative comparisons with six state-of-the-art methods on publicly available datasets.

scholarly journals A Generative Adversarial Network for Infrared and Visible Image Fusion Based on Semantic Segmentation

Entropy ◽  
2021 ◽  
Vol 23 (3) ◽  
pp. 376
Author(s):  
Jilei Hou ◽  
Dazhi Zhang ◽  
Wei Wu ◽  
Jiayi Ma ◽  
Huabing Zhou
Keyword(s):  
Image Fusion ◽  
Information Source ◽  
Semantic Segmentation ◽  
Input Image ◽  
Infrared Images ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Adversarial Network ◽  
Visible Images ◽  
High Level

This paper proposes a new generative adversarial network for infrared and visible image fusion based on semantic segmentation (SSGAN), which can consider not only the low-level features of infrared and visible images, but also the high-level semantic information. Source images can be divided into foregrounds and backgrounds by semantic masks. The generator with a dual-encoder-single-decoder framework is used to extract the feature of foregrounds and backgrounds by different encoder paths. Moreover, the discriminator’s input image is designed based on semantic segmentation, which is obtained by combining the foregrounds of the infrared images with the backgrounds of the visible images. Consequently, the prominence of thermal targets in the infrared images and texture details in the visible images can be preserved in the fused images simultaneously. Qualitative and quantitative experiments on publicly available datasets demonstrate that the proposed approach can significantly outperform the state-of-the-art methods.

scholarly journals Patch-Wise Infrared and Visible Image Fusion Using Spatial Adaptive Weights

2021 ◽  
Vol 11 (19) ◽  
pp. 9255
Author(s):  
Syeda Minahil ◽  
Jun-Hyung Kim ◽  
Youngbae Hwang
Keyword(s):  
Image Fusion ◽  
Infrared Image ◽  
Source Image ◽  
Infrared Images ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Significant Information ◽  
Adaptive Weights ◽  
Adversarial Network ◽  
Fusion Methods

In infrared (IR) and visible image fusion, the significant information is extracted from each source image and integrated into a single image with comprehensive data. We observe that the salient regions in the infrared image contain targets of interests. Therefore, we enforce spatial adaptive weights derived from the infrared images. In this paper, a Generative Adversarial Network (GAN)-based fusion method is proposed for infrared and visible image fusion. Based on the end-to-end network structure with dual discriminators, a patch-wise discrimination is applied to reduce blurry artifact from the previous image-level approaches. A new loss function is also proposed to use constructed weight maps which direct the adversarial training of GAN in a manner such that the informative regions of the infrared images are preserved. Experiments are performed on the two datasets and ablation studies are also conducted. The qualitative and quantitative analysis shows that we achieve competitive results compared to the existing fusion methods.

scholarly journals Advanced Driving Assistance Based on the Fusion of Infrared and Visible Images

Entropy ◽  
2021 ◽  
Vol 23 (2) ◽  
pp. 239
Author(s):  
Yansong Gu ◽  
Xinya Wang ◽  
Can Zhang ◽  
Baiyang Li
Keyword(s):  
Visual Information ◽  
State Of The Art ◽  
Infrared Image ◽  
Activity Level ◽  
Qualitative And Quantitative ◽  
Intensity Information ◽  
Driving Assistance ◽  
Fused Image ◽  
Visible Images ◽  
End To End

Obtaining key and rich visual information under sophisticated road conditions is one of the key requirements for advanced driving assistance. In this paper, a newfangled end-to-end model is proposed for advanced driving assistance based on the fusion of infrared and visible images, termed as FusionADA. In our model, we are committed to extracting and fusing the optimal texture details and salient thermal targets from the source images. To achieve this goal, our model constitutes an adversarial framework between the generator and the discriminator. Specifically, the generator aims to generate a fused image with basic intensity information together with the optimal texture details from source images, while the discriminator aims to force the fused image to restore the salient thermal targets from the source infrared image. In addition, our FusionADA is a fully end-to-end model, solving the issues of manually designing complicated activity level measurements and fusion rules existing in traditional methods. Qualitative and quantitative experiments on publicly available datasets RoadScene and TNO demonstrate the superiority of our FusionADA over the state-of-the-art approaches.

scholarly journals U-GAN Model for Infrared and Visible Images Fusion

2020 ◽  
Vol 38 (4) ◽  
pp. 904-912
Author(s):  
Zhuo Chen ◽  
Ming Fang ◽  
Xu Chai ◽  
Feiran Fu ◽  
Lihong Yuan
Keyword(s):  
Image Fusion ◽  
Network Architecture ◽  
Infrared Image ◽  
Data Sets ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Adversarial Network ◽  
Convolution Structure ◽  
Fusion Images ◽  
Visible Images

Infrared and visible image fusion is an effective method to solve the lack of single sensor imaging. The purpose is that the fusion images are suitable for human eyes and conducive to the next application and processing. In order to solve the problems of incomplete feature extraction, loss of details, and less samples of common data sets, it is not conducive to training, an end-to-end network architecture for image fusion is proposed. U-net is introduced into image fusion, and the final fusion result is obtained by using the generative adversarial network. Through its special convolution structure, the important feature information is extracted to the maximum extent, and the sample does not need to be cut to avoid the problem of reducing the fusion accuracy, but also to improve the training speed. Then the U-net extracted feature is confronted with the discriminator containing infrared image, and the generator model is obtained. The experimental results show that the present algorithm can obtain the fusion image with clear outline, prominent texture and obvious target. SD, SF, SSIM, AG and other indicators are obviously improved.

scholarly journals Infrared and visible image fusion using two-layer generative adversarial network

2021 ◽  
pp. 1-17
Author(s):  
Lei Chen ◽  
Jun Han ◽  
Feng Tian
Keyword(s):  
Ground Truth ◽  
Objective Evaluation ◽  
The Other ◽  
Data Sets ◽  
Generative Adversarial Network ◽  
Visible Image ◽  
Adversarial Network ◽  
Fused Image ◽  
Visible Images ◽  
Evaluation Parameters

Infrared (IR) images can distinguish targets from their backgrounds based on difference in thermal radiation, whereas visible images can provide texture details with high spatial resolution. The fusion of the IR and visible images has many advantages and can be applied to applications such as target detection and recognition. This paper proposes a two-layer generative adversarial network (GAN) to fuse these two types of images. In the first layer, the network generate fused images using two GANs: one uses the IR image as input and the visible image as ground truth, and the other with the visible as input and the IR as ground truth. In the second layer, the network transfer one of the two fused images generated in the first layer as input and the other as ground truth to GAN to generate the final fused image. We adopt TNO and INO data sets to verify our method, and by comparing with eight objective evaluation parameters obtained by other ten methods. It is demonstrated that our method is able to achieve better performance than state-of-arts on preserving both texture details and thermal information.

A Rapid Fusion Algorithm of Infrared and the Visible Images Based on Directionlet Transform

2010 ◽  
Vol 20-23 ◽  
pp. 45-51
Author(s):  
Xiang Li ◽  
Yue Shun He ◽  
Xuan Zhan ◽  
Feng Yu Liu
Keyword(s):  
Fusion Rule ◽  
Infrared Images ◽  
Fusion Algorithm ◽  
Visible Image ◽  
Regional Energy ◽  
Fused Image ◽  
Visible Images ◽  
Simulation Results ◽  
Edge Features ◽  
Intense Color

Direction transform; image fusion; infrared images; fusion rule; anisotropic Abstract Based on analysing the feature of infrared and the visible, this paper proposed an improved algorithm using Directionlet transform.The feature is like this: firstly, separate the color visible images to get the component images, and then make anisotropic decomposition for component images and inrared images, after analysing these images, process them according to regional energy rules ,finally incorporate the intense color to get the fused image. The simulation results shows that,this algorithm can effectively fuse infrared and the visible image, moreover, not only the fused images can maintain the environment details, but also underline the edge features, which applies to fusion with strong edges, therefore,this algorithm is of robust and convenient.

scholarly journals Generative Adversarial Network-Based Edge-Preserving Superresolution Reconstruction of Infrared Images

2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
Yuqing Zhao ◽  
Guangyuan Fu ◽  
Hongqiao Wang ◽  
Shaolei Zhang ◽  
Min Yue
Keyword(s):  
Infrared Image ◽  
Reconstruction Method ◽  
Infrared Images ◽  
Generative Adversarial Network ◽  
Edge Preserving ◽  
Edge Structure ◽  
Second Stage ◽  
Adversarial Network ◽  
Objective Quality ◽  
Image Edges

The convolutional neural network has achieved good results in the superresolution reconstruction of single-frame images. However, due to the shortcomings of infrared images such as lack of details, poor contrast, and blurred edges, superresolution reconstruction of infrared images that preserves the edge structure and better visual quality is still challenging. Aiming at the problems of low resolution and unclear edges of infrared images, this work proposes a two-stage generative adversarial network model to reconstruct realistic superresolution images from four times downsampled infrared images. In the first stage of the generative adversarial network, it focuses on recovering the overall contour information of the image to obtain clear image edges; the second stage of the generative adversarial network focuses on recovering the detailed feature information of the image and has a stronger ability to express details. The infrared image superresolution reconstruction method proposed in this work has highly realistic visual effects and good objective quality evaluation results.

Fusion of Infrared and Visible Image Based on Target Regions for Environment Perception

2011 ◽  
Vol 128-129 ◽  
pp. 589-593 ◽  
Author(s):  
Yi Feng Niu ◽  
Sheng Tao Xu ◽  
Wei Dong Hu
Keyword(s):  
Unmanned Aerial Vehicles ◽  
Target Recognition ◽  
Discrete Wavelet ◽  
Infrared Images ◽  
Target Information ◽  
Visible Image ◽  
Fused Image ◽  
Fusion Methods ◽  
Visible Images ◽  
Environment Perception

Infrared and visible image fusion is an important precondition to realize target perception for unmanned aerial vehicles (UAV) based on which UAV can perform various missions. The details in visible images are abundant, while the target information is more outstanding in infrared images. However, the conventional fusion methods are mostly based on region segmentation, and then the fused image for target recognition can’t be actually acquired. In this paper, a novel fusion method of infrared and visible image based on target regions in discrete wavelet transform (DWT) domain is proposed, which can gain more target information and preserve the details. Experimental results show that our method can generate better fused image for target recognition.

Sign in / Sign up

Export Citation Format

Share Document