Patch-Wise Infrared and Visible Image Fusion Using Spatial Adaptive Weights

In infrared (IR) and visible image fusion, the significant information is extracted from each source image and integrated into a single image with comprehensive data. We observe that the salient regions in the infrared image contain targets of interests. Therefore, we enforce spatial adaptive weights derived from the infrared images. In this paper, a Generative Adversarial Network (GAN)-based fusion method is proposed for infrared and visible image fusion. Based on the end-to-end network structure with dual discriminators, a patch-wise discrimination is applied to reduce blurry artifact from the previous image-level approaches. A new loss function is also proposed to use constructed weight maps which direct the adversarial training of GAN in a manner such that the informative regions of the infrared images are preserved. Experiments are performed on the two datasets and ablation studies are also conducted. The qualitative and quantitative analysis shows that we achieve competitive results compared to the existing fusion methods.

Download Full-text

Learning a Generative Model for Fusing Infrared and Visible Images via Conditional Generative Adversarial Network with Dual Discriminators

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2019/549 ◽

2019 ◽

Cited By ~ 12

Author(s):

Han Xu ◽

Pengwei Liang ◽

Wei Yu ◽

Junjun Jiang ◽

Jiayi Ma

Keyword(s):

Probability Distribution ◽

State Of The Art ◽

Infrared Image ◽

Infrared Images ◽

Generative Adversarial Network ◽

Visible Image ◽

Qualitative And Quantitative ◽

Adversarial Network ◽

Fused Image ◽

Visible Images

In this paper, we propose a new end-to-end model, called dual-discriminator conditional generative adversarial network (DDcGAN), for fusing infrared and visible images of different resolutions. Unlike the pixel-level methods and existing deep learning-based methods, the fusion task is accomplished through the adversarial process between a generator and two discriminators, in addition to the specially designed content loss. The generator is trained to generate real-like fused images to fool discriminators. The two discriminators are trained to calculate the JS divergence between the probability distribution of downsampled fused images and infrared images, and the JS divergence between the probability distribution of gradients of fused images and gradients of visible images, respectively. Thus, the fused images can compensate for the features that are not constrained by the single content loss. Consequently, the prominence of thermal targets in the infrared image and the texture details in the visible image can be preserved or even enhanced in the fused image simultaneously. Moreover, by constraining and distinguishing between the downsampled fused image and the low-resolution infrared image, DDcGAN can be preferably applied to the fusion of different resolution images. Qualitative and quantitative experiments on publicly available datasets demonstrate the superiority of our method over the state-of-the-art.

Download Full-text

Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network

Applied Sciences ◽

10.3390/app10020554 ◽

2020 ◽

Vol 10 (2) ◽

pp. 554 ◽

Cited By ~ 1

Author(s):

Dongdong Xu ◽

Yongcheng Wang ◽

Shuyan Xu ◽

Kaiguang Zhu ◽

Ning Zhang ◽

...

Keyword(s):

Image Fusion ◽

Objective Assessment ◽

Computational Cost ◽

Activity Level ◽

Source Image ◽

Residual Network ◽

Generative Adversarial Network ◽

Visible Image ◽

Adversarial Network ◽

Combined Images

Infrared and visible image fusion can obtain combined images with salient hidden objectives and abundant visible details simultaneously. In this paper, we propose a novel method for infrared and visible image fusion with a deep learning framework based on a generative adversarial network (GAN) and a residual network (ResNet). The fusion is accomplished with an adversarial game and directed by the unique loss functions. The generator with residual blocks and skip connections can extract deep features of source image pairs and generate an elementary fused image with infrared thermal radiation information and visible texture information, and more details in visible images are added to the final images through the discriminator. It is unnecessary to design the activity level measurements and fusion rules manually, which are now implemented automatically. Also, there are no complicated multi-scale transforms in this method, so the computational cost and complexity can be reduced. Experiment results demonstrate that the proposed method eventually gets desirable images, achieving better performance in objective assessment and visual quality compared with nine representative infrared and visible image fusion methods.

Download Full-text

A Generative Adversarial Network for Infrared and Visible Image Fusion Based on Semantic Segmentation

Entropy ◽

10.3390/e23030376 ◽

2021 ◽

Vol 23 (3) ◽

pp. 376

Author(s):

Jilei Hou ◽

Dazhi Zhang ◽

Wei Wu ◽

Jiayi Ma ◽

Huabing Zhou

Keyword(s):

Image Fusion ◽

Information Source ◽

Semantic Segmentation ◽

Input Image ◽

Infrared Images ◽

Generative Adversarial Network ◽

Visible Image ◽

Adversarial Network ◽

Visible Images ◽

High Level

This paper proposes a new generative adversarial network for infrared and visible image fusion based on semantic segmentation (SSGAN), which can consider not only the low-level features of infrared and visible images, but also the high-level semantic information. Source images can be divided into foregrounds and backgrounds by semantic masks. The generator with a dual-encoder-single-decoder framework is used to extract the feature of foregrounds and backgrounds by different encoder paths. Moreover, the discriminator’s input image is designed based on semantic segmentation, which is obtained by combining the foregrounds of the infrared images with the backgrounds of the visible images. Consequently, the prominence of thermal targets in the infrared images and texture details in the visible images can be preserved in the fused images simultaneously. Qualitative and quantitative experiments on publicly available datasets demonstrate that the proposed approach can significantly outperform the state-of-the-art methods.

Download Full-text

A Generative Adversarial Network with Dual Discriminators for Infrared and Visible Image Fusion Based on Saliency Detection

Mathematical Problems in Engineering ◽

10.1155/2021/4209963 ◽

2021 ◽

Vol 2021 ◽

pp. 1-9

Author(s):

Dazhi Zhang ◽

Jilei Hou ◽

Wei Wu ◽

Tao Lu ◽

Huabing Zhou

Keyword(s):

Image Fusion ◽

Loss Function ◽

Network Architecture ◽

Saliency Detection ◽

Infrared Image ◽

Saliency Map ◽

Generative Adversarial Network ◽

Visible Image ◽

Adversarial Network ◽

Salient Regions

Infrared and visible image fusion needs to preserve both the salient target of the infrared image and the texture details of the visible image. Therefore, an infrared and visible image fusion method based on saliency detection is proposed. Firstly, the saliency map of the infrared image is obtained by saliency detection. Then, the specific loss function and network architecture are designed based on the saliency map to improve the performance of the fusion algorithm. Specifically, the saliency map is normalized to [0, 1], used as a weight map to constrain the loss function. At the same time, the saliency map is binarized to extract salient regions and nonsalient regions. And, a generative adversarial network with dual discriminators is obtained. The two discriminators are used to distinguish the salient regions and the nonsalient regions, respectively, to promote the generator to generate better fusion results. The experimental results show that the fusion results of our method are better than those of the existing methods in both subjective and objective aspects.

Download Full-text

U-GAN Model for Infrared and Visible Images Fusion

Xibei Gongye Daxue Xuebao/Journal of Northwestern Polytechnical University ◽

10.1051/jnwpu/20203840904 ◽

2020 ◽

Vol 38 (4) ◽

pp. 904-912

Author(s):

Zhuo Chen ◽

Ming Fang ◽

Xu Chai ◽

Feiran Fu ◽

Lihong Yuan

Keyword(s):

Image Fusion ◽

Network Architecture ◽

Infrared Image ◽

Data Sets ◽

Generative Adversarial Network ◽

Visible Image ◽

Adversarial Network ◽

Convolution Structure ◽

Fusion Images ◽

Visible Images

Infrared and visible image fusion is an effective method to solve the lack of single sensor imaging. The purpose is that the fusion images are suitable for human eyes and conducive to the next application and processing. In order to solve the problems of incomplete feature extraction, loss of details, and less samples of common data sets, it is not conducive to training, an end-to-end network architecture for image fusion is proposed. U-net is introduced into image fusion, and the final fusion result is obtained by using the generative adversarial network. Through its special convolution structure, the important feature information is extracted to the maximum extent, and the sample does not need to be cut to avoid the problem of reducing the fusion accuracy, but also to improve the training speed. Then the U-net extracted feature is confronted with the discriminator containing infrared image, and the generator model is obtained. The experimental results show that the present algorithm can obtain the fusion image with clear outline, prominent texture and obvious target. SD, SF, SSIM, AG and other indicators are obviously improved.

Download Full-text

Reflection interference removal for infrared thermography images based on GAN

Insight - Non-Destructive Testing and Condition Monitoring ◽

10.1784/insi.2021.63.9.529 ◽

2021 ◽

Vol 63 (9) ◽

pp. 529-533

Author(s):

Jiali Zhang ◽

Yupeng Tian ◽

LiPing Ren ◽

Jiaheng Cheng ◽

JinChen Shi

Keyword(s):

Neural Networks ◽

Infrared Thermography ◽

Metal Surfaces ◽

Infrared Image ◽

Infrared Camera ◽

Infrared Images ◽

Generative Adversarial Network ◽

Adversarial Network ◽

Continuous Application ◽

Visible Images

Reflection in images is common and the removal of complex noise such as image reflection is still being explored. The problem is difficult and ill-posed, not only because there is no mixing function but also because there are no constraints in the output space (the processed image). When it comes to detecting defects on metal surfaces using infrared thermography, reflection from smooth metal surfaces can easily affect the final detection results. Therefore, it is essential to remove the reflection interference in infrared images. With the continuous application and expansion of neural networks in the field of image processing, researchers have tried to apply neural networks to remove image reflection. However, they have mainly focused on reflection interference removal in visible images and it is believed that no researchers have applied neural networks to remove reflection interference in infrared images. In this paper, the authors introduce the concept of a conditional generative adversarial network (cGAN) and propose an end-to-end trained network based on this with two types of loss: perceptual loss and adversarial loss. A self-built infrared reflection image dataset from an infrared camera is used. The experimental results demonstrate the effectiveness of this GAN for removing infrared image reflection.

Download Full-text

Infrared and Visible Image Fusion through Details Preservation

Sensors ◽

10.3390/s19204556 ◽

2019 ◽

Vol 19 (20) ◽

pp. 4556 ◽

Cited By ~ 2

Author(s):

Yaochen Liu ◽

Lili Dong ◽

Yuanyuan Ji ◽

Wenhai Xu

Keyword(s):

Image Fusion ◽

Subjective Evaluation ◽

Objective Assessment ◽

Decomposition Methods ◽

Source Image ◽

Fusion Method ◽

Visible Image ◽

Weighting Method ◽

Fused Image ◽

Fusion Methods

In many actual applications, fused image is essential to contain high-quality details for achieving a comprehensive representation of the real scene. However, existing image fusion methods suffer from loss of details because of the error accumulations of sequential tasks. This paper proposes a novel fusion method to preserve details of infrared and visible images by combining new decomposition, feature extraction, and fusion scheme. For decomposition, different from the most decomposition methods by guided filter, the guidance image contains only the strong edge of the source image but no other interference information so that rich tiny details can be decomposed into the detailed part. Then, according to the different characteristics of infrared and visible detail parts, a rough convolutional neural network (CNN) and a sophisticated CNN are designed so that various features can be fully extracted. To integrate the extracted features, we also present a multi-layer features fusion strategy through discrete cosine transform (DCT), which not only highlights significant features but also enhances details. Moreover, the base parts are fused by weighting method. Finally, the fused image is obtained by adding the fused detail and base part. Different from the general image fusion methods, our method not only retains the target region of source image but also enhances background in the fused image. In addition, compared with state-of-the-art fusion methods, our proposed fusion method has many advantages, including (i) better visual quality of fused-image subjective evaluation, and (ii) better objective assessment for those images.

Download Full-text