Infrared and visible image fusion methods and applications: A survey

AbstractImage fusion integrates information from multiple images (of the same scene) to generate a (more informative) composite image suitable for human and computer vision perception. The method based on multiscale decomposition is one of the commonly fusion methods. In this study, a new fusion framework based on the octave Gaussian pyramid principle is proposed. In comparison with conventional multiscale decomposition, the proposed octave Gaussian pyramid framework retrieves more information by decomposing an image into two scale spaces (octave and interval spaces). Different from traditional multiscale decomposition with one set of detail and base layers, the proposed method decomposes an image into multiple sets of detail and base layers, and it efficiently retains high- and low-frequency information from the original image. The qualitative and quantitative comparison with five existing methods (on publicly available image databases) demonstrate that the proposed method has better visual effects and scores the highest in objective evaluation.

Download Full-text

Infrared and Visible Image Fusion Techniques Based on Deep Learning: A Review

Electronics ◽

10.3390/electronics9122162 ◽

2020 ◽

Vol 9 (12) ◽

pp. 2162

Author(s):

Changqi Sun ◽

Cong Zhang ◽

Naixue Xiong

Keyword(s):

Deep Learning ◽

Image Fusion ◽

Research Work ◽

Image Features ◽

Future Research ◽

Learning Methods ◽

Visible Image ◽

Advantages And Disadvantages ◽

Fusion Methods ◽

Future Work

Infrared and visible image fusion technologies make full use of different image features obtained by different sensors, retain complementary information of the source images during the fusion process, and use redundant information to improve the credibility of the fusion image. In recent years, many researchers have used deep learning methods (DL) to explore the field of image fusion and found that applying DL has improved the time-consuming efficiency of the model and the fusion effect. However, DL includes many branches, and there is currently no detailed investigation of deep learning methods in image fusion. In this work, this survey reports on the development of image fusion algorithms based on deep learning in recent years. Specifically, this paper first conducts a detailed investigation on the fusion method of infrared and visible images based on deep learning, compares the existing fusion algorithms qualitatively and quantitatively with the existing fusion quality indicators, and discusses various fusions. The main contribution, advantages, and disadvantages of the algorithm. Finally, the research status of infrared and visible image fusion is summarized, and future work has prospected. This research can help us realize many image fusion methods in recent years and lay the foundation for future research work.

Download Full-text

Infrared and Visible Image Fusion through Details Preservation

Sensors ◽

10.3390/s19204556 ◽

2019 ◽

Vol 19 (20) ◽

pp. 4556 ◽

Cited By ~ 2

Author(s):

Yaochen Liu ◽

Lili Dong ◽

Yuanyuan Ji ◽

Wenhai Xu

Keyword(s):

Image Fusion ◽

Subjective Evaluation ◽

Objective Assessment ◽

Decomposition Methods ◽

Source Image ◽

Fusion Method ◽

Visible Image ◽

Weighting Method ◽

Fused Image ◽

Fusion Methods

In many actual applications, fused image is essential to contain high-quality details for achieving a comprehensive representation of the real scene. However, existing image fusion methods suffer from loss of details because of the error accumulations of sequential tasks. This paper proposes a novel fusion method to preserve details of infrared and visible images by combining new decomposition, feature extraction, and fusion scheme. For decomposition, different from the most decomposition methods by guided filter, the guidance image contains only the strong edge of the source image but no other interference information so that rich tiny details can be decomposed into the detailed part. Then, according to the different characteristics of infrared and visible detail parts, a rough convolutional neural network (CNN) and a sophisticated CNN are designed so that various features can be fully extracted. To integrate the extracted features, we also present a multi-layer features fusion strategy through discrete cosine transform (DCT), which not only highlights significant features but also enhances details. Moreover, the base parts are fused by weighting method. Finally, the fused image is obtained by adding the fused detail and base part. Different from the general image fusion methods, our method not only retains the target region of source image but also enhances background in the fused image. In addition, compared with state-of-the-art fusion methods, our proposed fusion method has many advantages, including (i) better visual quality of fused-image subjective evaluation, and (ii) better objective assessment for those images.

Download Full-text

Infrared and Visible Image Fusion Using Truncated Huber Penalty Function Smoothing and Visual Saliency Based Threshold Optimization

Electronics ◽

10.3390/electronics11010033 ◽

2021 ◽

Vol 11 (1) ◽

pp. 33

Author(s):

Chaowei Duan ◽

Yiliu Liu ◽

Changda Xing ◽

Zhisheng Wang

Keyword(s):

Image Fusion ◽

Penalty Function ◽

Visual Saliency ◽

Fusion Rule ◽

Visible Image ◽

Threshold Optimization ◽

Fused Image ◽

Fusion Methods ◽

Visible Images ◽

Image Representations

An efficient method for the infrared and visible image fusion is presented using truncated Huber penalty function smoothing and visual saliency based threshold optimization. The method merges complementary information from multimodality source images into a more informative composite image in two-scale domain, in which the significant objects/regions are highlighted and rich feature information is preserved. Firstly, source images are decomposed into two-scale image representations, namely, the approximate and residual layers, using truncated Huber penalty function smoothing. Benefiting from the edge- and structure-preserving characteristics, the significant objects and regions in the source images are effectively extracted without halo artifacts around the edges. Secondly, a visual saliency based threshold optimization fusion rule is designed to fuse the approximate layers aiming to highlight the salient targets in infrared images and remain the high-intensity regions in visible images. The sparse representation based fusion rule is adopted to fuse the residual layers with the goal of acquiring rich detail texture information. Finally, combining the fused approximate and residual layers reconstructs the fused image with more natural visual effects. Sufficient experimental results demonstrate that the proposed method can achieve comparable or superior performances compared with several state-of-the-art fusion methods in visual results and objective assessments.

Download Full-text

A Heterogeneous Image Fusion Method Based on DCT and Anisotropic Diffusion for UAVs in Future 5G IoT Scenarios

Wireless Communications and Mobile Computing ◽

10.1155/2020/8816818 ◽

2020 ◽

Vol 2020 ◽

pp. 1-11

Author(s):

Shuai Hao ◽

Beiyi An ◽

Hu Wen ◽

Xu Ma ◽

Keping Yu

Keyword(s):

Image Fusion ◽

Traffic Management ◽

Anisotropic Diffusion ◽

Weighted Averaging ◽

Base Layer ◽

Fusion Method ◽

Linear Superposition ◽

Visible Image ◽

Fusion Methods ◽

Visible Images

Unmanned aerial vehicles, with their inherent fine attributes, such as flexibility, mobility, and autonomy, play an increasingly important role in the Internet of Things (IoT). Airborne infrared and visible image fusion, which constitutes an important data basis for the perception layer of IoT, has been widely used in various fields such as electric power inspection, military reconnaissance, emergency rescue, and traffic management. However, traditional infrared and visible image fusion methods suffer from weak detail resolution. In order to better preserve useful information from source images and produce a more informative image for human observation or unmanned aerial vehicle vision tasks, a novel fusion method based on discrete cosine transform (DCT) and anisotropic diffusion is proposed. First, the infrared and visible images are denoised by using DCT. Second, anisotropic diffusion is applied to the denoised infrared and visible images to obtain the detail and base layers. Third, the base layers are fused by using weighted averaging, and the detail layers are fused by using the Karhunen–Loeve transform, respectively. Finally, the fused image is reconstructed through the linear superposition of the base layer and detail layer. Compared with six other typical fusion methods, the proposed approach shows better fusion performance in both objective and subjective evaluations.

Download Full-text

FuseVis: Interpreting Neural Networks for Image Fusion Using Per-Pixel Saliency Visualization

Computers ◽

10.3390/computers9040098 ◽

2020 ◽

Vol 9 (4) ◽

pp. 98

Author(s):

Nishant Kumar ◽

Stefan Gumhold

Keyword(s):

Neural Networks ◽

Image Fusion ◽

Clinical Application ◽

Medical Image ◽

Visual Analysis ◽

Input Image ◽

Visible Image ◽

Saliency Maps ◽

Fused Image ◽

Fusion Methods

Image fusion helps in merging two or more images to construct a more informative single fused image. Recently, unsupervised learning-based convolutional neural networks (CNN) have been used for different types of image-fusion tasks such as medical image fusion, infrared-visible image fusion for autonomous driving as well as multi-focus and multi-exposure image fusion for satellite imagery. However, it is challenging to analyze the reliability of these CNNs for the image-fusion tasks since no groundtruth is available. This led to the use of a wide variety of model architectures and optimization functions yielding quite different fusion results. Additionally, due to the highly opaque nature of such neural networks, it is difficult to explain the internal mechanics behind its fusion results. To overcome these challenges, we present a novel real-time visualization tool, named FuseVis, with which the end-user can compute per-pixel saliency maps that examine the influence of the input image pixels on each pixel of the fused image. We trained several image fusion-based CNNs on medical image pairs and then using our FuseVis tool we performed case studies on a specific clinical application by interpreting the saliency maps from each of the fusion methods. We specifically visualized the relative influence of each input image on the predictions of the fused image and showed that some of the evaluated image-fusion methods are better suited for the specific clinical application. To the best of our knowledge, currently, there is no approach for visual analysis of neural networks for image fusion. Therefore, this work opens a new research direction to improve the interpretability of deep fusion networks. The FuseVis tool can also be adapted in other deep neural network-based image processing applications to make them interpretable.

Download Full-text

Infrared and Visible Image Fusion with Hybrid Image Filtering

Mathematical Problems in Engineering ◽

10.1155/2020/1757214 ◽

2020 ◽

Vol 2020 ◽

pp. 1-17

Author(s):

Yongxin Zhang ◽

Deguang Li ◽

WenPeng Zhu

Keyword(s):

Thermal Radiation ◽

Image Fusion ◽

Image Filtering ◽

Divide And Conquer ◽

Base Layer ◽

Visible Image ◽

Composite Image ◽

Fusion Methods ◽

Visible Images ◽

Scene Information

Image fusion is an important technique aiming to generate a composite image from multiple images of the same scene. Infrared and visible images can provide the same scene information from different aspects, which is useful for target recognition. But the existing fusion methods cannot well preserve the thermal radiation and appearance information simultaneously. Thus, we propose an infrared and visible image fusion method by hybrid image filtering. We represent the fusion problem with a divide and conquer strategy. A Gaussian filter is used to decompose the source images into base layers and detail layers. An improved co-occurrence filter fuses the detail layers for preserving the thermal radiation of the source images. A guided filter fuses the base layers for retaining the background appearance information of the source images. Superposition of the fused base layer and fused detail layer generates the final fusion image. Subjective visual and objective quantitative evaluations comparing with other fusion algorithms demonstrate the better performance of the proposed method.

Download Full-text

Patch-Wise Infrared and Visible Image Fusion Using Spatial Adaptive Weights

Applied Sciences ◽

10.3390/app11199255 ◽

2021 ◽

Vol 11 (19) ◽

pp. 9255

Author(s):

Syeda Minahil ◽

Jun-Hyung Kim ◽

Youngbae Hwang

Keyword(s):

Image Fusion ◽

Infrared Image ◽

Source Image ◽

Infrared Images ◽

Generative Adversarial Network ◽

Visible Image ◽

Significant Information ◽

Adaptive Weights ◽

Adversarial Network ◽

Fusion Methods

In infrared (IR) and visible image fusion, the significant information is extracted from each source image and integrated into a single image with comprehensive data. We observe that the salient regions in the infrared image contain targets of interests. Therefore, we enforce spatial adaptive weights derived from the infrared images. In this paper, a Generative Adversarial Network (GAN)-based fusion method is proposed for infrared and visible image fusion. Based on the end-to-end network structure with dual discriminators, a patch-wise discrimination is applied to reduce blurry artifact from the previous image-level approaches. A new loss function is also proposed to use constructed weight maps which direct the adversarial training of GAN in a manner such that the informative regions of the infrared images are preserved. Experiments are performed on the two datasets and ablation studies are also conducted. The qualitative and quantitative analysis shows that we achieve competitive results compared to the existing fusion methods.

Download Full-text

An Improved Biologically-Inspired Image Fusion Method

International Journal of Pattern Recognition and Artificial Intelligence ◽

10.1142/s0218001418570045 ◽

2018 ◽

Vol 32 (08) ◽

pp. 1857004 ◽

Cited By ~ 1

Author(s):

Yuqing Wang ◽

Yong Wang

Keyword(s):

Receptive Field ◽

Image Fusion ◽

Color Image ◽

Infrared Image ◽

Fusion Method ◽

Fusion Model ◽

Biologically Inspired ◽

Visible Image ◽

Fusion Methods ◽

Image Fusion Method

A biologically inspired image fusion mechanism is analyzed in this paper. A pseudo-color image fusion method is proposed based on the improvement of a traditional method. The proposed model describes the fusion process using several abstract definitions which correspond to the detailed behaviors of neurons. Firstly, the infrared image and visible image are respectively ON against enhanced and OFF against enhanced. Secondly, we feed back the enhanced visible images given by the ON-antagonism system to the active cells in the center-surrounding antagonism receptive field. The fused [Formula: see text]VIS[Formula: see text]IR signal are obtained by feeding back the OFF-enhanced infrared image to the corresponding surrounding-depressing neurons. Then we feed back the enhanced visible signal from OFF-antagonism system to the depressing cells in the center-surrounding antagonism receptive field. The ON-enhanced infrared image is taken as the input signal of the corresponding active cells in the neurons, then the cell response of infrared-enhance-visible is produced in the process, it is denoted as [Formula: see text]IR[Formula: see text]VIS. The three kinds of signal are considered as R, G and B components in the output composite image. Finally, some experiments are performed in order to evaluate the performance of the proposed method. The information entropy, average gradient and objective image fusion measure are used to assess the performance of the proposed method objectively. Some traditional digital signal processing-based fusion methods are also evaluated for comparison in the experiments. In this paper, the Quantitative assessment indices show that the proposed fusion model is superior to the classical Waxman’s model, and some of its performance is better than the other image fusion methods.

Download Full-text

Infrared and visible image fusion based on optimal segmenting and contour extraction

SN Applied Sciences ◽

10.1007/s42452-020-04050-w ◽

2021 ◽

Vol 3 (3) ◽

Author(s):

Javad Abbasi Aghamaleki ◽

Alireza Ghorbani

Keyword(s):

Image Fusion ◽

Infrared Image ◽

Fusion Method ◽

Visible Image ◽

Output Image ◽

Contour Lines ◽

Fused Image ◽

Fusion Methods ◽

Visible Images ◽

The Individual

AbstractImage fusion is the combining process of complementary information of multiple same scene images into an output image. The resultant output image that is named fused image, produces more precise description of the scene than any of the individual input images. In this paper, we propose a novel simple and fast strategy for infrared (IR) and visible images based on local important areas of IR image. The fusion method is completed in three step approach. Firstly, only the segmented regions in the infrared image is extracted. Next, the image fusion is applied on segmented area and finally, contour lines are also used to improve the quality of the results of the second step of fusion method. Using a publicly available database, the proposed method is evaluated and compared to the other fusion methods. The experimental results show the effectiveness of the proposed method compared to the state of the art methods.

Download Full-text