scholarly journals FuseVis: Interpreting Neural Networks for Image Fusion Using Per-Pixel Saliency Visualization

Computers ◽  
2020 ◽  
Vol 9 (4) ◽  
pp. 98
Author(s):  
Nishant Kumar ◽  
Stefan Gumhold
Keyword(s):  
Neural Networks ◽  
Image Fusion ◽  
Clinical Application ◽  
Medical Image ◽  
Visual Analysis ◽  
Input Image ◽  
Visible Image ◽  
Saliency Maps ◽  
Fused Image ◽  
Fusion Methods

Image fusion helps in merging two or more images to construct a more informative single fused image. Recently, unsupervised learning-based convolutional neural networks (CNN) have been used for different types of image-fusion tasks such as medical image fusion, infrared-visible image fusion for autonomous driving as well as multi-focus and multi-exposure image fusion for satellite imagery. However, it is challenging to analyze the reliability of these CNNs for the image-fusion tasks since no groundtruth is available. This led to the use of a wide variety of model architectures and optimization functions yielding quite different fusion results. Additionally, due to the highly opaque nature of such neural networks, it is difficult to explain the internal mechanics behind its fusion results. To overcome these challenges, we present a novel real-time visualization tool, named FuseVis, with which the end-user can compute per-pixel saliency maps that examine the influence of the input image pixels on each pixel of the fused image. We trained several image fusion-based CNNs on medical image pairs and then using our FuseVis tool we performed case studies on a specific clinical application by interpreting the saliency maps from each of the fusion methods. We specifically visualized the relative influence of each input image on the predictions of the fused image and showed that some of the evaluated image-fusion methods are better suited for the specific clinical application. To the best of our knowledge, currently, there is no approach for visual analysis of neural networks for image fusion. Therefore, this work opens a new research direction to improve the interpretability of deep fusion networks. The FuseVis tool can also be adapted in other deep neural network-based image processing applications to make them interpretable.

scholarly journals A Review of Multimodal Medical Image Fusion Techniques

2020 ◽  
Vol 2020 ◽  
pp. 1-16 ◽  
Author(s):  
Bing Huang ◽  
Feng Yang ◽  
Mengxiao Yin ◽  
Xiaoying Mo ◽  
Cheng Zhong
Keyword(s):  
Image Fusion ◽  
Medical Image ◽  
Development Trend ◽  
Research Field ◽  
Imaging Modalities ◽  
Data Set ◽  
Medical Image Fusion ◽  
Multiple Imaging ◽  
Fused Image ◽  
Fusion Methods

The medical image fusion is the process of coalescing multiple images from multiple imaging modalities to obtain a fused image with a large amount of information for increasing the clinical applicability of medical images. In this paper, we attempt to give an overview of multimodal medical image fusion methods, putting emphasis on the most recent advances in the domain based on (1) the current fusion methods, including based on deep learning, (2) imaging modalities of medical image fusion, and (3) performance analysis of medical image fusion on mainly data set. Finally, the conclusion of this paper is that the current multimodal medical image fusion research results are more significant and the development trend is on the rise but with many challenges in the research field.

scholarly journals Infrared and Visible Image Fusion through Details Preservation

Sensors ◽  
2019 ◽  
Vol 19 (20) ◽  
pp. 4556 ◽  
Author(s):  
Yaochen Liu ◽  
Lili Dong ◽  
Yuanyuan Ji ◽  
Wenhai Xu
Keyword(s):  
Image Fusion ◽  
Subjective Evaluation ◽  
Objective Assessment ◽  
Decomposition Methods ◽  
Source Image ◽  
Fusion Method ◽  
Visible Image ◽  
Weighting Method ◽  
Fused Image ◽  
Fusion Methods

In many actual applications, fused image is essential to contain high-quality details for achieving a comprehensive representation of the real scene. However, existing image fusion methods suffer from loss of details because of the error accumulations of sequential tasks. This paper proposes a novel fusion method to preserve details of infrared and visible images by combining new decomposition, feature extraction, and fusion scheme. For decomposition, different from the most decomposition methods by guided filter, the guidance image contains only the strong edge of the source image but no other interference information so that rich tiny details can be decomposed into the detailed part. Then, according to the different characteristics of infrared and visible detail parts, a rough convolutional neural network (CNN) and a sophisticated CNN are designed so that various features can be fully extracted. To integrate the extracted features, we also present a multi-layer features fusion strategy through discrete cosine transform (DCT), which not only highlights significant features but also enhances details. Moreover, the base parts are fused by weighting method. Finally, the fused image is obtained by adding the fused detail and base part. Different from the general image fusion methods, our method not only retains the target region of source image but also enhances background in the fused image. In addition, compared with state-of-the-art fusion methods, our proposed fusion method has many advantages, including (i) better visual quality of fused-image subjective evaluation, and (ii) better objective assessment for those images.

scholarly journals Infrared and Visible Image Fusion Using Truncated Huber Penalty Function Smoothing and Visual Saliency Based Threshold Optimization

Electronics ◽  
2021 ◽  
Vol 11 (1) ◽  
pp. 33
Author(s):  
Chaowei Duan ◽  
Yiliu Liu ◽  
Changda Xing ◽  
Zhisheng Wang
Keyword(s):  
Image Fusion ◽  
Penalty Function ◽  
Visual Saliency ◽  
Fusion Rule ◽  
Visible Image ◽  
Threshold Optimization ◽  
Fused Image ◽  
Fusion Methods ◽  
Visible Images ◽  
Image Representations

An efficient method for the infrared and visible image fusion is presented using truncated Huber penalty function smoothing and visual saliency based threshold optimization. The method merges complementary information from multimodality source images into a more informative composite image in two-scale domain, in which the significant objects/regions are highlighted and rich feature information is preserved. Firstly, source images are decomposed into two-scale image representations, namely, the approximate and residual layers, using truncated Huber penalty function smoothing. Benefiting from the edge- and structure-preserving characteristics, the significant objects and regions in the source images are effectively extracted without halo artifacts around the edges. Secondly, a visual saliency based threshold optimization fusion rule is designed to fuse the approximate layers aiming to highlight the salient targets in infrared images and remain the high-intensity regions in visible images. The sparse representation based fusion rule is adopted to fuse the residual layers with the goal of acquiring rich detail texture information. Finally, combining the fused approximate and residual layers reconstructs the fused image with more natural visual effects. Sufficient experimental results demonstrate that the proposed method can achieve comparable or superior performances compared with several state-of-the-art fusion methods in visual results and objective assessments.

scholarly journals Medical image fusion based on variational and nonlinear structure tensor

2018 ◽  
Vol 189 ◽  
pp. 10021
Author(s):  
Xiaobei Wang ◽  
Rencan Nie ◽  
Xiaopeng Guo
Keyword(s):  
Image Fusion ◽  
Medical Image ◽  
Energy Functional ◽  
Structure Tensor ◽  
Fusion Method ◽  
Edge Information ◽  
Medical Image Fusion ◽  
Fused Image ◽  
New Energy ◽  
Fusion Methods

Medical image fusion plays an important role in detection and treatment of disease. Although numerous medical image fusion methods have been proposed, most of them decrease the contrast and lose the image information. In this paper, a novel MRI and CT image fusion method is proposed combining rolling guidance filter, structure tensor, and nonsubsampled shearlet transform (NSST). First, the rolling guidance filter and the sum-modified laplacian (SML) operator are introduced in the algorithm to construct the weight maps in non-linear domain, then the fused gradient is firstly obtained by a new weighted structure tensor fusion method, and the fused image is firstly acquired in NSST domain, finally, a new energy functional is defined to constrain the gradient and pixel information of the final fused image close to the pre-fused gradient and the pre-fused image, experimental results show that the proposed method can retain the edge information of source images effectively and preserve the reduction of contrast.

scholarly journals Multimodal Medical Image Fusion by Adaptive Manifold Filter

2015 ◽  
Vol 2015 ◽  
pp. 1-9 ◽  
Author(s):  
Peng Geng ◽  
Shuaiqi Liu ◽  
Shanna Zhuang
Keyword(s):  
Image Fusion ◽  
Medical Image ◽  
Low Frequency ◽  
Filter Method ◽  
Local Contrast ◽  
Complex Wavelet Transform ◽  
Image Guided Radiotherapy ◽  
Medical Image Fusion ◽  
Fused Image ◽  
Fusion Methods

Medical image fusion plays an important role in diagnosis and treatment of diseases such as image-guided radiotherapy and surgery. The modified local contrast information is proposed to fuse multimodal medical images. Firstly, the adaptive manifold filter is introduced into filtering source images as the low-frequency part in the modified local contrast. Secondly, the modified spatial frequency of the source images is adopted as the high-frequency part in the modified local contrast. Finally, the pixel with larger modified local contrast is selected into the fused image. The presented scheme outperforms the guided filter method in spatial domain, the dual-tree complex wavelet transform-based method, nonsubsampled contourlet transform-based method, and four classic fusion methods in terms of visual quality. Furthermore, the mutual information values by the presented method are averagely 55%, 41%, and 62% higher than the three methods and those values of edge based similarity measure by the presented method are averagely 13%, 33%, and 14% higher than the three methods for the six pairs of source images.

scholarly journals A Comprehensive Review of Multimodal Medical Image Fusion Techniques

2020 ◽  
pp. 407-410
Author(s):  
Jakir Hussain G K ◽  
Tamilanban R ◽  
Tamilselvan K S ◽  
Vinoth Saravanan M
Keyword(s):  
Image Fusion ◽  
Medical Image ◽  
Pedestrian Detection ◽  
Relevant Information ◽  
Medical Diagnostics ◽  
Optimization Techniques ◽  
Medical Image Fusion ◽  
Multiple Imaging ◽  
Fused Image ◽  
Fusion Methods

The multimodal image fusion is the process of combining relevant information from multiple imaging modalities. A fused image which contains recovering description than the one provided by any image fusion techniques are most widely used for real-world applications like agriculture, robotics and informatics, aeronautical, military, medical, pedestrian detection, etc. We try to give an outline of multimodal medical image fusion methods, developed during the period of time. The fusion of medical images in many combinations assists in utilizing it for medical diagnostics and examination. There is an incredible progress within the fields of deep learning, AI and bio-inspired optimization techniques. Effective utilization of these techniques is often used to further improve the effectiveness of image fusion algorithms.

scholarly journals The Fusion of MRI and CT Medical Images Using Variational Mode Decomposition

2021 ◽  
Vol 11 (22) ◽  
pp. 10975
Author(s):  
Srinivasu Polinati ◽  
Durga Prasad Bavirisetti ◽  
Kandala N V P S Rajesh ◽  
Ganesh R Naik ◽  
Ravindra Dhuli
Keyword(s):  
Image Fusion ◽  
Medical Image ◽  
Spatial Information ◽  
Variational Mode Decomposition ◽  
Intrinsic Mode Functions ◽  
Medical Image Fusion ◽  
Mode Decomposition ◽  
Fused Image ◽  
Fusion Methods ◽  
Magnetic Resonance Imaging Mri

In medical image processing, magnetic resonance imaging (MRI) and computed tomography (CT) modalities are widely used to extract soft and hard tissue information, respectively. However, with the help of a single modality, it is very challenging to extract the required pathological features to identify suspicious tissue details. Several medical image fusion methods have attempted to combine complementary information from MRI and CT to address the issue mentioned earlier over the past few decades. However, existing methods have their advantages and drawbacks. In this work, we propose a new multimodal medical image fusion approach based on variational mode decomposition (VMD) and local energy maxima (LEM). With the help of VMD, we decompose source images into several intrinsic mode functions (IMFs) to effectively extract edge details by avoiding boundary distortions. LEM is employed to carefully combine the IMFs based on the local information, which plays a crucial role in the fused image quality by preserving the appropriate spatial information. The proposed method’s performance is evaluated using various subjective and objective measures. The experimental analysis shows that the proposed method gives promising results compared to other existing and well-received fusion methods.

Image Fusion Techniques for Different Multimodality Medical Images Based on Various Conventional and Hybrid Algorithms for Disease Analysis

2020 ◽  
pp. 159-196
Author(s):  
Rajalingam B. ◽  
Priya R. ◽  
Bhavani R. ◽  
Santhoshkumar R.
Keyword(s):  
Image Fusion ◽  
Medical Image ◽  
Medical Images ◽  
Experimental Result ◽  
Accurate Information ◽  
Limited Information ◽  
Healthcare Applications ◽  
Medical Image Fusion ◽  
Fused Image ◽  
Fusion Methods

Image fusion is the process of combining two or more images to form a single fused image, which can provide more reliable and accurate information. Over the last few decades, medical imaging plays an important role in a large number of healthcare applications including diagnosis, treatment, etc. The different modalities of medical images contain complementary information of human organs and tissues, which help the physicians to diagnose the diseases. The multimodality medical images can provide limited information. These multimodality medical images cannot provide comprehensive and accurate information. This chapter proposed and examines some of the hybrid multimodality medical image fusion methods and discusses the most essential advantages and disadvantages of these methods. The hybrid multimodal medical image fusion algorithms are used to improve the quality of fused multimodality medical image. An experimental result of proposed hybrid fusion techniques provides the fused multimodal medical images of highest quality, shortest processing time, and best visualization.

scholarly journals Medical Image Fusion Based on Rolling Guidance Filter and Spiking Cortical Model

2015 ◽  
Vol 2015 ◽  
pp. 1-9 ◽  
Author(s):  
Liu Shuaiqi ◽  
Zhao Jie ◽  
Shi Mingzhu
Keyword(s):  
Image Fusion ◽  
Medical Image ◽  
Medical Images ◽  
Cortical Model ◽  
Image Guided Radiotherapy ◽  
Medical Image Fusion ◽  
Fused Image ◽  
Fusion Methods ◽  
Mean And Variance ◽  
The Mean

Medical image fusion plays an important role in diagnosis and treatment of diseases such as image-guided radiotherapy and surgery. Although numerous medical image fusion methods have been proposed, most of these approaches are sensitive to the noise and usually lead to fusion image distortion, and image information loss. Furthermore, they lack universality when dealing with different kinds of medical images. In this paper, we propose a new medical image fusion to overcome the aforementioned issues of the existing methods. It is achieved by combining with rolling guidance filter (RGF) and spiking cortical model (SCM). Firstly, saliency of medical images can be captured by RGF. Secondly, a self-adaptive threshold of SCM is gained by utilizing the mean and variance of the source images. Finally, fused image can be gotten by SCM motivated by RGF coefficients. Experimental results show that the proposed method is superior to other current popular ones in both subjectively visual performance and objective criteria.

scholarly journals Medical Image Fusion Using Transform Based Fusion Techniques

2019 ◽  
Vol 8 (2) ◽  
pp. 309-317
Keyword(s):  
Wavelet Transform ◽  
Image Fusion ◽  
Medical Image ◽  
Value Added ◽  
Discrete Wavelet ◽  
Discrete Cosine Transforms ◽  
Fused Image ◽  
Fusion Methods ◽  
Consistency Verification ◽  
Tomography Scan

The principal resolution of the image fusion is to merging indication from different images; CT (Computed Tomography) scan and an MRI (Magnetic Resonance Imaging) and to obtain more informative image. In this paper various transform based fusion methods like; discrete wavelet transform (DWT) and two specialisms of discrete cosine transform (DCT); DCT variance and DCT variance with consistency verification (DCT variance with CV) and stationary wavelet transform (SWT) image fusion procedures are instigated and associated in terms of image evidence. Fused outcomes attained from these fusion techniques are evaluated through distinctive evaluation metrics. A fused result accomplished from DCT variance with CV followed by DCT variance out performs DWT and SWT based image fusion methodologies. The potentiality of DCT features creates value-added evidence in the output fused image trailed by fused results proficient from DWT and SWT based image fusion methods. The discrete cosine transforms (DCT) stranded methods of image fusion are auxiliary accurate and concert leaning in real time solicitations by energy forte of DCT originated ideologies of stationary images. In this effort, a glowing systematic practice for fusion of multi-focus images based on DCT and its flavors are obtainable and demonstrated that DCT grounded fused outcomes exceed other fusion methodologies

Sign in / Sign up

Export Citation Format

Share Document