Fusion Based AER System Using Deep Learning Approach for Amplitude and Frequency Analysis

Automatic emotion recognition from Speech (AERS) systems based on acoustical analysis reveal that some emotional classes persist with ambiguity. This study employed an alternative method aimed at providing deep understanding into the amplitude–frequency, impacts of various emotions in order to aid in the advancement of near term, more effectively in classifying AER approaches. The study was undertaken by converting narrow 20 ms frames of speech into RGB or grey-scale spectrogram images. The features have been used to fine-tune a feature selection system that had previously been trained to recognise emotions. Two different Linear and Mel spectral scales are used to demonstrate a spectrogram. An inductive approach for in sighting the amplitude and frequency features of various emotional classes. We propose a two-channel profound combination of deep fusion network model for the efficient categorization of images. Linear and Mel- spectrogram is acquired from Speech-signal, which is prepared in the recurrence area to input Deep Neural Network. The proposed model Alex-Net with five convolutional layers and two fully connected layers acquire most vital features form spectrogram images plotted on the amplitude-frequency scale. The state-of-the-art is compared with benchmark dataset (EMO-DB). RGB and saliency images are fed to pre-trained Alex-Net tested both EMO-DB and Telugu dataset with an accuracy of 72.18% and fused image features less computations reaching to an accuracy 75.12%. The proposed model show that Transfer learning predict efficiently than Fine-tune network. When tested on Emo-DB dataset, the propȯsed system adequately learns discriminant features from speech spectrȯgrams and outperforms many stȧte-of-the-art techniques.

Data fusion and 3D visualization for optimized representation of neurovascular relationships in the posterior fossa

Background Reliable 3D visualization of neurovascular relationships in the posterior fossa at the surface of the brainstem is still critical due to artifacts of imaging. To assess neurovascular compression syndromes more reliably, a new approach of 3D visualization based on registration and fusion of high-resolution MR data is presented. Methods A total of 80 patients received MRI data with 3D-CISS and 3D-TOF at 3.0 Tesla. After registration and subsequent segmentation, the vascular information of the TOF data was fused into the CISS data. Two 3D visualizations were created for each patient, one before and one after fusion, which were verified with the intraoperative situation during microvascular decompression (MVD). The reproduction quality of vessels was evaluated with a rating system. Results In all cases, the presented approach compensated for typical limitations in the 3D visualization of neurovascular compression such as the partial or complete suppression of larger vessels, suppression of smaller vessels at the CSF margin, and artifacts from heart pulsation. In more than 95% of the cases of hemifacial spasm and glossopharyngeal neuralgia, accurate assessment of the compression was only possible after registration and fusion. In more than 50% of the cases with trigeminal neuralgia, the presented approach was crucial to finding the actually offending vessel. Conclusions 3D visualization of fused image data allows for a more complete representation of the vessel-nerve situation. The results from this approach are reproducible and the assessment of neurovascular compression is safer. It is a powerful tool for planning MVD.

Context-Fused Guidance for Image Captioning Using Sequence-Level Training

Recent image captioning models based on the encoder-decoder framework have achieved remarkable success in humanlike sentence generation. However, an explicit separation between encoder and decoder brings out a disconnection between the image and sentence. It usually leads to a rough image description: the generated caption only contains main instances but neglects additional objects and scenes unexpectedly, which reduces the caption consistency of the image. To address this issue, we proposed an image captioning system within context-fused guidance in this paper. It incorporates regional and global image representation as the compositional visual features to learn the objects and attributes in images. To integrate image-level semantic information, the visual concept is employed. To avoid misleading decoding, a context fusion gate is introduced to calculate the textual context by selectively aggregating the information of visual concept and word embedding. Subsequently, the context-fused image guidance is formulated based on the compositional visual features and textual context. It provides the decoder with informative semantic knowledge. Finally, a captioner with a two-layer LSTM architecture is constructed to generate captions. Moreover, to overcome the exposure bias, we train the proposed model through sequence decision-making. The experiments conducted on the MS COCO dataset show the outstanding performance of our work. The linguistic analysis demonstrates that our model improves the caption consistency of the image.

Analysis of Discrete Wavelet Transforms Variants for the Fusion of CT and MRI Images

Background: Medical image fusion methods are applied to a wide assortment of medical fields, for example, computer-assisted diagnosis, telemedicine, radiation treatment, preoperative planning, and so forth. Computed Tomography (CT) is utilized to scan the bone structure, while Magnetic Resonance Imaging (MRI) is utilized to examine the soft tissues of the cerebrum. The fusion of the images obtained from the two modalities helps radiologists diagnose the abnormalities in the brain and localize the position of the abnormality concerning the bone. Methods: Multimodal medical image fusion procedure contributes to the decrease of information vulnerability and improves the clinical diagnosis exactness. The motive is to protect salient features from multiple source images to produce an upgraded fused image. The CT-MRI image fusion study made it conceivable to analyze the two modalities straightforwardly. Several states of the art techniques are available for the fusion of CT & MRI images. The discrete wavelet transform (DWT) is one of the widely used transformation techniques for the fusion of images. However, the efficacy of utilization of the variants of wavelet filters for the decomposition of the images, which may improve the image fusion quality, has not been studied in detail. Therefore the objective of this study is to assess the utility of wavelet families for the fusion of CT and MRI images. In this paper investigation on the efficacy of 8 wavelet families (120 family members) on the visual quality of the fused CT & MRI image has been performed. Further, to strengthen the quality of the fused image, two quantitative performance evaluation parameters, namely classical and gradient information, have been calculated. Results: Experimental results demonstrate that amongst the 120 wavelet family members (8 wavelet families), db1, rbio1.1, and Haar wavelets have outperformed other wavelet family members in both qualitative and quantitative analysis. Conclusion: Quantitative and qualitative analysis shows that the fused image may help radiologists diagnose the abnormalities in the brain and localize the position of the abnormality concerning the bone more easily. For further improvement in the fused results, methods based on deep learning may be tested in the future.

Deep Learning-Based Thermal Image Analysis for Pavement Defect Detection and Classification Considering Complex Pavement Conditions

Automatic damage detection using deep learning warrants an extensive data source that captures complex pavement conditions. This paper proposes a thermal-RGB fusion image-based pavement damage detection model, wherein the fused RGB-thermal image is formed through multi-source sensor information to achieve fast and accurate defect detection including complex pavement conditions. The proposed method uses pre-trained EfficientNet B4 as the backbone architecture and generates an argument dataset (containing non-uniform illumination, camera noise, and scales of thermal images too) to achieve high pavement damage detection accuracy. This paper tests separately the performance of different input data (RGB, thermal, MSX, and fused image) to test the influence of input data and network on the detection results. The results proved that the fused image’s damage detection accuracy can be as high as 98.34% and by using the dataset after augmentation, the detection model deems to be more stable to achieve 98.35% precision, 98.34% recall, and 98.34% F1-score.

Infrared and Visible Image Fusion Using Truncated Huber Penalty Function Smoothing and Visual Saliency Based Threshold Optimization

An efficient method for the infrared and visible image fusion is presented using truncated Huber penalty function smoothing and visual saliency based threshold optimization. The method merges complementary information from multimodality source images into a more informative composite image in two-scale domain, in which the significant objects/regions are highlighted and rich feature information is preserved. Firstly, source images are decomposed into two-scale image representations, namely, the approximate and residual layers, using truncated Huber penalty function smoothing. Benefiting from the edge- and structure-preserving characteristics, the significant objects and regions in the source images are effectively extracted without halo artifacts around the edges. Secondly, a visual saliency based threshold optimization fusion rule is designed to fuse the approximate layers aiming to highlight the salient targets in infrared images and remain the high-intensity regions in visible images. The sparse representation based fusion rule is adopted to fuse the residual layers with the goal of acquiring rich detail texture information. Finally, combining the fused approximate and residual layers reconstructs the fused image with more natural visual effects. Sufficient experimental results demonstrate that the proposed method can achieve comparable or superior performances compared with several state-of-the-art fusion methods in visual results and objective assessments.

Artificial Intelligence-Based Deep Fusion Model for Pan-Sharpening of Remote Sensing Images

During the past two decades, many remote sensing image fusion techniques have been designed to improve the spatial resolution of the low-spatial-resolution multispectral bands. The main objective is fuse the low-resolution multispectral (MS) image and the high-spatial-resolution panchromatic (PAN) image to obtain a fused image having high spatial and spectral information. Recently, many artificial intelligence-based deep learning models have been designed to fuse the remote sensing images. But these models do not consider the inherent image distribution difference between MS and PAN images. Therefore, the obtained fused images may suffer from gradient and color distortion problems. To overcome these problems, in this paper, an efficient artificial intelligence-based deep transfer learning model is proposed. Inception-ResNet-v2 model is improved by using a color-aware perceptual loss (CPL). The obtained fused images are further improved by using gradient channel prior as a postprocessing step. Gradient channel prior is used to preserve the color and gradient information. Extensive experiments are carried out by considering the benchmark datasets. Performance analysis shows that the proposed model can efficiently preserve color and gradient information in the fused remote sensing images than the existing models.

Pan-Sharpening for Spectral Details Preservation Via Convolutional Sparse Coding in Non-Subsampled Shearlet Space

The optical satellite sensors encounter certain constraints on producing high-resolution multispectral (HRMS) images. Pan-sharpening (PS) is a remote sensing image fusion technique, which is an effective mechanism to overcome the limitations of available imaging products. The prevalent issue in PS algorithms is the imbalance between spatial quality and spectral details preservation, thereby producing intensity variations in the fused image. In this paper, a PS method is proposed based on convolutional sparse coding (CSC) implemented in the non-subsampled shearlet transform (NSST) domain. The source images, panchromatic (PAN) and multispectral (MS) images, are decomposed using NSST. The resultant high-frequency bands are fused using adaptive weights determined from chaotic grey wolf optimization (CGWO) algorithm. The CSC-based model is employed to fuse the low-frequency bands. Further, an iterative filtering mechanism is developed to enhance the quality of fused image. Four datasets with different geographical content like urban area, vegetation, etc. and eight existing algorithms are used for evaluation of the proposed PS method. The comprehensive visual and quantitative results approve that the proposed method accomplishes considerable improvement in spatial and spectral details equivalence in the pan-sharpened image.

A Critical Analysis of the Multi-Focus Images Fusion Based on Discrete Wavelet Transform and Computer Vision

This paper focused on the challenge of image fusion processing and lack of reliable image information and proposed multi-focus image fusion using discrete wavelet transforms and computer vision techniques for the fused image coefficient selection process. I made an in-depth analysis and improvement on the existing algorithms from the wavelet transform and the rules of multi-focus image fusion object features’ extractions. The wavelet transform uses authentic localization properties, and computer vision provides efficient processing time and is a powerful method to analyze object focus in the high-frequency precision and steps. The process of image fusion using wavelet transformation is the wavelet basis function and wavelet decomposition level in iterative experiments to enhance fused image information. The rules of multi-focus image fusions are the wavelet transformation on the features of the high-frequency coefficients, which enhance the fusion image features reliability on the frequency domain and regional contrast of the object.

Color multi-focus image fusion based on transfer learning

Multi-focus image fusion is a technique that integrates the focused areas in a pair or set of source images with the same scene into a fully focused image. Inspired by transfer learning, this paper proposes a novel color multi-focus image fusion method based on deep learning. First, color multi-focus source images are fed into VGG-19 network, and the parameters of convolutional layer of the VGG-19 network are then migrated to a neural network containing multilayer convolutional layers and multilayer skip-connection structures for feature extraction. Second, the initial decision maps are generated using the reconstructed feature maps of a deconvolution module. Third, the initial decision maps are refined and processed to obtain the second decision maps, and then the source images are fused to obtain the initial fused images based on the second decision maps. Finally, the final fused image is produced by comparing the Q ABF metrics of the initial fused images. The experimental results show that the proposed method can effectively improve the segmentation performance of the focused and unfocused areas in the source images, and the generated fused images are superior in both subjective and objective metrics compared with most contrast methods.