AttentionBased Deep Feature Fusion for the Scene Classification of HighResolution Remote Sensing Images

Scene classification of highresolution remote sensing images (HRRSI) is one of the most important means of landcover classification. Deep learning techniques, especially the convolutional neural network (CNN) have been widely applied to the scene classification of HRRSI due to the advancement of graphic processing units (GPU). However, they tend to extract features from the whole images rather than discriminative regions. The visual attention mechanism can force the CNN to focus on discriminative regions, but it may suffer from the influence of intraclass diversity and repeated texture. Motivated by these problems, we propose an attention-based deep feature fusion (ADFF) framework that constitutes three parts, namely attention maps generated by Gradientweighted Class Activation Mapping (GradCAM), a multiplicative fusion of deep features and the centerbased cross-entropy loss function. First of all, we propose to make attention maps generated by GradCAM as an explicit input in order to force the network to concentrate on discriminative regions. Then, deep features derived from original images and attention maps are proposed to be fused by multiplicative fusion in order to consider both improved abilities to distinguish scenes of repeated texture and the salient regions. Finally, the centerbased cross-entropy loss function that utilizes both the cross-entropy loss and center loss function is proposed to backpropagate fused features so as to reduce the effect of intraclass diversity on feature representations. The proposed ADFF architecture is tested on three benchmark datasets to show its performance in scene classification. The experiments confirm that the proposed method outperforms most competitive scene classification methods with an average overall accuracy of 94% under different training ratios.

Download Full-text

Retraction: Zhu R. et al. Attention-Based Deep Feature Fusion for the Scene Classification of High-Resolution Remote Sensing Images. Remote Sensing. 2019, 11(17), 1996

Remote Sensing ◽

10.3390/rs12040742 ◽

2020 ◽

Vol 12 (4) ◽

pp. 742 ◽

Cited By ~ 1

Author(s):

Ruixi Zhu ◽

Li Yan ◽

Nan Mo ◽

Yi Liu

Keyword(s):

Remote Sensing ◽

High Resolution ◽

Research Method ◽

Feature Fusion ◽

Scene Classification ◽

Remote Sensing Images ◽

Deep Feature

We have been made aware that the innovative contributions, research method and the majority of the content of this article [...]

Download Full-text

Deep Feature Fusion with Integration of Residual Connection and Attention Model for Classification of VHR Remote Sensing Images

Remote Sensing ◽

10.3390/rs11131617 ◽

2019 ◽

Vol 11 (13) ◽

pp. 1617 ◽

Cited By ~ 5

Author(s):

Jicheng Wang ◽

Li Shen ◽

Wenfan Qiao ◽

Yanshuai Dai ◽

Zhilin Li

Keyword(s):

Remote Sensing ◽

Feature Fusion ◽

Learning Ability ◽

Remote Sensing Images ◽

Convolutional Network ◽

Fully Convolutional Network ◽

Attention Model ◽

Low Level ◽

Deep Feature

The classification of very-high-resolution (VHR) remote sensing images is essential in many applications. However, high intraclass and low interclass variations in these kinds of images pose serious challenges. Fully convolutional network (FCN) models, which benefit from a powerful feature learning ability, have shown impressive performance and great potential. Nevertheless, only classification results with coarse resolution can be obtained from the original FCN method. Deep feature fusion is often employed to improve the resolution of outputs. Existing strategies for such fusion are not capable of properly utilizing the low-level features and considering the importance of features at different scales. This paper proposes a novel, end-to-end, fully convolutional network to integrate a multiconnection ResNet model and a class-specific attention model into a unified framework to overcome these problems. The former fuses multilevel deep features without introducing any redundant information from low-level features. The latter can learn the contributions from different features of each geo-object at each scale. Extensive experiments on two open datasets indicate that the proposed method can achieve class-specific scale-adaptive classification results and it outperforms other state-of-the-art methods. The results were submitted to the International Society for Photogrammetry and Remote Sensing (ISPRS) online contest for comparison with more than 50 other methods. The results indicate that the proposed method (ID: SWJ_2) ranks #1 in terms of overall accuracy, even though no additional digital surface model (DSM) data that were offered by ISPRS were used and no postprocessing was applied.

Download Full-text

SSDAN: Multi-Source Semi-Supervised Domain Adaptation Network for Remote Sensing Scene Classification

Remote Sensing ◽

10.3390/rs13193861 ◽

2021 ◽

Vol 13 (19) ◽

pp. 3861

Author(s):

Tariq Lasloum ◽

Haikel Alhichri ◽

Yakoub Bazi ◽

Naif Alajlan

Keyword(s):

Remote Sensing ◽

Loss Function ◽

Domain Adaptation ◽

Activation Function ◽

Cross Entropy ◽

Scene Classification ◽

Entropy Loss ◽

Training Approach ◽

Temperature Parameter ◽

And Control

We present a new method for multi-source semi-supervised domain adaptation in remote sensing scene classification. The method consists of a pre-trained convolutional neural network (CNN) model, namely EfficientNet-B3, for the extraction of highly discriminative features, followed by a classification module that learns feature prototypes for each class. Then, the classification module computes a cosine distance between feature vectors of target data samples and the feature prototypes. Finally, the proposed method ends with a Softmax activation function that converts the distances into class probabilities. The feature prototypes are also divided by a temperature parameter to normalize and control the classification module. The whole model is trained on both the unlabeled and labeled target samples. It is trained to predict the correct classes utilizing the standard cross-entropy loss computed over the labeled source and target samples. At the same time, the model is trained to learn domain invariant features using another loss function based on entropy computed over the unlabeled target samples. Unlike the standard cross-entropy loss, the new entropy loss function is computed on the model’s predicted probabilities and does not need the true labels. This entropy loss, called minimax loss, needs to be maximized with respect to the classification module to learn features that are domain-invariant (hence removing the data shift), and at the same time, it should be minimized with respect to the CNN feature extractor to learn discriminative features that are clustered around the class prototypes (in other words reducing intra-class variance). To accomplish these maximization and minimization processes at the same time, we use an adversarial training approach, where we alternate between the two processes. The model combines the standard cross-entropy loss and the new minimax entropy loss and optimizes them jointly. The proposed method is tested on four RS scene datasets, namely UC Merced, AID, RESISC45, and PatternNet, using two-source and three-source domain adaptation scenarios. The experimental results demonstrate the strong capability of the proposed method to achieve impressive performance despite using only a few (six in our case) labeled target samples per class. Its performance is already better than several state-of-the-art methods, including RevGrad, ADDA, Siamese-GAN, and MSCN.

Download Full-text

Scene Classification of Optical Remote Sensing Images Based on CNN Automatic Transfer

2018 IEEE International Conference on Automation, Electronics and Electrical Engineering (AUTEEE) ◽

10.1109/auteee.2018.8720785 ◽

2018 ◽

Author(s):

Jicheng Quan ◽

Chen Wu ◽

Hongwei Wang ◽

Zhiqiang Wang

Keyword(s):

Remote Sensing ◽

Optical Remote Sensing ◽

Scene Classification ◽

Remote Sensing Images

Download Full-text

Deep Discriminative Representation Learning with Attention Map for Scene Classification

Remote Sensing ◽

10.3390/rs12091366 ◽

2020 ◽

Vol 12 (9) ◽

pp. 1366 ◽

Cited By ~ 5

Author(s):

Jun Li ◽

Daoyu Lin ◽

Yang Wang ◽

Guangluan Xu ◽

Yunyan Zhang ◽

...

Keyword(s):

Remote Sensing ◽

Feature Fusion ◽

Representation Learning ◽

Classification Performance ◽

Great Success ◽

Scene Classification ◽

Remote Sensing Images ◽

Discriminative Ability ◽

Feature Representations ◽

Benchmark Datasets

In recent years, convolutional neural networks (CNNs) have shown great success in the scene classification of computer vision images. Although these CNNs can achieve excellent classification accuracy, the discriminative ability of feature representations extracted from CNNs is still limited in distinguishing more complex remote sensing images. Therefore, we propose a unified feature fusion framework based on attention mechanism in this paper, which is called Deep Discriminative Representation Learning with Attention Map (DDRL-AM). Firstly, by applying Gradient-weighted Class Activation Mapping (Grad-CAM) algorithm, attention maps associated with the predicted results are generated in order to make CNNs focus on the most salient parts of the image. Secondly, a spatial feature transformer (SFT) is designed to extract discriminative features from attention maps. Then an innovative two-channel CNN architecture is proposed by the fusion of features extracted from attention maps and the RGB (red green blue) stream. A new objective function that considers both center and cross-entropy loss are optimized to decrease the influence of inter-class dispersion and within-class variance. In order to show its effectiveness in classifying remote sensing images, the proposed DDRL-AM method is evaluated on four public benchmark datasets. The experimental results demonstrate the competitive scene classification performance of the DDRL-AM approach. Moreover, the visualization of features extracted by the proposed DDRL-AM method can prove that the discriminative ability of features has been increased.

Download Full-text

Scene classification of remote sensing images based on hierarchical sparse coding

The Journal of Engineering ◽

10.1049/joe.2018.8268 ◽

2018 ◽

Vol 2018 (16) ◽

pp. 1650-1657

Author(s):

Xu Jiaqing ◽

Lv Qi ◽

Liu Hongjun ◽

He Jie

Keyword(s):

Remote Sensing ◽

Sparse Coding ◽

Scene Classification ◽

Remote Sensing Images

Download Full-text

Learning Multi-Modality Features for Scene Classification of High-Resolution Remote Sensing Images

Journal of Computer and Communications ◽

10.4236/jcc.2018.611018 ◽

2018 ◽

Vol 06 (11) ◽

pp. 185-193

Author(s):

Feng’an Zhao ◽

Xiongmei Zhang ◽

Xiaodong Mu ◽

Zhaoxiang Yi ◽

Zhou Yang

Keyword(s):

Remote Sensing ◽

High Resolution ◽

Scene Classification ◽

Remote Sensing Images

Download Full-text

Attention-Based Pyramid Network for Segmentation and Classification of High-Resolution and Hyperspectral Remote Sensing Images

Remote Sensing ◽

10.3390/rs12213501 ◽

2020 ◽

Vol 12 (21) ◽

pp. 3501

Author(s):

Qingsong Xu ◽

Xin Yuan ◽

Chaojun Ouyang ◽

Yue Zeng

Keyword(s):

Remote Sensing ◽

High Resolution ◽

Image Classification ◽

Large Scale ◽

Feature Fusion ◽

Spectral Information ◽

Spatial Problem ◽

Remote Sensing Images ◽

Heavy Weight

Unlike conventional natural (RGB) images, the inherent large scale and complex structures of remote sensing images pose major challenges such as spatial object distribution diversity and spectral information extraction when existing models are directly applied for image classification. In this study, we develop an attention-based pyramid network for segmentation and classification of remote sensing datasets. Attention mechanisms are used to develop the following modules: (i) a novel and robust attention-based multi-scale fusion method effectively fuses useful spatial or spectral information at different and same scales; (ii) a region pyramid attention mechanism using region-based attention addresses the target geometric size diversity in large-scale remote sensing images; and (iii) cross-scale attention in our adaptive atrous spatial pyramid pooling network adapts to varied contents in a feature-embedded space. Different forms of feature fusion pyramid frameworks are established by combining these attention-based modules. First, a novel segmentation framework, called the heavy-weight spatial feature fusion pyramid network (FFPNet), is proposed to address the spatial problem of high-resolution remote sensing images. Second, an end-to-end spatial-spectral FFPNet is presented for classifying hyperspectral images. Experiments conducted on ISPRS Vaihingen and ISPRS Potsdam high-resolution datasets demonstrate the competitive segmentation accuracy achieved by the proposed heavy-weight spatial FFPNet. Furthermore, experiments on the Indian Pines and the University of Pavia hyperspectral datasets indicate that the proposed spatial-spectral FFPNet outperforms the current state-of-the-art methods in hyperspectral image classification.

Download Full-text