Hyperspectral Image Classification via a Novel Spectral–Spatial 3D ConvLSTM-CNN

In recent years, deep learning-based models have produced encouraging results for hyperspectral image (HSI) classification. Specifically, Convolutional Long Short-Term Memory (ConvLSTM) has shown good performance for learning valuable features and modeling long-term dependencies in spectral data. However, it is less effective for learning spatial features, which is an integral part of hyperspectral images. Alternatively, convolutional neural networks (CNNs) can learn spatial features, but they possess limitations in handling long-term dependencies due to the local feature extraction in these networks. Considering these factors, this paper proposes an end-to-end Spectral-Spatial 3D ConvLSTM-CNN based Residual Network (SSCRN), which combines 3D ConvLSTM and 3D CNN for handling both spectral and spatial information, respectively. The contribution of the proposed network is twofold. Firstly, it addresses the long-term dependencies of spectral dimension using 3D ConvLSTM to capture the information related to various ground materials effectively. Secondly, it learns the discriminative spatial features using 3D CNN by employing the concept of the residual blocks to accelerate the training process and alleviate the overfitting. In addition, SSCRN uses batch normalization and dropout to regularize the network for smooth learning. The proposed framework is evaluated on three benchmark datasets widely used by the research community. The results confirm that SSCRN outperforms state-of-the-art methods with an overall accuracy of 99.17%, 99.67%, and 99.31% over Indian Pines, Salinas, and Pavia University datasets, respectively. Moreover, it is worth mentioning that these excellent results were achieved with comparatively fewer epochs, which also confirms the fast learning capabilities of the SSCRN.

Download Full-text

HYPERSPECTRAL IMAGE CLASSIFICATION USING RESIDUAL 2D AND 3D CONVOLUTIONAL NEURAL NETWORK JOINT ATTENTION MODEL

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xliv-m-3-2021-187-2021 ◽

2021 ◽

Vol XLIV-M-3-2021 ◽

pp. 187-193

Author(s):

Q. Yuan ◽

Y. Ang ◽

H. Z. M. Shafri

Keyword(s):

Neural Network ◽

Remote Sensing ◽

Classification Accuracy ◽

Spatial Information ◽

Hyperspectral Image ◽

Spectral Features ◽

Hyperspectral Image Classification ◽

Spatial Features ◽

3D Cnn ◽

2D And 3D

Abstract. Hyperspectral image classification (HSIC) is a challenging task in remote sensing data analysis, which has been applied in many domains for better identification and inspection of the earth surface by extracting spectral and spatial information. The combination of abundant spectral features and accurate spatial information can improve classification accuracy. However, many traditional methods are based on handcrafted features, which brings difficulties for multi-classification tasks due to spectral intra-class heterogeneity and similarity of inter-class. The deep learning algorithm, especially the convolutional neural network (CNN), has been perceived promising feature extractor and classification for processing hyperspectral remote sensing images. Although 2D CNN can extract spatial features, the specific spectral properties are not used effectively. While 3D CNN has the capability for them, but the computational burden increases as stacking layers. To address these issues, we propose a novel HSIC framework based on the residual CNN network by integrating the advantage of 2D and 3D CNN. First, 3D convolutions focus on extracting spectral features with feature recalibration and refinement by channel attention mechanism. The 2D depth-wise separable convolution approach with different size kernels concentrates on obtaining multi-scale spatial features and reducing model parameters. Furthermore, the residual structure optimizes the back-propagation for network training. The results and analysis of extensive HSIC experiments show that the proposed residual 2D-3D CNN network can effectively extract spectral and spatial features and improve classification accuracy.

Download Full-text

3-D Convolution-Recurrent Networks for Spectral-Spatial Classification of Hyperspectral Images

Remote Sensing ◽

10.3390/rs11070883 ◽

2019 ◽

Vol 11 (7) ◽

pp. 883 ◽

Cited By ~ 10

Author(s):

Majid Seydgar ◽

Amin Alizadeh Naeini ◽

Mengmeng Zhang ◽

Wei Li ◽

Mehran Satari

Keyword(s):

Spatial Information ◽

Short Term Memory ◽

Structural Data ◽

Experimental Results ◽

Spatial Feature ◽

Computational Burden ◽

Spatial Classification ◽

Spatial Features ◽

Proposed Model

Nowadays, 3-D convolutional neural networks (3-D CNN) have attracted lots of attention in the spectral-spatial classification of hyperspectral imageries (HSI). In this model, the feed-forward processing structure reduces the computational burden of 3-D structural processing. However, this model as a vector-based methodology cannot analyze the full content of the HSI information, and as a result, its features are not quite discriminative. On the other hand, convolutional long short-term memory (CLSTM) can recurrently analyze the 3-D structural data to extract more discriminative and abstract features. However, the computational burden of this model as a sequence-based methodology is extremely high. In the meanwhile, the robust spectral-spatial feature extraction with a reasonable computational burden is of great interest in HSI classification. For this purpose, a two-stage method based on the integration of CNN and CLSTM is proposed. In the first stage, 3-D CNN is applied to extract low-dimensional shallow spectral-spatial features from HSI, where information on the spatial features are less than that of the spectral information; consequently, in the second stage, the CLSTM, for the first time, is applied to recurrently analyze the spatial information while considering the spectral one. The experimental results obtained from three widely used HSI datasets indicate that the application of the recurrent analysis for spatial feature extractions makes the proposed model robust against different spatial sizes of the extracted patches. Moreover, applying the 3-D CNN prior to the CLSTM efficiently reduces the model’s computational burden. The experimental results also indicated that the proposed model led to a 1% to 2% improvement compared to its counterpart models.

Download Full-text

Deficits in Verbal Long-Term Memory and Learning in Children with Poor Phonological Short-Term Memory Skills

Quarterly Journal of Experimental Psychology ◽

10.1080/17470210701273443 ◽

2008 ◽

Vol 61 (3) ◽

pp. 474-490 ◽

Cited By ~ 23

Author(s):

Susan E. Gathercole ◽

Josie Briscoe ◽

Annabel Thorn ◽

Claire Tiffany ◽

Keyword(s):

Spatial Information ◽

Short Term Memory ◽

Memory Function ◽

Verbal Material ◽

Long Term Memory ◽

Short Term ◽

Memory And Learning ◽

Term Memory ◽

Memory Group

Possible links between phonological short-term memory and both longer term memory and learning in 8-year-old children were investigated in this study. Performance on a range of tests of long-term memory and learning was compared for a group of 16 children with poor phonological short-term memory skills and a comparison group of children of the same age with matched nonverbal reasoning abilities but memory scores in the average range. The low-phonological-memory group were impaired on longer term memory and learning tasks that taxed memory for arbitrary verbal material such as names and nonwords. However, the two groups performed at comparable levels on tasks requiring the retention of visuo-spatial information and of meaningful material and at carrying out prospective memory tasks in which the children were asked to carry out actions at a future point in time. The results are consistent with the view that poor short-term memory function impairs the longer term retention and ease of learning of novel verbal material.

Download Full-text

The Contribution of the Interannual ENSO Cycle to the Spatial Pattern of Decadal ENSO-Like Variability*

Journal of Climate ◽

10.1175/jcli3365.1 ◽

2005 ◽

Vol 18 (12) ◽

pp. 2080-2092 ◽

Cited By ~ 92

Author(s):

Daniel J. Vimont

Keyword(s):

Spatial Structure ◽

Spatial Information ◽

Decadal Variability ◽

Enso Event ◽

Enso Cycle ◽

Physical Explanation ◽

Enso Variability ◽

Physical Mechanisms ◽

Spatial Features

Abstract A defining feature of Pacific decadal ENSO-like variability is the similarity between its spatial expression in sea surface temperature (SST) and the spatial structure of interannual ENSO variability. This similarity may indicate that the decadal variability is merely a long-term average over interannual ENSO variability. In contrast, subtle differences (namely the meridionally broadened tropical SST signature and emphasized midlatitude SST anomalies for the decadal ENSO-like pattern) may indicate that fundamentally different processes are responsible for generating variability on the decadal to interdecadal time scale. The present study attempts to reconcile the subtly different spatial structures of interannual ENSO and decadal ENSO-like variability by relating the decadal pattern to various SST patterns associated with the development of the interannual ENSO cycle. First, a statistical analysis is used to reconstruct the decadal ENSO-like SST pattern as a linear combination of interannual SST patterns. It is shown that the decadal ENSO-like pattern is well reconstructed in the absence of decadal spatial information. Next, these interannual patterns are physically interpreted in relation to the interannual ENSO cycle. The analysis reveals that the decadal ENSO-like SST pattern is obtained by averaging over three SST patterns associated with ENSO precursors, the peak of an ENSO event, and ENSO “leftovers.” The study provides a plausible physical explanation for the spatial structure of ENSO-like decadal variability as an average over variations in the interannual ENSO cycle. The results indicate that the prominent spatial features of decadal ENSO-like variability are generated by physical mechanisms that operate through the interannual ENSO cycle. This does not imply, however, that decadal processes are unimportant in altering the decadal properties of ENSO. Results may provide a framework for interpreting modeled decadal ENSO-like variability and for constraining plausible mechanisms of tropical decadal variability.

Download Full-text

Multiscale Information Fusion for Hyperspectral Image Classification Based on Hybrid 2D-3D CNN

Remote Sensing ◽

10.3390/rs13122268 ◽

2021 ◽

Vol 13 (12) ◽

pp. 2268

Author(s):

Hang Gong ◽

Qiuxia Li ◽

Chunlai Li ◽

Haishan Dai ◽

Zhiping He ◽

...

Keyword(s):

Classification Accuracy ◽

Spatial Information ◽

Hyperspectral Image ◽

Small Sample ◽

Hyperspectral Images ◽

Sample Problem ◽

Training Samples ◽

Small Sample Problem ◽

3D Cnn ◽

Hyperspectral Classification

Hyperspectral images are widely used for classification due to its rich spectral information along with spatial information. To process the high dimensionality and high nonlinearity of hyperspectral images, deep learning methods based on convolutional neural network (CNN) are widely used in hyperspectral classification applications. However, most CNN structures are stacked vertically in addition to using a onefold size of convolutional kernels or pooling layers, which cannot fully mine the multiscale information on the hyperspectral images. When such networks meet the practical challenge of a limited labeled hyperspectral image dataset—i.e., “small sample problem”—the classification accuracy and generalization ability would be limited. In this paper, to tackle the small sample problem, we apply the semantic segmentation function to the pixel-level hyperspectral classification due to their comparability. A lightweight, multiscale squeeze-and-excitation pyramid pooling network (MSPN) is proposed. It consists of a multiscale 3D CNN module, a squeezing and excitation module, and a pyramid pooling module with 2D CNN. Such a hybrid 2D-3D-CNN MSPN framework can learn and fuse deeper hierarchical spatial–spectral features with fewer training samples. The proposed MSPN was tested on three publicly available hyperspectral classification datasets: Indian Pine, Salinas, and Pavia University. Using 5%, 0.5%, and 0.5% training samples of the three datasets, the classification accuracies of the MSPN were 96.09%, 97%, and 96.56%, respectively. In addition, we also selected the latest dataset with higher spatial resolution, named WHU-Hi-LongKou, as the challenge object. Using only 0.1% of the training samples, we could achieve a 97.31% classification accuracy, which is far superior to the state-of-the-art hyperspectral classification methods.

Download Full-text

Research on Transformer Partial Discharge UHF Pattern Recognition Based on Cnn-lstm

Energies ◽

10.3390/en13010061 ◽

2019 ◽

Vol 13 (1) ◽

pp. 61 ◽

Cited By ~ 2

Author(s):

Xiu Zhou ◽

Xutao Wu ◽

Pei Ding ◽

Xiuguang Li ◽

Ninghui He ◽

...

Keyword(s):

Pattern Recognition ◽

Spatial Information ◽

Short Term Memory ◽

Partial Discharge ◽

Surface Discharge ◽

Recognition Ability ◽

Void Defects ◽

Memory Network ◽

Local Spatial Information

In view of the fact that the statistical feature quantity of traditional partial discharge (PD) pattern recognition relies on expert experience and lacks certain generalization, this paper develops PD pattern recognition based on the convolutional neural network (cnn) and long-term short-term memory network (lstm). Firstly, we constructed the cnn-lstm PD pattern recognition model, which combines the advantages of cnn in mining local spatial information of the PD spectrum and the advantages of lstm in mining the PD spectrum time series feature information. Then, the transformer PD UHF (Ultra High Frequency) experiment was carried out. The performance of the constructed cnn-lstm pattern recognition network was tested by using different types of typical PD spectrums. Experimental results show that: (1) for the floating potential defects, the recognition rates of cnn-lstm and cnn are both 100%; (2) cnn-lstm has better recognition ability than cnn for metal protrusion defects, oil paper void defects, and surface discharge defects; and (3) cnn-lstm has better overall recognition accuracy than cnn and lstm.

Download Full-text

Spectral and Spatial Classification of Hyperspectral Images Based on Random Multi-Graphs

Remote Sensing ◽

10.3390/rs10081271 ◽

2018 ◽

Vol 10 (8) ◽

pp. 1271 ◽

Cited By ~ 14

Author(s):

Feng Gao ◽

Qun Wang ◽

Junyu Dong ◽

Qizhi Xu

Keyword(s):

Image Classification ◽

Spatial Information ◽

Hyperspectral Image ◽

Small Sample ◽

Hyperspectral Data ◽

Hyperspectral Images ◽

High Dimensional ◽

Hyperspectral Image Classification ◽

Spatial Classification ◽

Spatial Features

Hyperspectral image classification has been acknowledged as the fundamental and challenging task of hyperspectral data processing. The abundance of spectral and spatial information has provided great opportunities to effectively characterize and identify ground materials. In this paper, we propose a spectral and spatial classification framework for hyperspectral images based on Random Multi-Graphs (RMGs). The RMG is a graph-based ensemble learning method, which is rarely considered in hyperspectral image classification. It is empirically verified that the semi-supervised RMG deals well with small sample setting problems. This kind of problem is very common in hyperspectral image applications. In the proposed method, spatial features are extracted based on linear prediction error analysis and local binary patterns; spatial features and spectral features are then stacked into high dimensional vectors. The high dimensional vectors are fed into the RMG for classification. By randomly selecting a subset of features to create a graph, the proposed method can achieve excellent classification performance. The experiments on three real hyperspectral datasets have demonstrated that the proposed method exhibits better performance than several closely related methods.

Download Full-text

Spectral-Spatial Hyperspectral Image Semisupervised Classification by Fusing Maximum Noise Fraction and Adaptive Random Multigraphs

Discrete Dynamics in Nature and Society ◽

10.1155/2021/9998185 ◽

2021 ◽

Vol 2021 ◽

pp. 1-11

Author(s):

Eryang Chen ◽

Ruichun Chang ◽

Kaibo Shi ◽

Ansheng Ye ◽

Fang Miao ◽

...

Keyword(s):

Spatial Information ◽

Hyperspectral Image ◽

Hyperspectral Data ◽

Spatial Features ◽

Training Samples ◽

Semisupervised Classification ◽

Ground Object ◽

Multiscale Local Binary Pattern ◽

Noise Fraction ◽

Reduced Data

Hyperspectral images (HSIs) contain large amounts of spectral and spatial information, and this provides the possibility for ground object classification. However, when using the traditional method, achieving a satisfactory classification result is difficult because of the insufficient labeling of samples in the training set. In addition, parameter adjustment during HSI classification is time-consuming. This paper proposes a novel fusion method based on the maximum noise fraction (MNF) and adaptive random multigraphs for HSI classification. Considering the overall spectrum of the object and the correlation of adjacent bands, the MNF was utilized to reduce the spectral dimension. Next, a multiscale local binary pattern (LBP) analysis was performed on the MNF dimension-reduced data to extract the spatial features of different scales. The obtained multiscale spatial features were then stacked with the MNF dimension-reduced spectral features to form multiscale spectral-spatial features (SSFs), which were sent into the RMG for HSI classification. Optimal performance was obtained by fusion. For all three real datasets, our method achieved competitive results with only 10 training samples. More importantly, the classification parameters corresponding to different hyperspectral data can be automatically optimized using our method.

Download Full-text

Variational Generative Adversarial Network with Crossed Spatial and Spectral Interactions for Hyperspectral Image Classification

Remote Sensing ◽

10.3390/rs13163131 ◽

2021 ◽

Vol 13 (16) ◽

pp. 3131

Author(s):

Zhongwei Li ◽

Xue Zhu ◽

Ziqi Xin ◽

Fangming Guo ◽

Xingshuai Cui ◽

...

Keyword(s):

Image Classification ◽

Spatial Information ◽

Hyperspectral Image ◽

Classification Performance ◽

Generative Adversarial Networks ◽

Hyperspectral Image Classification ◽

Generative Adversarial Network ◽

Adversarial Network ◽

Spatial Features ◽

Virtual Samples

Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) have been widely used in hyperspectral image classification (HSIC) tasks. However, the generated HSI virtual samples by VAEs are often ambiguous, and GANs are prone to the mode collapse, which lead the poor generalization abilities ultimately. Moreover, most of these models only consider the extraction of spectral or spatial features. They fail to combine the two branches interactively and ignore the correlation between them. Consequently, the variational generative adversarial network with crossed spatial and spectral interactions (CSSVGAN) was proposed in this paper, which includes a dual-branch variational Encoder to map spectral and spatial information to different latent spaces, a crossed interactive Generator to improve the quality of generated virtual samples, and a Discriminator stuck with a classifier to enhance the classification performance. Combining these three subnetworks, the proposed CSSVGAN achieves excellent classification by ensuring the diversity and interacting spectral and spatial features in a crossed manner. The superior experimental results on three datasets verify the effectiveness of this method.

Download Full-text

Shared long-term and short-term memory representational formats in occipital and parietal cortex

10.1101/2021.02.16.431528 ◽

2021 ◽

Author(s):

Vy A. Vo ◽

David W. Sutterer ◽

Joshua J. Foster ◽

Thomas C. Sprague ◽

Edward Awh ◽

...

Keyword(s):

Parietal Cortex ◽

Spatial Information ◽

Short Term Memory ◽

Memory Task ◽

Memory Systems ◽

Long Term Memory ◽

Short Term ◽

Term Memory ◽

Sensory Evoked

AbstractCurrent theories propose that the short-term retention of information in working memory (WM) and the recall of information from long-term memory (LTM) are supported by overlapping neural mechanisms in occipital and parietal cortex. Both are thought to rely on reinstating patterns of sensory activity evoked by the perception of the remembered item. However, the extent of the shared representations between WM and LTM are unclear, and it is unknown how WM and LTM representations may differ across cortical regions. We designed a spatial memory task that allowed us to directly compare the representations of remembered spatial information in WM and LTM. Critically, we carefully matched the precision of behavioral responses in these tasks. We used fMRI and multivariate pattern analyses to examine representations in (1) retinotopic cortex and (2) lateral parietal cortex (LPC) regions previously implicated in LTM. We show that visual memories were represented in a sensory-like code in both tasks across retinotopic regions in occipital and parietal cortex. LPC regions also encoded remembered locations in both WM and LTM, but in a format that differed from the sensory-evoked activity. These results suggest a striking correspondence in the format of WM and LTM representations across occipital and parietal cortex. On the other hand, we show that activity patterns in nearly all parietal regions, but not occipital regions, contained information that could discriminate between WM trials and LTM trials. Our data provide new evidence for theories of memory systems and the representation of mnemonic content.

Download Full-text