Machine learning, particularly deep learning (DL), has become a central and state-of-the-art method for several computer vision applications and remote sensing (RS) image processing. Researchers are continually trying to improve the performance of the DL methods by developing new architectural designs of the networks and/or developing new techniques, such as attention mechanisms. Since the attention mechanism has been proposed, regardless of its type, it has been increasingly used for diverse RS applications to improve the performances of the existing DL methods. However, these methods are scattered over different studies impeding the selection and application of the feasible approaches. This study provides an overview of the developed attention mechanisms and how to integrate them with different deep learning neural network architectures. In addition, it aims to investigate the effect of the attention mechanism on deep learning-based RS image processing. We identified and analyzed the advances in the corresponding attention mechanism-based deep learning (At-DL) methods. A systematic literature review was performed to identify the trends in publications, publishers, improved DL methods, data types used, attention types used, overall accuracies achieved using At-DL methods, and extracted the current research directions, weaknesses, and open problems to provide insights and recommendations for future studies. For this, five main research questions were formulated to extract the required data and information from the literature. Furthermore, we categorized the papers regarding the addressed RS image processing tasks (e.g., image classification, object detection, and change detection) and discussed the results within each group. In total, 270 papers were retrieved, of which 176 papers were selected according to the defined exclusion criteria for further analysis and detailed review. The results reveal that most of the papers reported an increase in overall accuracy when using the attention mechanism within the DL methods for image classification, image segmentation, change detection, and object detection using remote sensing images.

Effect of Attention Mechanism in Deep Learning-Based Remote Sensing Image Processing: A Systematic Literature Review

Deep Convolutional Neural Network (CNN) shows excellent effectiveness on hyperspectral image (HSI) classification. However, the architecture design of CNN requires abundant expert knowledge and experience, which poses great prohibition to its wide application in real-world engineering. To alleviate the issue, this paper proposes an evolving block-based CNN (EB-CNN) to search the optimal architecture based on the genetic algorithm automatically. Specifically, two kinds of basic blocks with totally six different configurations are first designed to construct the search space. Then, a flexible encoding strategy is devised for the genetic algorithm to allow different chromosomes to evolve with different lengths. In this manner, the width of each layer and the depth of the architecture can be simultaneously optimized. Furthermore, a novel swapping mutation operator is proposed for the genetic algorithm to speed up the search efficiency and save computing resources. With the above techniques, the proposed algorithm automatically seeks the optimal CNN architecture for HSI classification, leading to its better usability than handcrafted CNNs. At last, extensive experiments conducted on 5 commonly used HSI datasets demonstrate that the proposed EB-CNN achieves highly competitive or even better performance, as compared to state-of-the-art peer algorithms.

Evolving Block-Based Convolutional Neural Network for Hyperspectral Image Classification

In recent years, deep-learning-based hyperspectral image (HSI) classification methods have achieved significant development. The superior capability of feature extraction from these data-driven methods dramatically improves the classification performance. However, the previous methods usually require to retrain the network from scratch to obtain the capability of feature extraction adaptive for the target image when facing a new HSI to be classified, which is a time-consuming and redundant process. In this paper, we consider putting this process ahead and making the network have a robust capability of feature extraction with generalization through pre-training. Therefore, the network enables to directly extract features of the target HSI without re-training. For this purpose, we rethink the three-dimension (3D) HSI data from a perspective of spectral sequence, and we attempt to extract the spectral variation information as the spectrum motion feature. Then, we construct an unsupervised spectrum motion feature learning framework (SMF-UL), which can be pre-trained on mass unlabeled HSI data to learn the knowledge about perceiving spectral variation. Furthermore, to achieve the expansion of source data for pre-training, we develop an extendable training dataset construction method, which can integrate HSIs of different sizes, number of bands and sensors into a unified training set to utilize the rapidly growing mass unlabeled HSI data effectively. Finally, we use the trained network to directly extract the spectrum motion feature of the target HSI for classification, so the laborious re-training of the network can be avoided. Extensive experiments show that the proposed SMF-UL acquires the robust capability of feature extraction with generalization through unsupervised learning on mass unlabeled HSI data, and the classification performance of extracted spectrum motion feature is competitive to advanced in-domain and cross-domain methods, which shows its flexibility and superiority. The code of SMF-UL will be open at: https://github.com/sssssyf/SMF-UL. 

Perceiving Spectral Variation: Unsupervised Spectrum Motion Feature Learning for Hyperspectral Image Classification

Owing to their powerful feature extraction capabilities, deep learning-based methods have achieved significant progress in hyperspectral remote sensing classification. However, several issues still exist in these methods, including a lack of hyperspectral datasets for specific complicated scenarios and the need to improve the classification accuracy of land cover with limited samples. Thus, to highlight and distinguish effective features, we propose a hyperspectral classification framework based on a joint channel-space attention mechanism and generative adversarial network (JAGAN). To relearn feature-based weights, a higher priority was assigned to important features, which was developed by integrating a two-joint channel-space attention model to obtain the most valuable feature via the attention weight map. Additionally, two classifiers were designed in JAGAN: sigmoid was used to determine whether the input data were real or fake samples produced by the generator, while Softmax was adopted as a land cover classifier to yield the prediction type labels of the input samples. To test the classification performance of the JAGAN model, we used a self-constructed complex land cover dataset based on GaoFen-5 AHSI images, which consists of mixed landscapes of mining and agricultural areas from the urban-rural fringe. Compared with other methods, the proposed model achieved the highest overall classification accuracy of 86.09%, the highest kappa amount of 79.41%, the highest F1 score of 85.86%, and the highest average accuracy of 82.30%, indicating the JAGAN can effectively improve the classification accuracy for limited samples in complex regional environments using GF-5 AHSI images.

JAGAN: A Framework for Complex Land Cover Classification Using Gaofen-5 AHSI Images

Convolutional Neural Networks (CNN) have been rigorously studied for Hyperspectral Image Classification (HSIC) and are known to be effective in exploiting joint spatial-spectral information with the expense of lower generalization performance and learning speed due to the hard labels and non-uniform distribution over labels. Therefore, this paper proposed an idea to enhance the generalization performance of CNN for HSIC using soft labels that are a weighted average of the hard labels and uniform distribution over ground labels. The proposed method helps to prevent CNN from becoming over-confident. We empirically show that, in improving generalization performance, regularization also improves model calibration, which significantly improves beam-search. Several publicly available Hyperspectral datasets are used to validate the experimental evaluation, which reveals improved performance as compared to the state-of-the-art models with overall 99.29%, 99.97%, and 100.0% accuracy for Indiana Pines, Pavia University, and Salinas dataset, respectively.

/pdf/regularized-cnn-feature-hierarchy-for-hyperspectral-image-2y0qfkxr9j.pdf

Regularized CNN Feature Hierarchy for Hyperspectral Image Classification

Deep learning (DL) has become a hot topic in the research field of hyperspectral image (HSI) classification. However, with increasing depth and size of deep learning methods, its application in mobile and embedded vision applications has brought great challenges. In this article, we address a network architecture search (NAS)-guided lightweight spectral–spatial attention feature fusion network (LMAFN) for HSI classification. The overall architecture of the proposed network is guided by several conclusions of NAS, which achieves fewer parameters and lower computation cost with deeper network structure by exploiting multiscale Ghost grouped with efficient channel attention (ECA) module for adaptively adjusting the weights of different channels. It helps fully extract spectral–spatial discriminant features to avoid information loss of the dimension reduction operation. Specifically, a multilayer feature fusion method is proposed to extract the fusion information of the spectral–spatial features of each layer by considering complementary information of different hierarchical structures. Therefore, high-lever spectral–spatial attributes are gradually exploited along with the increase in layers and the fusion of layers. The experimental verification on three real HSI data sets demonstrates that the proposed framework presents more satisfying classification performance and efficiency with deeper network structure and lower parameter size.

NAS-Guided Lightweight Multiscale Attention Fusion Network for Hyperspectral Image Classification

Deep learning-based methods have demonstrated significant breakthroughs in the application of hyperspectral image (HSI) classification. However, some challenging issues still exist, such as the overfitting problem caused by the limitation of training size with high-dimensional feature and the efficiency of spectral-spatial (SS) exploitation. Therefore, to efficiently model the relative position of samples within the generative adversarial network (GAN) setting, we proposed a dual-channel SS fusion capsule generative adversarial network (DcCapsGAN) for HSI classification. Dual channels (1-D-CapsGAN and 2-D-CapsGAN) are constructed by integrating the capsule network (CapsNet) with GAN for eliminating the mode collapse and gradient disappearance problem caused by traditional GAN. Meanwhile, octave convolution and multiscale convolution are integrated into the proposed model for further reducing the parameters of the CapsNet and extracting multiscale features. To further boost the classification performance, the SS channel fusion model is constructed to composite and switch the feature information of different channels, thereby facilitating the accuracy and robustness of the whole classification performance. Three commonly used HSI data sets are utilized to investigate the performance of the proposed DcCapsGAN model, and the performance of the experiment demonstrates that the proposed model can efficiently improve the classification accuracy and performance.

Huang Runhu

Papers

NAS-Guided Lightweight Multiscale Attention Fusion Network for Hyperspectral Image Classification

Dual-Channel Capsule Generation Adversarial Network for Hyperspectral Image Classification