Advances in computing technology have fostered the development of new and powerful deep learning (DL) techniques, which have demonstrated promising results in a wide range of applications. Particularly, DL methods have been successfully used to classify remotely sensed data collected by Earth Observation (EO) instruments. Hyperspectral imaging (HSI) is a hot topic in remote sensing data analysis due to the vast amount of information comprised by this kind of images, which allows for a better characterization and exploitation of the Earth surface by combining rich spectral and spatial information. However, HSI poses major challenges for supervised classification methods due to the high dimensionality of the data and the limited availability of training samples. These issues, together with the high intraclass variability (and interclass similarity) –often present in HSI data– may hamper the effectiveness of classifiers. In order to solve these limitations, several DL-based architectures have been recently developed, exhibiting great potential in HSI data interpretation. This paper provides a comprehensive review of the current-state-of-the-art in DL for HSI classification, analyzing the strengths and weaknesses of the most widely used classifiers in the literature. For each discussed method, we provide quantitative results using several well-known and widely used HSI scenes, thus providing an exhaustive comparison of the discussed techniques. The paper concludes with some remarks and hints about future challenges in the application of DL techniques to HSI classification. The source codes of the methods discussed in this paper are available from: https://github.com/mhaut/hyperspectral_deeplearning_review .

Deep learning classifiers for hyperspectral imaging: A review

In recent years, deep learning (DL), a rebranding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, and natural language processing. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV, e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should not only be aware of advancements such as DL, but also be leading researchers in this area. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools, and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as they relate to (i) inadequate data sets, (ii) human-understandable solutions for modeling physical phenomena, (iii) big data, (iv) nontraditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial, and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.

Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community

Convolutional neural network (CNN) has demonstrated impressive ability to represent hyperspectral images and to achieve promising results in hyperspectral image classification. However, traditional CNN models can only operate convolution on regular square image regions with fixed size and weights, and thus, they cannot universally adapt to the distinct local regions with various object distributions and geometric appearances. Therefore, their classification performances are still to be improved, especially in class boundaries. To alleviate this shortcoming, we consider employing the recently proposed graph convolutional network (GCN) for hyperspectral image classification, as it can conduct the convolution on arbitrarily structured non-Euclidean data and is applicable to the irregular image regions represented by graph topological information. Different from the commonly used GCN models that work on a fixed graph, we enable the graph to be dynamically updated along with the graph convolution process so that these two steps can be benefited from each other to gradually produce the discriminative embedded features as well as a refined graph. Moreover, to comprehensively deploy the multiscale information inherited by hyperspectral images, we establish multiple input graphs with different neighborhood scales to extensively exploit the diversified spectral–spatial correlations at multiple scales. Therefore, our method is termed multiscale dynamic GCN (MDGCN). The experimental results on three typical benchmark data sets firmly demonstrate the superiority of the proposed MDGCN to other state-of-the-art methods in both qualitative and quantitative aspects.

Multiscale Dynamic Graph Convolutional Network for Hyperspectral Image Classification

Hyperspectral image classification has become a research focus in recent literature. However, well-designed features are still open issues that impact on the performance of classifiers. In this paper, a novel supervised deep feature extraction method based on siamese convolutional neural network (S-CNN) is proposed to improve the performance of hyperspectral image classification. First, a CNN with five layers is designed to directly extract deep features from hyperspectral cube, where the CNN can be intended as a nonlinear transformation function. Then, the siamese network composed by two CNNs is trained to learn features that show a low intraclass and high interclass variability. The important characteristic of the presented approach is that the S-CNN is supervised with a margin ranking loss function, which can extract more discriminative features for classification tasks. To demonstrate the effectiveness of the proposed feature extraction method, the features extracted from three widely used hyperspectral data sets are fed into a linear support vector machine (SVM) classifier. The experimental results demonstrate that the proposed feature extraction method in conjunction with a linear SVM classifier can obtain better classification performance than that of the conventional methods.

Supervised Deep Feature Extraction for Hyperspectral Image Classification

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

A novel deep learning classification method for hyperspectral data based on convolutional neural network is proposed in this paper. Deep learning means bringing multiple layers instead of one to the structure. Through convolution layers and pooling layers, the features in different layers are extracted from original spectral feature images. The key of this method is to restructure spectral feature images and choose convolution filters with a reasonable size, so that the spectral features of different land coverings in high dimensions can be extracted properly. In our experiments, proposed method was applied for hyperspectral data in several different situations, and preferable classification performance were obtained through relative parameters adjustment, which were given recommended scope during our comparative experiments.

Convolutional neural network based classification for hyperspectral data

Hyperspectral image with high dimensionality always increases the computational consumption, which challenges image processing. Deep learning models have achieved extraordinary success in various image processing domains, which are effective to improve classification performance. There remain considerable challenges in fully extracting abundant spectral information, such as the combination of spatial and spectral information. In this article, a novel unsupervised hyperspectral feature extraction architecture based on spatial revising variational autoencoder (AE) ( $U_{\text {Hfe}}\text {SRVAE}$ ) is proposed. The core concept of this method is extracting spatial features via designed networks from multiple aspects for the revision of the obtained spectral features. Multilayer encoder extracts spectral features, and then, latent space vectors are generated from the obtained means and standard deviations. Spatial features based on local sensing and sequential sensing are extracted using multilayer convolutional neural networks and long short-term memory networks, respectively, which can revise the obtained mean vectors. Besides, the proposed loss function guarantees the consistency of the probability distributions of various latent spatial features, which obtained from the same neighbor region. Several experiments are conducted on three publicly available hyperspectral data sets, and the experimental results show that $U_{\text {Hfe}}\text {SRVAE}$ achieves better classification results compared with comparison methods. The combination of spatial feature extraction models and deep AE models is designed based on the unique characteristics of hyperspectral images, which contributes to the performance of this method.

Spatial Revising Variational Autoencoder-Based Feature Extraction Method for Hyperspectral Images

Learning a local manifold representation based on improved neighborhood rough set and LLE for hyperspectral dimensionality reduction

Hyperspectral image processing is faced with difficulties considering its redundant features and complex information. Studies on hyperspectral feature extraction in the deep learning domain have become increasingly popular. The mainstream techniques fully consider the spatial information in local neighborhoods when extracting spectral features by constructing deep neural networks. Deep generative models simulate the intrinsic structure of samples by adequately training, showing their potential values for signal processing. In this article, a convolutional two-stream network (cs²GAN-FE) based on the improved Wasserstein generative adversarial network (WGAN) is proposed for unsupervised hyperspectral spatial-spectral feature extraction. The improved WGAN is composed of one generator and one discriminator; the former perceives real data distributions, and the latter determines the attribution of generated data. The designed two-stream strategy is not a simple extension of a one-stream strategy and considers both the static spectral-spatial information and the dynamic spectral reflectance variation in multiple bands. Intrinsic spatial-spectral features are extracted by the trained discriminator considering sample distributions and feature relationships. The loss function is also improved for the unique structure of cs²GAN-FE. Various state-of-the-art techniques are chosen for comparison. Experimental results show the feasibility and potential of this network. Besides, experiments with the random split and the disjointed split both show that the proposed method can outperform other comparison techniques.

Convolutional Two-Stream Generative Adversarial Network-Based Hyperspectral Feature Extraction

The large number of spectral bands in hyperspectral data provides abundant information to distinguish different land forms, however, the high dimensionality of hyperspectral data brings an extra computational burden. Hence band selection is important for hyperspectral data. The proposed hyperspectral clone selection algorithm based band selection method (CSABS) chooses an improved multi-dimensional mutual information method as the measure criterion, can select bands with richer information and lower redundancy from hyperspectral data and it is effective due to low reconstruction error. The clone selection algorithm is proposed as the search strategy where the adaptive clone and mutation operators are devised to speed up the convergence. This method not only take full advantage of multiple iterations of clone selection algorithm, but also has high accuracy. Experimental results on hyperspectral data demonstrate the effectiveness of the proposed band selection method.

Wenbo Yu

Papers

Convolutional neural network based classification for hyperspectral data

Spatial Revising Variational Autoencoder-Based Feature Extraction Method for Hyperspectral Images

Learning a local manifold representation based on improved neighborhood rough set and LLE for hyperspectral dimensionality reduction

Convolutional Two-Stream Generative Adversarial Network-Based Hyperspectral Feature Extraction

Mutual information and clone selection algorithm based hyperspectral band selection method