Advances in computing technology have fostered the development of new and powerful deep learning (DL) techniques, which have demonstrated promising results in a wide range of applications. Particularly, DL methods have been successfully used to classify remotely sensed data collected by Earth Observation (EO) instruments. Hyperspectral imaging (HSI) is a hot topic in remote sensing data analysis due to the vast amount of information comprised by this kind of images, which allows for a better characterization and exploitation of the Earth surface by combining rich spectral and spatial information. However, HSI poses major challenges for supervised classification methods due to the high dimensionality of the data and the limited availability of training samples. These issues, together with the high intraclass variability (and interclass similarity) –often present in HSI data– may hamper the effectiveness of classifiers. In order to solve these limitations, several DL-based architectures have been recently developed, exhibiting great potential in HSI data interpretation. This paper provides a comprehensive review of the current-state-of-the-art in DL for HSI classification, analyzing the strengths and weaknesses of the most widely used classifiers in the literature. For each discussed method, we provide quantitative results using several well-known and widely used HSI scenes, thus providing an exhaustive comparison of the discussed techniques. The paper concludes with some remarks and hints about future challenges in the application of DL techniques to HSI classification. The source codes of the methods discussed in this paper are available from: https://github.com/mhaut/hyperspectral_deeplearning_review .

Deep learning classifiers for hyperspectral imaging: A review

In recent years, deep learning (DL), a rebranding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, and natural language processing. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV, e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should not only be aware of advancements such as DL, but also be leading researchers in this area. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools, and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as they relate to (i) inadequate data sets, (ii) human-understandable solutions for modeling physical phenomena, (iii) big data, (iv) nontraditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial, and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.

Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community

Convolutional neural network (CNN) has demonstrated impressive ability to represent hyperspectral images and to achieve promising results in hyperspectral image classification. However, traditional CNN models can only operate convolution on regular square image regions with fixed size and weights, and thus, they cannot universally adapt to the distinct local regions with various object distributions and geometric appearances. Therefore, their classification performances are still to be improved, especially in class boundaries. To alleviate this shortcoming, we consider employing the recently proposed graph convolutional network (GCN) for hyperspectral image classification, as it can conduct the convolution on arbitrarily structured non-Euclidean data and is applicable to the irregular image regions represented by graph topological information. Different from the commonly used GCN models that work on a fixed graph, we enable the graph to be dynamically updated along with the graph convolution process so that these two steps can be benefited from each other to gradually produce the discriminative embedded features as well as a refined graph. Moreover, to comprehensively deploy the multiscale information inherited by hyperspectral images, we establish multiple input graphs with different neighborhood scales to extensively exploit the diversified spectral–spatial correlations at multiple scales. Therefore, our method is termed multiscale dynamic GCN (MDGCN). The experimental results on three typical benchmark data sets firmly demonstrate the superiority of the proposed MDGCN to other state-of-the-art methods in both qualitative and quantitative aspects.

Multiscale Dynamic Graph Convolutional Network for Hyperspectral Image Classification

Hyperspectral image classification has become a research focus in recent literature. However, well-designed features are still open issues that impact on the performance of classifiers. In this paper, a novel supervised deep feature extraction method based on siamese convolutional neural network (S-CNN) is proposed to improve the performance of hyperspectral image classification. First, a CNN with five layers is designed to directly extract deep features from hyperspectral cube, where the CNN can be intended as a nonlinear transformation function. Then, the siamese network composed by two CNNs is trained to learn features that show a low intraclass and high interclass variability. The important characteristic of the presented approach is that the S-CNN is supervised with a margin ranking loss function, which can extract more discriminative features for classification tasks. To demonstrate the effectiveness of the proposed feature extraction method, the features extracted from three widely used hyperspectral data sets are fed into a linear support vector machine (SVM) classifier. The experimental results demonstrate that the proposed feature extraction method in conjunction with a linear SVM classifier can obtain better classification performance than that of the conventional methods.

Supervised Deep Feature Extraction for Hyperspectral Image Classification

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

A novel deep learning classification method for hyperspectral data based on convolutional neural network is proposed in this paper. Deep learning means bringing multiple layers instead of one to the structure. Through convolution layers and pooling layers, the features in different layers are extracted from original spectral feature images. The key of this method is to restructure spectral feature images and choose convolution filters with a reasonable size, so that the spectral features of different land coverings in high dimensions can be extracted properly. In our experiments, proposed method was applied for hyperspectral data in several different situations, and preferable classification performance were obtained through relative parameters adjustment, which were given recommended scope during our comparative experiments.

Convolutional neural network based classification for hyperspectral data

Hyperspectral image with high dimensionality always increases the computational consumption, which challenges image processing. Deep learning models have achieved extraordinary success in various image processing domains, which are effective to improve classification performance. There remain considerable challenges in fully extracting abundant spectral information, such as the combination of spatial and spectral information. In this article, a novel unsupervised hyperspectral feature extraction architecture based on spatial revising variational autoencoder (AE) ( $U_{\text {Hfe}}\text {SRVAE}$ ) is proposed. The core concept of this method is extracting spatial features via designed networks from multiple aspects for the revision of the obtained spectral features. Multilayer encoder extracts spectral features, and then, latent space vectors are generated from the obtained means and standard deviations. Spatial features based on local sensing and sequential sensing are extracted using multilayer convolutional neural networks and long short-term memory networks, respectively, which can revise the obtained mean vectors. Besides, the proposed loss function guarantees the consistency of the probability distributions of various latent spatial features, which obtained from the same neighbor region. Several experiments are conducted on three publicly available hyperspectral data sets, and the experimental results show that $U_{\text {Hfe}}\text {SRVAE}$ achieves better classification results compared with comparison methods. The combination of spatial feature extraction models and deep AE models is designed based on the unique characteristics of hyperspectral images, which contributes to the performance of this method.

Spatial Revising Variational Autoencoder-Based Feature Extraction Method for Hyperspectral Images

Integration of spatial information has recently emerged as a powerful tool in improving the classification accuracy of hyperspectral image (HSI). However, partitioning homogeneous regions of the HSI remains a challenging task. This paper proposes a novel spectral-spatial classification method inspired by the support vector machine (SVM) and superpixel segmentation. Core ideas of the proposed method are twofold: 1) the HSI is first classified by the pixel-wise classifier (i.e. SVM); 2) a fast superpixel segmentation-based spatial processing is, for the first time, introduced in this study to refine the homogeneity and consistency of the classification maps. Experiments are conducted on two benchmark HSIs (i.e. the Indian Pines data and the Washington, D.C. Mall data) with different spectral and spatial resolutions. It is found that the proposed method yields more accurate classification results compared to the state-of-the-art techniques.

Spectral-spatial hyperspectral image classification via SVM and superpixel segmentation

Learning a local manifold representation based on improved neighborhood rough set and LLE for hyperspectral dimensionality reduction

Hyperspectral image unmixing techniques are developing to tackle the problem of mixed pixels caused by low spatial resolutions. In sparse unmixing, the redundant spectral library of materials is provided beforehand as a priori information to find the optimal representation by sparse linear regression. In order to improve the estimation of abundance distributions, the spatial correlation is taken into account and hypergraph learning is introduced to make full use of the underlying spatial-contextual information. Specifically, we find $K$ -nearest pixels of each pixel in spectral domains from a local region and construct a hypergraph to exploit the fact that spatial neighboring pixels have a high probability of sharing similar spectral information. Furthermore, a reweighted $\ell _1$ -norm minimization scheme is adopted instead to enhance the sparsity of estimated fractional abundances. The complicated large-scale regression problem is decomposed into subproblems to obtain the optimal solution within the framework of alternating direction method of multipliers. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithm.

Miao Zhang

Papers

Convolutional neural network based classification for hyperspectral data

Spatial Revising Variational Autoencoder-Based Feature Extraction Method for Hyperspectral Images

Spectral-spatial hyperspectral image classification via SVM and superpixel segmentation

Learning a local manifold representation based on improved neighborhood rough set and LLE for hyperspectral dimensionality reduction

Hypergraph Learning and Reweighted $\ell _1$ -Norm Minimization for Hyperspectral Unmixing