Advances in computing technology have fostered the development of new and powerful deep learning (DL) techniques, which have demonstrated promising results in a wide range of applications. Particularly, DL methods have been successfully used to classify remotely sensed data collected by Earth Observation (EO) instruments. Hyperspectral imaging (HSI) is a hot topic in remote sensing data analysis due to the vast amount of information comprised by this kind of images, which allows for a better characterization and exploitation of the Earth surface by combining rich spectral and spatial information. However, HSI poses major challenges for supervised classification methods due to the high dimensionality of the data and the limited availability of training samples. These issues, together with the high intraclass variability (and interclass similarity) –often present in HSI data– may hamper the effectiveness of classifiers. In order to solve these limitations, several DL-based architectures have been recently developed, exhibiting great potential in HSI data interpretation. This paper provides a comprehensive review of the current-state-of-the-art in DL for HSI classification, analyzing the strengths and weaknesses of the most widely used classifiers in the literature. For each discussed method, we provide quantitative results using several well-known and widely used HSI scenes, thus providing an exhaustive comparison of the discussed techniques. The paper concludes with some remarks and hints about future challenges in the application of DL techniques to HSI classification. The source codes of the methods discussed in this paper are available from: https://github.com/mhaut/hyperspectral_deeplearning_review .

Deep learning classifiers for hyperspectral imaging: A review

In recent years, deep learning (DL), a rebranding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, and natural language processing. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV, e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should not only be aware of advancements such as DL, but also be leading researchers in this area. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools, and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as they relate to (i) inadequate data sets, (ii) human-understandable solutions for modeling physical phenomena, (iii) big data, (iv) nontraditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial, and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.

Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community

Convolutional neural network (CNN) has demonstrated impressive ability to represent hyperspectral images and to achieve promising results in hyperspectral image classification. However, traditional CNN models can only operate convolution on regular square image regions with fixed size and weights, and thus, they cannot universally adapt to the distinct local regions with various object distributions and geometric appearances. Therefore, their classification performances are still to be improved, especially in class boundaries. To alleviate this shortcoming, we consider employing the recently proposed graph convolutional network (GCN) for hyperspectral image classification, as it can conduct the convolution on arbitrarily structured non-Euclidean data and is applicable to the irregular image regions represented by graph topological information. Different from the commonly used GCN models that work on a fixed graph, we enable the graph to be dynamically updated along with the graph convolution process so that these two steps can be benefited from each other to gradually produce the discriminative embedded features as well as a refined graph. Moreover, to comprehensively deploy the multiscale information inherited by hyperspectral images, we establish multiple input graphs with different neighborhood scales to extensively exploit the diversified spectral–spatial correlations at multiple scales. Therefore, our method is termed multiscale dynamic GCN (MDGCN). The experimental results on three typical benchmark data sets firmly demonstrate the superiority of the proposed MDGCN to other state-of-the-art methods in both qualitative and quantitative aspects.

Multiscale Dynamic Graph Convolutional Network for Hyperspectral Image Classification

Hyperspectral image classification has become a research focus in recent literature. However, well-designed features are still open issues that impact on the performance of classifiers. In this paper, a novel supervised deep feature extraction method based on siamese convolutional neural network (S-CNN) is proposed to improve the performance of hyperspectral image classification. First, a CNN with five layers is designed to directly extract deep features from hyperspectral cube, where the CNN can be intended as a nonlinear transformation function. Then, the siamese network composed by two CNNs is trained to learn features that show a low intraclass and high interclass variability. The important characteristic of the presented approach is that the S-CNN is supervised with a margin ranking loss function, which can extract more discriminative features for classification tasks. To demonstrate the effectiveness of the proposed feature extraction method, the features extracted from three widely used hyperspectral data sets are fed into a linear support vector machine (SVM) classifier. The experimental results demonstrate that the proposed feature extraction method in conjunction with a linear SVM classifier can obtain better classification performance than that of the conventional methods.

Supervised Deep Feature Extraction for Hyperspectral Image Classification

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

A novel deep learning classification method for hyperspectral data based on convolutional neural network is proposed in this paper. Deep learning means bringing multiple layers instead of one to the structure. Through convolution layers and pooling layers, the features in different layers are extracted from original spectral feature images. The key of this method is to restructure spectral feature images and choose convolution filters with a reasonable size, so that the spectral features of different land coverings in high dimensions can be extracted properly. In our experiments, proposed method was applied for hyperspectral data in several different situations, and preferable classification performance were obtained through relative parameters adjustment, which were given recommended scope during our comparative experiments.

Convolutional neural network based classification for hyperspectral data

The invention relates to a hyperspectral data classification method based on a multi-layer convolution network and data organization and folding. The method comprises: step one, pretreatment is carried out before expanding and classification of three-dimensional hyperspectral data and a data matrix including effective spectrum information and a tag vector are obtained; step two, feature dimension expanding is carried out on the data matrix, and column-based folding and reorganization are carried out on a feature dimension to obtain a reorganized three-dimensional hyperspectral data input matrix; step three, a multi-layer convolution network structure parameter and an initial value are set; and step four, a feature and an error are calculated layer by layer by using forward propagation and BP algorithms, a network weight and a bias value are updated, iteration is carried out continuously to obtain a network stablity parameter, and then a network model for classification and a parameter for classification are obtained. Compared with other methods, the provided method has advantages of clear principle, clear structure, short identification time, and high detection identification rate; and the method being an effective classification method for hyperspectral data is suitable for rapid target detection and classification identification application of hyperspectral images.

Hyperspectral data classification method based on multi-layer convolution network and data organization and folding

Hyperspectral image unmixing techniques are developing to tackle the problem of mixed pixels caused by low spatial resolutions. In sparse unmixing, the redundant spectral library of materials is provided beforehand as a priori information to find the optimal representation by sparse linear regression. In order to improve the estimation of abundance distributions, the spatial correlation is taken into account and hypergraph learning is introduced to make full use of the underlying spatial-contextual information. Specifically, we find $K$ -nearest pixels of each pixel in spectral domains from a local region and construct a hypergraph to exploit the fact that spatial neighboring pixels have a high probability of sharing similar spectral information. Furthermore, a reweighted $\ell _1$ -norm minimization scheme is adopted instead to enhance the sparsity of estimated fractional abundances. The complicated large-scale regression problem is decomposed into subproblems to obtain the optimal solution within the framework of alternating direction method of multipliers. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithm.

Hypergraph Learning and Reweighted $\ell _1$ -Norm Minimization for Hyperspectral Unmixing

In order to improve the classification accuracy of hyperspectral image, its internal relations and mutual relations should be paid attention to. This paper proposes a compound non-linear dimensionality reduction method with the help of non-negative matrix factorization (NMF) to show the relationships between samples within the same class and locality preserving projections (LPP) to show the relationships between classes. Combining with the advantages of the two algorithms, the proposed method makes data samples within a class closer and data samples between different classes farther after the projection by containing orthogonal constraints for datasets. Together with maximum-likelihood classifiers, all the methods mentioned above constitute the classification method for real hyperspectral data. Experimental results show its advantage in parameter adaption and classification accuracy.

Hyperspectral image classification method based on orthogonal NMF and LPP

According to the characters of complex hyperspectral data, sparsity technique is introduced to deep convolutional neural network to handle feature extraction and classification problems. Combining sparse unsupervised learning method with neural network model, it is possible to get a good, sparse representation of the spectral information so that deep CNN model could extract feature information hierarchically and effectively. EPLS algorithm is applied in this paper to combine population sparsity and lifetime sparsity with the advantages of extracting deep feature information of CNN model to get a fine classification model. In the experiment, two hyperspectral data sets are applied for the proposed method, and the results demonstrate fine classification performances of the model.

Peiyuan Jia

Papers

Convolutional neural network based classification for hyperspectral data

Hyperspectral data classification method based on multi-layer convolution network and data organization and folding

Hypergraph Learning and Reweighted $\ell _1$ -Norm Minimization for Hyperspectral Unmixing

Hyperspectral image classification method based on orthogonal NMF and LPP

Hyperspectral image feature extraction method based on sparse constraint convolutional neural network