The development of remote sensing images in recent years has made it possible to identify materials in inaccessible environments and study natural materials on a large scale. But hyperspectral images (HSIs) are a rich source of information with their unique features in various applications. However, several problems reduce the accuracy of HSI classification; for example, the extracted features are not effective, noise, the correlation of bands, and most importantly, the limited labeled samples. To improve accuracy in the case of limited training samples, we propose a multiscale dual-branch residual spectral&#x2013;spatial network with attention to the HSI classification model named MDBRSSN in this article. First, due to the correlation and redundancy between HSI bands, a principal component analysis operation is applied to preprocess the raw HSI data. Then, in MDBRSSN, a dual-branch structure is designed to extract the useful spectral&#x2013;spatial features of HSI. The advanced feature, multiscale abstract information extracted by the convolution neural network, is applied to image processing, which can improve complex hyperspectral data classification accuracy. In addition, the attention mechanisms applied separately to each branch enable MDBRSSN to optimize and refine the extracted feature maps. Such an MDBRSSN framework can learn and fuse deeper hierarchical spectral&#x2013;spatial features with fewer training samples. The purpose of designing the MDBRSSN model is to have high classification accuracy compared to state-of-the-art methods when the training samples are limited, which is proved by the results of the experiments in this article on four datasets. In Salinas, Pavia University, Indian Pines, and Houston 2013, the proposed model obtained 99.64&#x0025;, 98.93&#x0025;, 98.17&#x0025;, and 96.57&#x0025; overall accuracy using only 1&#x0025;, 1&#x0025;, 5&#x0025;, and 5&#x0025; of labeled data for training, respectively, which are much better compared to the state-of-the-art methods. 

Multiscale Dual-Branch Residual Spectral–Spatial Network With Attention for Hyperspectral Image Classification

Enhanced contextual representation with deep neural networks for land cover classification based on remote sensing images

In general, most of the existing convolutional neural network (CNN)-based deep-learning models suffer from spatial-information loss and inadequate feature-representation issues. This is due to their inability to capture multiscale-context information and the exclusion of semantic information throughout the pooling operations. In the early layers of a CNN, the network encodes simple semantic representations, such as edges and corners, while, in the latter part of the CNN, the network encodes more complex semantic features, such as complex geometric shapes. Theoretically, it is better for a CNN to extract features from different levels of semantic representation because tasks such as classification and segmentation work better when both simple and complex feature maps are utilized. Hence, it is also crucial to embed multiscale capability throughout the network so that the various scales of the features can be optimally captured to represent the intended task. Multiscale representation enables the network to fuse low-level and high-level features from a restricted receptive field to enhance the deep-model performance. The main novelty of this review is the comprehensive novel taxonomy of multiscale-deep-learning methods, which includes details of several architectures and their strengths that have been implemented in the existing works. Predominantly, multiscale approaches in deep-learning networks can be classed into two categories: multiscale feature learning and multiscale feature fusion. Multiscale feature learning refers to the method of deriving feature maps by examining kernels over several sizes to collect a larger range of relevant features and predict the input images’ spatial mapping. Multiscale feature fusion uses features with different resolutions to find patterns over short and long distances, without a deep network. Additionally, several examples of the techniques are also discussed according to their applications in satellite imagery, medical imaging, agriculture, and industrial and manufacturing systems.

/pdf/a-review-on-multiscale-deep-learning-applications-9mr96mhp.pdf

A Review on Multiscale-Deep-Learning Applications

Developments in space-based hyperspectral sensors, advanced remote sensing, and machine learning can help crop yield measurement, modelling, prediction, and crop monitoring for loss prevention and global food security. However, precise and continuous spectral signatures, important for large-area crop growth monitoring and early prediction of yield production with cutting-edge algorithms, can be only provided via hyperspectral imaging. Therefore, this article used new-generation Deutsches Zentrum für Luft- und Raumfahrt Earth Sensing Imaging Spectrometer (DESIS) images to classify the main crop types (hybrid corn, soybean, sunflower, and winter wheat) in Mezőhegyes (southeastern Hungary). A Wavelet-attention convolutional neural network (WA-CNN), random forest and support vector machine (SVM) algorithms were utilized to automatically map the crops over the agricultural lands. The best accuracy was achieved with the WA-CNN, a feature-based deep learning algorithm and a combination of two images with overall accuracy (OA) value of 97.89% and the user's accuracy producer's accuracy was from 97% to 99%. To obtain this, first, factor analysis was introduced to decrease the size of the hyperspectral image data cube. A wavelet transform was applied to extract important features and combined with the spectral attention mechanism CNN to gain higher accuracy in mapping crop types. Followed by SVM algorithm reported OA of 87.79%, with the producer's and user's accuracies of its classes ranging from 79.62% to 96.48% and from 79.63% to 95.73%, respectively. These results demonstrate the potentiality of DESIS data to observe the growth of different crop types and predict the harvest volume, which is crucial for farmers, smallholders, and decision-makers. 

/pdf/crop-type-classification-by-desis-hyperspectral-imagery-and-1r4x4fl1.pdf

Crop Type Classification by DESIS Hyperspectral Imagery and Machine Learning Algorithms

Deep learning has achieved promising performance in hyperspectral image (HSI) classification. Training deep models usually requires labeling massive HSIs, which, however, is prohibitively time-consuming and expensive. To fill in the gap, this article proposes a novel meta-learning method for HSI few-shot classification that conducts HSI classification with a few labeled samples. Specifically, we introduce the Earth mover’s distance (EMD) as a metric. The designed EMD metric learning module aims to calculate the similarity of paired embedding features by decomposing embedding features into a set of local representations. The EMD metric aims to find the optimal matching flows between local representations that have the minimum matching cost. Furthermore, we attempt to learn class prototype representation for each hyperspectral class using the EMD metric. The proposed network effectively learns general knowledge from base HSIs and transfers such knowledge to the classification of novel HSIs. We conduct HSI few-shot classification by training on three base HSIs and classification on three novel HSIs. Extensive experimental results on three novel HSI datasets demonstrate that the proposed model outperforms the existing state-of-the-art HSI methods, including two HSI few-shot methods.

Hyperspectral Image Few-Shot Classification Network Based on the Earth Mover’s Distance

Recently, hyperspectral image classification based on deep learning has achieved considerable attention. Many convolutional neural network classification methods have emerged and exhibited superior classification performance. However, most methods focus on extracting features by using fixed convolution kernels and layer-wise representation, resulting in feature extraction singleness. Additionally, the feature fusion process is rough and simple. Numerous methods get accustomed to fusing different levels of features by stacking modules hierarchically, which ignore the combination of shallow and deep spectral-spatial features. In order to overcome the preceding issues, a novel multiscale dual-branch feature fusion and attention network is proposed. Specifically, we design a multiscale feature extraction (MSFE) module to extract spatial-spectral features at a granular level and expand the range of receptive fields, thereby enhancing the MSFE ability. Subsequently, we develop a dual-branch feature fusion interactive module that integrates the residual connection's feature reuse property and the dense connection's feature exploration capability, obtaining more discriminative features in both spatial and spectral branches. Additionally, we introduce a novel shuffle attention mechanism that allows for adaptive weighting of spatial and spectral features, further improving classification performance. Experimental results on three benchmark datasets demonstrate that our model outperforms other state-of-the-art methods while incurring the lower computational cost.

/pdf/a-multiscale-dual-branch-feature-fusion-and-attention-2rgrz353mg.pdf

A Multiscale Dual-Branch Feature Fusion and Attention Network for Hyperspectral Images Classification

Hyperspectral image (HSI) has received considerable attention in the field of target detection due to its powerful ability to capture the spectral information of land covers, and plenty of detection algorithms have been explored. However, these methods generally leverage the difference between the spectrum of the target to be detected and the background spectrum to accomplish target detection, and so are susceptible to the problem of spectral variability. In this article, we propose a global to local hierarchical detection algorithm for HSI (G2LHTD). First, extended morphological attribute profile (EMAP) is first used to model global spatial texture information from HSI. Subsequently, a diverse-direction constrained energy minimization ( $\text{D}^{2}$ CEM) detector is developed to consider the spatial information within eight neighborhoods around each pixel in HSI, yielding comprehensive local spatial information. More substantially, to effectively discriminate the neighborhood information in diverse directions, we devise an adaptive neighborhood feature aggregation (ANFA) strategy, which will comprehensively evaluate the significance of neighborhood information in diverse directions. As a result, the spatial features of HSI can be comprehensively considered for hyperspectral target detection (HTD). Extensive experiments, conducted on four standard datasets, demonstrate the effectiveness of the proposed method. The codes of this work will be available at https://github.com/zhonghaochen/G2LHTD_Master for the sake of reproducibility.

Global to Local: A Hierarchical Detection Algorithm for Hyperspectral Image Target Detection

At present, deep learning classification researches of hyperspectral usually focus on optimizing the classification model. In essence, most of them did not take special measures for the characteristics of the small sample and imbalanced category distribution of hyperspectral itself. Aiming at the problems of small samples and imbalanced category distribution, we propose a dynamic data selection algorithm. For one thing, this algorithm can dynamically select the samples that need data augmentation most. For another, it can be nested in Stochastic gradient descent (SGD) and can be easily implemented. Furthermore, there will be differences between the original and the transformed sample because of data augmentation transformation, which obstructs trained models' performance. Aiming at the difference between the augmented sample and the original sample, we define the similarity score and introduce the Siamese training structure to obtain the similarity score by which we reduce the difference through the SGD algorithm. Experiments show that the method proposed in this article improves the classification results of the backbone training model when using data augmentation for training.

/pdf/dynamic-data-augmentation-method-for-hyperspectral-image-1n12vmp8eb.pdf

Dynamic Data Augmentation Method for Hyperspectral Image Classification Based on Siamese Structure

The task of hyperspectral images (HSIs) target detection is to identify whether the target spectral sequences present in the HSI. Recently, the topic of representation models has received much interest in hyperspectral target detection. The performance of representation models depends on whether the corresponding dictionary and sparse matrix can correctly recover the original spectrum. Therefore, the background dictionary of these models should contain the spectra of all classes except the target spectrum; i.e., the dictionary should be overcomplete. However, most representation models cannot satisfy this condition. Moreover, due to the potentially large spectral similarity between the target and the background, representation models perform poorly in background suppression. Aiming to solve these issues, a novel adaptively dictionary construction (ADC) strategy with background suppression sparse representation (BSSR) module is proposed in this letter, called adaptively dictionary construction for target detection (ADCTD). Specifically, the proposed ADC is adopted to segment the HSI into superpixels consisting of pixels with similar spectra. This process can be considered as an unsupervised coarse classification process, which can construct an overcomplete background dictionary. In addition, the BSSR is adopted to improve the separation of the target and background by a linear function. Experiments on three datasets demonstrate the superiority of the proposed ADCTD.

Adaptively Dictionary Construction for Hyperspectral Target Detection

In recent years, hyperspectral image classification (HSI) has attracted considerable attention. Various methods based on convolution neural networks have achieved outstanding classification results. However, most of them exited the defects of underutilization of spectral-spatial features, redundant information, and convergence difficulty. To address these problems, a novel 3D-2D multibranch feature fusion and dense attention network are proposed for HSI classification. Specifically, the 3D multibranch feature fusion module integrates multiple receptive fields in spatial and spectral dimensions to obtain shallow features. Then, a 2D densely connected attention module consists of densely connected layers and spatial-channel attention block. The former is used to alleviate the gradient vanishing and enhance the feature reuse during the training process. The latter emphasizes meaningful features and suppresses the interfering information along the two principal dimensions: channel and spatial axes. The experimental results on four benchmark hyperspectral images datasets demonstrate that the model can effectively improve the classification performance with great robustness.

Yiyan Zhang

Papers

A Multiscale Dual-Branch Feature Fusion and Attention Network for Hyperspectral Images Classification

Global to Local: A Hierarchical Detection Algorithm for Hyperspectral Image Target Detection

Dynamic Data Augmentation Method for Hyperspectral Image Classification Based on Siamese Structure

Adaptively Dictionary Construction for Hyperspectral Target Detection

A 3D-2D Multibranch Feature Fusion and Dense Attention Network for Hyperspectral Image Classification.