Deep learning (DL) has exhibited huge potentials for hyperspectral image (HSI) classification due to its powerful nonlinear modeling and end-to-end optimization characteristics. Although the superior performance of DL-based methods has been witnessed, some limitations can still be found. On the one hand, existing DL frameworks usually resorted to first-order statistical features, whereas they rarely considered second-order or higher order statistical features. On the other hand, the optimization of complex hyperparameters (e.g., the layer number and convolutional kernel size) is time-consuming and a very tough task, making the designed DL framework unexplainable. To overcome these challenges, we propose a novel attention-based second-order pooling network (A-SPN). First, a first-order feature operator is designed to model the spectral–spatial information of HSI. Second, an attention-based second-order pooling (A-SOP) operator is designed to model discriminative and representative features. Finally, a fully connected layer with softmax loss is used for classification. The proposed framework can obtain second-order statistical features in an end-to-end manner. In addition, A-SPN is free of complex hyperparameters tuning, making it more explainable and easily equipped for classification tasks. Experimental results based on three common hyperspectral data sets demonstrate that A-SPN outperforms other traditional and state-of-the-art DL-based HSI classification methods in terms of generalization performance with limited training samples, classification accuracy, convergence rate, and computational complexity.

Attention-Based Second-Order Pooling Network for Hyperspectral Image Classification

Machine learning, particularly deep learning (DL), has become a central and state-of-the-art method for several computer vision applications and remote sensing (RS) image processing. Researchers are continually trying to improve the performance of the DL methods by developing new architectural designs of the networks and/or developing new techniques, such as attention mechanisms. Since the attention mechanism has been proposed, regardless of its type, it has been increasingly used for diverse RS applications to improve the performances of the existing DL methods. However, these methods are scattered over different studies impeding the selection and application of the feasible approaches. This study provides an overview of the developed attention mechanisms and how to integrate them with different deep learning neural network architectures. In addition, it aims to investigate the effect of the attention mechanism on deep learning-based RS image processing. We identified and analyzed the advances in the corresponding attention mechanism-based deep learning (At-DL) methods. A systematic literature review was performed to identify the trends in publications, publishers, improved DL methods, data types used, attention types used, overall accuracies achieved using At-DL methods, and extracted the current research directions, weaknesses, and open problems to provide insights and recommendations for future studies. For this, five main research questions were formulated to extract the required data and information from the literature. Furthermore, we categorized the papers regarding the addressed RS image processing tasks (e.g., image classification, object detection, and change detection) and discussed the results within each group. In total, 270 papers were retrieved, of which 176 papers were selected according to the defined exclusion criteria for further analysis and detailed review. The results reveal that most of the papers reported an increase in overall accuracy when using the attention mechanism within the DL methods for image classification, image segmentation, change detection, and object detection using remote sensing images.

Effect of Attention Mechanism in Deep Learning-Based Remote Sensing Image Processing: A Systematic Literature Review

An extreme learning machine is a feed-forward neural network is a feature learning method used in image classification efficiently because of the faster learning rate, speed, good generalization ability, ease of implementation, and efficiency in classifications. This paper presents a new model for the classification of images by using an extreme learning machine classifier which works in two stages using deep learning techniques. A new framework is built by using the dropout technique of CNN for classifying images with less training time and also no need to over train neurons. The proposed model has a high accuracy of 98% using 1000 images. The proposed paper accomplishes the best results in image classification compare with other related methods on image classification.

Dropout technique for image classification based on extreme learning machine

The performance of hyperspectral image (HSI) classification is highly dependent on spatial and spectral information, and is heavily affected by factors such as data redundancy and insufficient spatial resolution. To overcome these challenges, many convolutional neural networks (CNN) especially 2D-CNN-based methods have been proposed for HSI classification. However, these methods produced insufficient results compared to 3D-CNN-based methods. On the other hand, the high computational complexity of the 3D-CNN-based methods is still a major concern that needs to be addressed. Therefore, this study introduces a consolidated convolutional neural network (C-CNN) to overcome the aforementioned issues. The proposed C-CNN is comprised of a three-dimension CNN (3D-CNN) joined with a two-dimension CNN (2D-CNN). The 3D-CNN is used to represent spatial–spectral features from the spectral bands, and the 2D-CNN is used to learn abstract spatial features. Principal component analysis (PCA) was firstly applied to the original HSIs before they are fed to the network to reduce the spectral bands redundancy. Moreover, image augmentation techniques including rotation and flipping have been used to increase the number of training samples and reduce the impact of overfitting. The proposed C-CNN that was trained using the augmented images is named C-CNN-Aug. Additionally, both Dropout and L2 regularization techniques have been used to further reduce the model complexity and prevent overfitting. The experimental results proved that the proposed model can provide the optimal trade-off between accuracy and computational time compared to other related methods using the Indian Pines, Pavia University, and Salinas Scene hyperspectral benchmark datasets.

/pdf/consolidated-convolutional-neural-network-for-hyperspectral-3fndf4cg.pdf

Consolidated Convolutional Neural Network for Hyperspectral Image Classification

The unmanned surface vehicles (USVs) have been regarded as a promising paradigm to automatically perform emergency tasks in a dynamic maritime traffic environment. However, the performance of maritime communication between USVs and offshore platforms becomes a critical challenge, and the efficiency of task allocation for USVs in the smart ocean is low. In this article, a novel task allocation scheme for USVs in the smart ocean Internet of Things (IoT) is proposed to improve the efficiency of task allocation. First, the offshore platform is developed to provide maritime communication for USVs in the smart ocean IoT. Second, the network resource allocation process between USVs and offshore platforms is modeled as the second price sealed auction game, where the optimal bidding strategy of USV is derived by the $Q$ -learning to maximize the utilities of USVs and offshore platforms. Third, the task allocation scheme is proposed to improve the number of allocated tasks. Finally, the performance of the proposed scheme is conducted based on extensive simulations. The simulation results show that the proposed scheme can significantly improve the number of allocated tasks compared with the conventional schemes.

Task Allocation With Unmanned Surface Vehicles in Smart Ocean IoT

Recently, convolution neural network (CNN)-based hyperspectral image (HSI) classification has enjoyed high popularity due to its appealing performance. However, using 2-D or 3-D convolution in a standalone mode may be suboptimal in real applications. On the one hand, the 2-D convolution overlooks the spectral information in extracting feature maps. On the other hand, the 3-D convolution suffers from heavy computation in practice and seems to perform poorly in scenarios having analogous textures along with consecutive spectral bands. To solve these problems, we propose a mixed CNN with covariance pooling for HSI classification. Specifically, our network architecture starts with spectral–spatial 3-D convolutions that followed by a spatial 2-D convolution. Through this mixture operation, we fuse the feature maps generated by 3-D convolutions along the spectral bands for providing complementary information and reducing the dimension of channels. In addition, the covariance pooling technique is adopted to fully extract the second-order information from spectral–spatial feature maps. Motivated by the channel-wise attention mechanism, we further propose two principal component analysis (PCA)-involved strategies, channel-wise shift and channel-wise weighting, to highlight the importance of different spectral bands and recalibrate channel-wise feature response, which can effectively improve the classification accuracy and stability, especially in the case of limited sample size. To verify the effectiveness of the proposed model, we conduct classification experiments on three well-known HSI data sets, Indian Pines, University of Pavia, and Salinas Scene. The experimental results show that our proposal, although with less parameters, achieves better accuracy than other state-of-the-art methods.

Hyperspectral Image Classification Using Mixed Convolutions and Covariance Pooling

Low-rank modeling has achieved great success in visual data completion. However, the low-rank assumption of original visual data may be in approximate mode, which leads to suboptimality for the recovery of underlying details, especially when the missing rate is extremely high. In this paper, we go further by providing a detailed analysis about the rank distributions in Hankel structured and clustered cases, and figure out both non-local similarity and patch-based structuralization play a positive role. This motivates us to develop a new Hankel low-rank tensor recovery method that is competent to truthfully capture the underlying details with sacrifice of slightly more computational burden. First, benefiting from the correlation of different spectral bands and the smoothness of local spatial neighborhood, we divide the visual data into overlapping 3D patches and group the similar ones into individual clusters exploring the non-local similarity. Second, the 3D patches are individually mapped to the structured Hankel tensors for better revealing low-rank property of the image. Finally, we solve the tensor completion model via the well-known alternating direction method of multiplier (ADMM) optimization algorithm. Due to the fact that size expansion happens inevitably in Hankelization operation, we further propose a fast randomized skinny tensor singular value decomposition (rst-SVD) to accelerate the periteration running efficiency. Extensive experimental results on real world datasets verify the superiority of our method compared to the state-of-the-art visual inpainting approaches.

Fast Tensor Nuclear Norm for Structured Low-Rank Visual Inpainting

Owing to the outstanding feature extraction capability, convolutional neural networks (CNNs) have been widely applied in hyperspectral image (HSI) classification problems and have achieved an impressive performance. However, it is well known that 2D convolution suffers from the absent consideration of spectral information, while 3D convolution requires a huge amount of computational cost. In addition, the cost of labeling and the limitation of computing resources make it urgent to improve the generalization performance of the model with scarcely labeled samples. To relieve these issues, we design an end-to-end 3D octave and 2D vanilla mixed CNN, namely Oct-MCNN-HS, based on the typical 3D-2D mixed CNN (MCNN). It is worth mentioning that two feature fusion operations are deliberately constructed to climb the top of the discriminative features and practical performance. That is, 2D vanilla convolution merges the feature maps generated by 3D octave convolutions along the channel direction, and homology shifting aggregates the information of the pixels locating at the same spatial position. Extensive experiments are conducted on four publicly available HSI datasets to evaluate the effectiveness and robustness of our model, and the results verify the superiority of Oct-MCNN-HS both in efficacy and efficiency.

3D Octave and 2D Vanilla Mixed Convolutional Neural Network for Hyperspectral Image Classification with Limited Samples

Tensor completion using patch-wise high order Hankelization and randomized tensor ring initialization

The invention discloses a hyperspectral remote sensing image classification method based on three-dimensional and two-dimensional hybrid convolution. The hyperspectral remote sensing image classification method comprises the steps of obtaining a to-be-classified hyperspectral remote sensing image; performing spectral dimension reduction by using a principal component analysis method; arranging spectral bands in the hyperspectral remote sensing image after dimension reduction from high to low along the middle of the channel to two sides of the channel according to the spectral information amount; endowing the spectral band with a corresponding weight according to the spectral information amount contained in the spectral band; taking cube data with a fixed space size from each pixel point inthe spectral band, extracting spectral-spatial features according to the cube data by utilizing three-dimensional convolution, and fusing spectral information by utilizing two-dimensional convolutionto obtain a final feature map; extracting second-order information from the feature map by adopting a covariance pooling method, and outputting a feature vector; and inputting the feature vector intoa three-layer full-connection network to obtain a prediction classification result. The method combines the advantages of three-dimensional convolution and two-dimensional convolution, and achieves the accurate classification of hyperspectral remote sensing images under low training samples.

Feng Yuchao

Papers

Hyperspectral Image Classification Using Mixed Convolutions and Covariance Pooling

Fast Tensor Nuclear Norm for Structured Low-Rank Visual Inpainting

3D Octave and 2D Vanilla Mixed Convolutional Neural Network for Hyperspectral Image Classification with Limited Samples

Tensor completion using patch-wise high order Hankelization and randomized tensor ring initialization

Hyperspectral remote sensing image classification method based on three-dimensional and two-dimensional hybrid convolution