Rapid technological advances in airborne hyperspectral and lidar systems paved the way for using machine learning algorithms to map urban environments. Both hyperspectral and lidar systems can discriminate among many significant urban structures and materials properties, which are not recognizable by applying conventional RGB cameras. In most recent years, the fusion of hyperspectral and lidar sensors has overcome challenges related to the limits of active and passive remote sensing systems, providing promising results in urban land cover classification. This paper presents principles and key features for airborne hyperspectral imaging, lidar, and the fusion of those, as well as applications of these for urban land cover classification. In addition, machine learning and deep learning classification algorithms suitable for classifying individual urban classes such as buildings, vegetation, and roads have been reviewed, focusing on extracted features critical for classification of urban surfaces, transferability, dimensionality, and computational expense.

/pdf/hyperspectral-and-lidar-data-applied-to-the-urban-land-cover-1zvls4g61n.pdf

Hyperspectral and Lidar Data Applied to the Urban Land Cover Machine Learning and Neural-Network-Based Classification: A Review

As one of the most devastating disasters to pine forests, pine wilt disease (PWD) has caused tremendous ecological and economic losses in China. An effective way to prevent large-scale PWD outbreaks is to detect and remove the damaged pine trees at the early stage of PWD infection. However, early infected pine trees do not show obvious changes in morphology or color in the visible wavelength range, making early detection of PWD tricky. Unmanned aerial vehicle (UAV)-based hyperspectral imagery (HI) has great potential for early detection of PWD. However, the commonly used methods, such as the two-dimensional convolutional neural network (2D-CNN), fail to simultaneously extract and fully utilize the spatial and spectral information, whereas the three-dimensional convolutional neural network (3D-CNN) is able to collect this information from raw hyperspectral data. In this paper, we applied the residual block to 3D-CNN and constructed a 3D-Res CNN model, the performance of which was then compared with that of 3D-CNN, 2D-CNN, and 2D-Res CNN in identifying PWD-infected pine trees from the hyperspectral images. The 3D-Res CNN model outperformed the other models, achieving an overall accuracy (OA) of 88.11% and an accuracy of 72.86% for detecting early infected pine trees (EIPs). Using only 20% of the training samples, the OA and EIP accuracy of 3D-Res CNN can still achieve 81.06% and 51.97%, which is superior to the state-of-the-art method in the early detection of PWD based on hyperspectral images. Collectively, 3D-Res CNN was more accurate and effective in early detection of PWD. In conclusion, 3D-Res CNN is proposed for early detection of PWD in this paper, making the prediction and control of PWD more accurate and effective. This model can also be applied to detect pine trees damaged by other diseases or insect pests in the forest.

/pdf/three-dimensional-convolutional-neural-network-model-for-2lu77k76zj.pdf

Three-Dimensional Convolutional Neural Network Model for Early Detection of Pine Wilt Disease Using UAV-Based Hyperspectral Images

Convolutional neural networks (CNNs) with excellent spatial feature extraction abilities have become popular in remote sensing (RS) image change detection (CD). However, CNNs often focus on the extraction of spatial information but ignore important spectral and temporal sequences for hyperspectral images (HSIs). In this article, we propose a joint spectral, spatial, and temporal transformer for hyperspectral image change detection (HSI-CD), named SST-Former. First, the SST-Former position-encodes each pixel on the cube to remember the spectral and spatial sequences. Second, a spectral transformer encoder structure is used to extract spectral sequence information. Then, a class token for storing the class information of a single temporal HSI concatenates the output of the spectral transformer encoder. The spatial transformer encoder is used to extract spatial texture information in the next step. Finally, the features of different temporal HSIs are sent as the input of temporal transformer, which is used to extract useful CD features between the current HSI pairs and obtain the binary CD result through multilayer perceptron (MLP). We evaluate the SST-Former on three HSI-CD datasets by numerous experiments, showing that it performs better than other excellent methods both visually and qualitatively. The codes of this work will be available at https://github.com/yanhengwang-heu/IEEE_TGRS_SSTFormer.

Spectral–Spatial–Temporal Transformers for Hyperspectral Image Change Detection

ABSTRACT We propose a fully octave convolution network with a pyramid attention mechanism (FOctConvPA) for whole Hyperspectral Image (HSI) classification. Because of the spatial relevance between pixels, effective extraction of global context and long-distance information can help improve the classification performance. However, global information flow is usually accompanied by the loss of local information and details. Therefore, in pursuit of solving this problem effectively, firstly, the proposed method based on the fully convolution network (FCN) uses the whole image as input to reserve original global information, and octave convolution is adopted to reduce spatial computing redundancy and highlight high-frequency edge details. Secondly, the information passes through the encoder and decoder in a resolution lossless form, so the loss of detail can be avoided as far as possible. Thirdly, we utilize the pyramid attention (PA) mechanism to achieve the purpose of global information flowing while long-distance dependencies are captured on both spatial and spectral dimensions. Finally, the mainstream information passing through the decoder is fused with initial information and symmetrical shallow information to improve classification accuracy and restore category boundary and details. Furthermore, to avoid the conflict between limited graphics processing unit (GPU) memory and the randomness of image size, a novel resolution reconstruction classification paradigm is proposed for our network. Experiments on four real benchmark hyperspectral datasets demonstrate that the proposed method could achieve better classification performance compared with other state-of-the-art deep learning models.

Resolution reconstruction classification: fully octave convolution network with pyramid attention mechanism for hyperspectral image classification

Snapshot hyperspectral imaging polarimetry with full spectropolarimetric resolution

Change detection (CD) aims to identify differences in scenes observed at different times. Hyperspectral image (HSI) is preferred for the understanding of land surface changes, since it can provide essential and unique features for CD. However, due to the high-dimensionality and limited data, the HSI-CD task is challenged. While model-driven CD methods are hard to achieve high accuracy due to the weak detection performance for fine changes, data-driven CD methods are hard to be generalized due to the limited datasets. The state-of-art method is to combine a single model-driven method with a data-driven convolutional neural network (CNN). Wherein the pseudolabels can be generated automatically by the model-driven method and then fed to CNN for training. However, the final detection accuracy is limited by the model-driven method which produces pseudolabels with one-sidedness and low accuracy. Therefore, the generation of credible pseudolabels is anticipated and crucial for such a combination. Herein, a novel strategy, the combination of two complementary model-driven methods, structural similarity (SSIM) and change vector analysis (CVA), is proposed to generate credible labels for training a subsequent CNN. The results show that the final accuracy is higher than that of SSIM and CVA. The main contributions of this article are threefold: First, a new paradigm for generating credible labels is proposed. Second, SSIM is used for the first time for HSI-CD tasks. Third, an unsupervised end-to-end framework is presented for the HSI-CD. Experimental results demonstrate the effectiveness of the proposed framework.

/pdf/unsupervised-hyperspectral-image-change-detection-via-deep-5bfyfv28cn.pdf

Unsupervised Hyperspectral Image Change Detection via Deep Learning Self-Generated Credible Labels

Hyperspectral images are widely used for classification due to its rich spectral information along with spatial information. To process the high dimensionality and high nonlinearity of hyperspectral images, deep learning methods based on convolutional neural network (CNN) are widely used in hyperspectral classification applications. However, most CNN structures are stacked vertically in addition to using a onefold size of convolutional kernels or pooling layers, which cannot fully mine the multiscale information on the hyperspectral images. When such networks meet the practical challenge of a limited labeled hyperspectral image dataset—i.e., “small sample problem”—the classification accuracy and generalization ability would be limited. In this paper, to tackle the small sample problem, we apply the semantic segmentation function to the pixel-level hyperspectral classification due to their comparability. A lightweight, multiscale squeeze-and-excitation pyramid pooling network (MSPN) is proposed. It consists of a multiscale 3D CNN module, a squeezing and excitation module, and a pyramid pooling module with 2D CNN. Such a hybrid 2D-3D-CNN MSPN framework can learn and fuse deeper hierarchical spatial–spectral features with fewer training samples. The proposed MSPN was tested on three publicly available hyperspectral classification datasets: Indian Pine, Salinas, and Pavia University. Using 5%, 0.5%, and 0.5% training samples of the three datasets, the classification accuracies of the MSPN were 96.09%, 97%, and 96.56%, respectively. In addition, we also selected the latest dataset with higher spatial resolution, named WHU-Hi-LongKou, as the challenge object. Using only 0.1% of the training samples, we could achieve a 97.31% classification accuracy, which is far superior to the state-of-the-art hyperspectral classification methods.

Multiscale Information Fusion for Hyperspectral Image Classification Based on Hybrid 2D-3D CNN

Polarized light scattering spectroscopy (PLSS) is a promising optical technique developed for the detection of cancer, which extracts the single scattering light to infer morphological information of epithelial cells. However, traditional PLSS uses either a rotatable polarizer or two orthogonal polarizers to purify the single scattering light, which makes it complicated and challenged to build a PLSS endoscope. Herein, we propose a snapshot PLSS with a single optical path to directly get the single scattering light for the first time. The single scattering light is encoded using the spectrally-modulated polarimetry and decoded using the continuous slide iterative method. Both the polystyrene microsphere solutions and the ex vivo gastric cancer samples are used to verify the method. The experimental results of the snapshot PLSS are consistent well with that of the traditional PLSS. The proposed method has a potential for the building of snapshot PLSS endoscope systems in future.

Snapshot polarized light scattering spectroscopy using spectrally-modulated polarimetry for early gastric cancer detection

Usually, the practical analysis states of an imaging polarimeter needs to be calibrated, with a set of standard polarization states, for the accurate reconstruction of Stokes parameters. However, it is really challenged to get the standard elements over wide field of view (FOV), broad waveband, large aperture, or other non-trivial conditions. Even if the system is well calibrated, the calibrated system will be disturbed in the vibration environment. To avoid the difficult from the standard polarization states, an iterative reconstruction method is presented at the first time to recover the polarization parameters from the data acquired by linear-Stokes polarimeters without polarimetric calibrations. Inspired from phase shifting interferometry, the method employs two least-squares iterative procedure and requires no any extra element for assistant. And we extend the method to a channeled linear imaging spectropolarimeter, channeled linear imaging spectropolarimeter can measure a two-dimensional distribution of spectrally-resolved linear Stokes parameters in a single-shot polarization modulation. However, the state-of-art reconstruction method, Fourier transform method (FTM), usually transforms the modulated spectrum into the frequency domain for further processing. As a result, there is channel crosstalk issue that limits available frequency bandwidth. In addition, FTM needs extra phase calibration to decode final spectra. We present a continuous slide iterative method (CSIM) in the spectral domain to avoid the use of the Fourier transform and phase calibration. It combines a sliding unit cell kernel in the spectral domain that provides unit cell tracking and a loop of twostep least-squares fit that estimates spatially-resolved polarized spectra.

Wenjing Wang

Papers

Unsupervised Hyperspectral Image Change Detection via Deep Learning Self-Generated Credible Labels

Multiscale Information Fusion for Hyperspectral Image Classification Based on Hybrid 2D-3D CNN

Snapshot hyperspectral imaging polarimetry with full spectropolarimetric resolution

Snapshot polarized light scattering spectroscopy using spectrally-modulated polarimetry for early gastric cancer detection

Iterative reconstruction for linear imaging spectropolarimetry without polarimetric calibration