Modern change detection (CD) has achieved remarkable success by the powerful discriminative ability of deep convolutions. However, high-resolution remote sensing CD remains challenging due to the complexity of objects in the scene. Objects with the same semantic concept may show distinct spectral characteristics at different times and spatial locations. Most recent CD pipelines using pure convolutions are still struggling to relate long-range concepts in space-time. Non-local self-attention approaches show promising performance via modeling dense relations among pixels, yet are computationally inefficient. Here, we propose a bitemporal image transformer (BIT) to efficiently and effectively model contexts within the spatial-temporal domain. Our intuition is that the high-level concepts of the change of interest can be represented by a few visual words, i.e., semantic tokens. To achieve this, we express the bitemporal image into a few tokens, and use a transformer encoder to model contexts in the compact token-based space-time. The learned context-rich tokens are then feedback to the pixel-space for refining the original features via a transformer decoder. We incorporate BIT in a deep feature differencing-based CD framework. Extensive experiments on three CD datasets demonstrate the effectiveness and efficiency of the proposed method. Notably, our BIT-based model significantly outperforms the purely convolutional baseline using only 3 times lower computational costs and model parameters. Based on a naive backbone (ResNet18) without sophisticated structures (e.g., FPN, UNet), our model surpasses several state-of-the-art CD methods, including better than four recent attention-based methods in terms of efficiency and accuracy. Our code is available at this https URL\_CD.

Remote Sensing Image Change Detection with Transformers

In Earth observation (EO), large-scale land-surface dynamics are traditionally analyzed by investigating aggregated classes. The increase in data with a very high spatial resolution enables investigations on a fine-grained feature level which can help us to better understand the dynamics of land surfaces by taking object dynamics into account. To extract fine-grained features and objects, the most popular deep-learning model for image analysis is commonly used: the convolutional neural network (CNN). In this review, we provide a comprehensive overview of the impact of deep learning on EO applications by reviewing 429 studies on image segmentation and object detection with CNNs. We extensively examine the spatial distribution of study sites, employed sensors, used datasets and CNN architectures, and give a thorough overview of applications in EO which used CNNs. Our main finding is that CNNs are in an advanced transition phase from computer vision to EO. Upon this, we argue that in the near future, investigations which analyze object dynamics with CNNs will have a significant impact on EO research. With a focus on EO applications in this Part II, we complete the methodological review provided in Part I.

Object Detection and Image Segmentation with Deep Learning on Earth Observation Data: A Review-Part II: Applications

Modern change detection (CD) has achieved remarkable success by the powerful discriminative ability of deep convolutions. However, high-resolution remote sensing CD remains challenging due to the complexity of objects in the scene. Objects with the same semantic concept may show distinct spectral characteristics at different times and spatial locations. Most recent CD pipelines using pure convolutions are still struggling to relate long-range concepts in space-time. Non-local self-attention approaches show promising performance via modeling dense relations among pixels, yet are computationally inefficient. Here, we propose a bitemporal image transformer (BIT) to efficiently and effectively model contexts within the spatial-temporal domain. Our intuition is that the high-level concepts of the change of interest can be represented by a few visual words, i.e., semantic tokens. To achieve this, we express the bitemporal image into a few tokens, and use a transformer encoder to model contexts in the compact token-based space-time. The learned context-rich tokens are then feedback to the pixel-space for refining the original features via a transformer decoder. We incorporate BIT in a deep feature differencing-based CD framework. Extensive experiments on three CD datasets demonstrate the effectiveness and efficiency of the proposed method. Notably, our BIT-based model significantly outperforms the purely convolutional baseline using only 3 times lower computational costs and model parameters. Based on a naive backbone (ResNet18) without sophisticated structures (e.g., FPN, UNet), our model surpasses several state-of-the-art CD methods, including better than four recent attention-based methods in terms of efficiency and accuracy. Our code is available at https://github.com/justchenhao/BIT\_CD. 

/pdf/remote-sensing-image-change-detection-with-transformers-3ha2h1q4.pdf

Remote Sensing Image Change Detection With Transformers

Hyperspectral and multispectral information processing systems and technologies have demonstrated its usefulness for the improvement of agricultural productivity and practices by providing useful information to farmers and crop managers on the factors affecting crop status and growth. These technologies are widely used in a range of agriculture applications such as crop management, crop yield forecasting, crop disease detection, and the monitoring of agriculture land usage, water, and soil conditions. Hyperspectral information sensing can acquire several hundred spectral bands that cover the electromagnetic spectrum of an observational scene in a single acquisition. The resulting hyperspectral data cube contains a large volume of spatial and spectral information. The hyperspectral sequence of images or video further increases the data generation velocity and volume which lead to the Big data challenges particularly in agricultural remote sensing applications. This paper is structured to first give a comprehensive review of representative studies to provide insights into significant research efforts in agriculture using Big data, machine learning and deep learning with the focus on frameworks or architectures, information processing and analytics with hyperspectral and multispectral data. The potential for utilizing Big data, machine learning and deep learning for hyperspectral and multispectral data in agriculture is very promising. The paper then further explores the potential of using ensemble machine learning and scalable parallel discriminant analysis which takes into consideration the spatial and spectral components for Big data in agriculture. To the best of our knowledge, no similar review study on agriculture with Big data, machine learning and deep learning for hyperspectral and multispectral information processing has been reported. Furthermore, the potential of ensemble machine learning and scalable parallel discriminant analysis has not been explored in agriculture information processing. Experiments and data analytics have been performed on hyperspectral data from agriculture for validation. The results have shown the good performance of our approach.

/pdf/big-data-and-machine-learning-with-hyperspectral-information-1irsy8d7z1.pdf

Big Data and Machine Learning With Hyperspectral Information in Agriculture

Machine learning, particularly deep learning (DL), has become a central and state-of-the-art method for several computer vision applications and remote sensing (RS) image processing. Researchers are continually trying to improve the performance of the DL methods by developing new architectural designs of the networks and/or developing new techniques, such as attention mechanisms. Since the attention mechanism has been proposed, regardless of its type, it has been increasingly used for diverse RS applications to improve the performances of the existing DL methods. However, these methods are scattered over different studies impeding the selection and application of the feasible approaches. This study provides an overview of the developed attention mechanisms and how to integrate them with different deep learning neural network architectures. In addition, it aims to investigate the effect of the attention mechanism on deep learning-based RS image processing. We identified and analyzed the advances in the corresponding attention mechanism-based deep learning (At-DL) methods. A systematic literature review was performed to identify the trends in publications, publishers, improved DL methods, data types used, attention types used, overall accuracies achieved using At-DL methods, and extracted the current research directions, weaknesses, and open problems to provide insights and recommendations for future studies. For this, five main research questions were formulated to extract the required data and information from the literature. Furthermore, we categorized the papers regarding the addressed RS image processing tasks (e.g., image classification, object detection, and change detection) and discussed the results within each group. In total, 270 papers were retrieved, of which 176 papers were selected according to the defined exclusion criteria for further analysis and detailed review. The results reveal that most of the papers reported an increase in overall accuracy when using the attention mechanism within the DL methods for image classification, image segmentation, change detection, and object detection using remote sensing images.

Effect of Attention Mechanism in Deep Learning-Based Remote Sensing Image Processing: A Systematic Literature Review

Multitemporal Earth observation capability plays an increasingly important role in crop monitoring. As the frequency of satellite acquisition of remote sensing images becomes higher, how to fully exploit the implicit phenological laws in dense multitemporal data is of increasing importance. In this article, we propose a CNN-transformer approach to perform the crop classification, in the model, we borrow the transformer architecture from the knowledge of NLP to dig into the pattern of multitemporal sequence. First, after unifying the spatial-spectral scale of each multiband data acquired from different sensors, we obtain the scale-consistent feature and position feature of multitemporal sequence. Second, with adopting multilayer encoder modules derived from the transformer, we mine deep correlation patterns of multitemporal sequence. Finally, the feed-forward layer and softmax layer serve as output layers of the model to predict crop categories. The proposed CNN-transformer approach is illustrated in a crop-rich agricultural region in central California, where 65 multitemporal profiles from multisensor Sentinel-2 A/B and Landsat-8 are obtained in 2018. Through multiband multiresolution fusion, sequence correlation extraction of multitemporal data and category feature extraction, the classification results show that the proposed method has a significant performance improvement compared with other traditional methods.

/pdf/a-cnn-transformer-hybrid-approach-for-crop-classification-34armczkrw.pdf

A CNN-Transformer Hybrid Approach for Crop Classification Using Multitemporal Multisensor Images

Crop classification based on multitemporal multisensor remote sensing imagery has a great significance. As more and more Earth observation satellites are launched, it becomes easier to obtain increasingly dense multitemporal data, and the category patterns hidden in multitemporal data can be more and more finely mined and expressed. However, the traditional classification methods treat the features of different temporal periods consistently, so the classification performances are not well enough, especially for the categories with subtle phenological differences. In this paper, we propose a temporal-attention CNN-GRU (Convolutional Neural Networks and Gated Recurrent Unit Networks) approach to distinguish subtle crop differences, and the temporal attention mechanism introduced in the model can achieve the effect of enhancing phenological differences and suppressing phenological similarity. Firstly, we use the GRU networks to model the temporal correlation of the multitemporal data. And then, the temporal attention layer utilize a query module to retrieve “what is the important information” over the whole temporal sequence, so we can obtain the attention weights for each temporal period. In the experiments, multitemporal samples are collected from Sentinel-2A/B and Landsat-8. Due to the different spatial resolutions of multiband images, transposed convolution which can extract the raw spatial structure information is used for multiband features fusion. Experimental results on multitemporal data show that the proposed approach achieve the best performance compared with conventional methods, especially for the categories with similar phenological laws.

/pdf/temporal-attention-networks-for-multitemporal-multisensor-2afov8hgh4.pdf

Temporal Attention Networks for Multitemporal Multisensor Crop Classification

Affected by geographical location and climatic conditions, crop classification in the Yunnan Plateau of China is greatly restricted by the low utilization rate of annual optical data, complex crop planting structure, and broken cultivated land. This paper combines monthly Sentinel-2 optical remote sensing data with Sentinel-1 radar data to minimize cloud interference to conduct crop classification for plateau areas. However, pixel classification will inevitably produce a “different spectrum of the same object, foreign objects in the same spectrum”. A principal component feature synthesis method is developed for multi-source remote sensing data (PCA-MR) to improve classification accuracy. In order to compare and analyze the classification effect of PCA-MR combined with multi-source remote sensing data, we constructed 11 classification scenarios using the Google Earth Engine platform and random forest algorithm (RF). The results show that: (1) the classification accuracy is 79.98% by using Sentinel-1 data and 91.18% when using Sentinel-2 data. When integrating Sentinel-1 and Sentinel-2 data, the accuracy is 92.31%. By analyzing the influence of texture features on classification under different feature combinations, it was found that optical texture features affected the recognition accuracy of rice to a lesser extent. (2) The errors will be reduced if the PCA-MR feature is involved in the classification, and the classification accuracy and Kappa coefficient are improved to 93.47% and 0.92, respectively.

/pdf/rice-and-greenhouse-identification-in-plateau-areas-1t91e78u.pdf

Rice and Greenhouse Identification in Plateau Areas Incorporating Sentinel-1/2 Optical and Radar Remote Sensing Data from Google Earth Engine

This paper proposes a superpixel spatial intuitionistic fuzzy C-means (SSIFCM) clustering algorithm to address the problems of misclassification, salt and pepper noise, and classification uncertainty arising in the pixel-level unsupervised classification of high spatial resolution remote sensing (HSRRS) images. To reduce information redundancy and ensure noise immunity and image detail preservation, we first use a superpixel segmentation to obtain the local spatial information of the HSRRS image. Secondly, based on the bias-corrected fuzzy C-means (BCFCM) clustering algorithm, the superpixel spatial intuitionistic fuzzy membership matrix is constructed by counting an intuitionistic fuzzy set and spatial function. Finally, to minimize the classification uncertainty, the local relation between adjacent superpixels is used to obtain the classification results according to the spectral features of superpixels. Four HSRRS images of different scenes in the aerial image dataset (AID) are selected to analyze the classification performance, and fifteen main existing unsupervised classification algorithms are used to make inter-comparisons with the proposed SSIFCM algorithm. The results show that the overall accuracy and Kappa coefficients obtained by the proposed SSIFCM algorithm are the best within the inter-comparison of fifteen algorithms, which indicates that the SSIFCM algorithm can effectively improve the classification accuracy of HSRRS image.

/pdf/a-superpixel-spatial-intuitionistic-fuzzy-c-means-clustering-9l06z0fb.pdf

A Superpixel Spatial Intuitionistic Fuzzy C-Means Clustering Algorithm for Unsupervised Classification of High Spatial Resolution Remote Sensing Images

Guokun Chen

Papers

A CNN-Transformer Hybrid Approach for Crop Classification Using Multitemporal Multisensor Images

Temporal Attention Networks for Multitemporal Multisensor Crop Classification

Rice and Greenhouse Identification in Plateau Areas Incorporating Sentinel-1/2 Optical and Radar Remote Sensing Data from Google Earth Engine

A Superpixel Spatial Intuitionistic Fuzzy C-Means Clustering Algorithm for Unsupervised Classification of High Spatial Resolution Remote Sensing Images