Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various datasets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning datasets and methods for scene classification is still lacking. In addition, almost all existing datasets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale dataset, termed "NWPU-RESISC45", which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This dataset contains 31,500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 (i) is large-scale on the scene classes and the total image number, (ii) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion, and (iii) has high within-class diversity and between-class similarity. The creation of this dataset will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed dataset and the results are reported as a useful baseline for future research.

Remote Sensing Image Scene Classification: Benchmark and State of the Art

Remote sensing image scene classification is an active and challenging task driven by many applications. More recently, with the advances of deep learning models especially convolutional neural networks (CNNs), the performance of remote sensing image scene classification has been significantly improved due to the powerful feature representations learnt through CNNs. Although great success has been obtained so far, the problems of within-class diversity and between-class similarity are still two big challenges. To address these problems, in this paper, we propose a simple but effective method to learn discriminative CNNs (D-CNNs) to boost the performance of remote sensing image scene classification. Different from the traditional CNN models that minimize only the cross entropy loss, our proposed D-CNN models are trained by optimizing a new discriminative objective function. To this end, apart from minimizing the classification error, we also explicitly impose a metric learning regularization term on the CNN features. The metric learning regularization enforces the D-CNN models to be more discriminative so that, in the new D-CNN feature spaces, the images from the same scene class are mapped closely to each other and the images of different classes are mapped as farther apart as possible. In the experiments, we comprehensively evaluate the proposed method on three publicly available benchmark data sets using three off-the-shelf CNN models. Experimental results demonstrate that our proposed D-CNN methods outperform the existing baseline methods and achieve state-of-the-art results on all three data sets.

When Deep Learning Meets Metric Learning: Remote Sensing Image Scene Classification via Learning Discriminative CNNs

Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various data sets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning data sets and methods for scene classification is still lacking. In addition, almost all existing data sets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale data set, termed “NWPU-RESISC45,” which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This data set contains 31 500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 1) is large-scale on the scene classes and the total image number; 2) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion; and 3) has high within-class diversity and between-class similarity. The creation of this data set will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed data set, and the results are reported as a useful baseline for future research.

In recent years, deep learning (DL), a rebranding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, and natural language processing. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV, e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should not only be aware of advancements such as DL, but also be leading researchers in this area. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools, and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as they relate to (i) inadequate data sets, (ii) human-understandable solutions for modeling physical phenomena, (iii) big data, (iv) nontraditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial, and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.

Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community

Scene classification of remote sensing images has drawn great attention because of its wide applications. In this paper, with the guidance of the human visual system (HVS), we explore the attention mechanism and propose a novel end-to-end attention recurrent convolutional network (ARCNet) for scene classification. It can learn to focus selectively on some key regions or locations and just process them at high-level features, thereby discarding the noncritical information and promoting the classification performance. The contributions of this paper are threefold. First, we design a novel recurrent attention structure to squeeze high-level semantic and spatial features into several simplex vectors for the reduction of learning parameters. Second, an end-to-end network named ARCNet is proposed to adaptively select a series of attention regions and then to generate powerful predictions by learning to process them sequentially. Third, we construct a new data set named OPTIMAL-31, which contains more categories than popular data sets and gives researchers an extra platform to validate their algorithms. The experimental results demonstrate that our model makes great promotion in comparison with the state-of-the-art approaches.

Scene Classification With Recurrent Attention of VHR Remote Sensing Images

Scene classification using local and global features with collaborative representation fusion

Recently, representation-based classifiers have gained increasing interest in hyperspectral image (HSI) clas sification. In this letter, based on our previously developed joint collaborative representation (JCR) classifier, an improved version, which is called weighted JCR (WJCR) classifier, is proposed. JCR adopts the same weights when extracting spatial and spectral features from surrounding pixels. Differing from JCR, WJCR attempts to utilize more appropriate weights by considering the similarity between the center pixel and its surroundings. Experimental results using two real HSIs demon strate that the proposed WJCR outperforms the original JCR and some other traditional classifiers, such as the support vector machine (SVM), the SVM with a composite kernel, and simultaneous orthogonal matching pursuit.

Hyperspectral Image Classification Using Weighted Joint Collaborative Representation

In this letter, a sparse representation-based nearest neighbor (SRNN) classifier is proposed. Unlike the traditional $k$ -nearest neighbor (NN) classifier that employs the Euclidean distance as similarity metric, the proposed SRNN considers sparse coefficients to determine the label of testing samples, since sparse coefficients can reflect the similarity between data and provide more discriminative information. A local SRNN (LSRNN) classifier is also proposed to utilize class-specific sparse coefficients to improve the performance. Furthermore, due to the fact that neighboring pixels tend to belong to the same class with high probability, a spatially joint version of LSRNN, called JSRNN, is developed to further improve LSRNN. The proposed SRNN, LSRNN, and JSRNN have been validated on several hyperspectral remote sensing image data sets. Experimental results demonstrate that the proposed classifiers increase the classification accuracy compared with the traditional $k$ -NN, local mean-based NN (LMNN) classifiers, and original sparse representation classifiers using representation residuals.

Sparse Representation-Based Nearest Neighbor Classifiers for Hyperspectral Imagery

Simultaneous orthogonal matching pursuit (SOMP) has been recently developed for hyperspectral image classification. It utilizes a joint sparsity model with the assumption that each pixel can be represented by a linear combination of labeled samples. We present an approach to improve the performance of SOMP based on a priori segmentation map. According to the map, we first build a local region where within-segment pixels are preserved while between-segment pixels are excluded. Hyperspectral pixels in the preserved region around the test pixel are then simultaneously represented by a linear combination of training samples, whose weights are recovered by solving a sparsity-constrained optimization problem. Finally, the label of the test pixel is determined to be the class that yields the minimal total residuals between the test samples and the approximations. Experimental results demonstrate that the proposed adaptive SOMP (ASOMP) is superior to some existing classifiers, such as the original SOMP and the recently proposed weighted-SOMP (WSOMP). For example, the ASOMP performed with an accuracy of 95.53% for the ROSIS University of Pavia data with 120 training samples per class, while SOMP obtained an accuracy of 87.61%, an improvement of approximately 8%.

Classification of hyperspectral urban data using adaptive simultaneous orthogonal matching pursuit

In this paper, convolutional neural networks (CNNs) is employed for remote-sensing scene classification, which fully utilizes the semantic features extracted from the images while ignoring some traditional features. Consider the limited labeled samples, CaffeNet model as the pre-trained architecture is adopted. By fine-tuning the pre-trained models, the proposed method is expected to be robust and efficient. Its performance is evaluated with two remote-sensing scene datasets. From the experimental results, the proposed CNN-based scene classification method does provide more excellent performance and be superior to several state-of-the-art methods.

Jinyi Zou

Papers

Scene classification using local and global features with collaborative representation fusion

Hyperspectral Image Classification Using Weighted Joint Collaborative Representation

Sparse Representation-Based Nearest Neighbor Classifiers for Hyperspectral Imagery

Classification of hyperspectral urban data using adaptive simultaneous orthogonal matching pursuit

Using CNN-based high-level features for remote sensing scene classification