In deep learning research, self-supervised learning (SSL) has received great attention, triggering interest within both the computer vision and remote sensing communities. While there has been big success in computer vision, most of the potential of SSL in the domain of Earth observation remains locked. In this article, we provide an introduction to and a review of the concepts and latest developments in SSL for computer vision in the context of remote sensing. Further, we provide a preliminary benchmark of modern SSL algorithms on popular remote sensing datasets, verifying the potential of SSL in remote sensing and providing an extended study on data augmentations. Finally, we identify a list of promising directions of future research in SSL for Earth observation (SSL4EO) to pave the way for the fruitful interaction of both domains.

Self-Supervised Learning in Remote Sensing: A review

In deep learning research, self-supervised learning (SSL) has received great attention, triggering interest within both the computer vision and remote sensing communities. While there has been big success in computer vision, most of the potential of SSL in the domain of Earth observation remains locked. In this article, we provide an introduction to and a review of the concepts and latest developments in SSL for computer vision in the context of remote sensing. Further, we provide a preliminary benchmark of modern SSL algorithms on popular remote sensing datasets, verifying the potential of SSL in remote sensing and providing an extended study on data augmentations. Finally, we identify a list of promising directions of future research in SSL for Earth observation (SSL4EO) to pave the way for the fruitful interaction of both domains. 

/pdf/self-supervised-learning-in-remote-sensing-a-review-2v8jveba.pdf

Deep learning has achieved great success in learning features from massive remote sensing images (RSIs). To better understand the connection between three feature learning paradigms, which are unsupervised feature learning (USFL), supervised feature learning (SFL), and self-SFL (SSFL), this article analyzes and compares them from the perspective of feature learning signals and gives a unified feature learning framework. Under this unified framework, we analyze the advantages of SSFL over the other two learning paradigms in RSI understanding tasks and give a comprehensive review of existing SSFL works in RS, including the pretraining dataset, SSFL signals, and evaluation methods. We further analyze the effects of SSFL signals and pretraining data on the learned features to provide insights into RSI feature learning. Finally, we briefly discuss some open problems and possible research directions.

Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works

Remote sensing image fusion on 3D scenarios: A review of applications for agriculture and forestry

Modern Earth observation satellites capture multi-exposure bursts of push-frame images that can be super-resolved via computational means. In this work, we propose a super-resolution method for such multi-exposure sequences, a problem that has received very little attention in the literature. The proposed method can handle the signal-dependent noise in the inputs, process sequences of any length, and be robust to inaccuracies in the exposure times. Furthermore, it can be trained end-to-end with self-supervision, without requiring ground truth high resolution frames, which makes it especially suited to handle real data. Central to our method are three key contributions: i) a base-detail decomposition for handling errors in the exposure times, ii) a noise-level-aware feature encoding for improved fusion of frames with varying signal-to-noise ratio and iii) a permutation invariant fusion strategy by temporal pooling operators. We evaluate the proposed method on synthetic and real data and show that it outperforms by a significant margin existing single-exposure approaches that we adapted to the multi-exposure case.

Self-Supervised Super-Resolution for Multi-Exposure Push-Frame Satellites

Recent constellations of optical satellites are adopting multi-image super-resolution (MISR) from bursts of push-frame images as a way to increase the resolution and reduce the noise of their products while maintaining a lower cost of operation. Most MISR techniques are currently based on the aggregation of samples from registered low resolution images. A promising research trend aimed at incorporating natural image priors in MISR consists in using data-driven neural networks. However, due to the unavailability of ground truth high resolution data, these networks cannot be trained on real satellite images. In this paper, we present a framework for training MISR algorithms from bursts of satellite images without requiring high resolution ground truth. This is achieved by adapting the recently proposed frame-to-frame framework to process bursts of satellite images. In addition we propose an architecture based on feature aggregation that allows to fuse a variable number of frames and is capable of handling degenerate samplings while also reducing noise. On synthetic datasets, the proposed self-supervision strategy attains results on par with those obtained with a supervised training. We applied our framework to real SkySat satellite image bursts leading to results that are more resolved and less noisy than the L1B product from Planet.

/pdf/self-supervised-multi-image-super-resolution-for-push-frame-1xehfue2sx.pdf

Self-supervised multi-image super-resolution for push-frame satellite images

The PROBA-V Super-Resolution challenge distributes real low-resolution image series and corresponding high-resolution targets to advance research on Multi-Image Super Resolution (MISR) for satellite images. However, in the PROBA-V dataset the low-resolution image corresponding to the high-resolution target is not identified. We argue that in doing so, the challenge ranks the proposed methods not only by their MISR performance, but mainly by the heuristics used to guess which image in the series is the most similar to the high-resolution target. We demonstrate this by improving the performance obtained by the two winners of the challenge only by using a different reference image, which we compute following a simple heuristic. Based on this, we propose PROBA-V-REF a variant of the PROBA-V dataset, in which the reference image in the low-resolution series is provided, and show that the ranking between the methods changes in this setting. This is relevant to many practical use cases of MISR where the goal is to super-resolve a specific image of the series, i.e. the reference is known. The proposed PROBA-V-REF should better reflect the performance of the different methods for this reference-aware MISR problem.

Ngoc Long Nguyen

Papers

Self-supervised multi-image super-resolution for push-frame satellite images

PROBA-V-REF: Repurposing the PROBA-V Challenge for Reference-Aware Super Resolution

Proba-V-ref: Repurposing the Proba-V challenge for reference-aware super resolution.