Large, pretrained models are commonly finetuned with imagery that is heavily augmented to mimic different conditions and scales, with the resulting models used for various tasks with imagery from a range of spatial scales. Such models overlook scale-specific information in the data for scale-dependent domains, such as remote sensing. In this paper, we present Scale-MAE, a pretraining method that explicitly learns relationships between data at different, known scales throughout the pretraining process. Scale-MAE pretrains a network by masking an input image at a known input scale, where the area of the Earth covered by the image determines the scale of the ViT positional encoding, not the image resolution. Scale-MAE encodes the masked image with a standard ViT backbone, and then decodes the masked image through a bandpass filter to reconstruct low/high frequency images at lower/higher scales. We find that tasking the network with reconstructing both low/high frequency images leads to robust multiscale representations for remote sensing imagery. Scale-MAE achieves an average of a $2.4 - 5.6\%$ non-parametric kNN classification improvement across eight remote sensing datasets compared to current state-of-the-art and obtains a $0.9$ mIoU to $1.7$ mIoU improvement on the SpaceNet building segmentation transfer task for a range of evaluation scales.

Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning

—Multispectral Sentinel-2 images are a valuable source of Earth observation data, however spatial resolution of their spectral bands limited to 10m, 20m, and 60m ground sam- pling distance remains insufﬁcient in many cases. This problem can be addressed with super-resolution, aimed at reconstructing a high-resolution image from a low-resolution observation. For Sentinel-2, spectral information fusion allows for enhancing the 20m and 60m bands to the 10m resolution. Also, there were attempts to combine multitemporal stacks of individual Sentinel- 2 bands, however these two approaches have not been combined so far. In this paper, we introduce DeepSent—a new deep network for super-resolving multitemporal series of multispectral Sentinel-2 images. It is underpinned with information fusion per- formed simultaneously in the spectral and temporal dimensions to generate an enlarged multispectral image. In our extensive ex- perimental study, we demonstrate that our solution outperforms other state-of-the-art techniques that realize either multitemporal or multispectral data fusion. Furthermore, we show that the advantage of DeepSent results from how these two fusion types are combined in a single architecture, which is superior to performing such fusion in a sequential manner. Importantly, we have applied our method to super-resolve real-world Sentinel-2 images, enhancing the spatial resolution of all the spectral bands to 3.3m nominal ground sampling distance, and we compare the outcome with very high-resolution WorldView-2 images. We will publish our implementation upon paper acceptance, and we expect it will increase the possibilities of exploiting super-resolved Sentinel-2 images in real-life applications.

Multitemporal and multispectral data fusion for super-resolution of Sentinel-2 images

Minerals are indispensable for a functioning modern society. Yet, their supply is limited causing a need for optimizing their exploration and extraction both from ores and recyclable materials. Typically, these processes must be meticulously adapted to the precise properties of the processed particles, an extensive characterization of their shapes, appearances as well as the overall material composition. Current approaches perform this analysis based on bulk segmentation and characterization of particles imaged with a micro CT, and rely on rudimentary postprocessing techniques to separate touching particles. However, due to their inability to reliably perform this separation as well as the need to retrain or reconfigure methods for each new image, these approaches leave untapped potential to be leveraged. Here, we propose ParticleSeg3D, an instance segmentation method that is able to extract individual particles from large micro CT images taken from mineral samples embedded in an epoxy matrix. Our approach is based on the powerful nnU-Net framework, introduces a particle size normalization, makes use of a border-core representation to enable instance segmentation and is trained with a large dataset containing particles of numerous different materials and minerals. We demonstrate that ParticleSeg3D can be applied out-of-the box to a large variety of particle types, including materials and appearances that have not been part of the training set. Thus, no further manual annotations and retraining are required when applying the method to new mineral samples, enabling substantially higher scalability of experiments than existing methods. Our code and dataset are made publicly available.

[Work in progress] Scalable, out-of-the box segmentation of individual particles from mineral samples acquired with micro CT

Multi-image super-resolution consists in fusing multiple images of the same scene to generate an image of higher spatial resolution. Recently, it has been demonstrated that for remotely sensed multispectral Sentine1-2 images, fusion performed in both spectral and temporal dimensions improves the reconstruction quality. However, the deep networks elaborated for this purpose are quite large and influence of the input data is difficult to interpret. In this paper, we show how to exploit transformer-based image fusion to reduce the number of trainable parameters by 30% and allow for increased interpretability with means of attention rollout. The reported experimental results performed over simulated and real-world data indicate that this does not affect the reconstruction quality, while at the same time the visualization tools may help in developing techniques for input data selection and preprocessing.

Transformer-based spectro-temporal fusion for Sentinel-2 super-resolution

—Insufﬁcient spatial resolution of satellite imagery, including Sentinel-2 data, is a serious limitation in many practical use cases. To mitigate this problem, super-resolution recon- struction is receiving considerable attention from the remote sensing community. When it is performed from multiple images captured at subsequent revisits, it may beneﬁt from information fusion, leading to enhanced reconstruction accuracy. One of the obstacles in multi-image super-resolution consists in the scarcity of real-life benchmark datasets—most of the research was performed for simulated data which do not fully reﬂect the operating conditions. In this letter, we introduce a new MuS2 benchmark for multi-image super-resolution reconstruction of Sentinel-2 images, with WorldView-2 imagery used as the high- resolution reference. Within MuS2, we publish the ﬁrst end-to-end evaluation procedure for this problem which we expect to help the researchers in advancing the state of the art in multi-image super-resolution for Sentinel-2 imagery.

MuS2: A Benchmark for Sentinel-2 Multi-Image Super-Resolution

—Cloud detection is a pivotal satellite image pre- processing step that can be performed both on the ground and on board a satellite to tag useful images. In the latter case, it can help to reduce the amount of data to downlink by pruning the cloudy areas, or to make a satellite more autonomous through data-driven acquisition re-scheduling of the cloudy areas. We approach this important task with nnU-Nets, a self-reconﬁgurable framework able to perform meta-learning of a segmentation network over various datasets. Our experiments, performed over Sentinel-2 and Landsat-8 multispectral images revealed that nnU-Nets deliver state-of-the-art cloud segmentation performance without any manual design. Our approach was ranked within the top 7% best solutions (across 847 participating teams) in the On Cloud N: Cloud Cover Detection Challenge, where we reached the Jaccard index of 0.882 over more than 10k unseen Sentinel-2 image patches (the winners obtained 0.897, whereas the baseline U-Net with the ResNet-34 backbone used as an encoder: 0.817, and the classic Sentinel-2 image thresholding: 0.652).

Self-Configuring nnU-Nets Detect Clouds in Satellite Images

Super-resolution reconstruction consists in generating a high-resolution image from a single low-resolution image or multiple images presenting the same area of interest. Existing state-of-the-art approaches to single-image and multi-image super-resolution are based on deep learning that requires large amounts of training data. They are commonly obtained by simulating low-resolution images from an original image treated as a high-resolution reference, but such simulation may not reflect the real-life operating conditions. Therefore, a serious obstacle in deploying super-resolution in real-world cases results from the lack of training data that would encompass real low-resolution images coupled with a real high-resolution reference. In this paper, we propose a new data augmentation technique underpinned with learning the relation between high and low resolution. This helps reduce the requirements concerned with the amount of real-life data necessary to train a super-resolution network, while providing higher-quality data for training, compared with the simulated low-resolution images. Our initial experimental results reported in the paper confirm that the proposed approach is suitable for multi-image super-resolution.

Data Augmentation for Multi-Image Super-Resolution

Cloud detection is a pivotal satellite image pre-processing step that can be performed both on the ground and on board a satellite to tag useful images. In the latter case, it can reduce the amount of data to downlink by pruning the cloudy areas, or to make a satellite more autonomous through data-driven acquisition re-scheduling. We approach this task with nnU-Nets, a self-reconfigurable framework able to perform meta-learning of a segmentation network over various datasets. Unfortunately, such models are commonly memory-inefficient due to their (very) large architectures. To benefit from them in on-board processing, we compress nnU-Nets with knowledge distillation into much smaller and compact U-Nets. Our experiments, performed over Sentinel-2 and Landsat-8 images revealed that nnU-Nets deliver state-of-the-art performance without any manual design. Our approach was ranked within the top 7% best solutions (across 847 teams) in the On Cloud N: Cloud Cover Detection Challenge, where we reached the Jaccard index of 0.882 over more than 10k unseen Sentinel-2 images (the winners obtained 0.897, the baseline U-Net with the ResNet-34 backbone: 0.817, and the classic Sentinel-2 image thresholding: 0.652). Finally, we showed that knowledge distillation enables to elaborate dramatically smaller (almost 280x) U-Nets when compared to nnU-Nets while still maintaining their segmentation capabilities.

Squeezing nnU-Nets with Knowledge Distillation for On-Board Cloud Detection

Cloud detection is one of the most important image pre-processing steps that can be performed on-board satellites. It may allow us to reduce the amount of data to analyze or downlink by pruning the cloudy areas, or to make the satellites more autonomous through data-driven image acquisition re-scheduling of the areas obscured by clouds. Thus, building the cloud detection algorithms that can be ultimately deployed in orbit became an important research avenue. In this paper, we investigate the robustness of the fully-convolutional neural networks for cloud detection against the atmospheric conditions that resemble real acquisition settings of the Intuition-1 mission. Our experiments, performed over the original and simulated Landsat-8 images, with the latter reflecting target conditions, shed more light on the performance of deep models and showed how can we verify their robustness in Earth observation tasks for which real images do not exist yet.

Maciej Ziaja

Papers

MuS2: A Benchmark for Sentinel-2 Multi-Image Super-Resolution

Self-Configuring nnU-Nets Detect Clouds in Satellite Images

Data Augmentation for Multi-Image Super-Resolution

Squeezing nnU-Nets with Knowledge Distillation for On-Board Cloud Detection

Are Cloud Detection U-Nets Robust Against in-Orbit Image Acquisition Conditions?