As a large amount of earth observation data is available on a global scale, it becomes possible to apply multimodal semantic segmentation technology to remote sensing scene analysis. However, the diversity of objects in large-scale scenes and the cross-modal gap between different images are still challenging in practical applications. To address these problems, we propose a Transformer-Induced Hierarchical Graph Network (GraFNet) for multimodal semantic segmentation in remote sensing scenes, which promotes the exploration of potential intra- and inter-modal relations by introducing a new modeling paradigm. Different from existing methods, GraFNet parses multimodal remote sensing images into semantic topological graphs, and exploits the structural information of land cover categories to learn joint representations. Specifically, an attentive heterogeneous information aggregation mechanism is presented to parse diverse objects in remote sensing scenes into semantic entities, and capture modality-specific object–object interaction patterns in a topology-aware environment. In addition, modality hierarchical dependency modeling is introduced to encode the interactive representation of cross-modal objects, and distinguish the modality-specific contribution to improve cross-modal compatibility. Extensive experiments on several multimodal remote sensing datasets demonstrate that the proposed GraFNet outperforms the state-of-the-art approaches, achieving F 1 /mIoU accuracy 91.1%/82.4% on the ISPRS Vaihingen dataset, 93.4%/88.4% on ISPRS Potsdam dataset, and 91.8%/84.0% on the MSAW dataset. 

Transformer-induced graph reasoning for multimodal semantic segmentation in remote sensing

Exploring carbon dioxide (CO2) emissions from human activities is essential for urban energy conservation and resource management. Remotely sensed nighttime lights from the Suomi NPP-VIIRS provide ...

Carbon dioxide (CO2) emissions from the service industry, traffic, and secondary industry as revealed by the remotely sensed nighttime light data

Observing traffic flow is of great significance to contemporary urban management. Overhead images, as represented by remote sensing images, provide a major source of information about traffic flow. However, the spatial resolutions of most common high-resolution remote sensing images are often limited to 0.5 m and even below, which makes it unrealistic to count vehicles by means of widely used object detection methods. Therefore, to explore the potential of remote sensing data for studying global urban development and management, this paper introduces a density map-based vehicle counting method for remote sensing imagery with limited resolution. Density map-based models regard the vehicle counting task as estimating the density of vehicle targets in terms of pixel values. We propose an improved CNN-based network, called Congested Scene Recognition Network Minus (CSRNet—), that generates a density map of vehicles from the input remote sensing imagery. A new dataset, RSVC2021, which was generated from the public DOTA and ITCVD datasets, is also introduced for network training and testing. A benchmark on the RSVC2021 dataset is accordingly established and CSRNet— is selected as the baseline model for subsequent experiments. A set of GF-2 time series images with a resolution of 1 m taken before, during and after the COVID-19 epidemic lockdown covering Wuhan city are applied for real-world application testing. The testing results on both the RSVC2021 dataset and real satellite images confirm that, in terms of both the counting values and the visualized density maps, the proposed method achieves good performance and exhibits considerable application potential in this task. The generating codes of RSVC2021 dataset will be publicly available at https://github.com/YinongGuo/RSVC2021-Dataset. 

Density Map-based vehicle counting in remote sensing images with limited resolution

Nighttime lights (NTL) create a unique footprint left by human activities, which can reflect the economic index and demographic characteristics of a country or region to some extent. It is of great significance to explore the impact of land features related to social–economic indexes to NTL intensity in urban areas. At present, there are few studies on the impact factors of high-resolution NTL remote sensing data to analyze the influence of NTL intensity variation at a fine scale. In this paper, taking Changchun, China as a case study, we selected the new generation of high spatial resolution (0.92 m) and multispectral bands NTL image JL1-3B data to evaluate the relationship between NTL intensity and related land features such as the normalized difference vegetation index (NDVI), land use types and point of information (POI) at the parcel level, and combined Luojia 1-01 images for comparative analysis. After screening features by the Gini index, 17 variables were selected to establish the best random forest (RF) regression model for the Luojia 1-01 and JL1-3B data, corresponding to out-of-bag (oob) scores of 0.8304 and 0.9054, respectively. The impact of features on NTL was determined by calculating the features contribution. It was found that JL1-3B data perform better on a finer scale and provide more information. In addition, JL1-3B data are less affected by light overflow effect and saturation, and they could provide more accurate information at smaller parcels. Through the impact analysis of land features on the two kinds of NTL data, it is proven that JL1-3B images can be used to study effectively the relationship between NTL and human activities information. This paper aims to establish a regression model between the radiance of two types of NTL data and land features by RF algorithm, to further excavate the main land features that impact radiance according to the feature contribution, and compare the performance of two types of NTL data in regression. The study is expected to provide a reference to the further application of NTL data such as land feature inversion, artificial surface monitoring and evaluation, geographic information point estimation, information mining, etc., and a more comprehensive cognition of land feature impact to urban social–economic indexes from a unique perspective, which can be used to assist urban planning and related decision-making.

https://www.mdpi.com/1424-8220/20/18/5447/pdf

The Impact Analysis of Land Features to JL1-3B Nighttime Light Data at Parcel Level: Illustrated by the Case of Changchun, China.

As a novel method of earth observation, video satellites can observe dynamic changes in ground targets in real time. To make use of satellite videos, target tracking in satellite videos has received extensive interest. However, this also faces a variety of new challenges such as global occlusion, low resolution, and insufficient information compared with traditional target tracking. To handle the abovementioned problems, a multi-feature correlation filter with motion estimation is proposed. First, we propose a motion estimation algorithm that combines a Kalman filter and an inertial mechanism to alleviate the boundary effects. This can also be used to track the occluded target. Then, we fuse a histogram of oriented gradient (HOG) features and optical flow (OF) features to improve the representation information of the target. Finally, we introduce a disruptor-aware mechanism to weaken the influence of background noise. Experimental results verify that our algorithm can achieve high tracking performance.

/pdf/satellite-video-tracking-by-multi-feature-correlation-2233nxs0.pdf

Satellite Video Tracking by Multi-Feature Correlation Filters with Motion Estimation

To realize energy conservation and environmental protection, solar street lights have been widely used in urban areas in China. To reasonably and effectively utilize solar street lights, the original street lights must be located, and the solar street light potential must be assessed. The Jilin1-03B (JL1-3B) satellite provides next-generation nighttime light data with a high spatial resolution and in three spectral bands. Consequently, the street lights can be extracted from the nighttime light data. We used the road network dataset from the open street map with a specific buffer to extract the road area as a constraint region. Next, the grayscale brightness of JL1-3B images was obtained by integrating all the three bands to locate the street light by using a local maximum algorithm. Then, the values of the original three bands were utilized to classify the types of street lights as high-pressure sodium (HPS) lamps or light-emitting diode lamps. Finally, we simulated the replacement of all the HPS lamps with solar street lights and assessed the corresponding solar energy potential by using the digital surface model data and hourly cloud cover data through the SHORTWAVE-C model. The accuracy of location of the street lights was approximately 90%. Replacing an HPS lamp by one solar street light for 20 years can save 1.85 × 104 kWh of electrical energy, 7.41 t of standard coal, 5.03 t of C emissions, 18.47 t of CO2 emissions, 0.55 t of SO2 emissions, and 0.28 t of NOX emissions.

/pdf/automated-extraction-of-street-lights-from-jl1-3b-nighttime-xfbm7os1cs.pdf

Automated Extraction of Street Lights From JL1-3B Nighttime Light Data and Assessment of Their Solar Energy Potential

Recently, satellite video has become an emerging means of earth observation, providing the possibility of tracking moving objects. However, the existing multi-object trackers are commonly designed for natural scenes without considering the characteristics of remotely sensed data. In addition, most trackers are composed of two independent stages of detection and re-identification (ReID), which means that they cannot be mutually promoted. To this end, we propose an end-to-end online framework, which is called TGraM, for multi-object tracking in satellite videos. It models multi-object tracking as a graph information reasoning procedure from the multi-task learning perspective. Specifically, a graph-based spatiotemporal reasoning module is presented to mine the potential high-order correlations between video frames. Furthermore, considering the inconsistency of optimization objectives between detection and ReID, a multi-task gradient adversarial learning strategy is designed to regularize each task-specific network. Additionally, aiming at the data scarcity in this field, a large-scale and high-resolution Jilin-1 satellite video dataset for multi-object tracking (AIR-MOT) is built for the experiments. Compared with state-of-the-art multi-object trackers, TGraM achieves efficient collaborative learning between detection and ReID, improving the tracking accuracy by 1.2 MOTA. The code and dataset will be available online (https://github.com/HeQibin/TGraM).

Beibei Li

Papers

Automated Extraction of Street Lights From JL1-3B Nighttime Light Data and Assessment of Their Solar Energy Potential

Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling