Deep learning (DL) algorithms have seen a massive rise in popularity for remote-sensing image analysis over the past few years. In this study, the major DL concepts pertinent to remote-sensing are introduced, and more than 200 publications in this field, most of which were published during the last two years, are reviewed and analyzed. Initially, a meta-analysis was conducted to analyze the status of remote sensing DL studies in terms of the study targets, DL model(s) used, image spatial resolution(s), type of study area, and level of classification accuracy achieved. Subsequently, a detailed review is conducted to describe/discuss how DL has been applied for remote sensing image analysis tasks including image fusion, image registration, scene classification, object detection, land use and land cover (LULC) classification, segmentation, and object-based image analysis (OBIA). This review covers nearly every application and technology in the field of remote sensing, ranging from preprocessing to mapping. Finally, a conclusion regarding the current state-of-the art methods, a critical conclusion on open challenges, and directions for future research are presented.

Deep learning in remote sensing applications: A meta-analysis and review

Point cloud learning has lately attracted increasing attention due to its wide applications in many areas, such as computer vision, autonomous driving, and robotics As a dominating technique in AI, deep learning has been successfully used to solve various 2D vision problems However, deep learning on point clouds is still in its infancy due to the unique challenges faced by the processing of point clouds with deep neural networks Recently, deep learning on point clouds has become even thriving, with numerous methods being proposed to address different problems in this area To stimulate future research, this paper presents a comprehensive review of recent progress in deep learning methods for point clouds It covers three major tasks, including 3D shape classification, 3D object detection and tracking, and 3D point cloud segmentation It also presents comparative results on several publicly available datasets, together with insightful observations and inspiring future research directions

Deep Learning for 3D Point Clouds: A Survey

Recent years have witnessed enormous progress in AI-related fields such as computer vision, machine learning, and autonomous vehicles. As with any rapidly growing field, it becomes increasingly difficult to stay up-to-date or enter the field as a beginner. While several survey papers on particular sub-problems have appeared, no comprehensive survey on problems, datasets, and methods in computer vision for autonomous vehicles has been published. This monograph attempts to narrow this gap by providing a survey on the state-of-the-art datasets and techniques. Our survey includes both the historically most relevant literature as well as the current state of the art on several specific topics, including recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning for autonomous driving. Towards this goal, we analyze the performance of the state of the art on several challenging benchmarking datasets, including KITTI, MOT, and Cityscapes. Besides, we discuss open problems and current research challenges. To ease accessibility and accommodate missing references, we also provide a website that allows navigating topics as well as methods and provides additional information.

/pdf/computer-vision-for-autonomous-vehicles-problems-datasets-557e7liccp.pdf

Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art

In recent years, deep learning (DL), a rebranding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, and natural language processing. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV, e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should not only be aware of advancements such as DL, but also be leading researchers in this area. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools, and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as they relate to (i) inadequate data sets, (ii) human-understandable solutions for modeling physical phenomena, (iii) big data, (iv) nontraditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial, and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.

Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community

In this work we address the task of the contextual classification of an airborne LiDAR point cloud. For that purpose, we integrate a Random Forest classifier into a Conditional Random Field (CRF) framework. It is a flexible approach for obtaining a reliable classification result even in complex urban scenes. In this way, we benefit from the consideration of context on the one hand and from the opportunity to use a large amount of features on the other hand. Considering the interactions in our experiments increases the overall accuracy by 2%, though a larger improvement becomes apparent in the completeness and correctness of some of the seven classes discerned in our experiments. We compare the Random Forest approach to linear models for the computation of unary and pairwise potentials of the CRF, and investigate the relevance of different features for the LiDAR points as well as for the interaction of neighbouring points. In a second step, building objects are detected based on the classified point cloud. For that purpose, the CRF probabilities for the classes are plugged into a Markov Random Field as unary potentials, in which the pairwise potentials are based on a Potts model. The 2D binary building object masks are extracted and evaluated by the benchmark ISPRS Test Project on Urban Classification and 3D Building Reconstruction. The evaluation shows that the main buildings (larger than 50 m 2 ) can be detected very reliably with a correctness larger than 96% and a completeness of 100%.

Contextual classification of lidar data and building object detection in urban areas

. For more than two decades, many efforts have been made to develop methods for extracting urban objects from data acquired by airborne sensors. In order to make the results of such algorithms more comparable, benchmarking data sets are of paramount importance. Such a data set, consisting of airborne image and laserscanner data, has been made available to the scientific community. Researchers were encouraged to submit results of urban object detection and 3D building reconstruction, which were evaluated based on reference data. This paper presents the outcomes of the evaluation for building detection, tree detection, and 3D building reconstruction. The results achieved by different methods are compared and analysed to identify promising strategies for automatic urban object extraction from current airborne sensor data, but also common problems of state-of-the-art methods.

/pdf/the-isprs-benchmark-on-urban-object-classification-and-3d-52ilshoznh.pdf

The ISPRS benchmark on urban object classification and 3D building reconstruction

For more than two decades, many efforts have been made to develop methods for extracting urban objects from data acquired by airborne sensors. In order to make the results of such algorithms more comparable, benchmarking data sets are of paramount importance. Such a data set, consisting of airborne image and laserscanner data, has been made available to the scientific community by ISPRS WGIII/4. Researchers were encouraged to submit their results of urban object detection and 3D building reconstruction, which were evaluated based on reference data. This paper presents the outcomes of the evaluation for building detection, tree detection, and 3D building reconstruction. The results achieved by different methods are compared and analysed to identify promising strategies for automatic urban object extraction from current airborne sensor data, but also common problems of state-of-the-art methods.

Results of the ISPRS benchmark on urban object detection and 3D building reconstruction

The DeepGlobe Building Extraction Challenge poses the problem of localizing all building polygons in the given satellite images. We can create polygons using an existing instance segmentation algorithm based on Mask R-CNN. However, polygons produced from instance segmentation have irregular shapes, which are far different from real building footprint boundaries and therefore cannot be directly applied to many cartographic and engineering applications. Hence, we present a method combining Mask R-CNN with building boundary regularization. Through the experiments, we find that the proposed method and Mask R-CNN achieve almost equivalent performance in terms of accuracy and completeness. However, compared to Mask R-CNN, our method produces better regularized polygons which are beneficial in many applications

/pdf/building-extraction-from-satellite-images-using-mask-r-cnn-grwe2jcoo5.pdf

Building Extraction from Satellite Images Using Mask R-CNN with Building Boundary Regularization

With rapid urbanization, highly accurate and semantically rich virtualization of building assets in 3D become more critical for supporting various applications, including urban planning, emergency response and location-based services. Many research efforts have been conducted to automatically reconstruct building models at city-scale from remotely sensed data. However, developing a fully-automated photogrammetric computer vision system enabling the massive generation of highly accurate building models still remains a challenging task. One the most challenging task for 3D building model reconstruction is to regularize the noises introduced in the boundary of building object retrieved from a raw data with lack of knowledge on its true shape. This paper proposes a data-driven modeling approach to reconstruct 3D rooftop models at city-scale from airborne laser scanning (ALS) data. The focus of the proposed method is to implicitly derive the shape regularity of 3D building rooftops from given noisy information of building boundary in a progressive manner. This study covers a full chain of 3D building modeling from low level processing to realistic 3D building rooftop modeling. In the element clustering step, building-labeled point clouds are clustered into homogeneous groups by applying height similarity and plane similarity. Based on segmented clusters, linear modeling cues including outer boundaries, intersection lines, and step lines are extracted. Topology elements among the modeling cues are recovered by the Binary Space Partitioning (BSP) technique. The regularity of the building rooftop model is achieved by an implicit regularization process in the framework of Minimum Description Length (MDL) combined with Hypothesize and Test (HAT). The parameters governing the MDL optimization are automatically estimated based on Min-Max optimization and Entropy-based weighting method. The performance of the proposed method is tested over the International Society for Photogrammetry and Remote Sensing (ISPRS) benchmark datasets. The results show that the proposed method can robustly produce accurate regularized 3D building rooftop models.

/pdf/implicit-regularization-for-reconstructing-3d-building-429x9fu0jb.pdf

Implicit Regularization for Reconstructing 3D Building Rooftop Models Using Airborne LiDAR Data.

. This paper proposes a new algorithm to generalize noisy polylines comprising a rooftop model by maximizing a shape regularity (orthogonality, symmetricity and directional simplications). The nature of remotely sensed data including airborne LiDAR often produce errors in localizing salient features (corners, lines and planes) due to weak contrast, occlusions, shadows and object complexity. A generalization or regularization process is well known algorithm for eliminating erroneous vertices while preserving significant information on rooftop shapes. Most of existing regularization methods achieves this goal base on a local process such as if-then rules due to lacking global objective functions or mainly focusing on minimising residuals between boundary observations and models. In this study, we implicitly derive rules to generate local hypothetical models. Those hypothesized models produce possible drawings of regular patterns that given rooftop vectors can possibly generate by combining global and local analysis of line directions and their connections. A final optimal model is globally selected through a gradient descent optimization. A BSP (Binary Tree Partitioning)-tree was used to produce initial rooftop vectors using ISPRS WGIII/4's benchmarking test sites in Veihngen. The proposed regularization algorithm was applied to reduce modelling errors produced by BSP-tree. An evaluation demonstrates the proposed algorithm is promising for updating of building database.

/pdf/an-implicit-regularization-for-3d-building-rooftop-modeling-3sc6ajrs3j.pdf

Jaewook Jung

Papers

The ISPRS benchmark on urban object classification and 3D building reconstruction

Results of the ISPRS benchmark on urban object detection and 3D building reconstruction

Building Extraction from Satellite Images Using Mask R-CNN with Building Boundary Regularization

Implicit Regularization for Reconstructing 3D Building Rooftop Models Using Airborne LiDAR Data.

An implicit regularization for 3d building rooftop modeling using airborne lidar data