Incremental Structure-from-Motion is a prevalent strategy for 3D reconstruction from unordered image collections. While incremental reconstruction systems have tremendously advanced in all regards, robustness, accuracy, completeness, and scalability remain the key problems towards building a truly general-purpose pipeline. We propose a new SfM technique that improves upon the state of the art to make a further step towards this ultimate goal. The full reconstruction pipeline is released to the public as an open-source implementation.

/pdf/structure-from-motion-revisited-10382wusrz.pdf

Structure-from-Motion Revisited

We address the problem of inferring the pose of an RGB-D camera relative to a known 3D scene, given only a single acquired image. Our approach employs a regression forest that is capable of inferring an estimate of each pixel's correspondence to 3D points in the scene's world coordinate frame. The forest uses only simple depth and RGB pixel comparison features, and does not require the computation of feature descriptors. The forest is trained to be capable of predicting correspondences at any pixel, so no interest point detectors are required. The camera pose is inferred using a robust optimization scheme. This starts with an initial set of hypothesized camera poses, constructed by applying the forest at a small fraction of image pixels. Preemptive RANSAC then iterates sampling more pixels at which to evaluate the forest, counting inliers, and refining the hypothesized poses. We evaluate on several varied scenes captured with an RGB-D camera and observe that the proposed technique achieves highly accurate relocalization and substantially out-performs two state of the art baselines.

/pdf/scene-coordinate-regression-forests-for-camera-44n6t221df.pdf

Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images

Visual localization enables autonomous vehicles to navigate in their surroundings and augmented reality applications to link virtual to real worlds. Practical visual localization approaches need to be robust to a wide variety of viewing condition, including day-night changes, as well as weather and seasonal variations, while providing highly accurate 6 degree-of-freedom (6DOF) camera pose estimates. In this paper, we introduce the first benchmark datasets specifically designed for analyzing the impact of such factors on visual localization. Using carefully created ground truth poses for query images taken under a wide variety of conditions, we evaluate the impact of various factors on 6DOF camera pose estimation accuracy through extensive experiments with state-of-the-art localization approaches. Based on our results, we draw conclusions about the difficulty of different conditions, showing that long-term localization is far from solved, and propose promising avenues for future work, including sequence-based localization approaches and the need for better local features. Our benchmark is available at visuallocalization.net.

/pdf/benchmarking-6dof-outdoor-visual-localization-in-changing-17ggv8ar37.pdf

Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions

UAV platforms are nowadays a valuable source of data for inspection, surveillance, mapping and 3D modeling issues. New applications in the short- and close-range domain are introduced, being the UAVs a low-cost alternatives to the classical manned a erial photogrammetry. Rotary or fixed wing UAVs, capable of performing the photogrammetric data acquisition with amateur or SLR digital cameras, can fly in manual, semi-automated and autonomous modes. With a typical photogrammetric pipeline, 3D results like DSM/DTM, contour lines, textured 3D models, vector data, etc. can be produced, in a reasonable automated way. The paper reports the latest developments of UAV image processing methods for photogrammetric applications, mapping and 3D modeling issues. Automation is nowadays necessary and feasible at the image orientation, DSM generation and orthophoto production stages, while accurate feature extraction is still an interactive procedure. New perspectives are also addressed.

/pdf/uav-photogrammetry-for-mapping-and-3d-modeling-current-3galh71j43.pdf

Uav photogrammetry for mapping and 3d modeling - current status and future perspectives -

This survey summarizes almost 50 years of research and development in the field of Augmented Reality AR. From early research in the1960's until widespread availability by the 2010's there has been steady progress towards the goal of being able to seamlessly combine real and virtual worlds. We provide an overview of the common definitions of AR, and show how AR fits into taxonomies of other related technologies. A history of important milestones in Augmented Reality is followed by sections on the key enabling technologies of tracking, display and input devices. We also review design guidelines and provide some examples of successful AR applications. Finally, we conclude with a summary of directions for future work and a review of some of the areas that are currently being researched.

/pdf/a-survey-of-augmented-reality-2ce6xet7y5.pdf

A Survey of Augmented Reality

Efficient view registration with respect to a given 3D reconstruction has many applications like inside-out tracking in indoor and outdoor environments, and geo-locating images from large photo collections. We present a fast location recognition technique based on structure from motion point clouds. Vocabulary tree-based indexing of features directly returns relevant fragments of 3D models instead of documents from the images database. Additionally, we propose a compressed 3D scene representation which improves recognition rates while simultaneously reducing the computation time and the memory consumption. The design of our method is based on algorithms that efficiently utilize modern graphics processing units to deliver real-time performance for view registration. We demonstrate the approach by matching hand-held outdoor videos to known 3D urban models, and by registering images from online photo collections to the corresponding landmarks.

/pdf/from-structure-from-motion-point-clouds-to-fast-location-366f2cowkb.pdf

From structure-from-motion point clouds to fast location recognition

Novel automated photogrammetry is based on four innovations. First is the cost-free increase of overlap between images when sensing digitally. Second is an improved radiometry. Third is multi-view matching. Fourth is the Graphics Processing Unit (GPU), making complex algorithms for image matching very practical. These innovations lead to improved automation of the photogrammetric workflow so that point clouds are created at sub-pixel accuracy, at very dense intervals, and in near real-time, thereby eroding the unique selling proposition of lidar scanners. Two test projects compare point clouds from aerial and street-side lidar systems with those created from images. We show that the photogrammetric accuracy compares well with the lidar-method, yet the density of surface points is much higher from images, and the throughput is commensurate with a fully automated all-digital approach. Beyond this density, we identify 15 additional advantages of the photogrammetric approach.

Point Clouds: Lidar versus 3D Vision

We present a fast and memory efficient method for localizing a mobile user's 6DOF pose from a single camera image. Our approach registers a view with respect to a sparse 3D point reconstruction. The 3D point dataset is partitioned into pieces based on visibility constraints and occlusion culling, making it scalable and efficient to handle. Starting with a coarse guess, our system only considers features that can be seen from the user's position. Our method is resource efficient, usually requiring only a few megabytes of memory, thereby making it feasible to run on low-end devices such as mobile phones. At the same time it is fast enough to give instant results on this device class.

Wide area localization on mobile phones

Micro Aerial Vehicles (MAVs) equipped with high resolution cameras have the ability of cost efficient and autonomous image acquisition from unconventional viewpoints. To fully exploit the limited flight-time of current MAVs view planning is essential for complete and precise 3D scene sampling. We propose a novel camera network design algorithm suitable for MAVs for close range photogrammetry that exploits prior knowledge of the surrounding. Our algorithm automatically determines a set of camera positions that guarantees important constraints for image based 3D reconstruction. On synthetic experiments we demonstrate that our camera network design obtains detailed 3D reconstructions with a reduced number of images at the desired accuracy level. Comparable results are also computed on an outdoor experiment using our MAV in autonomous

Photogrammetric Camera Network Design for Micro Aerial Vehicles

Highly accurate localization of a micro aerial vehicle (MAV) with respect to a scene is important for a wide range of applications, in particular surveillance and inspection. Most existing approaches to visual localization focus on indoor environments, while such tasks require outdoor navigation. Within this work, we introduce a novel algorithm for monocular visual localization for MAVs based on the concept of virtual views in 3D space. Under the assumption that significant parts of the scene do not alter their geometry and serve as natural landmarks, the accuracy of our visual approach outperforms consumer grade GPS systems. In an experimental setup we compare our approach to a state-of-the-art visual SLAM algorithm and evaluate the performance by geometric validation from an observer's view. As our method directly allows global registration, it is neither prone to drift nor bias. This makes it well suited for long-term autonomous navigation.

Arnold Irschara

Papers

From structure-from-motion point clouds to fast location recognition

Point Clouds: Lidar versus 3D Vision

Wide area localization on mobile phones

Photogrammetric Camera Network Design for Micro Aerial Vehicles

Natural landmark-based monocular localization for MAVs