This tutorial presents a hands-on view of the field of multi-view stereo with a focus on practical algorithms. Multi-view stereo algorithms are able to construct highly detailed 3D models from images alone. They take a possibly very large set of images and construct a 3D plausible geometry that explains the images under some reasonable assumptions, the most important being scene rigidity. The tutorial frames the multiview stereo problem as an image/geometry consistency optimization problem. It describes in detail its main two ingredients: robust implementations of photometric consistency measures, and efficient optimization algorithms. It then presents how these main ingredients are used by some of the most successful algorithms, applied into real applications, and deployed as products in the industry. Finally it describes more advanced approaches exploiting domain-specific knowledge such as structural priors, and gives an overview of the remaining challenges and future research directions.

/pdf/multi-view-stereo-a-tutorial-2jvvkwdg5s.pdf

Multi-View Stereo: A Tutorial

In the last few decades, Structure from Motion (SfM) and visual Simultaneous Localization and Mapping (visual SLAM) techniques have gained significant interest from both the computer vision and robotic communities. Many variants of these techniques have started to make an impact in a wide range of applications, including robot navigation and augmented reality. However, despite some remarkable results in these areas, most SfM and visual SLAM techniques operate based on the assumption that the observed environment is static. However, when faced with moving objects, overall system accuracy can be jeopardized. In this article, we present for the first time a survey of visual SLAM and SfM techniques that are targeted toward operation in dynamic environments. We identify three main problems: how to perform reconstruction (robust visual SLAM), how to segment and track dynamic objects, and how to achieve joint motion segmentation and reconstruction. Based on this categorization, we provide a comprehensive taxonomy of existing approaches. Finally, the advantages and disadvantages of each solution class are critically discussed from the perspective of practicality and robustness.

Visual SLAM and Structure from Motion in Dynamic Environments: A Survey

In this paper we study the problem of automatically generating polynomial solvers for minimal problems. The main contribution is a new method for finding small elimination templates by making use of the syzygies (i.e. the polynomial relations) that exist between the original equations. Using these syzygies we can essentially parameterize the set of possible elimination templates. We evaluate our method on a wide variety of problems from geometric computer vision and show improvement compared to both handcrafted and automatically generated solvers. Furthermore we apply our method on two previously unsolved relative orientation problems.

/pdf/efficient-solvers-for-minimal-problems-by-syzygy-based-1gskcopalq.pdf

Efficient Solvers for Minimal Problems by Syzygy-Based Reduction

Images taken under water usually suffer from the problems of quality degradation, such as low contrast, blurring details, color deviations, non-uniform illumination, etc. As an important problem in image processing and computer vision, the restoration and enhancement of underwater image are necessary for numerous practical applications. Over the last few decades, underwater image restoration and enhancement have been attracting an increasing amount of research effort. However, a comprehensive and in-depth survey of related achievements and improvements is still missing, especially the survey of underwater image dataset which is a key issue in underwater image processing and intelligent application. In this exposition, we first summarize more than 120 studies about the latest progress in underwater image restoration and enhancement, including the techniques, datasets, available codes, and evaluation metrics. We analyze the contributions and limitations of existing methods to facilitate the comprehensive understanding of underwater image restoration and enhancement. Furthermore, we provide detailed objective evaluations and analysis of the representative methods on five types of underwater scenarios, which verifies the applicability of these methods in different underwater conditions. Finally, we discuss the potential challenges and open issues of underwater image restoration and enhancement and suggest possible research directions in the future.

/pdf/an-in-depth-survey-of-underwater-image-enhancement-and-2h5bh2h7bf.pdf

An In-Depth Survey of Underwater Image Enhancement and Restoration

The increasingly common use of incidental satellite images for stereo reconstruction versus rigidly tasked binocular or trinocular coincident collection is helping to enable timely global-scale 3D mapping; however, reliable stereo correspondence from multi-date image pairs remains very challenging due to seasonal appearance differences and scene change. Promising recent work suggests that semantic scene segmentation can provide a robust regularizing prior for resolving ambiguities in stereo correspondence and reconstruction problems. To enable research for pairwise semantic stereo and multi-view semantic 3D reconstruction with incidental satellite images, we have established a large-scale public dataset including multi-view, multi-band satellite images and ground truth geometric and semantic labels for two large cities. To demonstrate the complementary nature of the stereo and segmentation tasks, we present lightweight public baselines adapted from recent state of the art convolutional neural network models and assess their performance.

Semantic Stereo for Incidental Satellite Images

If a picture is worth a thousand words, then images should be utilized together with other available data modalities when predicting the virality of online posts, such as tweets. In this paper, we re-visit the tweet popularity prediction problem by considering all data modalities: tweet language semantics, embedded images, author' social relationships, and the diffusion process of tweets. To model the content of tweets, we propose a joint-embedding neural network that combines visual, textual, and social cues together. Such content features can be either used for prediction directly, or for pre-conditioning a 'dynamics RNN', which models the message propagation process. A novel Poisson regression loss is optimized to train the network. We demonstrate that content based features can be used to improve upon social features and dynamics features via our joint-embedding regression model. Our model outperforms the state-of-the-art on multiple large-scale real-world datasets collected from Twitter.

Retweet Wars: Tweet Popularity Prediction via Dynamic Multimodal Regression

We propose two novel minimal solvers which advance the state of the art in satellite imagery processing. Our methods are efficient and do not rely on the prior existence of complex inverse mapping functions to correlate 2D image coordinates and 3D terrain. Our first solver improves on the stereo correspondence problem for satellite imagery, in that we provide an exact image-to-object space mapping (where prior methods were inaccurate). Our second solver provides a novel mechanism for 3D point triangulation, which has improved robustness and accuracy over prior techniques. Given the usefulness and ubiquity of satellite imagery, our proposed methods allow for improved results in a variety of existing and future applications.

https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Zheng_Minimal_Solvers_for_ICCV_2015_paper.pdf

Minimal Solvers for 3D Geometry from Satellite Imagery

We consider the problem of estimating the relative depth of a scene from a monocular image. The dark channel prior, used as a statistical observation of haze free images, has been previously leveraged for haze removal and relative depth estimation tasks. However, as a local measure, it fails to account for higher order semantic relationship among scene elements. We propose a dual channel prior used for identifying pixels that are unlikely to comply with the dark channel assumption, leading to erroneous depth estimates. We further leverage semantic segmentation information and patch match label propagation to enforce semantically consistent geometric priors. Experiments illustrate the quantitative and qualitative advantages of our approach when compared to state of the art methods.

Combining semantic scene priors and haze removal for single image depth estimation

Minimal Solvers for 3D Geometry from Satellite Imagery (Open Access)

We propose an efficient algorithm to jointly estimate geometry and semantics for a given geographical region observed by multiple satellite images. Our joint estimation leverages an efficient PatchMatch inference framework defined over lattice discretization of the environment. Our cost function relies on the local planarity assumption to model scene geometry and neural network classification to determine semantic (e.g. land use) labels for geometric structures. By utilizing the commonly available direct (i.e. space to image) rational polynomial coefficients (RPC) satellite camera models, our approach effectively circumvents the need for estimating or refining inverse RPC models. Experiments illustrate both the computational efficiency and high quality scene geometry estimates attained by our approach for satellite imagery. To further illustrate the generality of our representation and inference framework, experiments on standard benchmarks for ground-level imagery are also included.

Ke Wang

Papers

Retweet Wars: Tweet Popularity Prediction via Dynamic Multimodal Regression

Minimal Solvers for 3D Geometry from Satellite Imagery

Combining semantic scene priors and haze removal for single image depth estimation

Minimal Solvers for 3D Geometry from Satellite Imagery (Open Access)

Efficient joint stereo estimation and land usage classification for multiview satellite data