Recent work has shown that depth estimation from a stereo pair of images can be formulated as a supervised learning task to be resolved with convolutional neural networks (CNNs). However, current architectures rely on patch-based Siamese networks, lacking the means to exploit context information for finding correspondence in ill-posed regions. To tackle this problem, we propose PSMNet, a pyramid stereo matching network consisting of two main modules: spatial pyramid pooling and 3D CNN. The spatial pyramid pooling module takes advantage of the capacity of global context information by aggregating context in different scales and locations to form a cost volume. The 3D CNN learns to regularize cost volume using stacked multiple hourglass networks in conjunction with intermediate supervision. The proposed approach was evaluated on several benchmark datasets. Our method ranked first in the KITTI 2012 and 2015 leaderboards before March 18, 2018. The codes of PSMNet are available at: https://github.com/JiaRenChang/PSMNet.

/pdf/pyramid-stereo-matching-network-10jlru3yif.pdf

Pyramid Stereo Matching Network

As a fundamental and critical task in various visual applications, image matching can identify then correspond the same or similar structure/content from two or more images. Over the past decades, growing amount and diversity of methods have been proposed for image matching, particularly with the development of deep learning techniques over the recent years. However, it may leave several open questions about which method would be a suitable choice for specific applications with respect to different scenarios and task requirements and how to design better image matching methods with superior performance in accuracy, robustness and efficiency. This encourages us to conduct a comprehensive and systematic review and analysis for those classical and latest techniques. Following the feature-based image matching pipeline, we first introduce feature detection, description, and matching techniques from handcrafted methods to trainable ones and provide an analysis of the development of these methods in theory and practice. Secondly, we briefly introduce several typical image matching-based applications for a comprehensive understanding of the significance of image matching. In addition, we also provide a comprehensive and objective comparison of these classical and latest techniques through extensive experiments on representative datasets. Finally, we conclude with the current status of image matching technologies and deliver insightful discussions and prospects for future works. This survey can serve as a reference for (but not limited to) researchers and engineers in image matching and related fields.

/pdf/image-matching-from-handcrafted-to-deep-features-a-survey-27kn6w026w.pdf

Image Matching from Handcrafted to Deep Features: A Survey

Visibility algorithms for walkthrough and related applications have grown into a significant area, spurred by the growth in the complexity of models and the need for highly interactive ways of navigating them. In this survey, we review the fundamental issues in visibility and conduct an overview of the visibility culling techniques developed in the last decade. The taxonomy we use distinguishes point-based methods from-region methods. Point-based methods are further subdivided into object and image-precision techniques, while from-region approaches can take advantage of the cell-and-portal structure of architectural environments or handle generic scenes.

/pdf/a-survey-of-visibility-for-walkthrough-applications-1b5f1gxv54.pdf

A survey of visibility for walkthrough applications

Volume visualization is a method of extracting information from volumetric datasets through interactive graphics and imaging, and is concerned with the representation, manipulation, and rendering of these datasets [Gallagher 1995; Kaufman 1991; Rosenblum 1994]. Volume data are 3D entities that may have information inside them, may not consist of surfaces and edges, or may be too voluminous to be represented geometrically. Volume visualization encompasses an array of techniques for peering inside the dataset and for interactively extracting meaningful information from it using transformations, cuts, segmentation, translucency, measurements, and the like. The primary sources of volume data are three: sampled data of real objects or phenomena, computed data produced by a computer simulation, and modeled data generated from a geometric model. Examples of applications generating sampled data are medical imaging (e.g., CT, MRI), biology (e.g., confocal microscopy), geoscience (e.g., seismic measurements), industry (e.g., nondestructive inspection), and chemistry (e.g., electron density maps) [Kaufman 1991]. Some examples of applications generating computed datasets, typically by running a simulation on a supercomputer, are meteorology (e.g., storm prediction), computational fluid dynamics (e.g., water flow), and materials science (e.g., new materials). Recently, many traditional computer graphics applications, such as computer-aided design and flight simulation [Cohen and Shaked 1993; Kaufman et al. 1993], have been exploiting the advantages of volumetric techniques for modeling, manipulation, and visualization, an approach called volume graphics [Kaufman et al. 1993]. Volumetric data is typically a set S of samples (x, y, z, v), representing the value v of some property of the data at a 3D location (x, y, z). If v is simply a 0 or a 1, with 0 indicating background and 1 indicating the object, the data is called binary data. The data may instead be multivalued, with v representing some measurable property of the data, such as density, color, heat, or pressure. The value v may even be a vector, representing, for example, velocity at each location. In general, the samples may be taken at random locations in space, but in many cases S is isotropic, containing samples taken at regularly spaced intervals along three orthogonal axes. Since S is defined on a regular grid, a 3D array (called volume buffer, cubic frame buffer, 3D raster) is typically used to store the values. S is therefore referred to as the array of values S(x, y, z), which is defined only at grid locations. A function may be defined to describe the value at any continuous location by approximating v at a location (x, y, z) using some interpolation function to S, such as zero-order (nearest-neighbor), piecewise function known as first-order (trilinear), or higher-order interpolation. The region of constant value that surrounds each sample in zero-order interpolation is known as a volume cell (voxel for short), with each voxel being a rectangular cuboid having six faces, twelve edges, and eight corners. The terms, voxel, grid location,

Volume Visualization

Leveraging on the recent developments in convolutional neural networks (CNNs), matching dense correspondence from a stereo pair has been cast as a learning problem, with performance exceeding traditional approaches. However, it remains challenging to generate high-quality disparities for the inherently ill-posed regions. To tackle this problem, we propose a novel cascade CNN architecture composing of two stages. The first stage advances the recently proposed DispNet by equipping it with extra up-convolution modules, leading to disparity images with more details. The second stage explicitly rectifies the disparity initialized by the first stage; it couples with the first-stage and generates residual signals across multiple scales. The summation of the outputs from the two stages gives the final disparity. As opposed to directly learning the disparity at the second stage, we show that residual learning provides more effective refinement. Moreover, it also benefits the training of the overall cascade network. Experimentation shows that our cascade residual learning scheme provides state-of-the-art performance for matching stereo correspondence. By the time of the submission of this paper, our method ranks first in the KITTI 2015 stereo benchmark, surpassing the prior works by a noteworthy margin.

/pdf/cascade-residual-learning-a-two-stage-convolutional-neural-4v3facfp0x.pdf

Cascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching

We present an improved three-step pipeline for the stereo matching problem and introduce multiple novelties at each stage. We propose a new highway network architecture for computing the matching cost at each possible disparity, based on multilevel weighted residual shortcuts, trained with a hybrid loss that supports multilevel comparison of image patches. A novel post-processing step is then introduced, which employs a second deep convolutional neural network for pooling global information from multiple disparities. This network outputs both the image disparity map, which replaces the conventional winner takes all strategy, and a confidence in the prediction. The confidence score is achieved by training the network with a new technique that we call the reflective loss. Lastly, the learned confidence is employed in order to better detect outliers in the refinement step. The proposed pipeline achieves state of the art accuracy on the largest and most competitive stereo benchmarks, and the learned confidence is shown to outperform all existing alternatives.

/pdf/improved-stereo-matching-with-constant-highway-networks-and-528lwwviyx.pdf

Improved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning

We present an improved three-step pipeline for the stereo matching problem and introduce multiple novelties at each stage. We propose a new highway network architecture for computing the matching cost at each possible disparity, based on multilevel weighted residual shortcuts, trained with a hybrid loss that supports multilevel comparison of image patches. A novel post-processing step is then introduced, which employs a second deep convolutional neural network for pooling global information from multiple disparities. This network outputs both the image disparity map, which replaces the conventional "winner takes all" strategy, and a confidence in the prediction. The confidence score is achieved by training the network with a new technique that we call the reflective loss. Lastly, the learned confidence is employed in order to better detect outliers in the refinement step. The proposed pipeline achieves state of the art accuracy on the largest and most competitive stereo benchmarks, and the learned confidence is shown to outperform all existing alternatives.

We present a method for the generation of photo-realistic images of views over terrain datasets by mapping a digital aerial photograph on a perspective projection of a digital elevation map. We use high resolution for both digital maps to increase the quality and the realism of the image at the cost of the overhead of processing very large data bases. In the core of this paper we present an accelerated ray casting technique based on a new algorithm of traversing a pyramidal data structure. Unlike other known traversal techniques, the cost of a single step of the algorithm consists of a few additions, shifts and comparisons only.

Photo‐Realistic Imaging of Digital Terrains

The problem of detecting the visible and hidden areas of a topographic surface from a given viewpoint is well known, and has applications in GIS and other fields. The computational geometry literature is rich with solutions based on a polygonal representation of the surface. This work confronts the problem by working directly on the Digital Elevation Map which represents the terrain surface. We present an algorithm that processes discrete lines of sight from the viewpoint to the surface perimeter, and tests the unit-sized terrain elements along the discrete cross-sections defined by these lines. The algorithm is very efficient, performing On testing operations, consisting of a few additions and no more than one multiplication each, where n is the number of the terrain elements in the map.

/pdf/visibility-and-dead-zones-in-digital-terrain-maps-16bf0p33wh.pdf

Visibility and Dead-Zones in Digital Terrain Maps

The bulk minority carrier lifetime and interface recombination velocity in GaInP double-heterostructures (DHs) lattice matched to GaAs are extracted using time-resolved photoluminescence (PL) measured between 300 and 500K. Effective lifetimes show a strong dependence on temperature for samples with insufficiently strong confinement potentials due to significant thermionic emission losses out of the DHs. An increased PL signal from the underlying GaAs layer relative to GaInP's PL at high temperatures supports this hypothesis. The impact is a shorter recombination lifetime which can be wrongly interpreted as a high interface recombination velocity of up to 4500 cm/s at 500K. These effects are investigated experimentally using samples of different barrier heights based on the Al-content in (AlxGa1-x)0.51In0.49P. A larger barrier height is shown to inhibit thermionic emission out of the DH, thus revealing a more accurate interface recombination velocity of 300 cm/s at 500K. The results are then used to develop a correction procedure to extract a more accurate interface recombination velocity at the barriers of the DH. Optoelectronic device simulations are used to gain insight into carrier dynamics as a function of temperature and the DH's barrier height, and confirm a strong inhibition of the thermionic emission losses as the Al-content is maximized in the barrier.

https://iopscience.iop.org/article/10.1088/1361-6641/ab708e/pdf

Amit Shaked

Papers

Improved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning

Improved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning

Photo‐Realistic Imaging of Digital Terrains

Visibility and Dead-Zones in Digital Terrain Maps

Carrier recombination dynamics in Ga 0.51 In 0.49 P double-heterostructures up to 500 K