View synthesis

About: View synthesis is a research topic. Over the lifetime, 1701 publications have been published within this topic receiving 42333 citations.

Deep Learning Anthropomorphic 3D Point Clouds from a Single Depth Map Camera Viewpoint

Abstract: In footwear, fit is highly dependent on foot shape, which is not fully captured by shoe size. Scanners can be used to acquire better sizing information and allow for more personalized footwear matching, however when scanning an object, many images are usually needed for reconstruction. Semantics such as knowing the kind of object in view can be leveraged to determine the full 3D shape given only one input view. Deep learning methods have been shown to be able to reconstruct 3D shape from limited inputs in highly symmetrical objects such as furniture and vehicles. We apply a deep learning approach to the domain of foot scanning, and present a method to reconstruct a 3D point cloud from a single input depth map. Anthropomorphic body parts can be challenging due to their irregular shapes, difficulty for parameterizing and limited symmetries. We train a view synthesis based network and show that our method can produce foot scans with accuracies of 1.55 mm from a single input depth map.

Depth-guided patch-based disocclusion filling for view synthesis via Markov random field modelling

Abstract: In this paper, we propose a novel patch-based disocclusion filling method for view synthesis from video-plus-depth data. The proposed method treats disocclusion filling as a global optimization problem, where global (spatial) consistency among the patches is enforced via a Markov random field (MRF) model. The main idea of our method is to exploit disocclusion properties to limit and guide the search for candidate patches (labels) and to minimize efficiently the resulting MRF energy. In particular, we propose to constrain the label selection to local background regions in order to ensure that the disocclusions are filled with background information. Background is determined based on the locally estimated hard threshold on the depth values. The efficient minimization approach represents an extension of our previous method for general inpainting, where we propose to visit the MRF nodes from the background to the foreground disocclusion border and discard unnecessary labels. In this way, the number of labels is further reduced and the propagation of background information is additionally enforced. Finally, efficient inference is performed to obtain the final inpainting result. The proposed disocclusion filling method represents one step of the complete view synthesis framework that we also introduce in this paper. Experimental results show improvement of the proposed approach over related state-of-the-art methods both for small and big disocclusions.

Luminance coding in graph-based representation of multiview images

Abstract: Multi-view video transmission poses great challenges because of its data size and dimension. Therefore, how to design efficient 3D scene representations and coding (of luminance and geometry) has become a critical research topic. Recently, the graph-based representation (GBR) is introduced, which provides a lossless compression of multi-view geometry by connecting informative pixels among views. This representation has been shown as a promising alternative to the classical depth-based representation, where the view synthesis accuracy is hard to control. In this work, we study the luminance compression under GBR, which is not well considered in existing literature. With a proper structural reformulation, we show that the graph-based transform can be applied on the GBR paradigm, hence better extracting the correlation among pixels along graph connections. Moreover, we extend the popular SPIHT coding scheme to further improve coding efficiency. The experimental results show that our method leads to better RD coding performance as compared the classical luminance coding algorithms.

Projector View Synthesis and Virtual Texturing

Abstract: We propose a technique to project a synthesized texture onto 3D textureless real objects without contact, which can be applied for example, in medical or archaeological applications. Our system uses two cameras and one projector. By projecting a suitable set of light patterns onto the surface of a 3D real object and by capturing images with one or several cameras, a large number of correspondences can be determined and the 3D geometry can be estimated. Since we deal with textureless objects, traditional corner detection algorithms can not be applied, but our approach successfully determines both 3D geometry and the appropriate texture to be projected onto the 3D model.

Gaze correction using 3D video processing for videoconferencing

Abstract: In this paper, we propose a gaze correction method using 3D video processing techniques including depth estimation and virtual view synthesis. We design a set of color and depth cameras to reduce occlusion regions and improve the depth precision in the less-detailed region. The proposed algorithm deals with fully unsolved problems by fusing the depth data from both depth sensors. Furthermore, view synthesis is performed to create the gaze corrected image from the obtained depth information. Experimental results show our contribution is useful for videoconferencing.