Kinect-Variety Fusion: A Novel Hybrid Approach for Artifacts-Free 3DTV Content Generation
24 Aug 2014-pp 2275-2280
TL;DR: This paper presents a novel low-cost hybrid Kinect-variety based content generation scheme for 3DTV displays and demonstrates that proposed robust integration provides guarantees on the completeness and consistency of the algorithm.
Abstract: This paper presents a novel low-cost hybrid Kinect-variety based content generation scheme for 3DTV displays. The integrated framework constructs an efficient consistent image-space parameterization of 3D scene structure using only sparse depth information of few reference scene points. Under full-perspective camera model, the enforced Euclidean constraints simplify the synthesis of high quality novel multiview content for distinct camera motions. The algorithm does not rely on complete precise scene geometry information, and are unaffected by scene complex geometric properties, unconstrained environmental variations and illumination conditions. It, therefore, performs fairly well under a wider set of operation condition where the 3D range sensors fail or reliability of depth-based algorithms are suspect. The robust integration of vision algorithm and visual sensing scheme complement each other's shortcomings. It opens new opportunities for envisioning vision-sensing applications in uncontrolled environments. We demonstrate that proposed robust integration provides guarantees on the completeness and consistency of the algorithm. This leads to improved reliability on an extensive set of experimental results.
Citations
More filters
21 May 2018
TL;DR: A deep convolutional neural network architecture for high-precision depth estimation by jointly utilizing sparse 3D LiDAR and dense stereo depth information and is significantly more accurate than various baseline approaches.
Abstract: We present a deep convolutional neural network (CNN) architecture for high-precision depth estimation by jointly utilizing sparse 3D LiDAR and dense stereo depth information. In this network, the complementary characteristics of sparse 3D LiDAR and dense stereo depth are simultaneously encoded in a boosting manner. Tailored to the LiDAR and stereo fusion problem, the proposed network differs from previous CNNs in the incorporation of a compact convolution module, which can be deployed with the constraints of mobile devices. As training data for the LiDAR and stereo fusion is rather limited, we introduce a simple yet effective approach for reproducing the raw KITTI dataset. The raw LiDAR scans are augmented by adapting an off-the-shelf stereo algorithm and a confidence measure. We evaluate the proposed network on the KITTI benchmark and data collected by our multi-sensor acquisition system. Experiments demonstrate that the proposed network generalizes across datasets and is significantly more accurate than various baseline approaches.
59 citations
Cites background or methods from "Kinect-Variety Fusion: A Novel Hybr..."
...To estimate reliable depth information of scene, two kinds of techniques can be utilized, the use of active 3D scanners such as RGB-D sensors [24] or 3D LiDAR scanners [23] and the use of passive matching algorithms on stereo images [12]....
[...]
...For challenging outdoor scenarios, 3D LiDAR scanner [23] has been the most practical solution for 3D perception since the RGB-D sensor such as Kinect [24] frequently fails in the presence of sunlight [24] and provides a limited sensing range....
[...]
TL;DR: A deep sensor fusion framework for high-precision depth estimation using LiDAR point cloud and stereo images, and a simple but effective approach to generate pseudo ground truth labels from the raw KITTI dataset.
Abstract: We address the problem of 3D reconstruction from uncalibrated LiDAR point cloud and stereo images. Since the usage of each sensor alone for 3D reconstruction has weaknesses in terms of density and accuracy, we propose a deep sensor fusion framework for high-precision depth estimation. The proposed architecture consists of calibration network and depth fusion network, where both networks are designed considering the trade-off between accuracy and efficiency for mobile devices. The calibration network first corrects an initial extrinsic parameter to align the input sensor coordinate systems. The accuracy of calibration is markedly improved by formulating the calibration in the depth domain. In the depth fusion network, complementary characteristics of sparse LiDAR and dense stereo depth are then encoded in a boosting manner. Since training data for the LiDAR and stereo depth fusion are rather limited, we introduce a simple but effective approach to generate pseudo ground truth labels from the raw KITTI dataset. The experimental evaluation verifies that the proposed method outperforms current state-of-the-art methods on the KITTI benchmark. We also collect data using our proprietary multi-sensor acquisition platform and verify that the proposed method generalizes across different sensor settings and scenes.
56 citations
Cites background from "Kinect-Variety Fusion: A Novel Hybr..."
...For challenging outdoor scenarios, 3D LiDAR scanners [8] have become practical solutions for 3D perception since RGB-D sensors, such as Kinect 2 [7], often fail in the presence of sunlight [9] and provide limited sensing range....
[...]
TL;DR: A new hybrid Kinect-variety-based synthesis scheme that renders artifact-free multiple views for autostereoscopic/automultiscopic displays and provides guarantees on the completeness, optimality with respect to the inter-view consistency of the algorithm is presented.
Abstract: This paper presents a new hybrid Kinect-variety-based synthesis scheme that renders artifact-free multiple views for autostereoscopic/automultiscopic displays. The proposed approach does not explicitly require dense scene depth information for synthesizing novel views from arbitrary viewpoints. Instead, the integrated framework first constructs a consistent minimal image–space parameterization of the underlying 3D scene. The compact representation of scene structure is formed using only implicit sparse depth information of a few reference scene points extracted from raw RGB depth data. The views from arbitrary positions can be inferred by moving the novel camera in parameterized space by enforcing Euclidean constraints on reference scene images under a full-perspective projection model. Unlike the state-of-the-art depth image-based rendering (DIBR) methods, in which input depth map accuracy is crucial for high-quality output, our proposed algorithm does not depend on precise per-pixel geometry information. Therefore, it simply sidesteps to recover and refine the incomplete or noisy depth estimates with advanced filling or upscaling techniques. Our approach performs fairly well in unconstrained indoor/outdoor environments, where the performance of range sensors or dense depth-based algorithms could be seriously affected due to scene complex geometric conditions. We demonstrate that the proposed hybrid scheme provides guarantees on the completeness, optimality with respect to the inter-view consistency of the algorithm. In the experimental validation, we performed a quantitative evaluation as well as subjective assessment of the scene with complex geometric or surface properties. A comparison with the latest representative DIBR methods is additionally performed to demonstrate the superior performance of the proposed scheme.
13 citations
Cites methods from "Kinect-Variety Fusion: A Novel Hybr..."
...Once the image positions q0v , q1v , and q2v of reference scene points Q0, Q1, and Q2 are fixed at a defined novel viewpoint, the corresponding new image positions of all other points q ′ vs are inferred directly by solving the 6D variety Dv associated with a new camera using earlier computed structure coefficients ζ [15]....
[...]
...Our Kinect-based hybrid technique, in fact, presents an alternate, linear simplified formulation to solve this formidable FP variety problem [15]....
[...]
09 May 2019
TL;DR: Experiments indicate that the optimized color-guided filter for depth image denoising from different types of noises outperforms counterpart filters with guided and non-guided manners in terms of a variety of evaluation metrics.
Abstract: Color Guided Depth image denoising often suffers from the texture coping from the color image as well as the blurry effect at the depth discontinuities. Motivated by this, we propose an optimized color-guided filter for depth image denoising from different types of noises. This is a new framework that helps to mitigate the texture coping and enhance the depth discontinuities, especially in heavy noises. This framework consists of two parts namely depth driven color flattening model and patch synthesis-based Markov random field model. The first part which is a prepare step for the second part is used to mitigate the texture coping problem that faces all color guided methods. This first model consists of a modified joint bilateral filter which is used to mitigate the noise from the noisy depth image and an iterative guided bilateral filter that is proposed to flatten the colors in the color image for mitigating the texture coping problem. Based on the first part, Markov random field with an optimization technique is used for mitigating the blurry effect. Experiments indicate that our method outperforms counterpart filters with guided and non-guided manners in terms of a variety of evaluation metrics.
7 citations
Cites background from "Kinect-Variety Fusion: A Novel Hybr..."
...Depth images are crucial in a variety of visual communication and computer vision applications such as 3DTV broadcasting [1], 3D reconstruction [2] and visual saliency [3]....
[...]
References
More filters
Book•
01 Jan 2000TL;DR: In this article, the authors provide comprehensive background material and explain how to apply the methods and implement the algorithms directly in a unified framework, including geometric principles and how to represent objects algebraically so they can be computed and applied.
Abstract: From the Publisher:
A basic problem in computer vision is to understand the structure of a real world scene given several images of it. Recent major developments in the theory and practice of scene reconstruction are described in detail in a unified framework. The book covers the geometric principles and how to represent objects algebraically so they can be computed and applied. The authors provide comprehensive background material and explain how to apply the methods and implement the algorithms directly.
15,558 citations
01 Jan 2001
TL;DR: This book is referred to read because it is an inspiring book to give you more chance to get experiences and also thoughts and it will show the best book collections and completed collections.
Abstract: Downloading the book in this website lists can give you more advantages. It will show you the best book collections and completed collections. So many books can be found in this website. So, this is not only this multiple view geometry in computer vision. However, this book is referred to read because it is an inspiring book to give you more chance to get experiences and also thoughts. This is simple, read the soft file of the book and you get it.
14,282 citations
"Kinect-Variety Fusion: A Novel Hybr..." refers methods in this paper
...False correspondences appear due to local visual similarity of features are removed using RANSAC filtering [14]....
[...]
07 May 2006
TL;DR: A novel scale- and rotation-invariant interest point detector and descriptor, coined SURF (Speeded Up Robust Features), which approximates or even outperforms previously proposed schemes with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster.
Abstract: In this paper, we present a novel scale- and rotation-invariant interest point detector and descriptor, coined SURF (Speeded Up Robust Features). It approximates or even outperforms previously proposed schemes with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster.
This is achieved by relying on integral images for image convolutions; by building on the strengths of the leading existing detectors and descriptors (in casu, using a Hessian matrix-based measure for the detector, and a distribution-based descriptor); and by simplifying these methods to the essential. This leads to a combination of novel detection, description, and matching steps. The paper presents experimental results on a standard evaluation set, as well as on imagery obtained in the context of a real-life object recognition application. Both show SURF's strong performance.
13,011 citations
"Kinect-Variety Fusion: A Novel Hybr..." refers methods in this paper
...First to register RGB-D data, our system detects SURF feature descriptors in each color image and matches the local descriptors between all image pairs using approximate nearest neighbors search [3], [13]....
[...]
TL;DR: A novel scale- and rotation-invariant detector and descriptor, coined SURF (Speeded-Up Robust Features), which approximates or even outperforms previously proposed schemes with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster.
Abstract: This article presents a novel scale- and rotation-invariant detector and descriptor, coined SURF (Speeded-Up Robust Features). SURF approximates or even outperforms previously proposed schemes with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster. This is achieved by relying on integral images for image convolutions; by building on the strengths of the leading existing detectors and descriptors (specifically, using a Hessian matrix-based measure for the detector, and a distribution-based descriptor); and by simplifying these methods to the essential. This leads to a combination of novel detection, description, and matching steps. The paper encompasses a detailed description of the detector and descriptor and then explores the effects of the most important parameters. We conclude the article with SURF's application to two challenging, yet converse goals: camera calibration as a special case of image registration, and object recognition. Our experiments underline SURF's usefulness in a broad range of topics in computer vision.
12,449 citations
TL;DR: P can be chosen to I&E the centroid oC the triangle formed by X, y and z and Express each si E S in polar coordinates th origin P and 8 = 0 in the direction of zu~ arhitnry fixed half-line L from P.
Abstract: Step Find a point Pin the plane w%ch is in &he In&$x of Cl-l(s). At worst, this can be done in clfl sQp9 by te dting 3 element subsets of S for collineti@, discarding middle p&n& of collinear rets ar5b bq@rig when the fust noncollinear set (if there i# or&j. &y X, y arrcf z, is found. P can be chosen to I&E the centroid oC the triangle formed by X, y and z. Sfq 2: Express each si E S in polar coordinates th origin P and 8 = 0 in the direction of zu~ arhitnry fixed half-line L from P. This canversion can be done in c2n operations ior some rimed constant
1,741 citations
"Kinect-Variety Fusion: A Novel Hybr..." refers background in this paper
...2) For selecting other reference points q1 and q2, find the convex hull of matched points in any one of the reference view [5], and compute the counterclockwise sequence of extreme points on the lower and upper hull respectively....
[...]