Optical Flow Based Structure from Motion

TLDR: A new motion based segmentation algorithm able to automatically detect and reconstruct planar regions and extend the non-linear estimator to incorporate the optical flow covariance matrix (maximum-likelihood) and it is shown that it is possible to locally time integrate the reconstruction process for increased robustness.

Abstract: Reconstructing the 3D shape of a scene from its 2D images is a problem that has attracted a great deal of research 3D models are nowadays widely used for scientific visualization, entertainment and engineering tasks Most of the approaches developed by the computer vision community can be roughly classified as feature based or flow based, according to if the data they use is a set of features matches or an optical flow field While a dense optical flow field, due to its noisy nature, is not extremely suitable for tracking, finding corresponding features between different views of large baseline is still an open problem The system we develop in this thesis is of a hybrid type We track sparse features over sequences acquired at 25Hz from an hand held camera During the tracking good features can be selected as those laying in high textured areas: this guarantees higher precision in the estimation of features displacements Such displacements are used to approximate optical flow We demonstrate that this approximation is a good one for our working conditions Using this approach we bypass the matching problem of stereo and the complexity and time integration problems of the optical flow based reconstruction Time integration is obtained by an optimal predict-update procedure that merges measurements by re-weighting by the respective covariance measurements Most of the research effort of this thesis is focused on the robust estimation of structure and motion from a pair of images and the related optical flow field We test first a linear solution that has the appealing property of being of closed form but the problem of returning biased estimates We propose an non-linear refinement to the linear estimator showing convergence properties and improvements in bias and variance We further extend the non-linear estimator to incorporate the optical flow covariance matrix (maximum-likelihood) and, moreover, we show that, in the case of dense sequences, it is possible to locally time integrate the reconstruction process for increased robustness We experimentally investigate the possibility of introducing geometrical constraints in the structure and motion estimation Such constraints are of bilinear type, ie planes, lines and incidence of these primitives are used For this purpose we present a new motion based segmentation algorithm able to automatically detect and reconstruct planar regions To asses the efficacy of our solution the algorithms were tested on a variety of real and simulated sequences ISBN 91-7283-308-4 • TRITA-02-11 • ISSN 0348-2952 • ISRN KTH/NA/R 02-11