Discussion for direct versus features session

High frame-rate offers benefits of robust and accurate camera tracking for rapid motion. However, the benefits are generally understated arguing that it is not possible to operate on high frame-rates due to stringent processing budgets and that even today 1060Hz is treated as a standard real-time frame-rate range. How exactly does the choice of a given frame-rate varies as computational budget is changed? This thesis explores the possibilities of tracking at frame-rates higher than this range and argues that the computational cost per frame in trackers that use prediction is substantially reduced when the frame-rate is increased. Additionally, considering the physics of image formation, high frame-rate implies that the upper bound on the shutter time is reduced leading to less motion blur but more noise. On the other hand, low frame-rate often leads to motion blur but reduced noise in the images. Carefully considering the scene lighting that affects the image noise and the camera motion that affects the motion blur and putting these factors together, how are application-dependent performance requirements of accuracy, robustness and computational cost optimised as frame-rate varies? We study 3D camera tracking from a known rigid model as our test problem and analyse the fundamental image alignment approach to understand the choice of frame-rate that affects tracking. We systematically investigate this via a careful synthesis of photorealistic video using ray-tracing of detailed 3D scene, experimentally obtained photo-realistic reponse and noise models and rapid camera motions and later validate the conclusions with a well-controlled real experiment. The thesis provides quantitative conclusions about frame-rate selection, fundamental connections between frame-rate and image resolution and highlights the crucial role of full consideration of physical image formation process in pushing tracking performance.

Analysing high frame-rate camera tracking

Image-guided surgery needs an efficient and effective camera tracking system in order to perform augmented reality for overlaying preoperative models or label cancerous tissues on the 2D video images of the surgical scene. Tracking in endoscopic/laparoscopic scenes however is an extremely difficult task primarily due to tissue deformation, instrument invasion into the surgical scene and the presence of specular highlights. State of the art feature-based SLAM systems such as PTAM fail in tracking such scenes since the number of good features to track is very limited. When the scene is smoky and when there are instrument motions, it will cause feature-based tracking to fail immediately. The work of this thesis provides a systematic approach to this problem using dense vision. We initially attempted to register a 3D preoperative model with multiple 2D endoscopic/laparoscopic images using a dense method but this approach did not perform well. We subsequently proposed stereo reconstruction to directly obtain the 3D structure of the scene. By using the dense reconstructed model together with robust estimation, we demonstrate that dense stereo tracking can be incredibly robust even within extremely challenging endoscopic/laparoscopic scenes. Several validation experiments have been conducted in this thesis. The proposed stereo reconstruction algorithm has turned out to be the state of the art method for several publicly available ground truth datasets. Furthermore, the proposed robust dense stereo tracking algorithm has been proved highly accurate in synthetic environment (< 0.1 mm RMSE) and qualitatively extremely robust when being applied to real scenes in RALP prostatectomy surgery. This is an important step toward achieving accurate image-guided laparoscopic surgery.

/pdf/dense-vision-in-image-guided-surgery-c1f3ft0t7g.pdf

Dense Vision in Image-guided Surgery

The determination of the 3D motion of a camera and the 3D structure of the scene in which the camera is moving, known as the Structure from Motion (SFM) problem, is a central problem in computer vision. Specifically, the recursive (online) estimation is of major interest for robotics applications such as navigation and mapping. Many problems still hinder the deployment of SFM in real-life applications namely, (1) the robustness to noise, outliers and ambiguous motions, (2) the numerical tractability with a large number of features and (3) the cases of rapidly varying camera velocities. Towards solving those problems, this research presents the following four contributions that can be used individually, together, or combined with other approaches. A motion-only filter is devised by capitalizing on algebraic threading constraints. This filter efficiently integrates information over multiple frames achieving a performance comparable to the best state of the art filters. However, unlike other filter based approaches, it is not affected by large baselines (displacement between camera centers). An approach is introduced to incorporate, with only a small computational overhead, a large number of frame-to-frame features (i.e., features that are matched only in pairs of consecutive frames) in any analytic filter. The computational overhead grows linearly with the number of added frame-to-frame features and the experimental results show an increased accuracy and consistency. A novel filtering approach scalable to accommodate a large number of features is proposed. This approach achieves both the scalability of the state of the art filter in scalability and the accuracy of the state of the art filter in accuracy. A solution to the problem of prediction over large baselines in monocular Bayesian filters is presented. This problem is due to the fact that a simple prediction, using constant velocity models for example, is not suitable for large baselines, and the projections of the 3D points that are in the state vector can not be used in the prediction due to the need of preserving the statistical independence of the prediction and update steps.

Discussion for direct versus features session

References

Analysing high frame-rate camera tracking

Dense Vision in Image-guided Surgery

Recursive Estimation of Structure and Motion from Monocular Images

Related Papers (5)

An approach to a parallel shape analysis with the aid of the rough sets

About direct methods

Shape analysis in R2 space using mathematical morphology

Application of high-resolution computer graphics to pattern recognition analysis

A two dimensional context-dependent similarity measure