This article provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed in detail. Since any visual servo system must be capable of tracking image features in a sequence of images, we also include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control.

/pdf/a-tutorial-on-visual-servo-control-5cy6rje63g.pdf

A tutorial on visual servo control

We propose a non-iterative solution to the PnP problem--the estimation of the pose of a calibrated camera from n 3D-to-2D point correspondences--whose computational complexity grows linearly with n This is in contrast to state-of-the-art methods that are O(n 5) or even O(n 8), without being more accurate Our method is applicable for all n?4 and handles properly both planar and non-planar configurations Our central idea is to express the n 3D points as a weighted sum of four virtual control points The problem then reduces to estimating the coordinates of these control points in the camera referential, which can be done in O(n) time by expressing these coordinates as weighted sum of the eigenvectors of a 12×12 matrix and solving a small constant number of quadratic equations to pick the right weights Furthermore, if maximal precision is required, the output of the closed-form solution can be used to initialize a Gauss-Newton scheme, which improves accuracy with negligible amount of additional time The advantages of our method are demonstrated by thorough testing on both synthetic and real-data

/pdf/epnp-an-accurate-o-n-solution-to-the-pnp-problem-4x6uxro3qr.pdf

EPnP: An Accurate O(n) Solution to the PnP Problem

In this paper, we describe a method for finding the pose of an object from a single image. We assume that we can detect and match in the image four or more noncoplanar feature points of the object, and that we know their relative geometry on the object. The method combines two algorithms; the first algorithm,POS (Pose from Orthography and Scaling) approximates the perspective projection with a scaled orthographic projection and finds the rotation matrix and the translation vector of the object by solving a linear system; the second algorithm,POSIT (POS with ITerations), uses in its iteration loop the approximate pose found by POS in order to compute better scaled orthographic projections of the feature points, then applies POS to these projections instead of the original image projections. POSIT converges to accurate pose measurements in a few iterations. POSIT can be used with many feature points at once for added insensitivity to measurement errors and image noise. Compared to classic approaches making use of Newton's method, POSIT does not require starting from an initial guess, and computes the pose using an order of magnitude fewer floating point operations; it may therefore be a useful alternative for real-time operation. When speed is not an issue, POSIT can be written in 25 lines or less in Mathematica; the code is provided in an Appendix.

Model-based object pose in 25 lines of code

Model-based recognition and motion tracking depend upon the ability to solve for projection and model parameters that will best fit a 3-D model to matching 2-D image features. The author extends current methods of parameter solving to handle objects with arbitrary curved surfaces and with any number of internal parameters representing articulation, variable dimensions, or surface deformations. Numerical stabilization methods are developed that take account of inherent inaccuracies in the image measurements and allow useful solutions to be determined even when there are fewer matches than unknown parameters. The Levenberg-Marquardt method is used to always ensure convergence of the solution. These techniques allow model-based vision to be used for a much wider class of problems than was possible with previous methods. Their application is demonstrated for tracking the motion of curved, parameterized objects. >

/pdf/fitting-parameterized-three-dimensional-models-to-images-qh3tnmd54d.pdf

Fitting parameterized three-dimensional models to images

Determining the rigid transformation relating 2D images to known 3D geometry is a classical problem in photogrammetry and computer vision. Heretofore, the best methods for solving the problem have relied on iterative optimization methods which cannot be proven to converge and/or which do not effectively account for the orthonormal structure of rotation matrices. We show that the pose estimation problem can be formulated as that of minimizing an error metric based on collinearity in object (as opposed to image) space. Using object space collinearity error, we derive an iterative algorithm which directly computes orthogonal rotation matrices and which is globally convergent. Experimentally, we show that the method is computationally efficient, that it is no less accurate than the best currently employed optimization methods, and that it outperforms all tested methods in robustness to outliers.

/pdf/fast-and-globally-convergent-pose-estimation-from-video-f1440b89kr.pdf

Fast and globally convergent pose estimation from video images

A method for finding analytical solutions to the problem of determining the attitude of a 3D object in space from a single perspective image is presented. Its principle is based on the interpretation of a triplet of any image lines as the perspective projection of a triplet of linear ridges of the object model, and on the search for the model attitude consistent with these projections. The geometrical transformations to be applied to the model to bring it into the corresponding location are obtained by the resolution of an eight-degree equation in the general case. Using simple logical rules, it is shown on examples related to polyhedra that this approach leads to results useful for both location and recognition of 3D objects because few admissible hypotheses are retained from the interpolation of the three line segments. Line matching by the prediction-verification procedure is thus less complex. >

Determination of the attitude of 3D objects from a single perspective view

It has been demonstrated here how some geometrical features extracted from the brightness image of an object of revolution and coming from the perspective projection of lines or points situated on the surface of the object, can be used to find the spatial attitude of the object in the viewer coordinate system.

/pdf/spatial-localization-of-modelled-objects-of-revolution-in-3z0m476rzp.pdf

Spatial Localization Of Modelled Objects Of Revolution In Monocular Perspective Vision

For scene analysis by object identification, a new method is presented which involves a structural representation of the objects the components of which are local patterns. These local patterns are detected with a hypothesis accumulation technique analogous to the generalized Hough transform but adapted to polygonal contours as pattern representations. The object identification is controlled by a prediction-verification procedure. When a first local pattern is found by hypothesis accumulation, according to the structural model of the object, a second one is predicted, i.e. its direction and the window in which it is expected are given. Its detection is then verified by hypothesis accumulation with these research parameter values. Such an identification method gathers the advantages of both the prediction-verification and the hypothesis accumulation approaches which are respectively, the ability to make very few hypotheses for object recognition, and the capability of handling variations in polygonal contour segmentation. This method has been applied to 2D-scenes of partially observed pieces and the reported results prove its efficiency.

Prediction-Verification, Hypothesis Accumulation For Scene Analysis By Object Identification

This paper presents all the procedures up to now integrated in the \"ULYSSE\" software and designed for the automatic localization of a mobile robot in the area of the airlock of an \"ELECTRICITE DE FRANCE\" nuclear power plant. It is shown that if the initial pose of the robot is known, if a 3D geometrical model (even partial) is available, if a coarse estimation of each displacement is provided from odiometric devices. then the pose of the robot can be automatically determined by monocular vision and by matching the successive images and the CAD model. This matching occurs between the linear edges of the model and the image straight contours obtained after a polygonal approximation, and the attitude of the model is derived from the interpretation of the projection of at least three model edges.

/pdf/localization-of-autonomous-mobile-robots-by-monocular-vision-45ietqgliq.pdf

Marc Richetin

Papers

Determination of the attitude of 3D objects from a single perspective view

Spatial Localization Of Modelled Objects Of Revolution In Monocular Perspective Vision

Prediction-Verification, Hypothesis Accumulation For Scene Analysis By Object Identification

Localization of Autonomous Mobile Robots by Monocular Vision in Modelled Sites. Application to a Nuclear Power Plant.

About the integrability constraint in a shape-from-shading algorithm