SoftPOSIT: Simultaneous Pose and Correspondence
Determination
Philip David
, Daniel DeMenthon
, Ramani Duraiswami
, and Hanan Samet
University of Maryland Institute for Advanced Computer Studies,
College Park, MD 20742
Army Research Laboratory, 2800 Powder Mill Road, Adelphi, MD 20783-1197
Abstract
The problem of pose estimation arises in many areas of computer vision, including object recognition,
object tracking, site inspection and updating, and autonomous navigation when scene models are avail-
able. We present a new algorithm, called SoftPOSIT, for determining the pose of a 3D object from a
single 2D image when correspondences between model points and image points are not known. The
algorithm combines Gold’s iterative softassign algorithm [Gold 1996, Gold 1998] for computing corre-
spondences and DeMenthon’s iterative POSIT algorithm [DeMenthon 1995] for computing object pose
under a full-perspective camera model. Our algorithm, unlike most previous algorithms for pose de-
termination, does not have to hypothesize small sets of matches and then verify the remaining image
points. Instead, all possible matches are treated identically throughout the search for an optimal pose.
The performance of the algorithm is extensively evaluated in Monte Carlo simulations on synthetic data
The support of NSF grants EAR-99-05844 and IIS-00-86116 is gratefully acknowledged.
1
under a variety of levels of clutter, occlusion, and image noise. These tests show that the algorithm per-
forms well in a variety of difficult scenarios, and empirical evidence suggests that the algorithm has an
asymptotic run-time complexity that is better than previous methods by a factor of the number of image
points. The algorithm is being applied to a number of practical autonomous vehicle navigation problems
including the registration of 3D architectural models of a city to images, and the docking of small robots
onto larger robots.
1 Introduction
This paper presents an algorithm for solving the model-to-image registration problem, which is the task
of determining the position and orientation (the pose) of a three-dimensional object with respect to a
camera coordinate system, given a model of the object consisting of 3D reference points and a single 2D
image of these points. We assume that no additional information is available with which to constrain the
pose of the object or to constrain the correspondence of model features to image features. This is also
known as the simultaneous pose and correspondence problem.
Automatic registration of 3D models to images is an important problem. Applications include object
recognition, object tracking, site inspection and updating, and autonomous navigation when scene mod-
els are available. It is a difficult problem because it comprises two coupled problems, the correspondence
problem and the pose problem, each easy to solve only if the other has been solved first:
1. Solving the pose (or exterior orientation) problem consists of finding the rotation and translation
of the object with respect to the camera coordinate system. Given matching model and image fea-
tures, one can easily determine the pose that best aligns those matches. For three to five matches,
the pose can be found in closed form by solving sets of polynomial equations [Fischler 1981,
Haralick 1991, Horaud 1989, Yuan 1989]. For six or more matches, linear and nonlinear ap-
proximate methods are generally used [DeMenthon 1995, Fiore 2001, Hartley 2000, Horn 1986,
Lu 2000].
2. Solving the correspondence problem consists of finding matching image features and model fea-
2
tures. If the object pose is known, one can relatively easily determine the matching features. Pro-
jecting the model in the known pose into the original image, one can identify matches among
the model features that project sufficiently close to an image feature. This approach is typi-
cally used for pose verification, which attempts to determine how good a hypothesized pose is
[Grimson 1991].
The classic approach to solving these coupled problems is the hypothesize-and-test approach [Grimson 1990].
In this approach, a small set of image feature to model feature correspondences are first hypothesized.
Based on these correspondences, the pose of the object is computed. Using this pose, the model points
are back-projected into the image. If the original and back-projected images are sufficiently similar,
then the pose is accepted; otherwise, a new hypothesis is formed and this process is repeated. Perhaps
the best known example of this approach is the RANSAC algorithm [Fischler 1981] for the case that
no information is available to constrain the correspondences of model points to image points. When
three correspondences are used to determine a pose, a high probability of success can be achieved by the
RANSAC algorithm in
time when there are
image points and
object points (see “Model
point”
has been
changed
to
“object
point”
through-
out the
paper.
Previ-
ously,
we used
both
“model”
and
“object”
to mean
the same
thing.
Appendix A for details).
The problem addressed here is one that is encountered when taking a model-based approach to the
object recognition problem, and as such has received considerable attention. (The other main approach
to object recognition is the appearance-based approach [Murase 1995] in which multiple views of the
object are compared to the image. However, since 3D models are not used, this approach doesn’t pro-
vide accurate object pose.) Many investigators (e.g., [Cass 1994, Cass 1998, Ely 1995, Jacobs 1992,
Lamdan 1988, Procter 1997]) approximate the nonlinear perspective projection via linear affine ap-
proximations. This is accurate when the relative depths of object features are small compared to the
distance of the object from the camera. Among the pioneer contributions were Baird’s tree-pruning
method [Baird 1985], with exponential time complexity for unequal point sets, and Ullman’s alignment
method [Ullman 1989] with time complexity
.
The geometric hashing method [Lamdan 1988] determines an object’s identity and pose using a hash-
ing metric computed from a set of image features. Because the hashing metric must be invariant to
3
camera viewpoint, and because there are no view-invariant image features for general 3D point sets (for
either perspective or affine cameras) [Burns 1993], this method can only be applied to planar scenes.
In [DeMenthon 1993], we proposed an approach using binary search by bisection of pose boxes in
two 4D spaces, extending the research of [Baird 1985, Cass 1992, Breuel 1992] on affine transforms,
but it had high-order complexity. The approach taken by Jurie [Jurie 1999] was inspired by our work
and belongs to the same family of methods. An initial volume of pose space is guessed, and all of
the correspondences compatible with this volume are first taken into account. Then the pose volume is
recursively reduced until it can be viewed as a single pose. As a Gaussian error model is used, boxes
of pose space are pruned not by counting the number of correspondences that are compatible with the
box as in [DeMenthon 1993], but on the basis of the probability of having an object model in the image
within the range of poses defined by the box.
Among the researchers who have addressed the full perspective problem, Wunsch and Hirzinger
[Wunsch 1996] formalize the abstract problem in a way similar to the approach advocated here as the
optimization of an objective function combining correspondence and pose constraints. However, the
correspondence constraints are not represented analytically. Instead, each model feature is explicitly
matched to the closest lines of sight of the image features. The closest 3D points on the lines of sight
are found for each model feature, and the pose that brings the model features closest to these 3D points
is selected; this allows an easier 3D to 3D pose problem to be solved. The process is repeated until a
minimum of the objective function is reached.
The object recognition approach of Beis [Beis 1999] uses view-variant 2D image features to index
3D object models. Off-line training is performed to learn 2D feature groupings associated with large
numbers of views of the objects. Then, the on-line recognition stage uses new feature groupings to
index into a database of learned model-to-image correspondence hypotheses, and these hypotheses are
used for pose estimation and verification.
The pose clustering approach to model-to-image registration is similar to the classic hypothesize-and-
test approach. Instead of testing each hypothesis as it is generated, all hypotheses are generated and
clustered in a pose space before any back-projection and testing takes place. This later step is performed
4
only on poses associated with high-probability clusters. The idea is that hypotheses including only
correct correspondences should form larger clusters in pose space than hypotheses that include incorrect
correspondences. Olson [Olson 1997] gives a randomized algorithm for pose clustering whose time
complexity is
.
The method of Beveridge and Riseman [Beveridge 1992, Beveridge 1995] is also related to our ap-
proach. Random-start local search is combined with a hybrid pose estimation algorithm employing
both full-perspective and weak-perspective camera models. A steepest descent search in the space of
model-to-image line segment correspondences is performed. A weak-perspective pose algorithm is used
to rank neighboring points in this search space, and a full-perspective pose algorithm is used to update
the model’s pose after making a move to a new set of correspondences. The time complexity of this
algorithm was empirically determined to be
.
When there are
object points and
image points, the dimension of the solution space for this
problem is
!#"
since there are
correspondence variables and 6 pose variables. Each correspon-
dence variable has the domain
$&%(')*',+,+,+'
'-/.
representing a match of a object point to one of the
image points or to no image point (represented by
-
), and each pose variable has a continuous domain
determined by the allowed range of object translations and rotations. Most algorithms don’t explicitly
search this
0!1"
-dimensional space, but instead assume that pose is determined by correspondences or
that correspondences are determined by pose, and so search either an
-dimensional or a 6-dimensional
space. The SoftPOSIT approach is different in that its search alternates between these two spaces.
The SoftPOSIT approach to solving the model-to-image registration problem applies the formalism
proposed by Gold, Rangarajan and others [Gold 1996, Gold 1998] when they solved the correspon-
dence and pose problem in matching two images or two 3D models. We extend it to the more difficult
problem of registration between a 3D model and its perspective image, which they did not address.
The SoftPOSIT algorithm integrates an iterative pose technique called POSIT (Pose from Orthography
and Scaling with ITerations) [DeMenthon 1995], and an iterative correspondence assignment technique
called softassign [Gold 1996, Gold 1998] into a single iteration loop. A global objective function is
defined that captures the nature of the problem in terms of both pose and correspondence and combines
5