International Journal of Computer Vision 60(3), 241–265, 2004
c
2004 Kluwer Academic Publishers. Manufactured in The Netherlands.
On Symmetry and Multiple-View Geometry:
Structure, Pose, and Calibration from a Single Image
∗
WEI HONG, ALLEN YANG YANG, KUN HUANG AND YI MA
Department of Electrical & Computer Engineering, University of Illinois at Urbana-Champaign,
1308 West Main St., Urbana, IL 61801, USA
weihong@uiuc.edu
yangyang@uiuc.edu
kunhuang@uiuc.edu
yima@uiuc.edu
Received October 16, 2002; Revised March 17, 2004; Accepted March 17, 2004
Abstract. In this paper, we provide a principled explanation of how knowledge in global 3-D structural invariants,
typically captured by a group action on a symmetric structure, can dramatically facilitate the task of reconstructing
a 3-D scene from one or more images. More importantly, since every symmetric structure admits a “canonical”
coordinate frame with respect to which the group action can be naturally represented, the canonical pose between
the viewer and this canonical frame can be recovered too, which explains why symmetric objects (e.g., buildings)
provide us overwhelming clues to their orientation and position. We give the necessary and sufficient conditions
in terms of the symmetry (group) admitted by a structure under which this pose can be uniquely determined. We
also characterize, when such conditions are not satisfied, to what extent this pose can be recovered. We show how
algorithms from conventional multiple-view geometry, after properly modified and extended, can be directly applied
to perform such recovery, from all “hidden images” of one image of the symmetric structure. We also apply our
results to a wide range of applications in computer vision and image processing such as camera self-calibration,
image segmentation and global orientation, large baseline feature matching, image rendering and photo editing, as
well as visual illusions (caused by symmetry if incorrectly assumed).
Keywords: structure from symmetry, multiple-view geometry, symmetry group, reflective symmetry, rotational
symmetry, and translational symmetry
1. Introduction
One of the main goals of computer vision is the study
of how to infer three-dimensional (3-D) information
(e.g., shape, layout and motion) of a scene from its
two-dimensional (2-D) image(s). A particular thrust of
effort is to extract 3-D geometric information from 2-
D images by exploiting geometric relationships among
multiple images of the same set of features on a 3-D
∗
This work is supported by UIUC ECE/CSL startup fund and NSF
Career Award IIS-0347456.
object. This gives rise to the subject of multiple-view
geometry,aprimary focus of study in the computer
vision community for the past two decades or so. Un-
fortunately, certain relationships among features them-
selves have been, to a large extent, ignored or at least
under-studied. Some of those relationships, as we will
see from this paper, have significant impact on the way
that 3-D information can be (and should be) inferred
from images.
Before we proceed further, let us pause and exam
the images given in Fig. 1 below. What do they have
in common? Notice that these images are just a few
242 Hong et al.
Figure 1. Symmetry is in: architecture, machines, textures, crystals, molecules, ornaments, and nature, etc.
representatives of a common phenomenon exhibited in
nature or man-made environment: symmetry.Itisnot
so hard to convince ourselves that even from only a
single image, we are able to perceive clearly the 3-D
structure and relative pose (orientation and location)
of the object being seen, even though in the image the
shape of the object is distorted by the perspective pro-
jection. The reason is, simply put, there is symmetry at
play.
1
The goals of this paper are to provide a principled
explanation why symmetry could encode 3-D informa-
tion within a single perspective image and to develop
algorithms based on multiple-view geometry that ef-
ficiently extract the 3-D information from single im-
ages. There are two things which we want to point out
already:
1. Symmetry is not the only cue which encodes 3-D in-
formation through relationships among a set of fea-
tures (in one image or more images). For instance,
incidence relations among points, lines, and planes
may as well provide 3-D information to the viewer;
2. The concept of symmetry that we consider here is
not just the (bilateral) reflective symmetry, or the
(statistical) isotopic symmetry which has been stud-
ied in a certain extent in the computer vision litera-
ture. Instead it is a more general notion describing
global structural invariants of an object under the
action of any group of transformations. To clarify
this notion is one of the goals of this paper.
Symmetry, as a useful geometric cue to 3-D informa-
tion, has been extensively discussed in psychological
vision literature (Marr, 1982; Plamer, 1999). Neverthe-
less, its contribution to computational vision so far has
been explored often through statistical methods, such as
the study of isotropic textures (e.g., for the 4th image of
Fig. 1) (Gibson, 1950; Witkin, 1988; Zabrodsky et al.,
1995; Mukherjee et al., 1995; Malik and Rosenholtz,
1997; Rosenholtz and Malik, 1997; Leung and Malik,
1997). It is the works of Garding (1992, 1993) and
Malik and Rosenholtz (1997) that have provided peo-
ple a wide range of efficient algorithms for recov-
ering the shape (i.e. the slant and tilt) of a textured
plane based on the assumption of isotropy (or weak
isotropy). These methods are mainly based on collect-
ing statistical characteristics (e.g., the distribution of
edge directions) from sample patches of the texture
and comparing them with those of adjacent patches
against the isotropic hypothesis. Information about the
On Symmetry and Multiple-View Geometry 243
surface shape is then often conveniently encoded in the
discrepancy or variation of these characteristics.
But symmetry is by nature a geometric property!
Although in many cases the result of symmetry indeed
causes certain statistical homogeneity (like the 4th im-
age of Fig. 1), there are reasons to believe that more
accurate and reliable 3-D geometric information can
be retrieved if we can directly exploit this property
through geometric means. For example, for the texture
shown in the 4th image of Fig. 1, shouldn’t we directly
exploit the fact that the tiling is invariant under certain
proper translations parallel to the plane? To a large ex-
tent, such a geometric approach is complementary to
extant statistical approaches: if statistical homogeneity
can be exploited for shape recovery, so can geometric
homogeneity, especially in cases where symmetry is the
underlying cause for such homogeneity. Of course, for
cases where statistical methods no longer apply (e.g.,
the 5th image of Fig. 1), geometric methods remain as
the only option. One may call this approach as structure
from symmetry.
We are by no means the first to notice that symme-
try, especially reflective symmetry, can be exploited
by geometric means for retrieving 3-D geometric in-
formation. Mitsumoto et al. (1992) studied how to re-
construct a 3-D object using the mirror image based
on planar symmetry, Vetter and Poggio (1994) proved
that for any reflective symmetric 3-D object one non-
accidental 2-D model view is sufficient for recognition,
Zabrodsky and Weinshall (1997) used bilateral symme-
try assumption to improve 3-D reconstruction from im-
age sequences, and Zabrodsky et al. (1995) provided
a good survey on studies of reflective symmetry and
rotational symmetry in computer vision at the time.
In 3-D object and pose recognition, Rothwell et al.
(1993) pointed out that the assumption of reflective
symmetry can be used in the construction of projective
invariants and is able to eliminate certain restrictions
on the corresponding points. Cham and Cipolla (1996)
built the correspondences of contours from reflective
symmetry. For translational symmetry, Schaffalitzky
and Zisserman (2000) used it to detect the vanish-
ing lines and points. Liu et al. (1995) analyzed the
error of obtaining 3-D invariants derived from trans-
lational symmetry. In addition to isometric symme-
try, Liebowitz and Zisserman (1998), Criminisi and
Zisserman (1999) and Criminisi and Zisserman (2000)
showed that other knowledge (e.g., length ratio, vanish-
ing line, etc.) in 3-D also allows accurate reconstruction
of structural metric and camera pose.
For the detection of symmetry from images, Marola
(1989), Kiryati and Gofman (1998) and Mukherjee
et al. (1995) presented efficient algorithms to find axes
of reflective symmetry in 2-D images, Sun and Sherrah
(1997) discussed reflective symmetry detection in 3-D
space, and Zabrodsky et al. (1995) introduced a symme-
try distance to classify reflective and rotational symme-
try in 2-D and 3-D spaces (with some related comments
given in Kanatani (1997)). Carlsson (1998) and Gool
et al. (1996) derived methods to find 3-D symmetry
from invariants in the 2-D projection. Liu and Colline
(1998) proposed a method to classify any images with
translational symmetry into the 7 Frieze groups and 17
wallpaper groups.
However, there is still a lack of formal and unified
analysis as well as efficient algorithms which would
allow people to easily make use of numerous and dif-
ferent types of symmetry that nature offers. Is there a
unified approach to study 3-D information encoded in a
2-D perspective image of an object that exhibits certain
symmetry? This paper will try to provide a definite an-
swer to this question. Our work differs from previous
results in at least the following three aspects:
1. We study symmetry under perspective projection
based on existing theory of multiple-view geom-
etry.
2
We claim that in order to fully understand
such 3-D information encoded in a single image,
one must understand geometry among multiple im-
ages.
2. In addition to recover 3-D structure of a symmetric
object from its image, we show that any type of
symmetry is naturally equipped with a canonical
(world) coordinate frame, from which the viewer’s
relative pose to the object can be recovered.
3. We give the necessary and sufficient conditions in
terms of the symmetry group of the object under
which the canonical pose can be uniquely recov-
ered, and we characterize the inherent ambiguity for
each fundamental type of symmetry. Thus, for the
first time, geometric group theory and (perspective)
multiple-view geometry are elegantly and tightly
integrated.
During the development, an important principle asso-
ciated with images of symmetric objects will be ex-
amined with care: One image of a symmetric object is
equivalent to multiple images. This principle is how-
ever not entirely correct since, as we will see, often
relationships among such “images” will not be the
244 Hong et al.
same as those among conventional images. It in fact
requires careful modifications to existing theories and
algorithms in multiple-view geometry if they are to be
correctly applied to images of symmetric objects.
2. Problem Formulation
Before we formulate the problem in a more abstract
form, let us take a look at a simple example: a planar
board with a symmetric pattern as shown in Fig. 2. It is
easy to see that, from any generic viewpoint, there are
at least four equivalent vantage points (with only the
rotational symmetry considered, for now) which give
rise to an identical image. The only question is which
corners in the image correspond to the ones on the
board. In this sense, these images are in fact different
from the original one. We may call these images as
“hidden.”
3
For instance, in Fig. 2, we labeled in bracket
corresponding corner numbers for such a hidden image.
Figure 2. Left: a checker board whose symmetry includes reflection along the x and y axes and rotation about o by 90
◦
. Right: an image taken
at location o
1
. Notice that the image would appear to be exactly the same if it were taken at o
2
instead. g
0
is the relative pose of the board we
perceive from the image on the right.
Figure 3. I
x
: Corner correspondence between the original image of the board and an “image” with the board reflected in the x-axis by 180
◦
;
I
y
: Corner correspondence between the original image of the board and an “image” with the board reflected in the y-axis by 180
◦
.
In addition to the rotational symmetry, another kind
of symmetry, the reflective symmetry, can give rise to
a not so conventional type of hidden images, as shown
in Fig. 3. Notice that, in the figure, the two “hidden
images” with the four corners labeled by numbers in
bracket cannot be an image of the same board from any
(physically viable) vantage point!
4
Nevertheless, as we
will see below, just like the rotational symmetry, this
type of hidden images also encodes rich 3-D geometric
information about the object.
There is yet another type of symmetry “hidden” in
a pattern like a checker board. As shown in Fig. 4 be-
low, for a pattern that repeats a fundamental region
indefinitely along one or more directions, the so-called
“infinite rapport,” one would obtain exactly “the same”
image had the images been taken at vantage points that
differ from each other by multiples nT of one basic
translation T . Although all images would appear to be
the same, features (e.g., points, lines) in these images
correspond to different physical features in the world.
On Symmetry and Multiple-View Geometry 245
Figure 4. The checker pattern is repeated indefinitely along the
x-axis. Images taken at o
1
, o
2
, and o
3
will be the same.
Therefore, for an image like the 4th one in Fig. 1, it in
fact may give rise to many (in theory, possibly infinitely
many) “hidden images.” There is clearly a reason to
believe that it is these (many) hidden images that give
away the geometry of the plane (e.g., tilt, slant) to the
viewer’s eyes.
It is then not hard to imagine that the combination of
the rotational, reflective and translational symmetries
will give rise to all sorts of symmetric objects in 2-D
or 3-D space, many of which could be rather compli-
cated. In our man-made world, symmetric objects are
ubiquitous, under the names of “ornament,” “mosaic,”
“pattern,” or “tiling,” etc. Fascination about symmetric
objects can be traced back to ancient Egyptians and
Greeks.
5
Nevertheless, a formal mathematical inquiry
to symmetry is known as Hilbert’s 18th problem, and
a complete answer to it was not found till 1910 by
Bieberbach (1910). While in the appendix we briefly
review results of a complete list for 2-D and 3-D sym-
metric structures and groups, this paper will focus on
how to combine this knowledge about symmetry with
multiple-view geometry so as to infer 3-D information
of a symmetric object from its image(s).
In order to explain why symmetry gives away accu-
rate information about structure and location of a sym-
metric 3-D object from a single 2-D perspective image,
we will need a mathematical framework within which
all types of symmetries (that we have mentioned or not
mentioned in the above examples) can be uniformly
taken into account. Only if we can do that, will the in-
troduction of symmetry into multiple-view geometry
become natural and convenient.
Definition 1 (Symmetric structure and its group action).
A set of points S ⊂ R
3
is called a symmetric structure if
there exists a non-trivial subgroup G of the Euclidean
group E(3) that acts on it. That is, for any element
g ∈ G,itdefines a bijection (i.e. a one-to-one, onto)
map from S to itself:
g ∈ G : S → S.
Sometimes we say that S has a symmetry group G.Or
G is a group of symmetries of S.
In particular, we have g(S) = g
−1
(S) = S for any
g ∈ G. Mathematically, symmetric structures and
groups are equivalent ways to capture symmetry: any
symmetric structure is invariant under the action of its
symmetry group; and any group (here as a subgroup
of E(3)) defines a class of (3-D) structures that are in-
variant under this group action (see Appendix A). Here
we emphasize that G is in general a subgroup of the
Euclidean group E(3) but not the special one SE(3).
This is because many symmetric structures that we are
going to consider are invariant under reflection which
is an element in O(3) but not SO(3).
6
For simplicity,
in this paper we consider G to be a discontinuous (or
discrete) group.
7
Using the homogeneous representation of E(3), any
element g = (R, T )inG can be represented as a 4 × 4
matrix of the form
g =
RT
01
∈ R
4×4
, (1)
where R ∈ R
3×3
is an orthogonal matrix (“R” for both
rotation and reflection) and T ∈ R
3
is a vector (“T ” for
translation). Note that in order to represent G in this
way, aworld coordinate frame must have been chosen.
It is conventional to choose the origin of the world
coordinate frame to be the center of rotation and its
axes to line up with the axes of rotation and reflection
or direction of translation. Often the canonical world
coordinate frame results in the simplest representation
of the symmetry (Ma et al., 2003).
Now suppose that an image of a symmetric structure
S is taken at a vantage point g
0
= (R
0
, T
0
) ∈ SE(3)—
denoting the pose of the structure relative to the viewer
or the camera. Here g
0
is assumed to be represented
with respect to the canonical world coordinate frame
for the symmetry. If so, we call g
0
the canonical pose.
As we will see shortly, the canonical pose g
0
from the
viewer to the object can be uniquely determined from
a single image as long as symmetry admitted by the
object (or the scene) is “rich” enough.