A
General Imaging Model and a Method for Finding its Parameters
Michael
D.
Grossberg
and
Shree
K.
Nayar
Department
of
Computer Science, Columbia University
New York, New York
10027
{mdog, nayar} @cs.columbia.edu
Abstract
Linear perspective projection has served as the dominant
imaging model in computer vision. Recent developments in
image sensing make the perspective model highly restric-
tive. This paper presents a general imaging model that can
be used to represent an arbitrary imaging system. It is ob-
served that all imaging systems perform a mapping from in-
coming scene rays to photo-sensitive elements
on
the im-
age detector: This mapping can be conveniently described
using a
set
of virtual sensing elements called
raxels.
Rax-
els include geometric, radiometric and optical properties.
We present a novel calibration method that uses structured
light patterns to extract the raxel parameters of an arbi-
trary imaging system. Experimental results for perspective
as
well
as non-perspective imaging systems are included.
1
Introduction
Since the Renaissance, artists have been fascinated by the
visual manifestations
of
perspective projection. Geometers
have studied the properties
of
the
pinhole imaging model
and derived
a
large suite
of
projective invariants that provide
insights into the relation between
a
scene and its perspective
image. The field of optics has developed high quality imag-
ing lenses that closely adhere
to
the perspective model.
To-
day, perspective projection serves as the dominant imaging
model in computer vision and computer graphics.
Despite its great relevance, there are several reasons that
make the perspective model
far
too restrictive. In recent
years, the notion of a “vision sensor” has taken on
a
much
broader meaning.
A
variety of devices have been devel-
oped that sample the light field
[81
or the plenoptic func-
tion
[11
associated with
a
scene in interesting and useful
non-perspective ways. Figure
1
shows some examples of
such imaging systems that are widely used today. Figure
l(a) shows
a
catadioptric sensor that uses
a
combination of
lenses and mirrors. Even when such
a
sensor has a single
effective viewpoint
[211,
its projection modo1 can include
a
variety of non-perspective distortions (barrel, pincushion,
or more complex)[21. More interesting is the fact that cer-
tain applications (see
[221
for example) require the system
to
not have
a
single viewpoint but rather a locus
of
viewpoints
(catacaustic
[41).
Similarly, wide-angle lens systems
[201
curved mirror
/
camera
camera cluster
Q,
8
(c)
meniscus lens
E>
compound camera
lenses
3
Figure
1
:
Examples
of
non-perspective imaging systems:
(a)
a
catadioptric system,
(b)
a dioptric wide-angle system, (c)
an
imag-
ing
system
made
of
a
camera cluster,
and
(d)
a
compound camera
made
of
individual
sensing elements, each including
a
receptor
and
a lens.
In
all
of
these cases, the imaging model
of
the system devi-
ates from perspective projection.
like the one shown in Figure l(b), include severe projective
distortions and often have a locus of viewpoints (called
a
di-
acaustic). Recently, clusters of cameras, like the one shown
in Figure l(c), have become popular
[171[241.
It
is clear that
such
a
system includes multiple viewpoints
or
loci of
view-
points, each one associated with one of the cameras in the
cluster. Finally, in the case
of
insects, nature has evolved
eyes that have compound lenses
[71,
[61,
such
as
the one
shown in Figure I(d). These eyes are composed of thou-
sands
of
“ommatidia”, each ommatidium including
a
recep-
tor and lens. It is only a matter of time before we see solid-
state cameras with flexible imaging surfaces that include
a
large number
of
such ommatidia.
In this paper we address two questions that we believe are
fundamental to imaging:
0
Is
there an imaging model that
is
general enough to
represent any arbitrary imaging system? Note that we
0-7695-1143-0/01
$10.00
0
2001
IEEE
108
are not placing any restrictions on the properties of
the imaging system. It could be perspective or non-
perspective.
0
Given an unknown imaging system (a black box),
is
there
a
simple calibration method that can compute
the parameters of the imaging model? Note that for
the types
of
systems we wish to subsume in our imag-
ing model, conventional camera calibration techniques
will not suffice.
Such a general imaging model must be flexible enough to
cover the wide range of devices that are of interest to
us.
Yet, it should be specific enough in terms of its parameters
that it is useful in practice. Our approach is to exploit the
fact that
all
imaging systems perform
a
mapping from in-
coming scene rays to photo-sensitive elements on the image
detector. This mapping can be conveniently described by
a
ray
surjiuce
which is
a
surface in three-dimensional space
from which the rays are measured in various directions. The
smallest element of our imaging model is
a
virtual photo-
sensitive element that measures light in essentially a single
direction. We refer to these virtual elements
as
ray pixels,
or
ruxels.
It turns out that
a
convenient way
to
represent
the ray surface on which the raxels reside is the
caustic
of
the imaging system. In addition to its geometric parame-
ters, each raxel has its own radiometric response function
and local point spread function.
After describing the general imaging model and its proper-
ties, we present
a
simple method for finding the parameters
of the model for any arbitrary imaging system: It is im-
portant to note that, given the non-perspective nature of
a
general device, conventional calibration methods based on
known scene points
[251
or self-calibration techniques that
use unknown scene points
[51,
[lo],
[151,
cannot be directly
applied. Since we are interested in the mapping from rays to
image points, we need a
ruy-bused
calibration method. We
describe a simple and effective ray-based approach that uses
structured light patterns. This method allows a user to ob-
tain the geometric, radiometric, and optical parameters of an
arbitrarily complex imaging system in a matter of minutes.
2
General
Imaging
Model: Geometry
We will consider the imaging system shown in Figure
2
when formulating
our
mathematical model of the imaging
sensors mentioned in section
1.
The system includes
a
de-
tector with a large number of photo-sensitive elements (pix-
els). The detector could be an electronic chip, film,
or
any
other light sensitive device. The technology used to im-
plement it is not important. The imaging optics typically
include several elements. Even
a
relatively simple optical
component has about five individual lenses within it. In our
arbitrary system, there may be additional optical elements
such as mirrors, prisms, or beam-slitters. In fact, the system
109
could be comprised of multiple individual imaging systems,
each with its own imaging optics and image detector.
Irrespective of its specific design, the purpose of an imag-
ing system is to map incoming rays of light from the scene
onto pixels on the detector. Each pixel collects light energy
from a bundle of closely packed rays in any system that has
a non-zero aperture size. However, the bundle can be rep-
resented by a single chief (or principle) ray when studying
the geometric properties of the imaging system.
As
shown
in Figure
2,
the system maps the ray
Pi
to the pixel
i.
The
path that incoming ray traverses to the pixel can be arbitrar-
ily complex.
‘
photosensitive elements
’
arbitrary imaging system
-
captured light rays
I
Figure
2:
An imaging system directs incoming
light
rays
to
its
photo-sensitive elements (pixels). Each pixel collects light from a
bundle
of
rays
that
pass through the
finite
aperture
of
the imaging
system. However,
we
will
assume there is
a
correspondence
be-
tween each individual detector element
i
and
a
specific ray
of
light
Pi,
entering the system.
If the imaging system is perspective, all the incoming light
rays are projected directly onto to the detector plane through
a single point, namely, the effective pinhole of the perspec-
tive system. This is not true in an arbitrary system. For
instance, it is clear from Figure
2
that the captured rays do
not meet at
a
single effective viewpoint. The goal of this
section is to present a geometrical model that can represent
such imaging systems.
2.1
Raxels
It is convenient to represent the mapping from scene rays
to pixels in a form that easily lends itself to manipulation
and analysis. We can replace our physical pixels with an ab-
stract mathematical equivalent we refer to as a ray pixel or
a “raxel”.
A
raxel is
a
virtual photo-sensitive element that
measures the light energy of
a
compact bundle of rays which
can be represented as a single principle incoming ray.
A
similar abstraction, the pencigraph, was proposed by Mann
[
161
for the perspective case. The abstract optical model of
our
virtual raxel is shown in Figure
3.
Each raxel includes a
pixel that measures light energy and imaging optics (a lens)
that collects the bundle of rays around an incoming ray. In
this section, we will focus on the geometric properties
(lo-
cations and orientations) of raxels. However, each raxel can
posses its own radiometric (brightness and wavelength) re-
sponse as well as optical (point spread) properties. These
non-geometric properties will be discussed in subsequent
sections.
Figure
3:
(a)
A
raxel is
a
virtual
replacement
for
a real photo-
sensitive element.
It
may be placed along the line
of
a principle ray
of light entering the imaging system.
In
addition to location and
orientation,
a
raxel
may
have radiometric and optical parameters.
(b) The notation for a raxel used
in
this paper.
2.2
Plenoptic Function
What
does an imaging system see? The input to the sys-
tem is the plenoptic function
[l].
The plenoptic function
@(p,
q,
t,
A)
gives the intensity of light at each point
p
in
space, from direction
q,
at an instant of time
t
and wave-
length
A.
We specify position by
(px,
py,
pz)
and direction
by two angles
(q$,
qe).
Still images represent an integration
of light energy over
a
short time period, given by the ef-
fective shutter speed. Further, each photo-sensitive element
will average the plenoptic function across a range of wave-
lengths. Thus, we set aside time and wavelength by consid-
ering monochromatic still imaging. We will only consider
the plenoptic function as a function of position and direc-
tion:
@(p,
9).
We may view a raxel as a delta function'
d,,po,qo
over
p,
q
space,
as
it measures the value of plenoptic function
@(p,
q)
at
(PO,
90).
Hence the parameters for a raxel are just posi-
tion
po
and direction
90.
2.3
Pencils
of
Rays and Ray Surfaces
From
where
does the system (with its raxels) see the plenop-
tic function? Each point in the image corresponds to a ray.
Thus, the set of positions and directions determined by the
set of rays is the part of the domain
of
the plenoptic function
relevant to our system.2
'If
the raxel has
a
non-linear radiometric response then the response
2The related problem
of
representing the positions and directions cor-
must be linearized
for
the raxel to be
a
delta function.
responding to
a
light source was explored in
[
131.
The most general imaging system is described by a list
of these rays.
For
clarity, we assume
our
image is two-
dimen~ional.~ An image point is specified by
(x,y).
A
scene point
p
imaged at
(x,
y)
can be anywhere along the
corresponding ray.
To
specify the point in space we define a
parameter
r
along the ray. In the perspective case,
r
may be
chosen as scene depth.
A point
p(x,
y,
r)
imaged at
(2,
y)
at depth
r
is imaged
along a ray in the direction
q(x,
y,
r).
Thus, we see the
plenoptic function only from those points in the range of
p
and
q.
We may place a raxel anywhere along a ray.4 It will
be more convenient for representing the model to arrange
the raxels on a surface we call a
ray
surface.
For
example,
consider a sphere enclosing
our
imaging system,
as
shown
in Figure
4.
For
each photo-sensitive element
i
there is some
point
pi
on the sphere that received a ray in the direction
qi.
Thus
we can place our raxels on the sphere by assigning
them the positions and directions
(pi,
si).
It is important to
note that there could be several rays that enter the sphere at
the same point but with different directions (see
91,
q2
and
q3
in Figure
4).
Thus the direction
q
is not, in general, a
function of
p.
tf
Figure
4:
An
imaging system may be modeled as a set of raxels
on
a
sphere surrounding the imaging system. Each raxel
i
has
a
position
pi
on the sphere,
and
an orientation
q;
aligned
with
an
incoming ray. Multiple raxels may be located at the same point
(p1
=
p~
=
p3),
but
have different directions.
The choice of intersecting the incoming rays with a refer-
3Many
of
the arrays of photo-sensitive elements in the imaging devices
described in section
1
are
one
or
two-dimensional. Multi-camera systems
can be represented by two-dimensional arrays parameterized by
an
extra
parameter.
41ntensities usually do not change much along
a
ray
(particularly when
the medium is air) provided the the displacement is small with respect to
the total length of the ray.
110
ence sphere is arbitrary.
In
191
and [141, it was suggested
that the plenoptic function could
also
be restricted to
a
plane.
The important thing is to choose some reference surface
so
that each incoming ray intersects this surface at only one
point. If the incoming rays are parameterized by image co-
ordinates
(x,
y),
each ray will intersect
a
reference surface
at one point
p(x,
y).
We can write the ray surface
as
a
func-
tion of
(2,
y)
as:
We can express the position of
a
point along the ray
as
p(x,
y,
r)
=
p(z,
y)+
rq(x,
y).
This allows
us
to express
the relevant subset of the domain of the plenoptic function
as
the range of
In the case of an unknown imaging system
we
may mea-
sure
s(x,
y)
along some ray surface. In the case of
a
known
imaging system we are able to compute
s(z,
y)
a
priori.
Us-
ing equation 2 we may express one ray surface in terms of
another ray surface such
as
a
sphere or
a
plane,
3
Caustics
While convenient, the choices
of
a
plane or sphere
as
the
reference surface come with their drawbacks. The direction
q(z,
y)
of
a
raxel need not have any relation to the position
p(z,
y)
on
a general reference surface. There may be points
on the surface that do not have incoming rays. At other
points, several rays may pass through the same point.
For many imaging systems, there is a distinguished ray sur-
face which is defined
solely
by the geometry of the rays. In
Figure
5,
we see the envelope of the incoming rays form
a
curve. On such surfaces, direction
q
is
really
a
function of
position
p,
and the incomming rays are tangent to the sur-
face. This is
a
special case of
a
ray surface called
a
caustic.
We argue that caustics are the logical place
to
locate our
raxels.
Caustics have
a
number of different characterizations. The
caustic often forms a surface to which
all
incoming rays are
tangent. This does not happen in the perspective case, where
the caustic is
a
point. Caustics can
also
be viewed
as
geo-
metrical entities where there occurs
a
singularity (bunching)
of the light rays. As
a
result,
a
caustic formed by rays of illu-
mination generates
a
very bright region when
it
intersects
a
~urface.~ In our context of imaging systems, a caustic is the
locus of points where incoming rays most converge. Points
on the caustic can therefore be viewed as the ‘locus of view-
points” of the system. It is thus
a
natural place to locate the
raxels
of
our abstract imaging model.
’For
example, when light refracts through shallow water
of
a
pool,
bnght curves can be seen where the caustics intersect the bottom
Figure
5:
The caustic is a good candidate for the
ray
surface
of
an
imaging system as
it
is closely related to the geometry of the
incoming rays; the incoming
ray
directions are tangent to the caus-
tic.
3.1
Definition
of
a Caustic Surface
In section 2.3, we described the set of points and direc-
tions that can be detected by the imaging system. This set
is described by the range of
C(x,
y,
r)
from equation (2).
The position component functions are
9
=
px(z,y,~),
Y
=
py(x,y,r),
and
2
=
pz(z,y,r).
The map from
(x,
y,
r)
to
(A-,
Y,
2)
can be viewed
as
a change of coor-
dinates. The
caustic
surface
is defined
as
the locus
of
points
in
A-,
Y,
2
space where this change of coordinates is singu-
1
ar.
3.2
Computing Caustics
The ray-to-image mapping
of
an imaging system may be
obtained in two ways. One way, is to derive the mapping or
compute
it
numerically from optical components or parame-
ters of an imaging system which is known
a-priori.
Alterna-
tively,
a
calibration method can be used to obtain
a
discrete
form of the mapping (see section
5).
In either case, our goal
is to compute the caustic surface from the given mapping.
When this mapping is known in closed form, analytic meth-
ods can be used to derive the caustic surface
[41.
When
a
discrete approximation is given,
a
host
of
numerical meth-
ods [121,
[IS],
[261, may be used. The method we use com-
putes the caustic by finding
all
the points where the change
in
coordinates described above is singular
[21,
[31.
Equation (2) expresses
C
in terms of
a
known or measured
ray surface
s(x,
y).
The caustic is defined
as
the singular-
ities
in
the change from
(x,
y,
T)
coordinates to
(S,
Y,
2)
coordinates given by
p.
Singularities arise at those points
(A‘,
l’,
2)
where the Jacobian matrix
J
of the transforma-
tion does not have full rank. We can find these points by
111
computing the determinant of the Jacobian,
and setting it to zero. Since this is quadratic in
T
we can
solve for
r
explicitly in terms of
p,
q,
and their first deriva-
tives with respect to
z
and
y.
Plugging this back into
C
gives us an expression for the caustic ray surface parame-
terized by
(x,
y)
as
in equation
(1).
If the optical system
has translational or rotational symmetry then we may only
consider one parameter, for example
2,
in the image plane.
In this case Jacobian becomes linear in
T
and the solution
simplifies to:
(4)
3.3
Field
of
View
Some parameters used to specify the general perspective
camera model are derived from the ray surface representa-
tion in our general imaging model. Other parameters de-
pend on the perspective assumption and are ill defined in
our model. For example, field of view presents an ambigu-
ity, since in the non-perspective case the rays may
no
longer
form
a
simple rectangular cone. One candidate for
a
field
of view is the range of
q(x,
y)
over the image. This is the
same
as
the Gauss map
[l
11.
The Gauss map is
a
good ap-
proximation
to
the field of view when the scene points are
distant relative to the size
of
the imaging system.
Other geometric parameters of
a
perspective imaging sys-
tem, such
as
aspect ratio, and spherical aberration
[251,
may
no longer be separable in the general imaging model from
the parameterized ray surface.
4
Non-Geometric
Raxel
Parameters
Each raxel is treated
as
a
very narrow field
of
view, perspec-
tive imaging system. Many of the conventional parameters
associated with
a
perspective or near-perspective systems
may be attributed
to
a
raxel.
4.1
Local Focal Length and Point Spread
An arbitrary imaging system cannot be expected to have
a
single global focal length. However, each raxel may be
modeled
to
have its own focal length. We can compute each
raxel's focal length by measuring its point spread function
for several depths.
A
flexible approach models the point
spread
as
an elliptical Gaussian. Each ellipse has
a
major
fb,
as
well
as
a focal orientation
1c,
in the image.
4.2
Radiometry
The radiometric response,
g,
is expected to be smooth
and monotonic and can be modeled a polynomial
[191.
If
one can compute the radiometric response function of each
raxel, one can linearize the response with respect to scene
radiance, assuming the response is invertible.
We model the relation between raxel irradiance
E
as
scene
radiance
L
times
a
spatially varying attenuation factor,
h(z,
y),
corresponding to the image point
(2,
y).
We call
h(x,
y)
the
full-offfunction.
This factor takes into account
the finite size of the aperture and vignetting effects which
are linear in
L.
4.3
Complete Imaging Model
The general imaging model consists of
a
set of raxels pa-
rameterized by
x
and
y
in pixel coordinates. The parameters
associated with these raxels (see Figure
6),
are
(a)
position
and direction, that describe the ray surface of the caustic,
(b) major and minor focal lengths
as
well
as
a
focal orien-
tation, (c)
a
radiometric response function, and (d)
a
fall-off
constant for each pixel.
Figure
6:
Each raxel has the above parameters of image coor-
dinates, position and direction
in
space, major and minor focal
lengths,
focal
orientation, radiometric response,
and
fall-off fac-
tor. These parameters are measured
with
respect to a coordinate
frame
fixed
to
the imaging
system.
Camera parameters are separated into external and internal
parameters. The external parameters are specified by
a
co-
ordinate frame. The internal parameters (Figure
6)
are spec-
ified with respect to that coordinate frame. In particular, for
each raxel
i,
the parameters
pi,
qi
are measured with respect
to
a
single coordinate frame fixed to the imaging system. If
the system is rotated or translated, these parameters will not
change
but
the coordinate frame will.
In the case of perspective projection, the essential
[51
or
fundamental
[
101
matrix provides the relationship between
points in one image and lines in another image (of the same
scene). In the general imaging model, this correspondence
need no longer be projective. Nevertheless,
a
point in one
image still corresponds to
a
curve in the other, which may
be computed.
5
Finding
the
Model
Parameters
and minor axis. The major axis makes an angle
1c,
with the
x-axis in the image.6 Each raxel has two focal lengths,
fa,
hThe angle
$
is only defined
if
the
major
and minor
axis
have different
In
section
we
described
how
to
compute
Our
model
for
a
known optical system. In contrast, our goal in this section
lengths.
112