scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

A general imaging model and a method for finding its parameters

07 Jul 2001-Vol. 2, pp 108-115
TL;DR: A novel calibration method is presented that uses structured light patterns to extract the raxel parameters of an arbitrary imaging system and experimental results for perspective as well as ion-perspective imaging systems are included.
Abstract: Linear perspective projection has served as the dominant imaging model in computer vision. Recent developments in image sensing make the perspective model highly restrictive. This paper presents a general imaging model that can be used to represent an arbitrary imaging system. It is observed that all imaging systems perform a mapping from incoming scene rays to photo-sensitive elements on the image detector. This mapping can be conveniently described using a set of virtual sensing elements called raxels. Raxels include geometric, radiometric and optical properties. We present a novel calibration method that uses structured light patterns to extract the raxel parameters of an arbitrary imaging system. Experimental results for perspective as well as ion-perspective imaging systems are included.

Summary (2 min read)

1 Introduction

  • After describing the general imaging model and its properties, the authors present a simple method for finding the parameters of the model for any arbitrary imaging system:.
  • It is important to note that, given the non-perspective nature of a general device, conventional calibration methods based on known scene points [251 or self-calibration techniques that use unknown scene points [51, [lo] , [151, cannot be directly applied.
  • Since the authors are interested in the mapping from rays to image points, they need a ruy-bused calibration method.
  • The authors describe a simple and effective ray-based approach that uses structured light patterns.
  • This method allows a user to obtain the geometric, radiometric, and optical parameters of an arbitrarily complex imaging system in a matter of minutes.

2 General Imaging Model: Geometry

  • If the imaging system is perspective, all the incoming light rays are projected directly onto to the detector plane through a single point, namely, the effective pinhole of the perspective system.
  • The goal of this section is to present a geometrical model that can represent such imaging systems.

2.1 Raxels

  • Each raxel includes a pixel that measures light energy and imaging optics (a lens) that collects the bundle of rays around an incoming ray.
  • The authors will focus on the geometric properties (locations and orientations) of raxels.
  • Each raxel can posses its own radiometric (brightness and wavelength) response as well as optical (point spread) properties.
  • These non-geometric properties will be discussed in subsequent sections.

Figure 3: (a)

  • It may be placed along the line of a principle ray of light entering the imaging system.
  • In addition to location and orientation, a raxel may have radiometric and optical parameters.
  • The notation for a raxel used in this paper.

t f

  • Multiple raxels may be located at the same point (p1 = p~ = p3), but have different directions.
  • The choice of intersecting the incoming rays with a refer-3Many of the arrays of photo-sensitive elements in the imaging devices described in section 1 are one or two-dimensional.
  • 41ntensities usually do not change much along a ray (particularly when the medium is air) provided the the displacement is small with respect to the total length of the ray.
  • In 191 and [141, it was suggested that the plenoptic function could also be restricted to a plane.
  • The important thing is to choose some reference surface so that each incoming ray intersects this surface at only one point.

3 Caustics

  • 'For example, when light refracts through shallow water of a pool, bnght curves can be seen where the caustics intersect the bottom Figure 5 :.
  • The caustic is a good candidate for the ray surface of an imaging system as it is closely related to the geometry of the incoming rays; the incoming ray directions are tangent to the caustic.

4.1 Local Focal Length and Point Spread

  • An arbitrary imaging system cannot be expected to have a single global focal length.
  • Each raxel may be modeled to have its own focal length.
  • The authors can compute each raxel's focal length by measuring its point spread function for several depths.
  • A flexible approach models the point spread as an elliptical Gaussian.

4.3 Complete Imaging Model

  • In the case of perspective projection, the essential [51 or fundamental [ 101 matrix provides the relationship between points in one image and lines in another image (of the same scene).
  • In the general imaging model, this correspondence need no longer be projective.

5 Finding the Model Parameters

  • The major axis makes an angle 1c, with the x-axis in the image.
  • Each raxel has two focal lengths, f a , hThe angle $ is only defined if the major and minor axis have different.
  • In section the authors described how to compute The authors model for a known optical system.
  • In contrast, their goal in this section lengths.

Figure 7: (a)

  • If these positions are known, the direction of the ray qf may be determined for each pixel.
  • Now, the authors construct a calibration environment where the geometric and radiometric parameters can be efficiently estimated.
  • If a display has N locations, the authors can make each point distinct in logN images using simple grey coding or bit coding.
  • The authors may then compute the fall-off function across all the points.
  • The authors compute both the radiometric response function and the falloff from seventeen uniform brightness levels.

5.1 Experimental Apparatus

  • The laptop was oriented so as to give the maximum screen resolution along the axis of symmetry.
  • Figure 8 (b) shows a sample binary pattern as seen from the parabolic catadioptric system.
  • The perspective imaging system, consisting of just the camera itself, can be seen in Figure 1 l(a).

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

A
General Imaging Model and a Method for Finding its Parameters
Michael
D.
Grossberg
and
Shree
K.
Nayar
Department
of
Computer Science, Columbia University
New York, New York
10027
{mdog, nayar} @cs.columbia.edu
Abstract
Linear perspective projection has served as the dominant
imaging model in computer vision. Recent developments in
image sensing make the perspective model highly restric-
tive. This paper presents a general imaging model that can
be used to represent an arbitrary imaging system. It is ob-
served that all imaging systems perform a mapping from in-
coming scene rays to photo-sensitive elements
on
the im-
age detector: This mapping can be conveniently described
using a
set
of virtual sensing elements called
raxels.
Rax-
els include geometric, radiometric and optical properties.
We present a novel calibration method that uses structured
light patterns to extract the raxel parameters of an arbi-
trary imaging system. Experimental results for perspective
as
well
as non-perspective imaging systems are included.
1
Introduction
Since the Renaissance, artists have been fascinated by the
visual manifestations
of
perspective projection. Geometers
have studied the properties
of
the
pinhole imaging model
and derived
a
large suite
of
projective invariants that provide
insights into the relation between
a
scene and its perspective
image. The field of optics has developed high quality imag-
ing lenses that closely adhere
to
the perspective model.
To-
day, perspective projection serves as the dominant imaging
model in computer vision and computer graphics.
Despite its great relevance, there are several reasons that
make the perspective model
far
too restrictive. In recent
years, the notion of a “vision sensor” has taken on
a
much
broader meaning.
A
variety of devices have been devel-
oped that sample the light field
[81
or the plenoptic func-
tion
[11
associated with
a
scene in interesting and useful
non-perspective ways. Figure
1
shows some examples of
such imaging systems that are widely used today. Figure
l(a) shows
a
catadioptric sensor that uses
a
combination of
lenses and mirrors. Even when such
a
sensor has a single
effective viewpoint
[211,
its projection modo1 can include
a
variety of non-perspective distortions (barrel, pincushion,
or more complex)[21. More interesting is the fact that cer-
tain applications (see
[221
for example) require the system
to
not have
a
single viewpoint but rather a locus
of
viewpoints
(catacaustic
[41).
Similarly, wide-angle lens systems
[201
curved mirror
/
camera
camera cluster
Q,
8
(c)
meniscus lens
E>
compound camera
lenses
3
Figure
1
:
Examples
of
non-perspective imaging systems:
(a)
a
catadioptric system,
(b)
a dioptric wide-angle system, (c)
an
imag-
ing
system
made
of
a
camera cluster,
and
(d)
a
compound camera
made
of
individual
sensing elements, each including
a
receptor
and
a lens.
In
all
of
these cases, the imaging model
of
the system devi-
ates from perspective projection.
like the one shown in Figure l(b), include severe projective
distortions and often have a locus of viewpoints (called
a
di-
acaustic). Recently, clusters of cameras, like the one shown
in Figure l(c), have become popular
[171[241.
It
is clear that
such
a
system includes multiple viewpoints
or
loci of
view-
points, each one associated with one of the cameras in the
cluster. Finally, in the case
of
insects, nature has evolved
eyes that have compound lenses
[71,
[61,
such
as
the one
shown in Figure I(d). These eyes are composed of thou-
sands
of
“ommatidia”, each ommatidium including
a
recep-
tor and lens. It is only a matter of time before we see solid-
state cameras with flexible imaging surfaces that include
a
large number
of
such ommatidia.
In this paper we address two questions that we believe are
fundamental to imaging:
0
Is
there an imaging model that
is
general enough to
represent any arbitrary imaging system? Note that we
0-7695-1143-0/01
$10.00
0
2001
IEEE
108

are not placing any restrictions on the properties of
the imaging system. It could be perspective or non-
perspective.
0
Given an unknown imaging system (a black box),
is
there
a
simple calibration method that can compute
the parameters of the imaging model? Note that for
the types
of
systems we wish to subsume in our imag-
ing model, conventional camera calibration techniques
will not suffice.
Such a general imaging model must be flexible enough to
cover the wide range of devices that are of interest to
us.
Yet, it should be specific enough in terms of its parameters
that it is useful in practice. Our approach is to exploit the
fact that
all
imaging systems perform
a
mapping from in-
coming scene rays to photo-sensitive elements on the image
detector. This mapping can be conveniently described by
a
ray
surjiuce
which is
a
surface in three-dimensional space
from which the rays are measured in various directions. The
smallest element of our imaging model is
a
virtual photo-
sensitive element that measures light in essentially a single
direction. We refer to these virtual elements
as
ray pixels,
or
ruxels.
It turns out that
a
convenient way
to
represent
the ray surface on which the raxels reside is the
caustic
of
the imaging system. In addition to its geometric parame-
ters, each raxel has its own radiometric response function
and local point spread function.
After describing the general imaging model and its proper-
ties, we present
a
simple method for finding the parameters
of the model for any arbitrary imaging system: It is im-
portant to note that, given the non-perspective nature of
a
general device, conventional calibration methods based on
known scene points
[251
or self-calibration techniques that
use unknown scene points
[51,
[lo],
[151,
cannot be directly
applied. Since we are interested in the mapping from rays to
image points, we need a
ruy-bused
calibration method. We
describe a simple and effective ray-based approach that uses
structured light patterns. This method allows a user to ob-
tain the geometric, radiometric, and optical parameters of an
arbitrarily complex imaging system in a matter of minutes.
2
General
Imaging
Model: Geometry
We will consider the imaging system shown in Figure
2
when formulating
our
mathematical model of the imaging
sensors mentioned in section
1.
The system includes
a
de-
tector with a large number of photo-sensitive elements (pix-
els). The detector could be an electronic chip, film,
or
any
other light sensitive device. The technology used to im-
plement it is not important. The imaging optics typically
include several elements. Even
a
relatively simple optical
component has about five individual lenses within it. In our
arbitrary system, there may be additional optical elements
such as mirrors, prisms, or beam-slitters. In fact, the system
109
could be comprised of multiple individual imaging systems,
each with its own imaging optics and image detector.
Irrespective of its specific design, the purpose of an imag-
ing system is to map incoming rays of light from the scene
onto pixels on the detector. Each pixel collects light energy
from a bundle of closely packed rays in any system that has
a non-zero aperture size. However, the bundle can be rep-
resented by a single chief (or principle) ray when studying
the geometric properties of the imaging system.
As
shown
in Figure
2,
the system maps the ray
Pi
to the pixel
i.
The
path that incoming ray traverses to the pixel can be arbitrar-
ily complex.
photosensitive elements
arbitrary imaging system
-
captured light rays
I
Figure
2:
An imaging system directs incoming
light
rays
to
its
photo-sensitive elements (pixels). Each pixel collects light from a
bundle
of
rays
that
pass through the
finite
aperture
of
the imaging
system. However,
we
will
assume there is
a
correspondence
be-
tween each individual detector element
i
and
a
specific ray
of
light
Pi,
entering the system.
If the imaging system is perspective, all the incoming light
rays are projected directly onto to the detector plane through
a single point, namely, the effective pinhole of the perspec-
tive system. This is not true in an arbitrary system. For
instance, it is clear from Figure
2
that the captured rays do
not meet at
a
single effective viewpoint. The goal of this
section is to present a geometrical model that can represent
such imaging systems.
2.1
Raxels
It is convenient to represent the mapping from scene rays
to pixels in a form that easily lends itself to manipulation
and analysis. We can replace our physical pixels with an ab-
stract mathematical equivalent we refer to as a ray pixel or
a “raxel”.
A
raxel is
a
virtual photo-sensitive element that
measures the light energy of
a
compact bundle of rays which
can be represented as a single principle incoming ray.
A
similar abstraction, the pencigraph, was proposed by Mann

[
161
for the perspective case. The abstract optical model of
our
virtual raxel is shown in Figure
3.
Each raxel includes a
pixel that measures light energy and imaging optics (a lens)
that collects the bundle of rays around an incoming ray. In
this section, we will focus on the geometric properties
(lo-
cations and orientations) of raxels. However, each raxel can
posses its own radiometric (brightness and wavelength) re-
sponse as well as optical (point spread) properties. These
non-geometric properties will be discussed in subsequent
sections.
Figure
3:
(a)
A
raxel is
a
virtual
replacement
for
a real photo-
sensitive element.
It
may be placed along the line
of
a principle ray
of light entering the imaging system.
In
addition to location and
orientation,
a
raxel
may
have radiometric and optical parameters.
(b) The notation for a raxel used
in
this paper.
2.2
Plenoptic Function
What
does an imaging system see? The input to the sys-
tem is the plenoptic function
[l].
The plenoptic function
@(p,
q,
t,
A)
gives the intensity of light at each point
p
in
space, from direction
q,
at an instant of time
t
and wave-
length
A.
We specify position by
(px,
py,
pz)
and direction
by two angles
(q$,
qe).
Still images represent an integration
of light energy over
a
short time period, given by the ef-
fective shutter speed. Further, each photo-sensitive element
will average the plenoptic function across a range of wave-
lengths. Thus, we set aside time and wavelength by consid-
ering monochromatic still imaging. We will only consider
the plenoptic function as a function of position and direc-
tion:
@(p,
9).
We may view a raxel as a delta function'
d,,po,qo
over
p,
q
space,
as
it measures the value of plenoptic function
@(p,
q)
at
(PO,
90).
Hence the parameters for a raxel are just posi-
tion
po
and direction
90.
2.3
Pencils
of
Rays and Ray Surfaces
From
where
does the system (with its raxels) see the plenop-
tic function? Each point in the image corresponds to a ray.
Thus, the set of positions and directions determined by the
set of rays is the part of the domain
of
the plenoptic function
relevant to our system.2
'If
the raxel has
a
non-linear radiometric response then the response
2The related problem
of
representing the positions and directions cor-
must be linearized
for
the raxel to be
a
delta function.
responding to
a
light source was explored in
[
131.
The most general imaging system is described by a list
of these rays.
For
clarity, we assume
our
image is two-
dimen~ional.~ An image point is specified by
(x,y).
A
scene point
p
imaged at
(x,
y)
can be anywhere along the
corresponding ray.
To
specify the point in space we define a
parameter
r
along the ray. In the perspective case,
r
may be
chosen as scene depth.
A point
p(x,
y,
r)
imaged at
(2,
y)
at depth
r
is imaged
along a ray in the direction
q(x,
y,
r).
Thus, we see the
plenoptic function only from those points in the range of
p
and
q.
We may place a raxel anywhere along a ray.4 It will
be more convenient for representing the model to arrange
the raxels on a surface we call a
ray
surface.
For
example,
consider a sphere enclosing
our
imaging system,
as
shown
in Figure
4.
For
each photo-sensitive element
i
there is some
point
pi
on the sphere that received a ray in the direction
qi.
Thus
we can place our raxels on the sphere by assigning
them the positions and directions
(pi,
si).
It is important to
note that there could be several rays that enter the sphere at
the same point but with different directions (see
91,
q2
and
q3
in Figure
4).
Thus the direction
q
is not, in general, a
function of
p.
tf
Figure
4:
An
imaging system may be modeled as a set of raxels
on
a
sphere surrounding the imaging system. Each raxel
i
has
a
position
pi
on the sphere,
and
an orientation
q;
aligned
with
an
incoming ray. Multiple raxels may be located at the same point
(p1
=
p~
=
p3),
but
have different directions.
The choice of intersecting the incoming rays with a refer-
3Many
of
the arrays of photo-sensitive elements in the imaging devices
described in section
1
are
one
or
two-dimensional. Multi-camera systems
can be represented by two-dimensional arrays parameterized by
an
extra
parameter.
41ntensities usually do not change much along
a
ray
(particularly when
the medium is air) provided the the displacement is small with respect to
the total length of the ray.
110

ence sphere is arbitrary.
In
191
and [141, it was suggested
that the plenoptic function could
also
be restricted to
a
plane.
The important thing is to choose some reference surface
so
that each incoming ray intersects this surface at only one
point. If the incoming rays are parameterized by image co-
ordinates
(x,
y),
each ray will intersect
a
reference surface
at one point
p(x,
y).
We can write the ray surface
as
a
func-
tion of
(2,
y)
as:
We can express the position of
a
point along the ray
as
p(x,
y,
r)
=
p(z,
y)+
rq(x,
y).
This allows
us
to express
the relevant subset of the domain of the plenoptic function
as
the range of
In the case of an unknown imaging system
we
may mea-
sure
s(x,
y)
along some ray surface. In the case of
a
known
imaging system we are able to compute
s(z,
y)
a
priori.
Us-
ing equation 2 we may express one ray surface in terms of
another ray surface such
as
a
sphere or
a
plane,
3
Caustics
While convenient, the choices
of
a
plane or sphere
as
the
reference surface come with their drawbacks. The direction
q(z,
y)
of
a
raxel need not have any relation to the position
p(z,
y)
on
a general reference surface. There may be points
on the surface that do not have incoming rays. At other
points, several rays may pass through the same point.
For many imaging systems, there is a distinguished ray sur-
face which is defined
solely
by the geometry of the rays. In
Figure
5,
we see the envelope of the incoming rays form
a
curve. On such surfaces, direction
q
is
really
a
function of
position
p,
and the incomming rays are tangent to the sur-
face. This is
a
special case of
a
ray surface called
a
caustic.
We argue that caustics are the logical place
to
locate our
raxels.
Caustics have
a
number of different characterizations. The
caustic often forms a surface to which
all
incoming rays are
tangent. This does not happen in the perspective case, where
the caustic is
a
point. Caustics can
also
be viewed
as
geo-
metrical entities where there occurs
a
singularity (bunching)
of the light rays. As
a
result,
a
caustic formed by rays of illu-
mination generates
a
very bright region when
it
intersects
a
~urface.~ In our context of imaging systems, a caustic is the
locus of points where incoming rays most converge. Points
on the caustic can therefore be viewed as the ‘locus of view-
points” of the system. It is thus
a
natural place to locate the
raxels
of
our abstract imaging model.
’For
example, when light refracts through shallow water
of
a
pool,
bnght curves can be seen where the caustics intersect the bottom
Figure
5:
The caustic is a good candidate for the
ray
surface
of
an
imaging system as
it
is closely related to the geometry of the
incoming rays; the incoming
ray
directions are tangent to the caus-
tic.
3.1
Definition
of
a Caustic Surface
In section 2.3, we described the set of points and direc-
tions that can be detected by the imaging system. This set
is described by the range of
C(x,
y,
r)
from equation (2).
The position component functions are
9
=
px(z,y,~),
Y
=
py(x,y,r),
and
2
=
pz(z,y,r).
The map from
(x,
y,
r)
to
(A-,
Y,
2)
can be viewed
as
a change of coor-
dinates. The
caustic
surface
is defined
as
the locus
of
points
in
A-,
Y,
2
space where this change of coordinates is singu-
1
ar.
3.2
Computing Caustics
The ray-to-image mapping
of
an imaging system may be
obtained in two ways. One way, is to derive the mapping or
compute
it
numerically from optical components or parame-
ters of an imaging system which is known
a-priori.
Alterna-
tively,
a
calibration method can be used to obtain
a
discrete
form of the mapping (see section
5).
In either case, our goal
is to compute the caustic surface from the given mapping.
When this mapping is known in closed form, analytic meth-
ods can be used to derive the caustic surface
[41.
When
a
discrete approximation is given,
a
host
of
numerical meth-
ods [121,
[IS],
[261, may be used. The method we use com-
putes the caustic by finding
all
the points where the change
in
coordinates described above is singular
[21,
[31.
Equation (2) expresses
C
in terms of
a
known or measured
ray surface
s(x,
y).
The caustic is defined
as
the singular-
ities
in
the change from
(x,
y,
T)
coordinates to
(S,
Y,
2)
coordinates given by
p.
Singularities arise at those points
(A‘,
l’,
2)
where the Jacobian matrix
J
of the transforma-
tion does not have full rank. We can find these points by
111

computing the determinant of the Jacobian,
and setting it to zero. Since this is quadratic in
T
we can
solve for
r
explicitly in terms of
p,
q,
and their first deriva-
tives with respect to
z
and
y.
Plugging this back into
C
gives us an expression for the caustic ray surface parame-
terized by
(x,
y)
as
in equation
(1).
If the optical system
has translational or rotational symmetry then we may only
consider one parameter, for example
2,
in the image plane.
In this case Jacobian becomes linear in
T
and the solution
simplifies to:
(4)
3.3
Field
of
View
Some parameters used to specify the general perspective
camera model are derived from the ray surface representa-
tion in our general imaging model. Other parameters de-
pend on the perspective assumption and are ill defined in
our model. For example, field of view presents an ambigu-
ity, since in the non-perspective case the rays may
no
longer
form
a
simple rectangular cone. One candidate for
a
field
of view is the range of
q(x,
y)
over the image. This is the
same
as
the Gauss map
[l
11.
The Gauss map is
a
good ap-
proximation
to
the field of view when the scene points are
distant relative to the size
of
the imaging system.
Other geometric parameters of
a
perspective imaging sys-
tem, such
as
aspect ratio, and spherical aberration
[251,
may
no longer be separable in the general imaging model from
the parameterized ray surface.
4
Non-Geometric
Raxel
Parameters
Each raxel is treated
as
a
very narrow field
of
view, perspec-
tive imaging system. Many of the conventional parameters
associated with
a
perspective or near-perspective systems
may be attributed
to
a
raxel.
4.1
Local Focal Length and Point Spread
An arbitrary imaging system cannot be expected to have
a
single global focal length. However, each raxel may be
modeled
to
have its own focal length. We can compute each
raxel's focal length by measuring its point spread function
for several depths.
A
flexible approach models the point
spread
as
an elliptical Gaussian. Each ellipse has
a
major
fb,
as
well
as
a focal orientation
1c,
in the image.
4.2
Radiometry
The radiometric response,
g,
is expected to be smooth
and monotonic and can be modeled a polynomial
[191.
If
one can compute the radiometric response function of each
raxel, one can linearize the response with respect to scene
radiance, assuming the response is invertible.
We model the relation between raxel irradiance
E
as
scene
radiance
L
times
a
spatially varying attenuation factor,
h(z,
y),
corresponding to the image point
(2,
y).
We call
h(x,
y)
the
full-offfunction.
This factor takes into account
the finite size of the aperture and vignetting effects which
are linear in
L.
4.3
Complete Imaging Model
The general imaging model consists of
a
set of raxels pa-
rameterized by
x
and
y
in pixel coordinates. The parameters
associated with these raxels (see Figure
6),
are
(a)
position
and direction, that describe the ray surface of the caustic,
(b) major and minor focal lengths
as
well
as
a
focal orien-
tation, (c)
a
radiometric response function, and (d)
a
fall-off
constant for each pixel.
Figure
6:
Each raxel has the above parameters of image coor-
dinates, position and direction
in
space, major and minor focal
lengths,
focal
orientation, radiometric response,
and
fall-off fac-
tor. These parameters are measured
with
respect to a coordinate
frame
fixed
to
the imaging
system.
Camera parameters are separated into external and internal
parameters. The external parameters are specified by
a
co-
ordinate frame. The internal parameters (Figure
6)
are spec-
ified with respect to that coordinate frame. In particular, for
each raxel
i,
the parameters
pi,
qi
are measured with respect
to
a
single coordinate frame fixed to the imaging system. If
the system is rotated or translated, these parameters will not
change
but
the coordinate frame will.
In the case of perspective projection, the essential
[51
or
fundamental
[
101
matrix provides the relationship between
points in one image and lines in another image (of the same
scene). In the general imaging model, this correspondence
need no longer be projective. Nevertheless,
a
point in one
image still corresponds to
a
curve in the other, which may
be computed.
5
Finding
the
Model
Parameters
and minor axis. The major axis makes an angle
1c,
with the
x-axis in the image.6 Each raxel has two focal lengths,
fa,
hThe angle
$
is only defined
if
the
major
and minor
axis
have different
In
section
we
described
how
to
compute
Our
model
for
a
known optical system. In contrast, our goal in this section
lengths.
112

Citations
More filters
Journal ArticleDOI
TL;DR: Experimental results show the effectiveness of the proposed 3D panoramic sensor with a catadioptric imaging system, designed in a polar coordinate system to match the dimensions of a hyperbolic mirror.
Abstract: This paper aims to develop a 3D panoramic sensor with a catadioptric imaging system. First, the fringe pattern is designed in a polar coordinate system to match the dimensions of a hyperbolic mirror. Second, a variable-frequency fringe pattern is proposed to obtain the constant-frequency reconstruction in real space. Experiments are conducted to verify system accuracy and feasibility, and the experimental results show the effectiveness of the proposed 3D panoramic sensor.

6 citations

Journal ArticleDOI
11 Apr 2021-Sensors
TL;DR: In this article, a self-supervised monocular depth estimation using spatio-temporal consistency in the colon environment is proposed, where a loss function for reconstruction errors between adjacent predicted depths and a depth feedback network that uses predicted depth information of the previous frame to predict the depth of the next frame.
Abstract: A colonoscopy is a medical examination used to check disease or abnormalities in the large intestine. If necessary, polyps or adenomas would be removed through the scope during a colonoscopy. Colorectal cancer can be prevented through this. However, the polyp detection rate differs depending on the condition and skill level of the endoscopist. Even some endoscopists have a 90% chance of missing an adenoma. Artificial intelligence and robot technologies for colonoscopy are being studied to compensate for these problems. In this study, we propose a self-supervised monocular depth estimation using spatiotemporal consistency in the colon environment. It is our contribution to propose a loss function for reconstruction errors between adjacent predicted depths and a depth feedback network that uses predicted depth information of the previous frame to predict the depth of the next frame. We performed quantitative and qualitative evaluation of our approach, and the proposed FBNet (depth FeedBack Network) outperformed state-of-the-art results for unsupervised depth estimation on the UCL datasets.

6 citations

Proceedings ArticleDOI
23 Aug 2010
TL;DR: A novel calibration method for multi-camera platforms, based on multi-linear constraints, that can recover the relative orientation between the different cameras on the platform, even when there are no corresponding feature points between the cameras.
Abstract: We present a novel calibration method for multi-camera platforms, based on multi-linear constraints. The calibration method can recover the relative orientation between the different cameras on the platform, even when there are no corresponding feature points between the cameras, i.e. there are no overlaps between the cameras. It is shown that two translational motions in different directions are sufficient to linearly recover the rotational part of the relative orientation. Then two general motions, including both translation and rotation, are sufficient to linearly recover the translational part of the relative orientation. However, as a consequence of the speed-scale ambiguity the absolute scale of the translational part can not be determined if no prior information about the motions are known, e.g. from dead reckoning. It is shown that in case of planar motion, the vertical component of the translational part can not be determined. However, if at least one feature point can be seen in two different cameras, this vertical component can also be estimated. Finally, the performance of the proposed method is shown in simulated experiments.

6 citations


Cites methods from "A general imaging model and a metho..."

  • ...There exist general methods for calibrating these types of general cameras, [1], but they are not optimal in the specific case of a multi-camera platform....

    [...]

Proceedings ArticleDOI
15 Jun 2019
TL;DR: A novel ray-space projection model to transform sets of rays captured by multiple light field cameras in term of the Plucker coordinates is proposed and results on both synthetic and real light field data have verified the effectiveness and robustness of the proposed model.
Abstract: Light field essentially represents the collection of rays in space. The rays captured by multiple light field cameras form subsets of full rays in 3D space and can be transformed to each other. However, most previous approaches model the projection from an arbitrary point in 3D space to corresponding pixel on the sensor. There are few models on describing the ray sampling and transformation among multiple light field cameras. In the paper, we propose a novel ray-space projection model to transform sets of rays captured by multiple light field cameras in term of the Pl\"ucker coordinates. We first derive a $6\times6$ ray-space intrinsic matrix based on multi-projection-center (MPC) model. A homogeneous ray-space projection matrix and a fundamental matrix are then proposed to establish ray-ray correspondences among multiple light fields. Finally, based on the ray-space projection matrix, a novel camera calibration method is proposed to verify the proposed model. A linear constraint and a ray-ray cost function are established for linear initial solution and non-linear optimization respectively. Experimental results on both synthetic and real light field data have verified the effectiveness and robustness of the proposed model.

6 citations


Cites background from "A general imaging model and a metho..."

  • ...Grossberg and Nayar [6] define the image pixel as the light from a cone around a ray and propose a generalized camera model for calibration....

    [...]

References
More filters
Book
01 Jan 1959
TL;DR: In this paper, the authors discuss various topics about optics, such as geometrical theories, image forming instruments, and optics of metals and crystals, including interference, interferometers, and diffraction.
Abstract: The book is comprised of 15 chapters that discuss various topics about optics, such as geometrical theories, image forming instruments, and optics of metals and crystals. The text covers the elements of the theories of interference, interferometers, and diffraction. The book tackles several behaviors of light, including its diffraction when exposed to ultrasonic waves.

19,815 citations

Journal ArticleDOI
Roger Y. Tsai1
01 Aug 1987
TL;DR: In this paper, a two-stage technique for 3D camera calibration using TV cameras and lenses is described, aimed at efficient computation of camera external position and orientation relative to object reference coordinate system as well as the effective focal length, radial lens distortion, and image scanning parameters.
Abstract: A new technique for three-dimensional (3D) camera calibration for machine vision metrology using off-the-shelf TV cameras and lenses is described. The two-stage technique is aimed at efficient computation of camera external position and orientation relative to object reference coordinate system as well as the effective focal length, radial lens distortion, and image scanning parameters. The two-stage technique has advantage in terms of accuracy, speed, and versatility over existing state of the art. A critical review of the state of the art is given in the beginning. A theoretical framework is established, supported by comprehensive proof in five appendixes, and may pave the way for future research on 3D robotics vision. Test results using real data are described. Both accuracy and speed are reported. The experimental results are analyzed and compared with theoretical prediction. Recent effort indicates that with slight modification, the two-stage calibration can be done in real time.

5,940 citations

Book
03 Jan 1992
TL;DR: A new technique for three-dimensional camera calibration for machine vision metrology using off-the-shelf TV cameras and lenses using two-stage technique has advantage in terms of accuracy, speed, and versatility over existing state of the art.
Abstract: A new technique for three-dimensional (3D) camera calibration for machine vision metrology using off-the-shelf TV cameras and lenses is described. The two-stage technique is aimed at efficient computation of camera external position and orientation relative to object reference coordinate system as well as the effective focal length, radial lens distortion, and image scanning parameters. The two-stage technique has advantage in terms of accuracy, speed, and versatility over existing state of the art. A critical review of the state of the art is given in the beginning. A theoretical framework is established, supported by comprehensive proof in five appendixes, and may pave the way for future research on 3D robotics vision. Test results using real data are described. Both accuracy and speed are reported. The experimental results are analyzed and compared with theoretical prediction. Recent effort indicates that with slight modification, the two-stage calibration can be done in real time.

5,816 citations

Proceedings ArticleDOI
01 Aug 1996
TL;DR: This paper describes a sampled representation for light fields that allows for both efficient creation and display of inward and outward looking views, and describes a compression system that is able to compress the light fields generated by more than a factor of 100:1 with very little loss of fidelity.
Abstract: A number of techniques have been proposed for flying through scenes by redisplaying previously rendered or digitized views. Techniques have also been proposed for interpolating between views by warping input images, using depth information or correspondences between multiple images. In this paper, we describe a simple and robust method for generating new views from arbitrary camera positions without depth information or feature matching, simply by combining and resampling the available images. The key to this technique lies in interpreting the input images as 2D slices of a 4D function the light field. This function completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination. We describe a sampled representation for light fields that allows for both efficient creation and display of inward and outward looking views. We hav e created light fields from large arrays of both rendered and digitized images. The latter are acquired using a video camera mounted on a computer-controlled gantry. Once a light field has been created, new views may be constructed in real time by extracting slices in appropriate directions. Since the success of the method depends on having a high sample rate, we describe a compression system that is able to compress the light fields we have generated by more than a factor of 100:1 with very little loss of fidelity. We also address the issues of antialiasing during creation, and resampling during slice extraction. CR Categories: I.3.2 [Computer Graphics]: Picture/Image Generation — Digitizing and scanning, Viewing algorithms; I.4.2 [Computer Graphics]: Compression — Approximate methods Additional keywords: image-based rendering, light field, holographic stereogram, vector quantization, epipolar analysis

4,426 citations

Book
01 Mar 1986
TL;DR: Robot Vision as discussed by the authors is a broad overview of the field of computer vision, using a consistent notation based on a detailed understanding of the image formation process, which can provide a useful and current reference for professionals working in the fields of machine vision, image processing, and pattern recognition.
Abstract: From the Publisher: This book presents a coherent approach to the fast-moving field of computer vision, using a consistent notation based on a detailed understanding of the image formation process. It covers even the most recent research and will provide a useful and current reference for professionals working in the fields of machine vision, image processing, and pattern recognition. An outgrowth of the author's course at MIT, Robot Vision presents a solid framework for understanding existing work and planning future research. Its coverage includes a great deal of material that is important to engineers applying machine vision methods in the real world. The chapters on binary image processing, for example, help explain and suggest how to improve the many commercial devices now available. And the material on photometric stereo and the extended Gaussian image points the way to what may be the next thrust in commercialization of the results in this area. Chapters in the first part of the book emphasize the development of simple symbolic descriptions from images, while the remaining chapters deal with methods that exploit these descriptions. The final chapter offers a detailed description of how to integrate a vision system into an overall robotics system, in this case one designed to pick parts out of a bin. The many exercises complement and extend the material in the text, and an extensive bibliography will serve as a useful guide to current research. Errata (164k PDF)

3,783 citations