Journal of Applied Statistics, Vol. 25, No. 2, 1998, 155± 171
A review of i mage-warping methods
C. A. GL ASBEY
1
& K. V. MARDIA
2
,
1
B iomathematics and Statistics Scotland ,
King’s B uildings, Edinburgh, UK a nd
2
Department of Statistics, University of Leeds, UK
SUMM ARY
Image war ping is a transformation which maps all positions in one image
plane t o positions in a second plane. It arises in many image analysis problems, whether
in order to remove optical distortions introduced by a came ra or a particular vie wi ng
perspective, to register an image with a map or t emplate, or to align two or more images.
The choice of war p is a compromise between a smooth distortion and one which achieves
a good match. Smoothness can be ensured by assuming a parametric for m for the warp
or by constraining it using diŒerential equations. M atching can be speci® ed by points to
be brough t into alignment, by local measures of correlation between images, or by the
coincidence of edges . Parametric and non-parametric approaches to war ping, and matching
criteria, are reviewed.
1 Introduction
Warping of images is an important stage in ma ny applications of image analysis. It
may be needed to remove optical distortions introduced by a camera or viewing
perspective (Heikkila & Silven, 1997; Tang & Suen, 1993), to register an image
with a reference grid such as a ma p, or to align two or more images (Brown, 1992).
For example, matching is important i n reconstructing three-dimensional shape
from either a series of two-dimensional sections or stereoscopic pairs of images.
Much e
Œ
ort has been expended in developing algorithms for registering satellite
images with both geographic information systems and with other forms of remote
sensing system, such as optical sensors and synthetic aperture radar (see, for
example, Richards, 1986, Chapter 2). Recently, there has been considerable
interest in registering images produced by medical sensing systems with body atlas
information (Colchester & Hawkes, 1991, Section 3). Combined images produced
using di
Œ
erent imaging modalities also have great potential. For example, X- ray
Correspondence
: K . V. Mardia, Department of Statistics, U niversity of Leeds, Leeds LS2 9JT, U K. Tel:
01 13 233 5101.
02 66-476 3/98 /020155-17 $7.00 199 8 Carfax Publishing Ltd
156
C. A. Glasbey & K. V. M ardia
FIG. 1. N otation for image warping.
images reveal structure, whereas magnetic resonance images reveal functionality,
so their synthesis generat es more informative images (for example, see Hurn
et al
.,
1996; M ardia & Little, 1994).
A related problem is the warping of one-dimensional signals to bring them into
alignment. This is sometimes referred to as dynamic time warping. Dynamic
programming methods have been applied to speech processing (Sakoe & Chiba,
1978), handwriting analysis (Burr, 1983), alignment of boundaries of ice ¯ oes
(McCo nnell
et al
., 1991) and of tracks in electrophoresis gels (Skovgaard
et al
.,
1995). Wang and Gasser (1997) consider ed theoretical issues. Where features on
two curves are already matched, the problem simpli® es to one of monotone
regression (see, for example, Ramsay, 1988).
A warping is a pair of two-dimensional functions,
u
(
x
,
y
) and
v
(
x
,
y
), which map
a position (
x
,
y
) in one image, where
x
denotes column number and
y
denotes row
number, to position (
u
,
v
) in another image (see Fig. 1). There have been many
approaches to ® nding an appropriate warp, but a common theme is the compromise
between insisting the d istortion is smooth and achieving a good match. In some
recently published cases the warp seems unnecessarily rough (Conrad sen & Peder-
sen, 1992, Fig. 8(b); Grenander & Miller, 1994, Fig. 7(f )). Sm oothness can be
ensured by assuming a par ametric form for the warp, such as the a
ne transforma-
tion, or by penalizing roughn ess, such as by using thin-plate s plines. Depending
on the application, matching might be speci® ed by points which must be brought
into alignment, by local measures of correlation between images , or by the
coincidence of edges. Overviews of ge ometric transformations are given by Wolberg
(1988), Bookstein (1991), Brown (1992) and Tang and Suen (1993).
In S ection 2, we will progress through para metric models, from linear to non-
linear ones, elucidating their properties. Then , in Section 3, we will consider a
rang e of non-parametric models. Matching criteria will be covered as they arise in
di
Œ
erent applications.
2 Param etric transform ations
Figure 2 shows a hierarchy of parametric transformations. In many appli cations, it
is important to use a transformation which is no more general than it need be. We
will consider the properties of each transformation in turn.
Image-warping methods
157
FIG. 2. A hierarchy of transformations. Arrows denote models which generalize others.
2.1 Translation
If the only di
Œ
erence between two images is one of translation, eithe r along rows,
along columns or a combination of both, then
u
5
x
+
b
00
and
v
5
y
+
b
00
performs the required mapping. Here, the top-left corner (0, 0) o f one image
158
C. A. Glasbey & K. V. M ardia
matches location (
a
00
,
b
00
) in the other image. Agreement betwee n images may be
measured by
S
5
R
n
x
x
5
0
R
n
y
y
5
0
(
I
x
,
y
2
I Â
u
,
v
)
2
where
I
x
,
y
denotes pixel value (
x
,
y
) in one image and
I Â
u
,
v
denotes pixel value (
u
,
v
)
in the other image. In general,
u
and
v
will not be integers, and
I Â
u
,
v
is assigned the
value of
I Â
[
u
+ 1/2],[
v
+ 1 /2]
, where [
.
] is used to denote the integer part of a number.
Alternatively, bilinear interpolation co uld be used, with
I Â
u
,
v
5
([
u
]
+
1 2
u
)([
v
]
+
1 2
v
)
I Â
[
u
], [
v
]
+
(
u
2 [
u
])([
v
]
+
1 2
v
)
I Â
[
u
+ 1 ],[
v
]
+
([
u
]
+
1 2
u
)(
v
2 [
v
])
I Â
[
u
], [
v
+ 1]
+
(
u
2 [
u
])(
v
2 [
v
])
I Â
[
u
+ 1 ],[
v
+ 1]
There are potential problems with
S
: if the dimensions of the second image (
n
u
,
n
v
)
are such t hat
n
u
<
n
x
and
n
v
<
n
y
, then some values of (
u
,
v
) will lie outside this
range if (
a
00
,
b
00
)
¹
(0, 0). The simplest way around thi s di
culty is to restrict the
summations to the area of o verlap of the two images after transfo rmation. But then
it is also a go od idea to standardize
S
by dividing by the area of overlap, because
otherwise the measure takes a mi nimum value of zero when the two images do not
overlap at all. Note that, although the transformation is functionally invertible,
because (
x
,
y
) can be expressed as a translation of (
u
,
v
), in general a di
Œ
erent
result would be obtained if
S Â
5
R
n
u
u
5
0
R
n
v
v
5
0
(
I
x
,
y
2
I Â
u
,
v
)
2
were minimized instead of
S
. One way to retain equivalence in the two images
would be to minimize
S
+
S Â
instead.
There are many other meas ures of agreement (see, for example, Rosenfeld &
Kak, 1982, Section 9.4), such as the covariance, which is proport ional to
R
n
x
x
5
0
R
n
y
y
5
0
(
I
x
,
y
2
I
Å
)(
I Â
u
,
v
2
I
Å
Â
)
where
I
Å
denotes the average pixel value in the ® rst image. This is a linear
convolution, which can be evaluated very e
ciently for all possible changes in
origin simultaneously using a fast Fourier transform (Glasbey & Horgan, 1995 ,
Section 3.2). The covariances are given by
&
2 1
{& (
I
2
I
Å
)&
C
(
I Â
2
I
Å
Â
)}
where & denotes the Fouri er transform, &
C
denotes the complex conjugate of the
transform and &
2 1
is the inverse transform.
S
can be computed in an analogous
way. W hen shifts of location are all that are involved, this problem is sometimes
referred to as template matching. Attention has been given to e
cient computa-
tions; for example, by searching ® rst for optimal values of
a
00
and
b
00
using a coarse
grid, and then on a ® ner grid (Goshtasby
et al
., 1984).
Phase correlation, de® ned as
&
2 1
{
& (
I
)&
C
(
I Â
)
½
& (
I
)&
C
(
I Â
)
½
}
Image-warping methods
159
where
½ ½
denotes modulus, was proposed by Kuglin and Hines (1975). It performs
better than the covariance in cases where the di
Œ
erences between
I
and
I Â
(after
the appropria te shift in location) occur only at a subset of frequencies. Thi s would
be the case, for example, if the trend in illu mination di
Œ
ers between the images,
but not if signi® cant levels of white noise are present. It is also possible to formulate
criteria, intermediate between correlation and phase correlation, which are optimal
when image s agree only at a subset of frequencies.
2.2 Procr ustes transfor mation
If ther e is some change i n magni® cation between the images, and a ro tation of h
degrees, then
u
5
cx
cos h
+
cy
sin h
+
a
00
and
v
5
2
cx
sin h
+
cy
cos h
+
b
00
where the constant
c
performs a rescaling. A value of
c
5
1 corresponds to no
change in magni® cation, whereas
c
>
1 is an enlargement and
c
<
1 is a shrinkage.
In computer vision, location, scale and rotation are know n collectively as pose.
Once these di
Œ
erences have been removed, what remai ns is known as shape. This
is the transformation used in Procrustes analysis (Goodall, 1991) to align labelled
set s of points. It is a four-parameter transformat ion and can be uniquely de® ned
from two points in the two images, or a least-squares solution can be obtained if
there ar e more than two points. Extensions to more than two images are relatively
straightforward. The simplest approach is to align all images with the ® rst one.
However, it is usually somewhat arbi trary to single out one image in this way. In
generalized Procrustes analysis, all sets of points are matched to the average
con® guration. In other applications, such an approach could lead to much heavier
computations.
It is possible to restrict the transfor mation to fewer parameters in speci® c
situations, by omitting either the scaling or the rotation terms (see Fig. 2). For
example, Glasbey and Martin (1996) used translation and scaling to align micro-
scope images of algal cells obtained using di
Œ
erent modalities, na mely bright® eld,
phase contrast and di
Œ
erentia l interference contrast (DIC). This warping was both
necessary and su
cient to compensate f or changes in image alignment resulting
from imperfect centration of the di
Œ
erent lens systems and from slight di
Œ
erences
in magni® cation between objective lenses with the same nominal magni® cation.
Matching criteria such as co rrelation, which is based simpl y on image intensities,
did not perform well because, in bright® eld microscopy, algal cells appear dark,
whereas in DIC one side of cells is dark and the other side is light. Positions where
intensities change rapidly at the edges of cells do coincide, and were therefore used
in the matching criterion. Edge information has also been used by Bajcsy and
Kovacic (1989) and M oshfeghi (1991) to align medical images. Prewitt’ s edge ® lter
(see, for example, Glasbey & Horgan, 1995, Section 3.4) was applied to all
the microscope images . Then the cross-correlation between gradient images was
computed using Fourier methods for each of a range of di
Œ
erences in magni® cations
between the images. Translation and magni® cation parameters which produced
the highest cross-correlation were selected.
2.3 A ne transformation
The a
ne transformation is a six-parameter generalization of the Procrustes
transformation. It permits di
Œ
erent stretching along rows and columns of an image,