![Figure 12. The Gauss-Newton steepest descent inverse compositional algorithm estimates the parameter updates p as constant multiples of the gradient ∑ x[ ∂G ∂p ] T rather than multiplying the gradient with the inverse Hessian. The computation of the constant multiplication factor c requires the Hessian however, estimated here with the Gauss-Newton approximation. The steepest descent inverse compositional algorithm is slightly faster than the original inverse compositional algorithm and takes time O(n N + n2) per iteration.](/figures/figure-12-the-gauss-newton-steepest-descent-inverse-1hfld27y.webp)
International Journal of Computer Vision 56(3), 221–255, 2004
c
2004 Kluwer Academic Publishers. Manufactured in The Netherlands.
Lucas-Kanade 20 Years On: A Unifying Framework
SIMON BAKER AND IAIN MATTHEWS
The Robotics Institute, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, USA
simonb@cs.cmu.edu
iainm@cs.cmu.edu
Received July 10, 2002; Revised February 6, 2003; Accepted February 7, 2003
Abstract. Since the Lucas-Kanade algorithm was proposed in 1981 image alignment has become one of the most
widely used techniques in computer vision. Applications range from optical flow and tracking to layered motion,
mosaic construction, and face coding. Numerous algorithms have been proposed and a wide variety of extensions
have been made to the original formulation. We present an overview of image alignment, describing most of the
algorithms and their extensions in a consistent framework. We concentrate on the inverse compositional algorithm,
an efficient algorithm that we recently proposed. We examine which of the extensions to Lucas-Kanade can be used
with the inverse compositional algorithm without any significant loss of efficiency, and which cannot. In this paper,
Part 1 in a series of papers, we cover the quantity approximated, the warp update rule, and the gradient descent
approximation. In future papers, we will cover the choice of the error function, how to allow linear appearance
variation, and how to impose priors on the parameters.
Keywords: image alignment, Lucas-Kanade, a unifying framework, additive vs. compositional algorithms, for-
wards vs. inverse algorithms, the inverse compositional algorithm, efficiency, steepest descent, Gauss-Newton,
Newton, Levenberg-Marquardt
1. Introduction
Image alignment consists of moving, and possibly de-
forming, a template to minimize the difference between
the template and an image. Since the first use of im-
age alignment in the Lucas-Kanade optical flow al-
gorithm (Lucas and Kanade, 1981), image alignment
has become one of the most widely used techniques
in computer vision. Besides optical flow, some of its
other applications include tracking (Black and Jepson,
1998; Hager and Belhumeur, 1998), parametric and
layered motion estimation (Bergen et al., 1992), mo-
saic construction (Shum and Szeliski, 2000), medical
image registration (Christensen and Johnson, 2001),
and face coding (Baker and Matthews, 2001; Cootes
et al., 1998).
The usual approach to image alignment is gradi-
ent descent. A variety of other numerical algorithms
such as difference decomposition (Gleicher, 1997) and
linear regression (Cootes et al., 1998) have also been
proposed, but gradient descent is the defacto standard.
Gradient descent can be performed in variety of dif-
ferent ways, however. One difference between the var-
ious approaches is whether they estimate an additive
increment to the parameters (the additive approach
(Lucas and Kanade, 1981)), or whether they estimate
an incremental warp that is then composed with the
current estimate of the warp (the compositional ap-
proach (Shum and Szeliski, 2000)). Another difference
is whether the algorithm performs a Gauss-Newton, a
Newton, a steepest-descent, or a Levenberg-Marquardt
approximation in each gradient descent step.
We propose a unifying framework for image align-
ment, describing the various algorithms and their ex-
tensions in a consistent manner. Throughout the frame-
work we concentrate on the inverse compositional
222 Baker and Matthews
algorithm, an efficient algorithm that we recently pro-
posed (Baker and Matthews, 2001). We examine which
of the extensions to Lucas-Kanade can be applied to
the inverse compositional algorithm without any sig-
nificant loss of efficiency, and which extensions require
additional computation. Wherever possible we provide
empirical results to illustrate the various algorithms and
their extensions.
In this paper, Part 1 in a series of papers, we be-
gin in Section 2 by reviewing the Lucas-Kanade algo-
rithm. We proceed in Section 3 to analyze the quan-
tity that is approximated by the various image align-
ment algorithms and the warp update rule that is used.
We categorize algorithms as either additive or compo-
sitional, and as either forwards or inverse.Weprove
the first order equivalence of the various alternatives,
derive the efficiency of the resulting algorithms, de-
scribe the set of warps that each alternative can be
applied to, and finally empirically compare the algo-
rithms. In Section 4 we describe the various gradient de-
scent approximations that can be used in each iteration,
Gauss-Newton, Newton, diagonal Hessian, Levenberg-
Marquardt, and steepest-descent (Press et al., 1992).
We compare these alternatives both in terms of speed
and in terms of empirical performance. We conclude
in Section 5 with a discussion. In future papers in this
series (which will be made available on our website
http://www.ri.cmu.edu/projects/project
515.html), we
will cover the choice of the error function, how to al-
low linear appearance variation, and how to add priors
on the parameters.
2. Background: Lucas-Kanade
The original image alignment algorithm was the Lucas-
Kanade algorithm (Lucas and Kanade, 1981). The goal
of Lucas-Kanade is to align a template image T (x)toan
input image I (x), where x = (x, y)
T
is a column vector
containing the pixel coordinates. If the Lucas-Kanade
algorithm is being used to compute optical flow or to
track an image patch from time t = 1totime t = 2,
the template T (x)isanextracted sub-region (a 5 × 5
window, maybe) of the image at t = 1 and I (x)isthe
image at t = 2.
Let W(x; p) denote the parameterized set of allowed
warps, where p = ( p
1
,... p
n
)
T
is a vector of parame-
ters. The warp W(x; p) takes the pixel x in the coordi-
nate frame of the template T and maps it to the sub-pixel
location W(x; p)inthe coordinate frame of the image
I .Ifweare computing optical flow, for example, the
warps W(x; p) might be the translations:
W(x; p) =
x + p
1
y + p
2
(1)
where the vector of parameters p = (p
1
, p
2
)
T
is then
the optical flow. If we are tracking a larger image patch
moving in 3D we may instead consider the set of affine
warps:
W(x; p) =
(1 + p
1
) · x + p
3
· y + p
5
p
2
· x + (1 + p
4
) · y + p
6
=
1 + p
1
p
3
p
5
p
2
1 + p
4
p
6
x
y
1
(2)
where there are 6 parameters p = (p
1
, p
2
, p
3
, p
4
, p
5
,
p
6
)
T
as, for example, was done in Bergen et al. (1992).
(There are other ways to parameterize affine warps.
Later in this framework we will investigate what is
the best way.) In general, the number of parameters n
may be arbitrarily large and W(x; p) can be arbitrar-
ily complex. One example of a complex warp is the
set of piecewise affine warps used in Active Appear-
ance Models (Cootes et al., 1998; Baker and Matthews,
2001) and Active Blobs (Sclaroff and Isidoro, 1998).
2.1. Goal of the Lucas-Kanade Algorithm
The goal of the Lucas-Kanade algorithm is to mini-
mize the sum of squared error between two images,
the template T and the image I warped back onto the
coordinate frame of the template:
x
[
I (W(x; p)) − T (x)
]
2
. (3)
Warping I back to compute I (W(x; p)) requires inter-
polating the image I at the sub-pixel locations W(x; p).
The minimization of the expression in Eq. (3) is per-
formed with respect to p and the sum is performed
over all of the pixels x in the template image T (x).
Minimizing the expression in Eq. (1) is a non-linear
optimization task even if W(x; p)islinear in p because
the pixel values I (x) are, in general, non-linear in x.
In fact, the pixel values I (x) are essentially un-related
to the pixel coordinates x.Tooptimize the expression
in Eq. (3), the Lucas-Kanade algorithm assumes that
a current estimate of p is known and then iteratively
Lucas-Kanade 20 Years On: A Unifying Framework 223
solves for increments to the parameters p; i.e. the
following expression is (approximately) minimized:
x
[I (W(x; p + p)) − T (x)]
2
(4)
with respect to p, and then the parameters are up-
dated:
p ← p + p. (5)
These two steps are iterated until the estimates of the
parameters p converge. Typically the test for conver-
gence is whether some norm of the vector p is below
a threshold ; i.e. p≤.
2.2. Derivation of the Lucas-Kanade Algorithm
The Lucas-Kanade algorithm (which is a Gauss-
Newton gradient descent non-linear optimization al-
gorithm) is then derived as follows. The non-linear ex-
pression in Eq. (4) is linearized by performing a first
order Taylor expansion on I (W(x; p + p)) to give:
x
I (W(x; p)) + ∇I
∂W
∂p
p − T (x)
2
. (6)
In this expression, ∇ I = (
∂ I
∂x
,
∂ I
∂y
)isthe gradient of
image I evaluated at W(x; p); i.e. ∇ I is computed
in the coordinate frame of I and then warped back onto
the coordinate frame of T using the current estimate of
the warp W(x; p). The term
∂W
∂p
is the Jacobian of the
warp. If W(x; p) = (W
x
(x; p), W
y
(x; p))
T
then:
∂W
∂p
=
∂W
x
∂p
1
∂W
x
∂p
2
...
∂W
x
∂p
n
∂W
y
∂p
1
∂W
y
∂p
2
...
∂W
y
∂p
n
. (7)
We follow the notational convention that the partial
derivatives with respect to a column vector are laid out
as a row vector. This convention has the advantage that
the chain rule results in a matrix multiplication, as in
the expression in Eq. (6). For example, the affine warp
in Eq. (2) has the Jacobian:
∂W
∂p
=
x 0 y 010
0 x 0 y 01
. (8)
Minimizing the expression in Eq. (6) is a least squares
problem and has a closed from solution which can be
Figure 1. The Lucas-Kanade algorithm (Lucas and Kanade, 1981)
consists of iteratively applying Eqs. (10) and (5) until the estimates
of the parameters p converge. Typically the test for convergence
is whether some norm of the vector p is below a user specified
threshold . Because the gradient ∇ I must be evaluated at W(x; p)
and the Jacobian
∂W
∂p
must be evaluated at p, all 9 steps must be
repeated in every iteration of the algorithm.
derived as follows. The partial derivative of the expres-
sion in Eq. (6) with respect to p is:
2
x
∇I
∂W
∂p
T
I (W(x; p)) + ∇ I
∂W
∂p
p − T (x)
(9)
where we refer to ∇I
∂W
∂p
as the steepest descent im-
ages. (See Section 4.3 for why.) Setting this expression
to equal zero and solving gives the closed form solution
for the minimum of the expression in Eq. (6) as:
p = H
−1
x
∇I
∂W
∂p
T
[T (x) − I (W(x; p))]
(10)
where H is the n × n (Gauss-Newton approximation
to the) Hessian matrix:
H =
x
∇I
∂W
∂p
T
∇I
∂W
∂p
. (11)
For reasons that will become clear later we refer to
x
[∇I
∂W
∂p
]
T
[T (x) − I(W(x; p))] as the steepest de-
scent parameter updates. Equation (10) then expresses
the fact that the parameter updates p are the steepest
descent parameter updates multiplied by the inverse
of the Hessian matrix. The Lucas-Kanade algorithm
(Lucas and Kanade, 1981) then consists of iteratively
applying Eqs. (10) and (5). See Figs. 1 and 2 for a sum-
mary. Because the gradient ∇I must be evaluated at
224 Baker and Matthews
Figure 2.Aschematic overview of the Lucas-Kanade algorithm (Lucas and Kanade, 1981). The image I is warped with the current estimate
of the warp in Step 1 and the result subtracted from the template in Step 2 to yield the error image. The gradient of I is warped in Step 3, the
Jacobian is computed in Step 4, and the two combined in Step 5 to give the steepest descent images. In Step 6 the Hessian is computed from
the steepest descent images. In Step 7 the steepest descent parameter updates are computed by dot producting the error image with the steepest
descent images. In Step 8 the Hessian is inverted and multiplied by the steepest descent parameter updates to get the final parameter updates p
which are then added to the parameters p in Step 9.
W(x; p) and the Jacobian
∂W
∂p
at p, they both in gen-
eral depend on p.For some simple warps such as the
translations in Eq. (1) and the affine warps in Eq. (2)
the Jacobian can sometimes be constant. See for ex-
ample Eq. (8). In general, however, all 9 steps of the
algorithm must be repeated in every iteration because
the estimates of the parameters p vary from iteration to
iteration.
2.3. Requirements on the Set of Warps
The only requirement on the warps W(x; p)isthat they
are differentiable with respect to the warp parameters p.
This condition is required to compute the Jacobian
∂W
∂p
.
Normally the warps are also (piecewise) differentiable
with respect to x,but even this condition is not strictly
required.
Lucas-Kanade 20 Years On: A Unifying Framework 225
Table 1. The computation cost of one iteration of the Lucas-Kanade algorithm. If n is the number of warp
parameters and N is the number of pixels in the template T , the cost of each iteration is O(n
2
N + n
3
). The
most expensive step by far is Step 6, the computation of the Hessian, which alone takes time O(n
2
N ).
Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Total
O(nN)O(N )O(nN)O(nN)O(nN)O(n
2
N )O(nN)O(n
3
)O(n)O(n
2
N + n
3
)
2.4. Computational Cost of the
Lucas-Kanade Algorithm
Assume that the number of warp parameters is n and the
number of pixels in T is N . Step 1 of the Lucas-Kanade
algorithm usually takes time O(nN). For each pixel x
in T we compute W(x; p) and then sample I at that
location. The computational cost of computing W(x; p)
depends on W but for most warps the cost is O(n) per
pixel. Step 2 takes time O(N ). Step 3 takes the same
time as Step 1, usually O(nN). Computing the Jacobian
in Step 4 also depends on W but for most warps the cost
is O(n) per pixel. The total cost of Step 4 is therefore
O(nN). Step 5 takes time O(nN), Step 6 takes time
O(n
2
N ), and Step 7 takes time O(nN). Step 8 takes
time O(n
3
)toinvert the Hessian matrix and time O(n
2
)
to multiply the result by the steepest descent parameter
updated computed in Step 7. Step 9 just takes time
O(n)toincrement the parameters by the updates. The
total computational cost of each iteration is therefore
O(n
2
N +n
3
), the most expensive step being Step 6. See
Table 1 for a summary of these computational costs.
3. The Quantity Approximated and the Warp
Update Rule
In each iteration Lucas-Kanade approximately mini-
mizes
x
[
I (W(x; p + p)) − T (x)
]
2
with respect to
p and then updates the estimates of the parameters in
Step 9 p ← p + p. Perhaps somewhat surprisingly
iterating these two steps is not the only way to minimize
the expression in Eq. (3). In this section we outline 3
alternative approaches that are all provably equivalent
to the Lucas-Kanade algorithm. We then show empiri-
cally that they are equivalent.
3.1. Compositional Image Alignment
The first alternative to the Lucas-Kanade algorithm is
the compositional algorithm.
3.1.1. Goal of the Compositional Algorithm. The
compositional algorithm, used most notably by Shum
and Szeliski (2000), approximately minimizes:
x
[I (W(W(x; p); p)) − T (x)]
2
(12)
with respect to p in each iteration and then updates
the estimate of the warp as:
W(x; p) ← W(x; p) ◦ W(x; p), (13)
i.e. the compositional approach iteratively solves for
an incremental warp W(x; p) rather than an additive
update to the parameters p.Inthis context, we refer to
the Lucas-Kanade algorithm in Eqs. (4) and (5) as the
additive approach to contrast it with the compositional
approach in Eqs. (12) and (13). The compositional and
additive approaches are proved to be equivalent to first
order in p in Section 3.1.5. The expression:
W(x; p) ◦ W(x; p) ≡ W(W(x; p); p) (14)
is the composition of 2 warps. For example, if W(x; p)
is the affine warp of Eq. (2) then:
W(x; p) ◦ W(x; p)
=
(1 + p
1
) · ((1 + p
1
) · x + p
3
· y + p
5
)
+ p
3
· (p
2
· x + (1 + p
4
) · y + p
6
)
+ p
5
p
2
· ((1 + p
1
) · x + p
3
· y + p
5
)
+ (1 + p
4
) · (p
2
· x + (1 + p
4
) · y
+ p
6
) + p
6
,
(15)