Entropy Minimization for Shadow Removal
Graham D. Finlayson
University of East Anglia, Norwich, UK
graham@cmp.uea.ac.uk
∗
Mark S. Drew
Simon Fraser University, Vancouver, Canada
mark@cs.sfu.ca
†
Cheng Lu
Micron Technology, Inc., San Jose, CA, U.S.A
chenglu@micron.com
Nov. 20, 2007
Abstract.
Recently, a method for removing shadows from colour images was developed [Finlayson, Hordley, Lu, and Drew,
PAMI2006] that relies upon finding a special direction in a 2D chromaticity feature space. This “invariant direction” is that
for which particular colour features, when projected into 1D, produce a greyscale image which is approximately invariant
to intensity and colour of scene illumination. Thus shadows, which are in essence a particular type of lighting, are greatly
attenuated. The main approach to finding this special angle is a camera calibration: a colour target is imaged under many
different lights, and the direction that best makes colour patch images equal across illuminants is the invariant direction. Here,
we take a different approach. In this work, instead of a camera calibration we aim at finding the invariant direction from
evidence in the colour image itself. Specifically, we recognize that producing a 1D projection in the correct invariant direction
will result in a 1D distribution of pixel values that have smaller entropy than projecting in the wrong direction. The reason
is that the correct projection results in a probability distribution spike, for pixels all the same except differing by the lighting
that produced their observed RGB values and therefore lying along a line with orientation equal to the invariant direction.
Hence we seek that projection which produces a type of intrinsic, independent of lighting reflectance-information only image
by minimizing entropy, and from there go on to remove shadows as previously. To be able to develop an effective description
of the entropy-minimization task, we go over to the quadratic entropy, rather than Shannon’s definition. Replacing the observed
pixels with a kernel density probability distribution, the quadratic entropy can be written as a very simple formulation, and can
be evaluated using the efficient Fast Gauss Transform. The entropy, written in this embodiment, has the advantage that it is
more insensitive to quantization than is the usual definition. The resulting algorithm is quite reliable, and the shadow removal
step produces good shadow-free colour image results whenever strong shadow edges are present in the image. In most cases
studied, entropy has a strong minimum for the invariant direction, revealing a new property of image formation.
Keywords: Illumination, reflectance, intrinsic images, illumination invariants, color, shadows, entropy, quadratic entropy
1. Introduction
Illumination conditions confound many computer vision algorithms. In particular, shadows in an image
can cause segmentation, tracking, or recognition algorithms to fail. An illumination-invariant image is
therefore of great utility in a wide range of problems in both computer vision and computer graphics.
An interesting feature of this problem is that shadows are approximately but accurately described as a
change of lighting (Finlayson et al., 2002). Hence, it is possible to cast the problem of removing shadows
from images into an equivalent statement about removing (and possibly later restoring) the effects of
lighting in imagery.
∗
This work was supported by the Leverhulme Trust.
†
This work was supported by the Natural Sciences and Engineering Research Council of Canada
c
° 2009 Kluwer Academic Publishers. Printed in the Netherlands.
ijcv08.tex; 29/04/2009; 8:42; p.1
2 Finlayson et al.
Removal of outdoor cast shadows has been addressed before in the literature but typically not based
on a photometric approach. For example, (Cho et al., 2005) uses background subtraction in YCbCr
colour space and gradient thresholding to extract moving blobs in colour to detect cast shadows in traffic
surveillance video. The objective is to find a moving blob that is not in a shadow region. In (Liu et al.,
2006), another method based on time-varying data and a Gaussian mixture model uses the early video-
based model in (Stauder et al., 1999) to partially remove effects of lighting from sequences (and cf.
(Weiss, 2001)) but the proposed insensitivity to illumination depends on a slowly changing penumbra
and does not work for strong shadows. In fact, a substantial amount of work on shadow detection has
been concerned with moving shadows (Prati et al., 2003; Nadimi and Bhanu, 2004; Martel-Brisson
and Zaccarin, 2007), whereas here we concentrate on single still images. In this paper we focus on a
physically-based rather than image-processing approach in order to gain understanding of the underlying
image formation process.
A method was recently devised (Finlayson et al., 2002; Finlayson and Hordley, 2001; Finlayson and
Drew, 2001; Drew et al., 2003; Finlayson et al., 2006) for the recovery of an invariant image from
a 3-band colour image. The invariant image, originally 1D greyscale but subsequently derived as a 2D
chromaticity, is independent of lighting, and also has shading removed: it forms a type of intrinsic image,
independent of illumination conditions, that may be used as a guide in recovering colour images that are
independent of illumination conditions. While the essential definition of an intrinsic image is one that
captures full reflectance information (Barrow and Tenenbaum, 1978), including albedo information, here
we claim only to capture only chromaticity information, not full reflectance. Nevertheless, invariance to
illuminant colour and intensity means that such images are free of shadows as well, to a good degree
(Finlayson et al., 2006). Although shadow removal is not always perfect, the effect of shadows is so
greatly attenuated that many algorithms can easily benefit from the new method; e.g., a shadow-free
active contour based tracking method shows that the snake can without difficulty follow an object and
not its shadow, using the new approach to illumination colour invariance (Jiang and Drew, 2003; Jiang
and Drew, 2007). In place of standard luminance images used in vision, if in an application the effects of
lighting would usefully be removed then arguably the greyscale version of the invariant image should be
used instead.
The method works in a very simple way: Suppose we form chromaticity band-ratios, e.g., G/R, B/G
for a colour 3-band RGB image, and suppose we further take logarithms. An interesting feature to note is
that, under simplifying assumptions set out below, the scatterplot values for pixels from the same surface,
but under different lighting fall on a straight line; and every such line, for different surfaces, has the same
slope. This remarkable fact still hold true approximately even when the guiding, simplifying assumptions
are broken. Since shadowing is a result of a difference in lighting, we can use this physics-based insight
to devise a shadow-removal scheme. This paper uses evidence internal to any particular image, based on
an entropy measure, to find the slope of such lines. Projection orthogonal to this special direction results
in a 1D greyscale image that has shadows approximately removed. We also derive a 2D colour version
of such an invariant image.
The method devised finds an intrinsic reflectivity image motivated by the assumptions of Lambertian
reflectance, approximately Planckian lighting, and fairly narrowband camera sensors. Nevertheless, the
method still works well when these assumptions do not hold. A crucial piece of information is the angle
for an “invariant direction” in a log-chromaticity space. Originally, this information was gleaned via a
preliminary calibration routine, using the camera involved to capture images of a colour target under
different lights. Subsequently, it was shown in principle (Finlayson et al., 2004) that we can in fact
dispense with the calibration step, by recognizing a simple but important fact: the correct projection is
that which minimizes entropy in the resulting invariant greyscale image. In this paper, the entropy based
ijcv08.tex; 29/04/2009; 8:42; p.2
Entropy Minimization for Shadow Removal 3
method is examined in detail, and in order to carry out an efficient search over smooth values that are
not subject to quantization problems, we replace the Shannon’s entropy measure, used previously, by
a Quadratic Entropy measure such that a Gaussian mixture model of the probability density function
(pdf) produces an analytic formula. We show that such quadratic entropy values are much smoother and
usually produce only a single minimum, making this approach the most efficient. The quadratic entropy
can be evaluated in linear time using a Fast Gauss Transform, leading to a simple method for finding the
invariant direction. Shadow removal in full colour, by means of comparing edges in the original and in
the invariant image and then subsequent re-integration, follows.
The paper is organized as follows. In §2, we briefly recapitulate the motivation for a projection-
based definition of an illuminant invariant, and set out the relevant equations. Section 3 looks at how the
entropy minimization scheme plays out for a set of synthetic colour patches, on the one hand, and then
for a set of actual paint patches in a calibration chart. Section 4 considers the issue of how an effective
entropy-minimization algorithm should proceed, and argues that an efficient approach is possible, based
on replacing the definition of entropy by the quadratic form of Renyi’s entropy. Finally, we apply the
method devised to unsourced images, from unknown cameras under unknown lighting, with unknown
processing having been applied. Results are again strikingly good, leading us to conclude, in §8, that the
method indeed holds great promise for developing a stand-alone approach to removing shadows from
(and therefore conceivably re-lighting) any image, e.g. images from consumer cameras.
2. The Invariant Image
2.1. LOG-CHROMATICITY PROJECTION AND ENTROPY MINIMIZATION
In order to motivate the study, we first briefly set out the strategy for developing an illumination invariant
image, and the rationale determining entropy minimization as the key insight for finding such an image.
Consider a calibration scheme, for a particular colour camera, wherein a target composed of coloured
patches (or just images of a rather colourful scene) are imaged under different illuminants — the more
illuminants the better. Then knowledge that these are registered images of the same scene, under differing
lighting, is put to use by plotting the capture RGB values, for each of the patches used, as the lighting
changes. If pixels are first transformed from 3D RGB triples into a 2D band-ratio chromaticity colour
space, {G/R, B/R} say, and then logarithms are taken, the values across different illuminants tend to
fall on straight lines in a 2D scatter plot. And in fact all such lines are parallel, for a given camera
(Finlayson and Hordley, 2001), as illustrated in Fig. 1(a).
So change of illumination simply amounts to movement along such a line. Thus it is straightforward
to devise a 1D, greyscale, illumination-invariant image by projecting the 2D chromaticity points into a
direction perpendicular to all such lines. The result is hence a greyscale image that is independent of
lighting, and is, therefore, a type of intrinsic image (Barrow and Tenenbaum, 1978) that portrays only
the inherent reflectance properties in the scene. Since shadows are mostly due to change in the illuminant
intensity and colour — i.e., differing lighting — such an image also has shadows removed.
Below, we discuss the restrictions on this straight-line model, but it may be useful to look at shadows
and lighting colour in an example. Fig. 2(a) shows a typical consumer-grade camera TIFF image, with a
strong shadow present. Here, the image processing software applied is typically aimed at a “preferred”
(i.e., pleasing) rendition, rather than photometric accuracy, and the number of processing steps in the
camera software can be substantial (Ramanath et al., 2005).
The standard definition of chromaticity, i.e., colour contents without intensity, is defined in an L
1
norm: r = {r, g, b} ≡ {R, G, B}/(R + G + B). Fig. 2(b) shows this colour content for the image.
ijcv08.tex; 29/04/2009; 8:42; p.3
4 Finlayson et al.
log(G/R)
log(B/R)
Greyscale image
Invariant direction
(a)
log(G/R)
log(B/R)
Wrong invariant direction
Greyscale image
(b)
Figure 1. Intuition for finding best direction via minimizing the entropy. (a); Log-ratio feature space values for paint patches
fall along parallel lines, as lighting is changed. Each patch corresponds to a single probability peak when projected in the
direction orthogonal to the direction of lighting change. (b): Projecting in the wrong direction leads to a 1D pdf which is less
peaked, and hence of larger entropy.
Notice that the colour of the shadow is basically a deep blue; since this is an outdoor shot on a clear day,
this is not surprising in that the light for shadowed pixels is mostly from the sky dome, whereas light for
non-shadowed pixels is comprised of both sky-light as well as direct sunlight. Thus shadowing is seen to
be an effect due to change of lighting colour as well as intensity.
The invariant greyscale is shown in Fig. 2(e) where we see the shadow is no longer present. In (Drew
et al., 2003), a 2D-colour chromaticity version of the invariant image, as in Fig. 2(f), is recovered by
projecting orthogonal to the lighting direction and keeping the 2D colour location information, and also
putting back an appropriate amount of lighting along the lighting direction. While 2(f) looks flat and the
colours somewhat false, intrinsic images created this way are useful in computer vision: e.g. see (Jiang
and Drew, 2003; Jiang and Drew, 2007).
We can use the greyscale or the pseudo-colour invariant as a guide that allows us to determine which
colours in the original, RGB, colour image are intrinsic to the scene or are simply artifacts of the shadows
due to lighting. Forming the gradient of the image’s colour channels, we can guide a thresholding step via
the difference between edges in the original and in the invariant image (Finlayson et al., 2002; Finlayson
et al., 2006). Forming a further derivative, and then integrating back, we can produce a result that is a
3-band colour image which contains all the original salient information in the image, except that the
shadows are removed, as in Fig. 2(g). Although this method is based on the invariant image, which has
shading removed, nonetheless its output is a colour image, including shading. It is worth pointing out
that we have found that in implementing this process, a 2D-colour, chromaticity, illumination-invariant
image is more well-behaved than the greyscale variant, and thus gives slightly better shadow removal.
Of course these applications sit on top of a well calibrated imaging system. We measure how the
camera responds to light and find the invariant direction accordingly. However, often in vision tasks we
do not know the provenance of the images or even if we do have a calibrated camera this calibration
does change over time. Thus, the problem we consider, and solve, in this paper is the determination of
the invariant image from unsourced imagery — images that arise from cameras that are not calibrated.
ijcv08.tex; 29/04/2009; 8:42; p.4
Entropy Minimization for Shadow Removal 5
(a) (b)
0 50 100 150
3
3.5
4
Shannon Entropy
Angle
(c)
0 50 100 150
2
3
4
5
6
7
Information Potential
Angle
(d)
(e) (f) (g)
Figure 2. Colour and intensity shift in shadows: (a): Original image; (b): L
1
chromaticity image; (c): Shannon’s entropy plot
(we seek the minimum); (d): quadratic entropy plot (we seek the maximum of the quantity plotted); (e): greyscale 1D invariant;
(f): 2D invariant L
1
chromaticity; (g): re-integrated 3D colour image.
ijcv08.tex; 29/04/2009; 8:42; p.5
