Illumination-Effects Compensation
in
Facial Images
A.Z.
Kouzani
School of Informatics and Engineering, Flinders University of
SA
GPO Box 2100, Adelaide,
SA
5001, Australia
egazk@flinders .edu. au
Abstract-Based on the concepts of the lin-
ear object classes and the principal compo-
nents analysis, an illumination-effects com-
pensation method is presented in this paper
to transform an arbitrary-lit face image whose
illumination effects are pre-determined, into
a
front-lit face image.
I.
INTRODUCTION
The appearance of
a
person is highly dependent on
the lighting conditions. Often slight changes in illu-
mination produce large changes in the person’s ap-
pearance. In face recognition, since the face images
in the known face database are taken under front-
lit illumination conditions, recognition of
a
face im-
age taken under
a
different illumination condition be-
comes difficult. Compensating for image variations
caused by illumination changes is therefore crucial
and can improve the recognition results. This com-
pensation is performed in two modules: illumination-
effects determination and illumination-effects com-
pensation. In the illumination-effects determination
module, illumination effects are determined in the in-
put image. In the illumination-effects compensation
module, the face image is compensated for the de-
termined illumination effects by synthesising
a
face
image under front-lit illumination conditions.
Several methods, e.g. neural networks, can be used
for illumination-effects determination [5]. This paper
deals with the illumination-effects compensation and
assumes that the illumination effects in the images
are known.
There are
a
few existing methods for illumination-
effects compensation [5]. Among these methods,
Brunelli’s illumination-compensation method
[2]
is
probably the best method. This method has been
implemented and tested by the author of this pa-
per for comparative evaluation purpose. The results
indicate that Brunelli’s method is not sufficiently ro-
bust for compensation of illumination effects. There
are several factors that make this method unreliable.
These factors are given below.
1. The feature locator is found to be sensitive to
changes in illumination.
2. The optical flow algorithm on which it is based,
fails to find correspondence between the refer-
ence image, taken under front-lit illumination
conditions, and the input image with arbitrary
illumination effects. Figure
1
shows an exam-
ple of a reference face image taken under front-
lit illumination, an input image containing some
illumination effects, the geometry-adj usted im-
age under Brunelli’s method, and the correspon-
dence vector of the reference and the adjusted
images.
3.
Given
a
perfect correspondence vector for the
reference and the input images, the final stage
of Brunelli’s method still cannot perform
a
good
illumination compensation. The residuals are
large and the corrected image is poor.
To obtain
a
better performance for compensa-
tion of illumination effects,
a
method, called the
Illumination-Effects Compensation Method (IECM)
,
is proposed in the following section using the theo-
ries of the Linear Object Classes (LOC)
[7]
and the
Principle Components Analysis (PCA)
[6].
This paper is organised as follows. In Section
11,
the proposed illumination-effects compensation
method is presented. In Section
111,
experimental
results are given. These results are then discussed in
Section IV. Finally, concluding remarks are given in
Section
V.
11.
PROPOSED
ILLUMINATION-EFFECTS
COMPENSATION METHOD
Given
a
3D object, there are
a
variety of differ-
ent ways to perform visible-surface determination,
illumination, and shading in computer graphics
[4].
The process of creating 2D images from 3D mod-
els is called
rendering.
Different rendering meth-
ods such as z-buffer, list-priority algorithm, radiosity,
ray-tracing, etc. can be found in
[4].
To use these
techniques and generate
a
face image under front-lit
illumination from
a
face image with arbitrary illumi-
nation effects, 3D models should be generated from
2D images, which is
a
difficult task for face images
containing illumination effects.
Vetter and Poggio
[7]
proposed
a
simpler tech-
0-7803-5731-0/91k%10.00 01999
IEEE
VI--840
Fig.
1.
Brunelli’s correspondence extraction method. (a) Reference face image.
(b)
Input image. (c) Geometry adjusted image.
(d) Correspondence vector
of
the reference and the adjusted images. (e) Enlarged part of the correspondence vector.
nique, called
linear object classes,
that can be used
under restricted conditions. This technique is based
on
2D
models and does not need any depth informa-
tion. This technique can provide additional artificial
example images of an object when only a single im-
age is given, and is originally proposed for generating
a
novel pose of an object from a different pose of the
object. The image transformations that are specific
to the relevant object class are learnt from example
poses of other prototypical objects of the same class.
There are 3D objects whose 3D shapes can be
represented as
a
linear combination of
a
sufficiently
small number
of
prototypical objects. Linear object
classes have the property that new poses of any ob-
ject of the class under uniform affine
3D
transforma-
tions can be generated exactly if the corresponding
transformed poses are known for the set of proto-
types.
Vetter-Poggio’s method involves computation of
the correspondence between the images of the ob-
jects, linear decomposition of the correspondence
field of the new images into the correspondence fields
of
the examples, similar decomposition of new tex-
ture into the example textures, and synthesis of the
new image. This method has only been used for syn-
thesising face images of novel pose.
It
has not been
applied to the synthesis of face images
of
novel illu-
mination.
The main limitation
of
this method, however,
is
the existence of linear object classes and the com-
pleteness of the available examples. This is equiv-
alent to whether object classes can be modelled
through linear object classes. Presently there is no
final answer to this question, apart from simple ob-
jects where the dimensionality is given through their
mathematical definitions. The application of the
method to a small example set of human faces, pro-
vides preliminary promising results
at
least for some
faces. Based on their experiments, Vetter and Pog-
gio have concluded in
[7]
that the linear object class
method may be a satisfactory approximation even
for complex objects such as faces. Given
a
linear
space such as the face space, one can choose among
different sets
of
basis vectors that will span the same
space. The basis set used by Vetter and Poggio is the
original set of images themselves. However, other po-
tential basis set can be used
as
well.
One popular method for choosing the basis set is
VI-841
applying the PCA to the example set. Since the ba-
sis set found by the PCA is orthogonal (and can be
easily normalised to be made orthonormal), recon-
struction of new images can be performed if
suffi-
cient number of example face images are used. This
orthogonality produces
a
more stable set of linear
coefficients
[l].
This may produce a better approxi-
mation of
a
different face image.
The IECM synthesises
a
front-lit face image from
an arbitrary-lit input face image. The method
utilises the theories of the LOC and the PCA. The
proposed method is described as follows.
Algorithm
1:
(Illumination-Effects Compen-
sation)
The illumination-effects compensation process is
performed in two stages as described in the following.
Training:
In the training stage the following op-
erations are performed.
1.
One training set for each class of illumination
effects is constructed using as many example im-
ages
as
possible.
2.
An image is manually selected from each train-
ing set and is named the reference image of
the set. Although this selection is an arbitrary
choice, the employed principle is that the face
should be located in the centre of the image.
3.
All images of each training set are aligned based
on the associated reference image.
4.
The PCA is applied to each training set, and
the basis images are generated.
Synthesis:
Given the illumination effects, the
following operations are performed in the synthesis
stage.
1.
A
set of weights
w
is calculated based on the
input image
Ii,
and the basis images. This is
done by projecting the input image onto each
basis image of the training set representing sim-
ilar illumination effects as that of the input im-
age.
2.
A face image
Iout
is synthesised under front-
lit illumination using the calculated weights and
the basis images of the training set representing
front-lit illumination.
3.
A set
of
weights
w
is calculated based on
the synthesised image
lout
and the basis im-
ages. This is done by projecting the image onto
each basis image of the training set representing
front-lit illumination.
4.
A face image
I;,
is synthesised under the illu-
mination effects similar to the input image using
the calculated weights
w
and the basis images
of the training set representing the similar illu-
mination effects as that of the input image.
5.
The sum of squared difference error between
Iin
and
Ti,
is obtained using
where
pk
is the mean image of the training Set
with the similar illumination effects as that of
the input image,
N
is the number of example
images in that training set, and
bi
represents the
basis images obtained by performing the PCA
on that training set.
6.
The error
E
is minimised by modifying
w
using
the conjugate-gradient algorithm
[8].
7.
A jump is performed to Step
2
unless the error
stops decreasing or
a
fixed number of iterations
are reached.
To develop
a
practical system using the process
described above, example face images that contain
different lighting conditions are required. Using
a
3D
head database of
63
people, face images with
66
illumination-effects per person are generated and
grouped into
66
illumination-effects classes. This
classification is done based on the lighting conditions
of each face image. In each image, specific direction
and distance of
a
single light source are implemented.
The longitudinal and latitudinal of the light source
directions are within
15"
-
75"
degrees of the camera
axis. Alignment is performed to adjust scale, orien-
tation, and position of faces within images.
In order to minimise the error function, any stan-
dard minimisation algorithm could be used. There
are several different methods for optimisation. They
can be divided into two main
groups:
gradient and
non-gradient methods. The gradient methods are
characterised by the fact that they use derivatives
of the object function, either explicitly or numeri-
cally obtained, to find the optimal solution. The
non-gradient methods, however, have other ways of
finding
a
search path to the optimal solution.
The conjugate-gradient method
[3]
is
a
minimisa-
tion technique to find the lowest energy. The method
is similar to
a
steepest-descent minimisation. The
procedure for the steepest-descent minimisation is
as follows: (i) the gradient is calculated in all di-
rections, (ii) the path of steepest descent is chosen
-
this is in the form of
a
straight line but makes
measurements of the energy
at
small increments, (iii)
when the energy stops decreasing the process is re-
peated. The conjugate-gradient minimisation adopts
VI
-
842
the same procedure, but also uses information from
the previous iteration steps to select the optimal
route.
111.
EXPERIMENTAL
RESULTS
Four experiments are carried out to evaluate the
performance of the IECM. Both the IECM and
Brunelli’s method
[2]
are implemented and their per-
formances are compared. The test set used in these
experiments is built as follows.
Fig.
2.
Four
illumination
masks
used
in creation
of
the test
set.
Test
Set:
A set of front-view face images of 80
subjects taken under front-lit illumination conditions
are collected by the author. The face images are
128
x
128 gray-level, each containing only the face
area of
a
subject. The images are all aligned. Four
128
x
128 gray-level images are generated by com-
puter as illumination masks. In each mask, a specific
illumination effect is implemented. These illumina-
tion masks are shown in Figure
2.
Using the
80
face
images and the four illumination masks,
a
test set of
400
face images are created. This test set consists
of five subsets. Subset
1
contains
80
original front-lit
face images. Each of Subsets 2-5 contains 80 face im-
ages generated by superimposing each illumination
mask on each face image of Subset
1.
Before su-
perimposing an illumination mask on
a
face image,
the illumination mask is weighted using one of the
five pre-selected weights
-
1.0, 0.93, 0.89, 0.84, and
0.8,
randomly. The pre-selection has been conducted
in such a way as to degrade the synthesised images’
brightnesses by up to
20%.
The weighting operation
is performed in order to include more variations in
illumination condition by producing different illumi-
nation effects from each mask, randomly. Example
images of Subset
4
are demonstrated in Figure
3.
Four experiments are performed on this test set.
In each experiment, the face images of Subset
1
and
one of Subsets 2-5 are used. In the following, the
procedure for Experiment
1
is described. The proce-
dure for Experiments 2-4 are the same except that
one
of
Subsets 3-5 is used instead of Subset
2.
The percentage errors obtained from Experiments
1-4 are summarised in Table
I
for Brunelli’s method
and for the IECM. A discussion of the obtained re-
sults is given in the next subsection.
IV. DISCUSSIONS
According to Table
I,
the proposed IECM outper-
forms Brunelli’s method in all experiments. While
the percentage error for Brunelli’s method rises sig-
nificantly to
83%
when three light sources are used,
the percentage error for the IECM increases only
slightly to 23% for the same experiment. The IECM
achieves significantly lower percentage errors in all
experiments for the synthesised images than those of
Brunelli’s method. In other words, the corrected im-
ages generated by the IECM have higher similarities
with their original images than those generated by
Brunelli’s method.
The
IECM
is confronted with
a
problem which is
the completeness of the available example face im-
ages. The results discussed above are based on the
experiments performed on a small test set. The test
set
did not contain all the possible illumination
ef-
fects. A simple solution to the completeness problem
would be the inclusion of more face images contain-
ing all possible illumination effects in the ensembles
of face images.
V. CONCLUSIONS
An illumination-effects compensation method
(IECM) is proposed in this paper for dealing with
face image variations that are due to illumination
changes. In the IECM, the determined illumination
effects in the face image are compensated for by syn-
thesising
a
face image under front-lit illumination
conditions. The IECM utilises the theories of the
linear object classes and the principal components
analysis.
VI
-
843
Fig.
3.
Example images from Subset
4.
Brunelli
TABLE
I
PERCENTAGE ERRORS OBTAINED FROM
EXPERIMENTS
1-4
”
3
1
1
and
2 62
2
1
and
3
65
11
Method
11
ExDeriment
I
Subsets
I
Pcrcentarre
Error
11
J
1
1
and
2
14
IECM
2
1
and
3
17
3
4
1
1
and
4
1
71
I
1
and
5
I
83
1
and
4
17
1
and
5 23
E
ACKNOWLEDGEMENTS
[31
The first author would like to thank
DEETYA,
Australia, and FUSA, for providing him with schol-
arships to undertake this research.
[4]
[51
PI
REFERENCES
D.
Beymer, “Vectorizing face images by interleaving shape
and texture computations”, Tech. Rep.
1537,
MIT AI
[7]
Lab., September
1995.
R. Brunelli, “Estimation
of
pose and illumination direc-
tion for face processing”, Tech. Rep. TR-AI
1499,
Mas-
sachusetts Institute of Technology, November
1994.
[8]
R.
Fletcher and
C.
Reeves, “Function minimization by
conjugate gradients”,
Computer Journal,
pp.
81-84,
1964.
J.D.
Foley, A.V. Dam,
S.K.
Feiner, and J.F. Hughes,
Computer Graphics
:
Principles and Practice,
Addison-
Wesley,
2
edition,
1996.
A.Z. Kouzani,
Invariant Face Recognition,
PhD thesis,
School of Informatics and Engineering, Flinders University
of
SA, Adelaide, South Australia, Australia,
1999.
G.J.
McLachlan,
Discriminant Analysis and Statistical
Pattern
Recognition,
Wiley, New York,
1992.
T. Vetter and
T.
Poggio, “Linear object classes and image
synthesis from
a
single example image”, Tech. Rep.
1531,
MIT AI Lab.,
1995.
P. Viola, “Alignment by maximization of mutual informa-
tion’’, Tech. Rep.
1548,
MIT AI Lab.,
1995.
VI
-
844