TROPERHCRAESERPAIDI
HAAR LOCAL BINARY PATTERN FEATURE
FOR FAST ILLUMINATION INVARIANT FACE
DETECTION
Anindya Roy Sébastien Marcel
Idiap-RR-28-2009
SEPTEMBER 2009
Centre du Parc, Rue Marconi 19, P.O. Box 592, CH - 1920 Martigny
T +41 27 721 77 11 F +41 27 721 77 12 info@idiap.ch www.idiap.ch
ROY, MARCEL: HLBP FEATURE FOR FAST ILLUMINATION INVARIANT FACE DETECTION 1
Haar Local Binary Pattern Feature
for Fast Illumination Invariant
Face Detection
Anindya Roy
Anindya.Roy@idiap.ch
École Polytechnique Fédérale de
Lausanne
Lausanne, Switzerland
Sébastien Marcel
Sebastien.Marcel@idiap.ch
Idiap Research Institute
Martigny, Switzerland
Abstract
Face detection is the first step in many visual processing systems like face recogni-
tion, emotion recognition and lip reading. In this paper, we propose a novel feature called
Haar Local Binary Pattern (HLBP) feature for fast and reliable face detection, particu-
larly in adverse imaging conditions. This binary feature compares bin values of Local
Binary Pattern histograms calculated over two adjacent image subregions. These subre-
gions are similar to those in the Haar masks, hence the name of the feature. They capture
the region-specific variations of local texture patterns and are boosted using AdaBoost in
a framework similar to that proposed by Viola and Jones. Preliminary results obtained on
several standard databases show that it competes well with other face detection systems,
especially in adverse illumination conditions.
1 Introduction
The main challenge for a face detection system is to successfully detect faces in an arbitrary
image, irrespective of variations in illumination conditions, background, pose, scale, expres-
sion and the identity of the person. Numerous approaches have been proposed to counter
these issues. Most of these approaches can be organized in three categories: feature-based
approaches [5], appearance-based approaches [23] and boosting-based approaches [20].
The third approach, which involves the boosting of simple local features called Haar fea-
tures in a cascade architecture, was introduced in 2001 by Viola and Jones [20]. It has
become very popular since then because it shows very good results both in terms of accuracy
and speed ( with the use of Integral Image concept ), and is quite suitable for real-time appli-
cations. Since the initial work of Viola and Jones, most of the research in face detection has
focused on the improvement of their cascade architecture. Related works can be classified
in mainly two possible directions: alternative boosting algorithms [10], [18] or alternative
architecture designs [9], [17].
However, most of these boosting-based methods which are derived from the Haar feature
set have a common limitation. This is the vulnerability of the Haar feature set to variations
in illumination conditions, for example, where there is a strong side illumination either from
c
2009. The copyright of this document resides with its authors.
It may be distributed unchanged freely in print or electronic forms.
2ROY, MARCEL: HLBP FEATURE FOR FAST ILLUMINATION INVARIANT FACE DETECTION
left or right, or the dynamic range of the image intensity varies from region to region over
the face (ref. Sec. 2.3, Fig.6). Thus, there is a need to improve the robustness of the system
to take into account these illumination variations, but retaining the richness of the feature
set, and the advantages of efficient feature selection by boosting and fast evaluation of the
features using the Integral Image concept.
The Local Binary Pattern (LBP) introduced by Ojala et al. [12] is one such operator
which is robust to monotonic illumination variations (ref. Fig. 1). Thus, various face detec-
tion systems have been proposed using LBP or its variants, such as Improved Local Binary
Patterns (ILBP) [7], Multi-Block Local Binary Patterns [25], the Modified Census Trans-
form (MCT) [4], [14] and the Locally Assembled Binary (LAB) features [22].
In this paper, we propose a new type of feature called the Haar Local Binary Pattern
(HLBP) feature which combines the advantages of both Haar and LBP. This feature com-
pares the LBP label counts in two adjacent image subregions, i.e. it indicates whether the
number of times a particular LBP label occurs in one region is greater or lesser than the num-
ber of times it occurs in another region, offset by a certain threshold. These two subregions
are represented by a set of masks similar to Haar masks [20]. Thus, our features are able to
capture the region-specific variation of local texture patterns. This makes our features more
robust to illumination variations, which may be quite complex and concentrated over certain
subregions of the image only (strong side illumination), compared to Haar and LBP individ-
ually. Since each LBP label count is actually a particular bin value of the spatial histogram
[24], our features are also robust to slight variations in location and pose.
To our knowledge, this is the first time individual LBP label counts have been combined
with Haar features for face detection. Since each HLBP feature is linked with exactly one
LBP label, there is no need to consider the entire LBP histogram in training and test, as in
[4]. Thus our system is more efficient in terms of storage requirements as well as speed
(ref. Sec.3.2). This makes it more suitable for use on mobile devices for instance. We use
a variation of the Integral Histogram [21] to calculate our features, which further increases
the speed.
We tested our proposed approach using several standard databases against two standard
face detection systems. The first is the baseline system based on Haar features [20]. The
second is the system based on MCT [4] which is one of the best performing systems repre-
senting the state of the art today.
1
The rest of the paper is organized as follows: we first introduce the proposed HLBP
features in Sec. 2. We report the experiments and discuss the results in Sec. 3. Finally,
conclusions are given in Sec. 4.
2 The Proposed Framework : Face Detection using HLBP
features
In the current work, we unite the two popular concepts of Boosted Haar features [20] and
Local Binary Patterns [12], so as to use the advantages of both in the task of face detection.
1
A public demonstration of the MCT-based face detection system can be found at
http://www.idiap.ch/onlinefacedetector .
ROY, MARCEL: HLBP FEATURE FOR FAST ILLUMINATION INVARIANT FACE DETECTION 3
Figure 1: LBP robustness to monotonic gray-scale transformations. On the top row, the
original image (left) as well as several images (right) obtained by varying the brightness,
contrast and illumination. The bottom row shows the corresponding LBP images which are
almost identical. Please see Fig. 6 for more complex illumination changes considered in our
experiments.
2.1 General Boosting Framework
The central concept of our framework (as in the Viola and Jones’ face detector) is to use
boosting, that linearly combines simple weak classifiers f
j
(I) to build a strong ensemble,
F(I) as follows :
F(I) =
n
∑
j=1
α
j
f
j
(I). (1)
The selection of weak classifiers f
j
(I) as well as the estimation of the weights α
j
are learned
by the boosting procedure. An input image I is detected as a face if F(I) is higher than
a certain threshold Θ which is also given by the boosting procedure [20] and is rejected
otherwise. Each weak classifier f
j
is associated with a weak feature, called the Haar feature
in Viola and Jones’ system. Here, instead of the Haar feature, we use a different set of weak
features which we call Haar Local Binary Pattern (HLBP) features.
2.2 The proposed HLBP features
We assume that our input is an N × M 8-bit gray-level image, which can be represented as
an N × M matrix I, each of whose elements satisfy, 0 ≤ I(x, y) ≤ 2
8
. In the first stage, we
calculate the LBP image I
LBP
[12] from the original input image I. The LBP operator can be
applied at different scales. However, after extensive preliminary testing, we have found the
LBP
4,1
operator as the optimal LBP operator in our case. At a given pixel position (x
c
, y
c
),
the LBP
4,1
operator is defined as an ordered set of binary comparisons of pixel intensities
between the center pixel (x
c
, y
c
) and its four surrounding pixels, {(x
i
, y
i
)}
3
i=0
(ref. Fig. 2).
The decimal form of the resulting 4-bit word is called the LBP code or LBP label of the
center pixel and can be expressed as,
I
LBP
(x
c
, y
c
) =
3
∑
n=0
s(I(x
n
, y
n
) − I(x
c
, y
c
))2
n
. (2)
where I(x
c
, y
c
) is the gray-level value of the center pixel (x
c
, y
c
) and {I(x
n
, y
n
)}
3
n=0
are the
gray-level values of the 4 surrounding pixels. The function s(x) is defined as,
s(x) =
(
1 if x ≥ 0,
0 if x < 0.
(3)