![Fig. 8(b) and (c) shows sample cases when the two classifiers are in conflict but the proposed sequential fusion algorithm correctly accepts the subjects while the other fusion algorithms (sum rule, SVM fusion, and PLR fusion) provide incorrect results. Finally, there are few cases [sample shown in Fig. 8(d)] when both the classifiers reject a genuine subject. In such cases, the fusion algorithms cannot do much to improve the performance and, therefore, a 100% accuracy is not achieved.](/figures/fig-8-b-and-c-shows-sample-cases-when-the-two-classifiers-8oeu5o4j.webp)
![Fig. 8(b) and (c) shows sample cases when the two classifiers are in conflict but the proposed sequential fusion algorithm correctly accepts the subjects while the other fusion algorithms (sum rule, SVM fusion, and PLR fusion) provide incorrect results. Finally, there are few cases [sample shown in Fig. 8(d)] when both the classifiers reject a genuine subject. In such cases, the fusion algorithms cannot do much to improve the performance and, therefore, a 100% accuracy is not achieved.](/figures/fig-8-sample-cases-from-the-labeled-faces-in-the-wild-1hj2mi6y.webp)
![Fig. 3. Illustrating examples of quality vector on images from the LFW database [9].](/figures/fig-3-illustrating-examples-of-quality-vector-on-images-from-3fg7tu9l.webp)
470 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 5, NO. 3, SEPTEMBER 2010
On the Dynamic Selection of Biometric Fusion
Algorithms
Mayank Vatsa, Member, IEEE, Richa Singh, Member, IEEE, Afzel Noore, Member, IEEE, and
Arun Ross, Member, IEEE
Abstract—Biometric fusion consolidates the output of multiple
biometric classifiers to render a decision about the identity of an
individual. We consider the problem of designing a fusion scheme
when 1) the number of training samples is limited, thereby af-
fecting the use of a purely density-based scheme and the likelihood
ratio test statistic; 2) the output of multiple matchers yields con-
flicting results; and 3) the use of a single fusion rule may not be
practical due to the diversity of scenarios encountered in the probe
dataset. To address these issues, a dynamic reconciliation scheme
for fusion rule selection is proposed. In this regard, the contribu-
tion of this paper is two-fold: 1) the design of a sequential fusion
technique that uses the likelihood ratio test-statistic in conjunction
with a support vector machine classifier to account for errors in
the former; and 2) the design of a dynamic selection algorithm that
unifies the constituent classifiers and fusion schemes in order to op-
timize both verification accuracy and computational cost. The case
study in multiclassifier face recognition suggests that the proposed
algorithm can address the issues listed above. Indeed, it is observed
that the proposed method performs well even in the presence of
confounding covariate factors thereby indicating its potential for
large-scale face recognition.
Index Terms—Biometrics, face verification, match score fusion,
support vector machine (SVM).
I. INTRODUCTION
T
HE paradigm of information fusion, which entails the
consolidation of evidence presented by multiple sources,
has been successfully used to enhance the recognition accuracy
of biometric systems. The use of multiple pieces of evidence in
order to deduce or verify human identity is often referred to as
multibiometrics. While biometric fusion can be accomplished
at several different levels in a biometric system [18]—viz.,
data-level, feature-level, score-level, rank-level, and deci-
sion-level—fusion at the match score level has been extensively
studied in the literature. Fusion at the match score level involves
Manuscript received November 15, 2009; revised June 14, 2010; accepted
June 16, 2010. Date of publication July 08, 2010; date of current version Au-
gust 13, 2010. This work was supported in part by research grants from the Army
Research Laboratory (Award W911NF-10-02-0021). The work of A. Ross was
also supported by NSF CAREER Grant IIS 0642554. The associate editor co-
ordinating the review of this manuscript and approving it for publication was
Prof. Davide Maltoni.
M. Vatsa and R. Singh are with the Indraprastha Institute of Information
Technology (IIIT) Delhi, New Delhi 110078, India (e-mail: mayank@iiitd.ac.in;
rsingh@iiitd.ac.in).
A. Noore and A. Ross are with the Lane Department of Computer Science
and Electrical Engineering, West Virginia University, Morgantown, WV 26505
USA (e-mail: afzel.noore@mail.wvu.edu; arun.ross@mail.wvu.edu).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIFS.2010.2056683
combining the match scores generated by multiple classifiers
(or matchers) in order to render a decision about the identity
of the subject. There are different schemes for performing
score level fusion based on different models. These include
density-based fusion schemes where the model is based on
estimating density functions for the genuine and impostor score
distributions; transformation-based fusion schemes where the
model is based on estimating normalization functions; and
classifier-based fusion schemes where the model is a classifier.
While match score fusion has been demonstrated to be effec-
tive [18], [22], its matching performance is compromised under
several scenarios.
1) Density-based score fusion schemes [18] which use the
likelihood ratio test to formulate the fusion rule can be af-
fected by the use of incorrect density functions for the gen-
uine and impostor scores. The use of parametric methods of
density estimation can be based on the assumption of incor-
rect models (e.g., Gaussian densities for both genuine and
impostor scores) that can lead to suboptimal fusion rules;
the use of nonparametric methods, on the other hand, is
affected by the availability of a small number of training
samples (especially genuine scores) thereby impacting the
feasibility of designing an effective fusion rule.
2) Classifier-based fusion schemes [2] are susceptible to over-
training on one hand and classifier bias on the other [4],
[27]. Further, a pure data-driven approach will not be able
to accommodate scenarios that are not represented in the
training data. For example, when conflicting scores from
multiple matchers are presented to the fusion classifier,
then, in the absence of sufficient training samples repre-
senting such a scenario, an incorrect decision may be reg-
ularly rendered.
Training and using a single fusion rule—whether it be the
simple sum rule or the likelihood ratio-based fusion rule—on
the entire probe dataset may not be appropriate for the rea-
sons stated above. Further, component classifiers can render
conflicting decisions that can impact the performance of fu-
sion schemes such as the simple sum rule. To address these is-
sues and, subsequently, improve the verification performance of
a biometric system, we propose a sequential fusion algorithm
which combines a density-based fusion scheme with a classi-
fier-based scheme. The first contribution lies in using a sup-
port vector machine (SVM) classifier in conjunction with the
likelihood ratio test statistic. The likelihood ratio aspect of the
algorithm helps in modeling the underlying class distribution
using simple Gaussian mixture models; the statistical and geo-
metrical properties of SVM [14], [15], [23] ensures that there is
1556-6013/$26.00 © 2010 IEEE
VATSA et al.: ON THE DYNAMIC SELECTION OF BIOMETRIC FUSION ALGORITHMS 471
Fig. 1. Block diagram illustrating the steps involved in the proposed sequential match score fusion algorithm.
a “correction” of the decision rendered by the likelihood ratio
test statistic. By employing a simple model to characterize the
genuine and impostor density functions, the requirement for a
large number of training samples is avoided.
The sequential nature of the proposed fusion algorithm makes
it computationally expensive. The fusion algorithm may not be
required if the probe image is of high quality and exhibits suf-
ficient biometric information useful for recognition using only
one biometric classifier. Further, simple fusion rules such as sum
rule with minimum/maximum (min/max) normalization can be
used for most of the probe cases when multiclassifier biometric
output is not highly conflicting. One way to improve the ver-
ification accuracy, without increasing the computational cost,
is to develop a context switching scheme that dynamically se-
lects the most appropriate classifier or fusion algorithm for the
given probe. The second contribution of this work is the design
of an algorithm for the dynamic selection of constituent uni-
modal biometric classifiers or match score fusion algorithms
that not only improves the verification accuracy but also de-
creases the computational cost of the system. In a two-class,
biclassifier biometric system, the dynamic selection algorithm
uses quality information (not based on match scores) to select
one of four options: 1) first biometric classifier only, 2) second
biometric classifier only, 3) sum rule with min/max normaliza-
tion, and 4) sequential match score fusion. The selected option
is then used to render the final decision.
The performance of the proposed algorithm is evaluated in
the context of a face recognition application to mitigate the
effect of covariate factors such as pose, expression, illumi-
nation, and occlusion. Match scores computed from two face
recognition algorithms, namely local binary pattern (LBF) [3]
and neural network architecture-based 2-D log polar Gabor
transform (2DG-NN) [20], are fused and the verification
performance is compared with existing match score fusion
algorithms. Experiments indicate that the proposed fusion
architecture efficiently improves the verification performance
without increasing the computational cost.
II. P
ROPOSED SEQUENTIAL MATCH SCORE
FUSION ALGORITHM
Fig. 1 shows the steps involved in the proposed fusion al-
gorithm that consists of two steps: 1) match score fusion and
2) classification. First, the match scores are transformed into be-
lief assignments using density estimation schemes. In the next
step, a belief model is used for fusion and finally, the likeli-
hood ratio test statistic and SVM are used for classification.
Throughout the paper, we use
to represent the first biometric
classifier and
to represent the second biometric classifier.
A. Match Score Fusion
For a two class problem, let
, where
represents the genuine class and represents the impostor
class. The first step in the sequential fusion algorithm is to trans-
form match scores into belief assignments. A multivariate den-
sity estimation technique is used to compute belief assignments
induced by the match scores because previous literature has
shown the usefulness of mixture models in biometrics [18]. The
multivariate Gaussian density function [7] can be written as
(1)
where
is a vector with components, is the mean vector,
and
is the covariance matrix. Let be the conditional
joint density of
match scores and . is computed
using
(2)
where
, , and are the mean vector, covari-
ance matrix, and weight factor, respectively, corresponding to
the
th mixture component in the conditional joint density. Also,
and is the number of mixture com-
ponents used to model the density. A recursive algorithm [29]
is used to estimate the parameters of the mixture model.
Let
be the match score vector, where
is the match score computed by the th biometric classifier or
matcher. To mitigate the effect of curse-of-dimensionality and
for faster computation, we assume independence among con-
stituent matchers and compute the marginal density
of the th classifier. The belief assignment for the th clas-
sifier is computed using
(3)
where
is the verification accuracy prior of the th
classifier that is used as the ancillary information to estimate
the beliefs. With the help of (3), the belief assignments for
individual biometric classifiers are computed. For example,
in a two-class two-classifier biometric system, we compute
and .
The belief assignments of biometric classifiers are then fused
using the proportional conflict redistribution rule [6]. In this
rule, redistribution of the conflicts is performed only on those
472 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 5, NO. 3, SEPTEMBER 2010
elements which are involved in each conflict and is done ac-
cording to the proportion/weight of each classifier. The belief
assignments of classifiers
and are fused using
(4)
Here
, , , and and are the belief model
weight factors
. and denote the
belief assignments of classifier 1 and classifier 2, respec-
tively, computed using (3).
is a vector with values
1
representing the fused belief.
In (4), the first term denotes the degree of conflict between the
classifiers and the formulation effectively combines the beliefs
of multiclassifier match scores.
B. Classification
First, the fused belief assignments induced from
match scores are converted into the likelihood ratio
. Next, the likelihood ratio
is used as input to the SVM classifier for decision making
as shown in (5). Utilizing the SVM with likelihood ratio for
decision-making ensures that the algorithm is less prone to
over-fitting and addresses the nonlinearity in the biometric
match scores
if
otherwise.
(5)
Here
is the decision threshold chosen for a specific false accept
rate (using the concept of SVM regression). The advantage of
this approach is its control over the false accept and false reject
rates, and it also satisfies the Neyman–Pearson criteria [10] for
decision making.
III. D
YNAMIC SELECTION OF
CONSTITUENT BIOMETRIC
CLASSIFIERS AND
FUSION
ALGORITHMS
When encountering a good quality gallery-probe pair,
2
an ef-
ficient classifier can verify the identity without the need for fu-
sion. For cases when the two biometric classifiers have minor
conflicts, the sum rule with min/max normalization [18] can ef-
fectively fuse the match scores and yield correct results with
much less time complexity. The sequential fusion rule is used
to perform fusion when individual classifiers are prone to gen-
erate conflicting or ambiguous decisions, i.e., cases involving
uncertainties. In our previous research, we introduced an adap-
tive framework that reconciles match score fusion algorithms
to improve the verification performance both in terms of accu-
racy and time [24]. The concept behind the framework is to
dynamically select an optimal fusion algorithm for the given
1
m
(
)=
m
(
)
m
(
)+
w m
(
)
m
(
)
=
m
(
)+
m
(
)+
w m
(
)
m
(
)
=m
(
)+
m
(
)
;
m
(
)=
m
(
)
m
(
)+
w m
(
)
m
(
)
=
m
(
)+
m
(
)+
w m
(
)
m
(
)
=m
(
)+
m
(
)
.
2
The term gallery-probe pair is used to denote that, in the verification mode,
a probe is compared against a gallery.
probe image. In other words, the algorithm selects a complex fu-
sion algorithm only when there is uncertainty in the constituent
match scores; otherwise, it selects a simple fusion algorithm.
In this paper, we extend the framework to reconcile constituent
biometric classifiers (e.g., two face recognition algorithms in a
multiclassifier system) with the proposed sequential fusion al-
gorithm and the sum rule in order to optimize both verification
accuracy and computational time. Fig. 2 illustrates the steps in-
volved in the proposed dynamic selection algorithm. The algo-
rithm is explained in the context of face recognition but it can
be easily generalized to any multibiometric scenario.
Input to the dynamic selection algorithm is a quality vector
which is a quantitative representation of biometric information
pertaining to the gallery-probe pair. In the context of face recog-
nition, the quality vector consists of quality score, visual activity
level, and pose of the face image. The quality vector
is
computed using the following approach.
• To encode the facial edge information and noise present
in the image, a redundant discrete wavelet transformation
(RDWT)-based quality assessment algorithm [25] is used
that provides both frequency and spatial information. A
face image
of size is decomposed into three levels
of the RDWT, i.e.,
. Let repre-
sent the approximation, horizontal, vertical, and diagonal
subbands, respectively. The RDWT decomposition can be
written as
(6)
The image quality score
is computed using (7).
(7)
where
(8)
and
(9)
Here,
and are the mean and standard deviation of
the RDWT coefficients of the
th subband and the th level,
respectively, and
denotes the gradient operator. Finally,
the quality score
is normalized in the range using
min/max normalization [18] (0 represents the worst quality
and 1 the best quality) and used as the first element in the
quality vector.
• Image properties such as brightness and contrast can be
encoded using the visual activity level which is computed
VATSA et al.: ON THE DYNAMIC SELECTION OF BIOMETRIC FUSION ALGORITHMS 473
Fig. 2. Dynamic selection of biometric classifiers and fusion algorithms in the context of a face recognition application.
Fig. 3. Illustrating examples of quality vector on images from the LFW database [9].
using (10), shown at the bottom of the page. Activity level
is then normalized in the range and used as the
second element in the quality vector. A higher activity level
represents properly illuminated and contrast normalized
image.
• In face recognition, pose variations can reduce the amount
of overlapping biometric features required for recognition.
Therefore, it is important to include the head position or
angle as a pose parameter in the quality vector. In this re-
search, a fast single view algorithm [13] is used for esti-
mating the pose of a face image. The output of the algo-
rithm is the pose angle
which serves as the third element
in the quality vector.
Fig. 3 shows examples of the image quality vector on the
LFW face database [9]. In the dynamic selection algorithm, if
the quality of gallery-probe pair is high then the constituent clas-
sifiers are used; if not, the fusion rules are chosen. The proposed
algorithm uses three SVMs to select from the two classifiers
(10)
474 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 5, NO. 3, SEPTEMBER 2010
Fig. 4. Illustrating the steps involved in match score fusion of a multiclassifier face recognition system.
and the two fusion algorithms. In this research, we use LBP [3]
and 2DG-NN [20] based face recognition algorithms as the con-
stituent classifiers, and the sum rule with min/max normaliza-
tion and the proposed sequential fusion as the two fusion algo-
rithms. As shown in Fig. 2, the first SVM, denoted as SVM
,is
used to select between the classifiers and the fusion rules. If the
classifiers are selected, then the second SVM, denoted as SVM
,
is used to choose between LBP and 2DG-NN face recognition
algorithms. If the option pertaining to fusion rules is selected,
then the match scores from LBP and 2DG-NN are computed
and the third SVM, denoted as SVM
, is used to select between
the sum rule and sequential fusion. The dynamic selection algo-
rithm is divided into two stages: training the SVMs and dynamic
selection of algorithms for every probe instance.
1) Training SVMs: Three SVMs are independently trained
using the labeled training database. The training procedure is
explained as follows.
a) SVM
is trained using the labeled training data .
Here,
is the quality vector belonging to the th training
gallery-probe pair, i.e.,
.
is the respective label such that is as-
signed when the gallery-probe pair is of high quality and
can be correctly matched using individual classifiers and
is assigned to the pair that requires match score fusion.
At the end of the training stage, a nonlinear decision hy-
perplane is learned that can select between the individual
classifiers and match score fusion.
b) SVM
is trained using the labeled training data ,
where
is the quality vector belonging to the th
training gallery-probe pair and
. In this
case,
indicates the gallery-probe pair that can be
matched using the LBP classifier and
is assigned
to the data that requires matching using the 2DG-NN
classifier. A nonlinear decision hyperplane is learned that
can select either the LBP or the 2DG-NN.
c) SVM
is trained using the labeled training data .
Here,
is the th training data vector that contains
match scores and verification accuracy priors pertaining
to the two classifiers, and
is the label such
that
belongs to match scores that should be fused
using the sum rule with min/max normalization and
belongs to the match scores that should be fused using
the sequential fusion algorithm. The SVM is trained such
that an output of SVM
indicates the use of the sum
rule and SVM
indicates the use of the sequential
fusion algorithm.
2) Dynamic Selection of Algorithms: For probe verification,
the trained SVMs are used to dynamically select the most ap-
propriate algorithm depending on the quality vector.
a) The quality vectors pertaining to both the gallery and
probe images are provided as input to the trained SVMs.
The SVM
classifier selects between using a single clas-
sifier and fusion.
b) Depending on the classification result of the SVM
clas-
sifier, SVM
and SVM
are used to select one of the four
options: 1) LBP, 2) 2DG-NN, 3) sum rule with min/max
normalization, and 4) sequential fusion.
IV. R
EDUCING THE EFFECT OF COVARIATE FACTORS IN FACE
RECOGNITION USING MATCH SCORE FUSION
There are several global, local, nonlinear, appearance-based,
texture-based, and feature-based face recognition algorithms
[11], [26], [28]. These algorithms independently attempt to
reduce the effect of covariate factors such as expression, illu-
mination, pose, and occlusion on the recognition performance.
However, most of the existing algorithms are optimized to
mitigate the effect of specific covariates. For example, the
neural network architecture-based 2DG-NN algorithm [20] can
tolerate variations in expression, illumination, and occlusion
whereas local facial features can handle pose and expression
variations. It is our assertion that the performance of a face
recognition system can be greatly enhanced if information from
multiple algorithms is fused and a final decision is obtained
using the fused information. In this section, we use the sequen-
tial fusion and dynamic selection algorithms to fuse the match
scores computed from a nonlinear face recognition algorithm
and a local facial feature based algorithm to mitigate the effect
of covariate factors.
As shown in Fig. 4, two face classifiers (
and ) are used
for feature extraction and matching. The match scores com-
puted using these classifiers are combined using the proposed
sequential fusion and dynamic selection algorithms. First, the
face region from the input image is detected using the triangle-
based face detection algorithm [21]. The size of the detected
face image is normalized to 128
96. Next, the following algo-
rithms are used for feature extraction and matching.