HAL Id: hal-01247885
https://hal.inria.fr/hal-01247885
Submitted on 22 Dec 2015
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entic research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diusion de documents
scientiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
What else does your biometric data reveal? A survey on
soft biometrics
Antitza Dantcheva, Petros Elia, Arun Ross
To cite this version:
Antitza Dantcheva, Petros Elia, Arun Ross. What else does your biometric data reveal? A survey on
soft biometrics. IEEE Transactions on Information Forensics and Security, Institute of Electrical and
Electronics Engineers, 2015, 11 (3), pp.441 - 467. �10.1109/TIFS.2015.2480381�. �hal-01247885�
1
What Else Does Your Biometric Data Reveal? A
Survey on Soft Biometrics
Antitza Dantcheva, Petros Elia, Arun Ross
Abstract—Recent research has explored the possibility of
extracting ancillary information from primary biometric traits,
viz., face, fingerprints, hand geometry and iris. This ancillary
information includes personal attributes such as gender, age,
ethnicity, hair color, height, weight, etc. Such attributes are
known as soft biometrics and have applications in surveillance and
indexing biometric databases. These attributes can be used in a
fusion framework to improve the matching accuracy of a primary
biometric system (e.g., fusing face with gender information), or
can be used to generate qualitative descriptions of an individual
(e.g., “young Asian female with dark eyes and brown hair”). The
latter is particularly useful in bridging the semantic gap between
human and machine descriptions of biometric data. In this paper,
we provide an overview of soft biometrics and discuss some of the
techniques that have been proposed to extract them from image
and video data. We also introduce a taxonomy for organizing
and classifying soft biometric attributes, and enumerate the
strengths and limitations of these attributes in the context of an
operational biometric system. Finally, we discuss open research
problems in this field. This survey is intended for researchers
and practitioners in the field of biometrics.
Index Terms—Soft biometrics, Biometrics, Computer Vision,
Gender, Age, Ethnicity, Race, Cosmetics, Privacy, Semantics,
Visual Attributes
I. INTRODUCTION
A. Biometrics
Biometrics is the science of recognizing individuals based
on their physical, behavioral, and physiological attributes such
as fingerprint, face, iris, gait and voice [111]. A classical
biometric system acquires biometric data from an individual
(e.g., a fingerprint image), extracts a set of features from
the data, and compares this feature set with templates in the
database in order to verify a claimed identity or to determine
an identity.
While biometric data is typically used to recognize individ-
uals, it is possible to deduce other types of attributes of an
individual from the same data. For example, attributes such
as age, gender, ethnicity, height, hair color and eye color
can be deduced from data collected for biometric recognition
purposes. Recent work [270] has established the possibility
Copyright (c) 2013 IEEE. Personal use of this material is permitted.
However, permission to use this material for any other purposes must be
obtained from the IEEE by sending a request to pubs-permissions@ieee.org
During the preparation of this manuscript, A. Dantcheva was supported
as a Postdoctoral Fellow at West Virginia University and Michigan State
University, and as an ERCIM “Alain Bensoussan” Fellow at INRIA funded by
the European Union Seventh Framework Programme (FP7/2007-2013) under
grant agreement # 246016. A. Ross was funded in part by US NSF CAREER
Award # IIS 0642554.
A. Dantcheva is with the STARS team of Inria, France. A. Ross is with the
Department of Computer Science and Engineering, Michigan State University.
P. Elia is with the Mobile Communications Department at Eurecom, France.
E-mail: Antitza.Dantcheva@inria.fr, elia@eurecom.fr, rossarun@cse.msu.edu
of computing the body mass index (BMI) from face images,
thereby suggesting the possibility of assessing health from
biometric data.
B. Soft Biometrics
These additionally deduced attributes, while not necessarily
unique to an individual, can be used in a variety of applica-
tions. Further, they can be used in conjunction with primary
biometric traits in order to improve or expedite recognition
performance.
Fig. 1. Anthropometry card of Alphonse Bertillon, who originated the
criminal identification system based on profile and full-face photos, and key
body measurements (1892). These key measurements include body height,
body weight, build, complexion, head length, head width, cheek width,
measurements of right ear and left foot, as well as “peculiar marks” such
as birthmarks, scars, and tattoos.
It is perhaps this latter application that has led to these
attributes being referred to as soft biometrics [109], [110],
[180] or light biometrics [4]. In this context, soft biometrics
can be traced back to Bertillon [211] (see Figure 1), who
brought to the fore the idea of using anatomical, morphological
and anthropometrical characteristics for person identification.
These attributes have also been referred to as semantics [223],
[207], in reference to their semantic interpretation (e.g., de-
scribing a face as “young male”).
1) Scope and benefits: Various researchers have attempted
to define the scope of soft biometrics. Jain et al. [109] defined
soft biometrics to be the set of characteristics that provide
some information for recognizing individuals, but that are not
capable of distinguishing between individuals, mainly due to
their lack of distinctiveness and permanence. Samangooei et al.
[223], as well as Reid and Nixon [208], further associated soft
To appear in IEEE Transactions on Information Forensics and Security (TIFS), 2015
2
biometrics with labels which people use to describe each other:
an association that nicely bridges the gap between human and
machine descriptions of biometric data.
Combining the above with the ideas in Dantcheva et al. [42],
and keeping in mind that such soft traits can go beyond person
recognition, one could define soft biometrics as follows. Soft
biometric traits are physical, behavioral, or material acces-
sories, which are associated with an individual, and which can
be useful for recognizing an individual. These attributes are
typically gleaned from primary biometric data, are classifiable
in pre-defined human understandable categories, and can be
extracted in an automated manner.
Age: 25-45
G
ender: female
H
air color: blond
Hair
style: long, hair
down
B
ody type:
Hourglass
shape
C
lothes: black T-shirt
A
ccessories: airport
cart
Fig. 2. Importance of soft biometrics. Typical video surveillance scenario:
when faces are of low resolution, appear in different poses, and are either
occluded or not visible, other attributes such as age, gender, hair color and
style, height, body type, clothes and accessories can be used for identification
and re-identification. Image obtained from PETS 2007 [63].
!
"
!
Fig. 3. Ancillary information, referred to as soft biometrics,
can be gleaned
from the following biometric modalities: face, iris, fingerprint, gait, body,
hand, voice.
a) Benefits of soft biometrics: Soft biometrics are often
descriptive and have a semantic representation. In addition -
as noted by Jain et al. [110] - they can be inexpensive to
compute, discerned at a distance in a crowded environment,
and require less or no cooperation of the observed subjects.
To elaborate, we note the following benefits.
Human understandable interpretation: Soft biometric at-
tributes have a semantic interpretation, in the sense that that
they can provide a description that can be readily understood
by humans; for example the description “young, tall, female”.
This makes them particularly useful in applications such as
video surveillance, where they are directly compatible with
how humans perceive their surroundings [210], [67], [43],
[204], [47], [257], [45], [46]. In other words, when a human
attempts to verbally describe a person, obvious characteristics
regarding the person’s appearance such as gender, age, height
and clothes color are often used (e.g., in police reports).
This allows soft biometrics to be used in applications where
traditional biometrics may be insufficient, as is argued, for
example, by Klontz and Jain [125] in the case of the 2013
Boston bombings.
Robustness to low data quality: Some soft biometric at-
tributes can be deduced from low-quality biometric data (see
Figure 2. In this context, such attributes can be extracted,
when primary biometric data is not conclusive, due to poor
acquisition quality. For example, if the input iris image is
of poor quality, one could utilize the surrounding periocular
information to perform recognition, rather than relying on the
iris itself.
Consent-free acquisition: Soft biometrics can often be cap-
tured without the consent and cooperation of the observed
subject. For example, information about a person’s height or
gender can be deduced from a distance.
Privacy: Since soft biometric traits are not distinctive, they
only provide a partial description of a person (such as “female,
tall, young”). This limitation has positive privacy ramifications
when it comes to extracting and storing such soft biometric
data.
2) Taxonomy: With the aforementioned scope and benefits
in mind, it is worth identifying a taxonomy that can facilitate
organization and categorization of soft-biometric traits. This
taxonomy is based on utility, and it considers four groups of
attributes: demographic, anthropometric, medical, and mate-
rial and behavioral attributes. This categorization - and the
more refined sub-categorization based on the modalities of
face, iris, body, gait, fingerprint and hand (Figure 3) - will
also help us structure the exposition of the state-of-art in the
rest of this survey paper.
TABLE I
SOFT BIOMETRIC TAXONOMY WITH FOUR GROUPS: I) DEMOGRAPHIC, II)
ANTHROPOMETRIC AND GEOMETRIC, III) MEDICAL, IV) MATERIAL AND
BEHAVIORAL.
Demographic attributes age, gender, ethnicity
eye-, hair-, skin-color
Anthropometric body geometry
and geometric attributes and facial geometry
Medical attributes health condition, BMI/
body weight, wrinkles
Material and behavioral Hat, scarf, bag, clothes,
attributes lenses, glasses
The above taxonomy need not necessarily result in disjoint
gro
ups and it is certainly not a unique taxonomy. For example,
taxonomies in Jain et al. [110] and Dantcheva et al. [42] (see
also [39]) have different definitions that categorize traits based
on their ability to distinguish between individuals as well as
To appear in IEEE Transactions on Information Forensics and Security (TIFS), 2015
3
their variability over time.
C. Domains of application
Automated soft biometric extraction has a number of ap-
plications: in the area of security where algorithms can locate
a person-of-interest based on a specific set of soft biometric
attributes; in image-tagging and video indexing where photo
or video album management can be performed based, for
example, on age, gender, and clothing; in human-computer-
interaction where data and personalized avatars can be auto-
matically designed according to the user’s external appearance
(e.g. hair- and skin-color, age and gender); in forensics where
artists can amend sketches of the suspect or the victim based
on old pictures; and in surveillance where suspects can be
located based on semantic descriptions. Other applications in-
clude age-specific access control where, for example, children
can be prevented from watching certain movies, accessing
certain web sites, or entering bars or liquor-stores. There are
industrial systems
1
that extract demographic information of
customers for customizing advertisements or for collecting
aggregate data about consuming habits (e.g., based on age,
gender, ethnicity). In addition, Electronic Customer Rela-
tionship Management (ECRM) can use soft biometrics-based
categorization for effectively managing customers by offering
customized products and services. For example, age or gender
specific advertisement can be presented for consumer goods
such as mobile phones, fashion, and food. In cosmetology, it
is of interest to estimate the rejuvenating effect of decorative
cosmetics and cosmetic surgery by computing the perceived
age of an individual from their face image.
In video retrieval systems [93], [252], [258], [190], soft
biometric traits can be used to locate specific individuals in
a video stream either by verbal descriptions (e.g., “individual
with a red shirt”) or by automatically extracting soft biometric
features from an input image and using these features to locate
a matching individual in the video stream.
Finally, in health monitoring, soft biometrics are envisioned
to play a major role in early diagnosis of illness, sickness
prevention and health maintenance. Such traits include body
weight / body mass index, skin abnormalities, and wrinkles.
We will expand on this possibly later on in the paper.
Below, we describe the various contexts in which soft
biometric traits can be used.
1) Uni-modal system: Often applications might require the
extraction of a single soft biometric trait (e.g. gender in a
gender-personalized advertising campaign), in a so called uni-
modal soft biometric system. Such a system generally contains
the “preprocessing”, “feature extraction” and “classification”
modules, with the main focus being on the choice of repre-
sentation (feature extraction).
2) Fusion with primary biometric traits: Here, the goal is to
improve the recognition accuracy of a biometric system. Such
an approach was proposed by Jain et al. [110], who considered
a hybrid system that combined fingerprint identification with
soft biometric attributes such as age, gender and height, to
improve the overall matching accuracy.
1
http://www.quividi.com/
Let W = {w
1
, w
2
, ..., w
n
} be the set of n subjects enrolled
in the database, and let x be the feature vector corresponding
to the primary biometric system. The output of the primary
biometric system is of the form P (w
i
|x), i = 1, 2, ..., n, where
P (w
i
|x) is the probability that the input data belongs to subject
w
i
given the feature vector x. Let y = {y
1
, y
2
, ..., y
m
} be
the soft biometric feature vector. Then the updated probability
P (w
i
|x, y) that the subject in question is w
i
, can be calculated
using the Bayes rule to be
P (w
i
|x, y) =
p(y|w
i
)P (w
i
|x)
P
n
j=1
p(y|w
j
)P (w
j
|x)
(1)
where, p(y|w
i
), i = 1, 2, ..., n represents the conditional prob-
ability of the random vector y given subject w
i
.
Other notable research on fusing soft biometrics and clas-
sical biometrics, include the works in [229], [112], [109], [1],
[289], [189].
3) Search space reduction: Soft biometrics can also be used
to expedite the search in large biometric databases by filtering
out subjects. A number of attributes such as age, gender, hair
and skin color have been proposed for efficient filtering of face
databases [129], [130], [103]. Furthermore, an analysis of the
filtering-gain versus filtering-reliability tradeoff in using soft
biometric traits to prune large databases was presented in [41].
D. Visual attributes
The computer vision community refers to describable visual
attributes as any visual and contextual information that is
helpful in representing an image (cf. Scheirer et al. [229]). In
this approach, semantically meaningful labels are employed
towards image retrieval and object categorization. In the
context of human recognition, this semantic information can
describe gender [228], ethnicity [229], accessories [21], cloth-
ing style [238], and facial-feature-shapes [228]. Related work
include fusion of attributes by Scheirer et al. [227], pruning
of large-scale datasets by Russakovsky and Fei-Fei [220], as
well as studies on similarities between faces or objects based
on relative attributes by Parikh and Grauman [187] and Zhou
et al. [297]. Other pertinent literature include [62], [131],
[52], [137], [156], [21]. Of specific interest are the “zero-
shot” learning approaches, where previously unseen objects
are described using attributes of objects encountered in the
training set (cf. Parikh and Grauman [187]).
E. Structure of paper
The survey provides a review of salient techniques for
extracting soft biometrics from modalities such as face, body,
fingerprint, iris, and voice. While an exhaustive survey of all
soft biometric traits is not possible due to the richness of
the field (for example, we do not expand on traits relating to
the ear, or to saccadic movements), we try to offer a holistic
view of most of these traits. In this way, this survey paper
is significantly different from other introductory overviews
(see [42], [210], [67], [213], [106] and Table II) that have
focused on specific soft biometric traits such as gender, age
or ethnicity.
To appear in IEEE Transactions on Information Forensics and Security (TIFS), 2015
4
The structure of this survey is based on the aforementioned
taxonomy of soft biometrics. We discuss soft biometric traits
that are heavily used as demographic attributes (Section II),
as anthropometric (geometric) attributes (Section III), med-
ical attributes (Section IV), and as miscellaneous material
and behavioral attributes (Section V). Finally in Section VI
we discuss open research problems that are currently being
addressed in the field of soft biometrics.
TABLE II
EXISTING INTRODUCTORY OVERVIEWS ON GENDER AND AGE ESTIMATION
TECHNIQUES.
Modality Scientific work Year
Gender Ng et al. [177] 2012
Gender Khan et al. [120] 2011
Gender Bekios-Calfa et al. [18] 2011
Gender Ramanathan et al. [201] 2009
Gender M¨akinen and Raisamo [157] 2008
Gender M¨akinen and Raisamo [158] 2008
Age Guo [90] 2012
Age Fu et al. [68] 2010
Ethnicity Fu et al. [65] 2014
II. DEMOGRAPHIC ATTRIBUTES
The term demographics, in addition to referring to the
quantifiable statistics of a given population, refers to attributes
such as age, gender, ethnicity and race that are widely used
in common population statistics. Since the early publications
in [164], [243], research on this class of soft biometrics has
been embraced by the computer vision community.
A. Gender Estimation
The traditional definition of sex refers to the biological
characteristics that differentiate men and women, as opposed
to gender, which is related to the social and cultural dis-
tinctions between the sexes. However, very often, the terms
“sex” and “gender” have been used interchangeably in the
biometrics literature. Consequently, we do not make any
explicit distinction between the two terms in this article.
Gender estimation remains a challenging task, which is
inherently associated with different biometric modalities in-
cluding fingerprint, face, iris, voice, body shape, gait, signa-
ture, DNA, as well as clothing, hair, jewelry and even body
temperature (see [165]). The forensic literature [148] suggests
that the skull, specifically the chin and the jawbone, as well
as the pelvis, are the most significant indicators of the gender
of a person; in juveniles, these shape-based features have been
recorded to provide classification accuracy of 91% − 99%. It
has been argued (see for example the work by Loth and Is-
can [148]) that there is no single skeletal feature that definitely
reveals the evidence of sexual dimorphism, and that there is
in fact a cross-gender metric overlap of up to 85%, which
can be attributed to environmental influences and pathologic
conditions, such as diet and occupational stress. In spite of
this, forensic experts argue [128] that near 100% gender
determination accuracy can be attained by visual examination
of the entirety of the skeleton.
Humans are generally quite good at gender recognition, as
they have been programmed - from an evolutionary standpoint
- to classify gender from early on in their lives [185]. As
pointed out by Edelman et al. [54], humans perform face
image-based gender classification with an error rate of about
11%, which is commensurate to that of a neural network
algorithm performing the same task (at that point in time).
Despite this, automated gender recognition from biometric
data remains to be a challenge and is impacted by other soft
biometrics, for example, age and race; gender dimorphism is
accentuated only in adults, and varies across different races.
1) Gender from face: In gender recognition from face,
feature-based approaches extract and analyze a specific set of
discriminative facial features (patches) in order to identify the
gender of a person. This is a particularly challenging problem,
as is implied from the fact that female and male average facial
shapes are generally found to be very similar.
One of the primary challenges in face-based gender recog-
nition is the step of feature selection, where one must ju-
dicially select the type of considered features in order to
improve gender recognition rates. Towards understanding this
feature selection process, different types of strategies have
been attempted, such as the work in Sun et al. [245] that
employed genetic algorithms for eigen-feature selection. Other
approaches focus on specific facial features, such as the
approach by Zhang et al. [292] that focused on the eye brow
and the jaw region.
Another challenge comes in unconstrained settings where
the face image is affected by changes in illumination, pose,
etc. While in more constrained settings face-based gender
estimation has been reported to achieve classification rates of
up to 99.3% (see Table III), this performance significantly
decreases in more realistic and unconstrained settings.
The majority of gender classification methods contain two
steps succeeding face detection, namely feature extraction and
pattern classification.
Feature extraction: Notable efforts include the early work
by Moghaddam et al. [169] and the work by Baluja et al.
[15] who used raw pixel intensities as inputs to SVM and
Adaboost classifiers, in order to achieve a 96% success rate on
low resolution images. Interesting work can also be found in
Cao et al. [22] who investigated facial metrology for pertinent
gender traits, which resulted in error rates that were observed
to be between 3.8% and 5.7% lower than that of appearance-
based methods. Other feature extraction approaches are found
in the work of Saatci and Town in [221], who presented an
active appearance model (AAM) based geometric-approach for
extracting gender and expression (using a SVM classifier with
a radial basis kernel), as well as recent approaches that use
SIFT [254], LBP [281], [157], semi-supervised discriminant
analysis (SDA) [19] or combinations of different features [83],
[265].
Classification: A number of classification methods have
been used for gender estimation, and a useful comparative
guide of these classification methods can be found in M¨akinen
and Raisamo [157]. One interesting conclusion of their work
To appear in IEEE Transactions on Information Forensics and Security (TIFS), 2015