Understanding Kin Relationships in a Photo
TL;DR: Experimental results have shown that the proposed algorithms can effectively annotate the kin relationships among people in an image and semantic context can further improve the accuracy.
Abstract: There is an urgent need to organize and manage images of people automatically due to the recent explosion of such data on the Web in general and in social media in particular. Beyond face detection and face recognition, which have been extensively studied over the past decade, perhaps the most interesting aspect related to human-centered images is the relationship of people in the image. In this work, we focus on a novel solution to the latter problem, in particular the kin relationships. To this end, we constructed two databases: the first one named UB KinFace Ver2.0, which consists of images of children, their young parents and old parents, and the second one named FamilyFace. Next, we develop a transfer subspace learning based algorithm in order to reduce the significant differences in the appearance distributions between children and old parents facial images. Moreover, by exploring the semantic relevance of the associated metadata, we propose an algorithm to predict the most likely kin relationships embedded in an image. In addition, human subjects are used in a baseline study on both databases. Experimental results have shown that the proposed algorithms can effectively annotate the kin relationships among people in an image and semantic context can further improve the accuracy.
Citations
More filters
16 Jun 2012
TL;DR: This paper proposes a new neighborhood repulsed metric learning (NRML) method for kinship verification, and proposes a multiview NRM-L method to seek a common distance metric to make better use of multiple feature descriptors to further improve the verification performance.
Abstract: Kinship verification from facial images is an interesting and challenging problem in computer vision, and there are very limited attempts on tackle this problem in the literature. In this paper, we propose a new neighborhood repulsed metric learning (NRML) method for kinship verification. Motivated by the fact that interclass samples (without a kinship relation) with higher similarity usually lie in a neighborhood and are more easily misclassified than those with lower similarity, we aim to learn a distance metric under which the intraclass samples (with a kinship relation) are pulled as close as possible and interclass samples lying in a neighborhood are repulsed and pushed away as far as possible, simultaneously, such that more discriminative information can be exploited for verification. To make better use of multiple feature descriptors to extract complementary information, we further propose a multiview NRML (MNRML) method to seek a common distance metric to perform multiple feature fusion to improve the kinship verification performance. Experimental results are presented to demonstrate the efficacy of our proposed methods. Finally, we also test human ability in kinship verification from facial images and our experimental results show that our methods are comparable to that of human observers.
423 citations
Cites background or methods from "Understanding Kin Relationships in ..."
...[60], [61] proposed a transfer subspace learning approach for kinship verification....
[...]
...isting methods were only evaluated on relatively small data sets (150 and 200 pairs in [17] and [61], respectively), which...
[...]
TL;DR: A discriminative deep multi-metric learning method to jointly learn multiple neural networks, under which the correlation of different features of each sample is maximized, and the distance of each positive pair is reduced and that of each negative pair is enlarged.
Abstract: This paper presents a new discriminative deep metric learning (DDML) method for face and kinship verification in wild conditions. While metric learning has achieved reasonably good performance in face and kinship verification, most existing metric learning methods aim to learn a single Mahalanobis distance metric to maximize the inter-class variations and minimize the intra-class variations, which cannot capture the nonlinear manifold where face images usually lie on. To address this, we propose a DDML method to train a deep neural network to learn a set of hierarchical nonlinear transformations to project face pairs into the same latent feature space, under which the distance of each positive pair is reduced and that of each negative pair is enlarged. To better use the commonality of multiple feature descriptors to make all the features more robust for face and kinship verification, we develop a discriminative deep multi-metric learning method to jointly learn multiple neural networks, under which the correlation of different features of each sample is maximized, and the distance of each positive pair is reduced and that of each negative pair is enlarged. Extensive experimental results show that our proposed methods achieve the acceptable results in both face and kinship verification.
264 citations
Cites background from "Understanding Kin Relationships in ..."
...Many metric learning algorithms have been proposed over the past decade [4], [6], [10], [11], [13], [21], [29]–[32]....
[...]
TL;DR: Experimental results show the effectiveness of the proposed discriminative multimetric learning method for kinship verification via facial image analysis over the existing single-metric and multimetricLearning methods.
Abstract: In this paper, we propose a new discriminative multimetric learning method for kinship verification via facial image analysis. Given each face image, we first extract multiple features using different face descriptors to characterize face images from different aspects because different feature descriptors can provide complementary information. Then, we jointly learn multiple distance metrics with these extracted multiple features under which the probability of a pair of face image with a kinship relation having a smaller distance than that of the pair without a kinship relation is maximized, and the correlation of different features of the same face sample is maximized, simultaneously, so that complementary and discriminative information is exploited for verification. Experimental results on four face kinship data sets show the effectiveness of our proposed method over the existing single-metric and multimetric learning methods.
207 citations
Cites background from "Understanding Kin Relationships in ..."
...In this section, we briefly review two related topics: 1) kinship verification, and 2) metric learning....
[...]
TL;DR: Experimental results on four publicly available kinship datasets show the superior performance of the proposed PDFL methods over both the state-of-the-art kinship verification methods and human ability in the kinships verification task.
Abstract: In this paper, we propose a new prototype-based discriminative feature learning (PDFL) method for kinship verification. Unlike most previous kinship verification methods which employ low-level hand-crafted descriptors such as local binary pattern and Gabor features for face representation, this paper aims to learn discriminative mid-level features to better characterize the kin relation of face images for kinship verification. To achieve this, we construct a set of face samples with unlabeled kin relation from the labeled face in the wild dataset as the reference set. Then, each sample in the training face kinship dataset is represented as a mid-level feature vector, where each entry is the corresponding decision value from one support vector machine hyperplane. Subsequently, we formulate an optimization function by minimizing the intraclass samples (with a kin relation) and maximizing the neighboring interclass samples (without a kin relation) with the mid-level features. To better use multiple low-level features for mid-level feature learning, we further propose a multiview PDFL method to learn multiple mid-level features to improve the verification performance. Experimental results on four publicly available kinship datasets show the superior performance of the proposed methods over both the state-of-the-art kinship verification methods and human ability in our kinship verification task.
171 citations
01 Nov 2014
TL;DR: A new large margin multi-metric learning (LM\(^3\)L) method for face and kinship verification in the wild that jointly learns multiple distance metrics under which the correlations of different feature representations of each sample are maximized.
Abstract: Metric learning has been widely used in face and kinship verification and a number of such algorithms have been proposed over the past decade. However, most existing metric learning methods only learn one Mahalanobis distance metric from a single feature representation for each face image and cannot deal with multiple feature representations directly. In many face verification applications, we have access to extract multiple features for each face image to extract more complementary information, and it is desirable to learn distance metrics from these multiple features so that more discriminative information can be exploited than those learned from individual features. To achieve this, we propose a new large margin multi-metric learning (LM\(^3\)L) method for face and kinship verification in the wild. Our method jointly learns multiple distance metrics under which the correlations of different feature representations of each sample are maximized, and the distance of each positive is less than a low threshold and that of each negative pair is greater than a high threshold, simultaneously. Experimental results show that our method can achieve competitive results compared with the state-of-the-art methods.
168 citations
Cites background from "Understanding Kin Relationships in ..."
...Face and Kinship Verification in the Wild: In recent years, many approaches have been proposed for face and kinship verification in the wild, and they can be mainly classified into two categories: feature-based [7, 10, 37, 38] and model-based [17, 18, 33, 34]....
[...]
References
More filters
TL;DR: The relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as covariate shift are discussed.
Abstract: A major assumption in many machine learning and data mining algorithms is that the training and future data must be in the same feature space and have the same distribution. However, in many real-world applications, this assumption may not hold. For example, we sometimes have a classification task in one domain of interest, but we only have sufficient training data in another domain of interest, where the latter data may be in a different feature space or follow a different data distribution. In such cases, knowledge transfer, if done successfully, would greatly improve the performance of learning by avoiding much expensive data-labeling efforts. In recent years, transfer learning has emerged as a new learning framework to address this problem. This survey focuses on categorizing and reviewing the current progress on transfer learning for classification, regression, and clustering problems. In this survey, we discuss the relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as covariate shift. We also explore some potential future issues in transfer learning research.
18,616 citations
"Understanding Kin Relationships in ..." refers background or methods in this paper
...In [32], several practical instances have been introduced to illustrate the role of transfer learning, e....
[...]
...In terms of transfer learning [32], method in [27] con-...
[...]
...Sometimes, however, transfer learning may fail due to the large discrepancy between the source and target data sets [32], such as children and old parents images....
[...]
TL;DR: A near-real-time computer system that can locate and track a subject's head, and then recognize the person by comparing characteristics of the face to those of known individuals, and that is easy to implement using a neural network architecture.
Abstract: We have developed a near-real-time computer system that can locate and track a subject's head, and then recognize the person by comparing characteristics of the face to those of known individuals. The computational approach taken in this system is motivated by both physiology and information theory, as well as by the practical requirements of near-real-time performance and accuracy. Our approach treats the face recognition problem as an intrinsically two-dimensional (2-D) recognition problem rather than requiring recovery of three-dimensional geometry, taking advantage of the fact that faces are normally upright and thus may be described by a small set of 2-D characteristic views. The system functions by projecting face images onto a feature space that spans the significant variations among known face images. The significant features are known as "eigenfaces," because they are the eigenvectors (principal components) of the set of faces; they do not necessarily correspond to features such as eyes, ears, and noses. The projection operation characterizes an individual face by a weighted sum of the eigenface features, and so to recognize a particular face it is necessary only to compare these weights to those of known individuals. Some particular advantages of our approach are that it provides for the ability to learn and later recognize new faces in an unsupervised manner, and that it is easy to implement using a neural network architecture.
14,562 citations
"Understanding Kin Relationships in ..." refers background in this paper
...tion, such as PCA [4], LDA [6], LPP [7], NPE [8], MFA [9],...
[...]
...Face recognition without context information, namely, pairwise face recognition, has been extensively studied during the past decade by exploration of following techniques: face detection and alignment [2], [3], subspace learning [4]–[10], invariant feature extraction [11]–[13], metric learning method [14], attributes based classifier [15] and face synthesis and hallucination [16], [17]....
[...]
...6 illustrates the idea of transfer subspace learning proposed in this paper which can be mathematically formulated as (2) where is a general subspace learning objective function, such as PCA [4], LDA [6], LPP [7], NPE [8], MFA [9], and DLA [10], and and represent the distribution of the source and target samples respectively. is the Bregman divergence-based regularization that measures the distance between two different distributions in the projected sub- space . is the weight for the regularization....
[...]
TL;DR: In this paper, a face detection framework that is capable of processing images extremely rapidly while achieving high detection rates is described. But the detection performance is limited to 15 frames per second.
Abstract: This paper describes a face detection framework that is capable of processing images extremely rapidly while achieving high detection rates. There are three key contributions. The first is the introduction of a new image representation called the “Integral Image” which allows the features used by our detector to be computed very quickly. The second is a simple and efficient classifier which is built using the AdaBoost learning algorithm (Freund and Schapire, 1995) to select a small number of critical visual features from a very large set of potential features. The third contribution is a method for combining classifiers in a “cascade” which allows background regions of the image to be quickly discarded while spending more computation on promising face-like regions. A set of experiments in the domain of face detection is presented. The system yields face detection performance comparable to the best previous systems (Sung and Poggio, 1998; Rowley et al., 1998; Schneiderman and Kanade, 2000; Roth et al., 2000). Implemented on a conventional desktop, face detection proceeds at 15 frames per second.
13,037 citations
TL;DR: A face recognition algorithm which is insensitive to large variation in lighting direction and facial expression is developed, based on Fisher's linear discriminant and produces well separated classes in a low-dimensional subspace, even under severe variations in lighting and facial expressions.
Abstract: We develop a face recognition algorithm which is insensitive to large variation in lighting direction and facial expression. Taking a pattern classification approach, we consider each pixel in an image as a coordinate in a high-dimensional space. We take advantage of the observation that the images of a particular face, under varying illumination but fixed pose, lie in a 3D linear subspace of the high dimensional image space-if the face is a Lambertian surface without shadowing. However, since faces are not truly Lambertian surfaces and do indeed produce self-shadowing, images will deviate from this linear subspace. Rather than explicitly modeling this deviation, we linearly project the image into a subspace in a manner which discounts those regions of the face with large deviation. Our projection method is based on Fisher's linear discriminant and produces well separated classes in a low-dimensional subspace, even under severe variation in lighting and facial expressions. The eigenface technique, another method based on linearly projecting the image space to a low dimensional subspace, has similar computational requirements. Yet, extensive experimental results demonstrate that the proposed "Fisherface" method has error rates that are lower than those of the eigenface technique for tests on the Harvard and Yale face databases.
11,674 citations
"Understanding Kin Relationships in ..." refers background in this paper
...tion, such as PCA [4], LDA [6], LPP [7], NPE [8], MFA [9],...
[...]
...6 illustrates the idea of transfer subspace learning proposed in this paper which can be mathematically formulated as (2) where is a general subspace learning objective function, such as PCA [4], LDA [6], LPP [7], NPE [8], MFA [9], and DLA [10], and and represent the distribution of the source and target samples respectively. is the Bregman divergence-based regularization that measures the distance between two different distributions in the projected sub- space . is the weight for the regularization....
[...]
01 Oct 2008
TL;DR: The database contains labeled face photographs spanning the range of conditions typically encountered in everyday life, and exhibits “natural” variability in factors such as pose, lighting, race, accessories, occlusions, and background.
Abstract: Most face databases have been created under controlled conditions to facilitate the study of specific parameters on the face recognition problem. These parameters include such variables as position, pose, lighting, background, camera quality, and gender. While there are many applications for face recognition technology in which one can control the parameters of image acquisition, there are also many applications in which the practitioner has little or no control over such parameters. This database, Labeled Faces in the Wild, is provided as an aid in studying the latter, unconstrained, recognition problem. The database contains labeled face photographs spanning the range of conditions typically encountered in everyday life. The database exhibits “natural” variability in factors such as pose, lighting, race, accessories, occlusions, and background. In addition to describing the details of the database, we provide specific experimental paradigms for which the database is suitable. This is done in an effort to make research performed with the database as consistent and comparable as possible. We provide baseline results, including results of a state of the art face recognition system combined with a face alignment system. To facilitate experimentation on the database, we provide several parallel databases, including an aligned version.
5,742 citations
"Understanding Kin Relationships in ..." refers background or result in this paper
...Similar to “Labeled Faces in the Wild” [18], these unconstrained facial images show a large range of variations, including pose, lighting, expression, age, gender, background, race, color saturation, and image quality, etc....
[...]
...The performance of the face recognition algorithm is dramatically degraded when a large-scale database is considered [18]....
[...]
...Meanwhile, other uncontrollable factors are evident in the real-world, as described in [18]....
[...]