Aiding face recognition with social context association rule based re-ranking
TL;DR: The results show that association rules extracted from social context can be used to augment face recognition and improve the identification performance.
Abstract: Humans are very efficient at recognizing familiar face images even in challenging conditions. One reason for such capabilities is the ability to understand social context between individuals. Sometimes the identity of the person in a photo can be inferred based on the identity of other persons in the same photo, when some social context between them is known. This research presents an algorithm to utilize cooccurrence of individuals as the social context to improve face recognition. Association rule mining is utilized to infer multi-level social context among subjects from a large repository of social transactions. The results are demonstrated on the G-album and on the SN-collection pertaining to 4675 identities prepared by the authors from a social networking website. The results show that association rules extracted from social context can be used to augment face recognition and improve the identification performance.
Citations
More filters
TL;DR: A comprehensive review of techniques incorporating ancillary information in the biometric recognition pipeline is presented in this paper, where the authors provide a comprehensive overview of the role of information fusion in biometrics.
151 citations
TL;DR: In this paper, a hierarchical kinship verification via representation learning (KVRL) framework is utilized to learn the representation of different face regions in an unsupervised manner, and a compact representation of facial images of kin is extracted as an output from the learned model and a multi-layer neural network is employed to verify the kin accurately.
Abstract: Kinship verification has a number of applications such as organizing large collections of images and recognizing resemblances among humans. In this paper, first, a human study is conducted to understand the capabilities of human mind and to identify the discriminatory areas of a face that facilitate kinship-cues. The visual stimuli presented to the participants determine their ability to recognize kin relationship using the whole face as well as specific facial regions. The effect of participant gender and age and kin-relation pair of the stimulus is analyzed using quantitative measures such as accuracy, discriminability index $d'$ , and perceptual information entropy. Utilizing the information obtained from the human study, a hierarchical kinship verification via representation learning (KVRL) framework is utilized to learn the representation of different face regions in an unsupervised manner. We propose a novel approach for feature representation termed as filtered contractive deep belief networks ( fc DBN). The proposed feature representation encodes relational information present in images using filters and contractive regularization penalty. A compact representation of facial images of kin is extracted as an output from the learned model and a multi-layer neural network is utilized to verify the kin accurately. A new WVU kinship database is created, which consists of multiple images per subject to facilitate kinship verification. The results show that the proposed deep learning framework (KVRL- fc DBN) yields the state-of-the-art kinship verification accuracy on the WVU kinship database and on four existing benchmark data sets. Furthermore, kinship information is used as a soft biometric modality to boost the performance of face verification via product of likelihood ratio and support vector machine based approaches. Using the proposed KVRL- fc DBN framework, an improvement of over 20% is observed in the performance of face verification.
81 citations
17 Jul 2019
TL;DR: KVQA is introduced – the first dataset for the task of (world) knowledge-aware VQA and is the largest dataset for exploring V QA over large Knowledge Graphs (KG), which consists of 183K question-answer pairs involving more than 18K named entities and 24K images.
Abstract: Visual Question Answering (VQA) has emerged as an important problem spanning Computer Vision, Natural Language Processing and Artificial Intelligence (AI). In conventional VQA, one may ask questions about an image which can be answered purely based on its content. For example, given an image with people in it, a typical VQA question may inquire about the number of people in the image. More recently, there is growing interest in answering questions which require commonsense knowledge involving common nouns (e.g., cats, dogs, microphones) present in the image. In spite of this progress, the important problem of answering questions requiring world knowledge about named entities (e.g., Barack Obama, White House, United Nations) in the image has not been addressed in prior research. We address this gap in this paper, and introduce KVQA – the first dataset for the task of (world) knowledge-aware VQA. KVQA consists of 183K question-answer pairs involving more than 18K named entities and 24K images. Questions in this dataset require multi-entity, multi-relation, and multi-hop reasoning over large Knowledge Graphs (KG) to arrive at an answer. To the best of our knowledge, KVQA is the largest dataset for exploring VQA over KG. Further, we also provide baseline performances using state-of-the-art methods on KVQA.
75 citations
Cites background from "Aiding face recognition with social..."
...There have also been works demonstrating utility of context in improving face identifica- tion often in restricted settings (Bharadwaj, Vatsa, and Singh 2014; Lin et al. 2010; O’Hare and Smeaton 2009)....
[...]
Posted Content•
TL;DR: The purpose of this article is to provide readers a comprehensive overview of the role of information fusion in biometrics with specific focus on three questions: what to fusion, when to fuse, and how to fuse.
Abstract: The performance of a biometric system that relies on a single biometric modality (e.g., fingerprints only) is often stymied by various factors such as poor data quality or limited scalability. Multibiometric systems utilize the principle of fusion to combine information from multiple sources in order to improve recognition accuracy whilst addressing some of the limitations of single-biometric systems. The past two decades have witnessed the development of a large number of biometric fusion schemes. This paper presents an overview of biometric fusion with specific focus on three questions: what to fuse, when to fuse, and how to fuse. A comprehensive review of techniques incorporating ancillary information in the biometric recognition pipeline is also presented. In this regard, the following topics are discussed: (i) incorporating data quality in the biometric recognition pipeline; (ii) combining soft biometric attributes with primary biometric identifiers; (iii) utilizing contextual information to improve biometric recognition accuracy; and (iv) performing continuous authentication using ancillary information. In addition, the use of information fusion principles for presentation attack detection and multibiometric cryptosystems is also discussed. Finally, some of the research challenges in biometric fusion are enumerated. The purpose of this article is to provide readers a comprehensive overview of the role of information fusion in biometrics.
47 citations
Cites background or methods from "Aiding face recognition with social..."
...[46] proposed a social context based re770...
[...]
...rank lists from multiple matchers have been fused using techniques like Borda count, logistic regression, and highest rank method [64, 65, 66, 67, 46]....
[...]
...occurrence of various parts or attributes of an object or face” [46]....
[...]
...[46] Social context based re-ranking algorithm using association rules 2014 Hochreiter et al....
[...]
TL;DR: A novel person recognition approach is presented, that relies on the knowledge of individuals’ social behavior to enhance the performance of a traditional biometric system.
Abstract: The goal of a biometric recognition system is to make a human-like decisions on individual’s identity by recognizing their physiological and/or behavioral traits. Nevertheless, the decision-making process by either a human or a biometric recognition system can be highly complicated due to low quality of data or an uncertain environment. Human brain has an advantage over computer system due to its ability to perform a massive parallel processing of auxiliary information, such as visual cues, cognitive and social interactions, contextual, and spatio-temporal data. Similarly to a human brain, social behavioral cues can aid the reliable decision-making of an automated biometric system. In this paper, a novel person recognition approach is presented, that relies on the knowledge of individuals’ social behavior to enhance the performance of a traditional biometric system. The social behavioral information of individuals’ has been mined from an online social network and fused with traditional face and ear biometrics. Experimental results on individual’s and semi-real databases demonstrate significant performance gain in the proposed method over traditional biometric system.
34 citations
Cites background from "Aiding face recognition with social..."
...is an emerging direction in biometric research [35], [44]....
[...]
...[35] demonstrates a significant recognition performance improvement in challenging environment by fusing social contextual information with face biometric....
[...]
References
More filters
TL;DR: Examination of the application of Bayesian Attribute Networks to combine descriptive attributes and produce accurate weighting factors to apply to match scores from face recognition systems based on incomplete observations made at match time shows that incorporating descriptive attributes into the matching process significantly enhances face identification over the baseline.
Abstract: For identity related problems, descriptive attributes can take the form of any information that helps represent an individual, including age data, describable visual attributes, and contextual data. With a rich set of descriptive attributes, it is possible to enhance the base matching accuracy of a traditional face identification system through intelligent score weighting. If we can factor any attribute differences between people into our match score calculation, we can deemphasize incorrect results, and ideally lift the correct matching record to a higher rank position. Naturally, the presence of all descriptive attributes during a match instance cannot be expected, especially when considering non-biometric context. Thus, in this paper, we examine the application of Bayesian Attribute Networks to combine descriptive attributes and produce accurate weighting factors to apply to match scores from face recognition systems based on incomplete observations made at match time. We also examine the pragmatic concerns of attribute network creation, and introduce a Noisy-OR formulation for streamlined truth value assignment and more accurate weighting. Experimental results show that incorporating descriptive attributes into the matching process significantly enhances face identification over the baseline by up to 32.8%.
45 citations
"Aiding face recognition with social..." refers background in this paper
...The term context has been used in object recognition [27], person detection and also in face recognition research [17, 24] to imply acceptable co-occurrence of various parts or attributes of an object or face....
[...]
16 Apr 2013
TL;DR: A unified framework that employs bootstrapping to automatically learn adaptive rules to integrate heterogeneous contextual information, along with facial features, together is proposed, which demonstrates the effectiveness of the proposed approach in improving recall while maintaining very high precision of face clustering.
Abstract: Automatic face clustering, which aims to group faces referring to the same people together, is a key component for face tagging and image management. Standard face clustering approaches that are based on analyzing facial features can already achieve high-precision results. However, they often suffer from low recall due to the large variation of faces in pose, expression, illumination, occlusion, etc. To improve the clustering recall without reducing the high precision, we leverage the heterogeneous context information to iteratively merge the clusters referring to same entities. We first investigate the appropriate methods to utilize the context information at the cluster level, including using of "common scene", people co-occurrence, human attributes, and clothing. We then propose a unified framework that employs bootstrapping to automatically learn adaptive rules to integrate this heterogeneous contextual information, along with facial features, together. Experimental results on two personal photo collections and one real-world surveillance dataset demonstrate the effectiveness of the proposed approach in improving recall while maintaining very high precision of face clustering.
38 citations
"Aiding face recognition with social..." refers background in this paper
...Face clustering approaches incorporate contextual constraints such as same-day (a person on a given day is wearing the same clothing throughout), Person-Exclusive constraint (a person can occur in a photo only once), and co-occurrence [30] to improve recall....
[...]
Proceedings Article•
[...]
TL;DR: This is the first work, which covers such a system end-to-end, from offline crawling up to augmentation on the mobile device, and the complete system runs in real time on a state-of-the-art mobile phone.
Abstract: In this paper we present a fully automatic system for face augmentation on mobile devices. A user can point his mobile phone to a person and the system recognizes his or her face. A tracking algorithm overlays information about the identified person on the screen, thereby achieving an augmented reality effect. The tracker is running on the mobile client, while the recognition is running on a server. The database on the server is built by a fully autonomous crawling method, which taps social networks. For this work we collected 300 000 images from Facebook. The social context gained during this social network analysis is also used to improve the face recognition. The complete system runs in real time on a state-of-the-art mobile phone and is fully automatic, from offline crawling up to augmentation on the mobile device. It can be used to display more information about the identified persons or as a user interface for mixed reality application. To the best of our knowledge this is the first work, which covers such a system end-to-end.
32 citations
TL;DR: The independence assumption is reconsiders, new features and methods for recognizing pairs of individuals in group photographs are introduced, and a marked improvement when these features are used in joint decision making vs. independent decision making is demonstrated.
Abstract: Face recognition systems classically recognize people individually. When presented with a group photograph containing multiple people, such systems implicitly assume statistical independence between each detected face. We question this basic assumption and consider instead that there is a dependence between face regions from the same image; after all, the image was acquired with a single camera, under consistent lighting (distribution, direction, spectrum), camera motion, and scene/camera geometry. Such naturally occurring commonalities between face images can be exploited when recognition decisions are made jointly across the faces, rather than independently. Furthermore, when recognizing people in isolation, some features such as color are usually uninformative in unconstrained settings. But by considering pairs of people, the relative color difference provides valuable information. This paper reconsiders the independence assumption, introduces new features and methods for recognizing pairs of individuals in group photographs, and demonstrates a marked improvement when these features are used in joint decision making vs. independent decision making. While these features alone are only moderately discriminative, we combine these new features with state-of-art attribute features and demonstrate effective recognition performance. Initial experiments on two datasets show promising improvements in accuracy.
27 citations
24 Mar 2014
TL;DR: This work introduces a method for extracting the social network structure for the persons appearing in a set of video clips and incorporates a novel active clustering technique to create more accurate identity clusters based on feedback from the user about ambiguously matched faces.
Abstract: We introduce a method for extracting the social network structure for the persons appearing in a set of video clips. Individuals are unknown, and are not matched against known enrollments. An identity cluster representing an individual is formed by grouping similar-appearing faces from different videos. Each identity cluster is represented by a node in the social network. Two nodes are linked if the faces from their clusters appeared together in one or more video frames. Our approach incorporates a novel active clustering technique to create more accurate identity clusters based on feedback from the user about ambiguously matched faces. The final output consists of one or more network structures that represent the social group(s), and a list of persons who potentially connect multiple social groups. Our results demonstrate the efficacy of the proposed clustering algorithm and network analysis techniques.
24 citations
"Aiding face recognition with social..." refers background in this paper
...[2] use active learning to generate social constraints to improve face clustering and Hochreiter et al....
[...]