Local binary pattern (LBP) is a nonparametric descriptor, which efficiently summarizes the local structures of images. In recent years, it has aroused increasing interest in many areas of image processing and computer vision and has shown its effectiveness in a number of applications, in particular for facial image analysis, including tasks as diverse as face detection, face recognition, facial expression analysis, and demographic classification. This paper presents a comprehensive survey of LBP methodology, including several more recent variations. As a typical application of the LBP approach, LBP-based facial image analysis is extensively reviewed, while its successful extensions, which deal with various tasks of facial image analysis, are also highlighted.

/pdf/local-binary-patterns-and-its-application-to-facial-image-1kaldi7ot7.pdf

Local Binary Patterns and Its Application to Facial Image Analysis: A Survey

In this paper, we propose a novel face photo-sketch synthesis and recognition method using a multiscale Markov Random Fields (MRF) model. Our system has three components: 1) given a face photo, synthesizing a sketch drawing; 2) given a face sketch drawing, synthesizing a photo; and 3) searching for face photos in the database based on a query sketch drawn by an artist. It has useful applications for both digital entertainment and law enforcement. We assume that faces to be studied are in a frontal pose, with normal lighting and neutral expression, and have no occlusions. To synthesize sketch/photo images, the face region is divided into overlapping patches for learning. The size of the patches decides the scale of local face structures to be learned. From a training set which contains photo-sketch pairs, the joint photo-sketch model is learned at multiple scales using a multiscale MRF model. By transforming a face photo to a sketch (or transforming a sketch to a photo), the difference between photos and sketches is significantly reduced, thus allowing effective matching between the two in face sketch recognition. After the photo-sketch transformation, in principle, most of the proposed face photo recognition approaches can be applied to face sketch recognition in a straightforward way. Extensive experiments are conducted on a face sketch database including 606 faces, which can be downloaded from our Web site (http://mmlab.ie.cuhk.edu.hk/facesketch.html).

https://www.ee.cuhk.edu.hk/~xgwang/papers/wangTPAMI08.pdf

Face Photo-Sketch Synthesis and Recognition

In many computer vision systems, the same object can be observed at varying viewpoints or even by different sensors, which brings in the challenging demand for recognizing objects from distinct even heterogeneous views. In this work we propose a Multi-view Discriminant Analysis (MvDA) approach, which seeks for a single discriminant common space for multiple views in a non-pairwise manner by jointly learning multiple view-specific linear transforms. Specifically, our MvDA is formulated to jointly solve the multiple linear transforms by optimizing a generalized Rayleigh quotient, i.e., maximizing the between-class variations and minimizing the within-class variations from both intra-view and inter-view in the common space. By reformulating this problem as a ratio trace problem, the multiple linear transforms are achieved analytically and simultaneously through generalized eigenvalue decomposition. Furthermore, inspired by the observation that different views share similar data structures, a constraint is introduced to enforce the view-consistency of the multiple linear transforms. The proposed method is evaluated on three tasks: face recognition across pose, photo versus. sketch face recognition, and visual light image versus near infrared image face recognition on Multi-PIE, CUFSF and HFB databases respectively. Extensive experiments show that our MvDA achieves significant improvements compared with the best known results.

Multi-View Discriminant Analysis

This paper presents a novel way to perform multi-modal face recognition. We use Partial Least Squares (PLS) to linearly map images in different modalities to a common linear subspace in which they are highly correlated. PLS has been previously used effectively for feature selection in face recognition. We show both theoretically and experimentally that PLS can be used effectively across modalities. We also formulate a generic intermediate subspace comparison framework for multi-modal recognition. Surprisingly, we achieve high performance using only pixel intensities as features. We experimentally demonstrate the highest published recognition rates on the pose variations in the PIE data set, and also show that PLS can be used to compare sketches to photos, and to compare images taken at different resolutions.

/pdf/bypassing-synthesis-pls-for-face-recognition-with-pose-low-55e312cxqp.pdf

Bypassing synthesis: PLS for face recognition with pose, low-resolution and sketch

This paper comprehensively surveys the development of face hallucination (FH), including both face super-resolution and face sketch-photo synthesis techniques. Indeed, these two techniques share the same objective of inferring a target face image (e.g. high-resolution face image, face sketch and face photo) from a corresponding source input (e.g. low-resolution face image, face photo and face sketch). Considering the critical role of image interpretation in modern intelligent systems for authentication, surveillance, law enforcement, security control, and entertainment, FH has attracted growing attention in recent years. Existing FH methods can be grouped into four categories: Bayesian inference approaches, subspace learning approaches, a combination of Bayesian inference and subspace learning approaches, and sparse representation-based approaches. In spite of achieving a certain level of development, FH is limited in its success by complex application conditions such as variant illuminations, poses, or views. This paper provides a holistic understanding and deep insight into FH, and presents a comparative analysis of representative methods and promising future directions.

https://www.tpbin.com/Uploads/Subjects/6153ac35-26a8-4179-9cf9-368614e71179.pdf

A Comprehensive Survey to Face Hallucination

Most face recognition systems focus on photo-based face recognition. In this paper, we present a face recognition system based on face sketches. The proposed system contains two elements: pseudo-sketch synthesis and sketch recognition. The pseudo-sketch generation method is based on local linear preserving of geometry between photo and sketch images, which is inspired by the idea of locally linear embedding. The nonlinear discriminate analysis is used to recognize the probe sketch from the synthesized pseudo-sketches. Experimental results on over 600 photo-sketch pairs show that the performance of the proposed method is encouraging.

/pdf/a-nonlinear-approach-for-face-sketch-synthesis-and-29iy7ujqi3.pdf

A nonlinear approach for face sketch synthesis and recognition

In this paper, we propose a real-time face detection approach based on local structure and texture of the objects in gray-level images. Our strategy is to map the local spatial structures and image textures of face class into binary patterns, and use these binary patterns as local descriptors. Boosting based face detector is constructed using these local descriptors, and cascade scheme is employed to further improve the efficiency of the face detector. Compared to the existing face detection approaches, our proposed method has two advantages: (1) it is robust to illumination changes to some extend, for the features use the information of local relationship instead of the original gray values; (2) the computational cost is very low, both in training procedure and evaluation step. The experimental results show that the proposed method can meet the demand of real-time applications with a satisfied detection performance

/pdf/learning-local-descriptors-for-face-detection-47n43h2x5k.pdf

Learning Local Descriptors for Face Detection

http://mmlab.ie.cuhk.edu.hk/2008/gk19.pdf

A new extension of kernel feature and its application for visual recognition

Inthis paper, wepropose arealtime facedetection approach based onlocal structure andtexture oftheobjects ingray-level images. Ourstrategy istomapthelocal spatial structures andimagetextures offace class into binary patterns, andusethese binary patterns aslocal descriptors. Boosting based facedetector isconstructed using these localdescriptors, andcascade scheme isemployed tofurther improve theefficiency oftheface detector. Compared tothe existing facedetection approaches, ourproposed method hastwoadvantages: (1)itisrobust toillumination changes tosomeextend, forthefeatures usetheinformation oflocal relationship instead oftheoriginal grayvalues; (2)thecomputational costisverylow, bothintraining procedure and evaluation step. Theexperimental results showthat theproposed method canmeetthedemandofrealtime applications withasatisfied detection performance.

http://ieeexplore.ieee.org/iel5/10203/32544/01521576.pdf

Hongliang Jin

Papers

A nonlinear approach for face sketch synthesis and recognition

Learning Local Descriptors for Face Detection

A new extension of kernel feature and its application for visual recognition

Learning LocalDescriptors forFaceDetection