Light field based face recognition via a fused deep representation

TLDR: This paper proposes the first deep CNN solution for light field based face recognition, exploiting the richer information available in a lenslet light field image, and for the first time, the exploitation of disparity maps together with 2D-RGB images and depth maps has been considered in the context of a fusion scheme to improve the face recognition performance.

Abstract: The emergence of light field cameras opens new frontiers in terms of biometric recognition. This paper proposes the first deep CNN solution for light field based face recognition, exploiting the richer information available in a lenslet light field image. Additionally, for the first time, the exploitation of disparity maps together with 2D-RGB images and depth maps has been considered in the context of a fusion scheme to further improve the face recognition performance. The proposed solution uses the 2D-RGB central sub-aperture view as well as the disparity and depth maps extracted from the full set of sub-aperture images associated to a lenslet light field. After, feature extraction is performed using a VGG-Face deep descriptor for texture and independently fine-tuned models for disparity and depth maps. Finally, the extracted features are concatenated to be fed into an SVM classifier. A comprehensive set of experiments has been conducted with the IST-EURECOM light field face database, showing the superior performance of the fused deep representation for varied and challenging recognition tasks.