The capacity to estimate the head pose of another person is a common human ability that presents a unique challenge for computer vision systems. Compared to face detection and recognition, which have been the primary foci of face-related vision research, identity-invariant head pose estimation has fewer rigorously evaluated systems or generic solutions. In this paper, we discuss the inherent difficulties in head pose estimation and present an organized survey describing the evolution of the field. Our discussion focuses on the advantages and disadvantages of each approach and spans 90 of the most innovative and characteristic papers that have been published on this topic. We compare these systems by focusing on their ability to estimate coarse and fine head pose, highlighting approaches that are well suited for unconstrained environments.

Head Pose Estimation in Computer Vision: A Survey

https://www.cse.unr.edu/~mircea/Publications/mn_cviu07.pdf

Vision-based hand pose estimation: A review

Communication is an important issue in man-to-robot interaction. Signs can be used to interact with machines by providing user instructions or commands. Embodiment of the present invention include human detection, human body parts detection, hand shape analysis, trajectory analysis, orientation determination, gesture matching, and the like. Many types of shapes and gestures are recognized in a non-intrusive manner based on computer vision. A number of applications become feasible by this sign-understanding technology, including remote control of home devices, mouse-less (and touch-less) operation of computer consoles, gaming, and man-robot communication to give instructions among others. Active sensing hardware is used to capture a stream of depth images at a video rate, which is consequently analyzed for information extraction.

Sign based human-machine interaction

A method is presented for robust tracking in highly cluttered environments. The method makes effective use of 3D depth sensing technology, resulting in illumination-invariant tracking. A few applications using tracking are presented including face tracking and hand tracking.

Visual Tracking Using Depth Data

We present a multi-PC/camera system that can perform 3D reconstruction and ellipsoid fitting of moving humans in real time. The system consists of five cameras. Each camera is connected to a PC which locally extracts the silhouettes of the moving person in the image captured by the camera. The five silhouette images are then sent, via local network, to a host computer to perform 3D voxel-based reconstruction by an algorithm called SPOT. Ellipsoids are then used to fit the reconstructed data. By using a simple and user-friendly interface, the user can display and observe, in real time and from any view-point, the 3D models of the moving human body. With a rate of higher than 15 frames per second, the system is able to capture non-intrusively, a sequence of human motions.

A real time system for robust 3D voxel reconstruction of human motions

We propose a new technique for simultaneously executing face deformation modeling and 3D head pose estimation. Previous methods for estimating 3D head pose require a preliminary training stage for the head model, and cannot start tracking the head pose until this stage is complete. In contrast, our proposed method can acquire and refine a user’s deformable head model in parallel with tracking the user’s head pose. This allows progressive improvement in the accuracy of the estimation of head pose and face deformation.

Our technique consists of three main steps. In the first step we estimate the 3D head pose using a head model that is obtained automatically. The second step finds true positions of feature points by using the resulting poses of the first step. Finally, the basis vectors of face deformation are calculated from the true positions of feature points to acquire a new deformable head model as a linear combination of the basis vectors.

The iteration of the three steps refines the deformable head model, thus improving the accuracy of head pose estimation progressively. The improvement has been successfully demonstrated via experiments.

Real-time modeling of face deformation for 3d head pose estimation

This paper describes a two-handed drawing tool developed on our augmented desk system. Using our real-time finger tracking method, a user can draw and manipulate objects interactively by his/her own finger/hand. Based on the former work on two-handed interaction, different roles are assigned to each hand. The right hand is used to draw and to manipulate objects. Using gesture recognition, primitive objects can be drawn by users' handwriting. On the other hand, the left hand is used to manipulate menus and to assist the right hand. By closing all left hand fingers, users can initiate the appearance of structural radial menus around their left hands, and can select appropriate items by using a left hand finger. The left hand is also used to assist in the performance of drawing tasks, e.g., specifying the center of a circle or top-left corner of a rectangle, or specifying the object to be copied.

Two-handed drawing on augmented desk system

PROBLEM TO BE SOLVED: To provide an animation editing system which can make gestures with the hands and fingers execute physical operations. SOLUTION: A user 160 executes each operation for editing animations by means of the user's gestures with the hands and fingers on an animation editing screen which is projected to a screen 150 from a projector 130 connected to an image-projecting computer 110. A camera 140 connected to an image-processing computer 120 picks up the gestures, and the image-processing computer 120 recognizes the positions of the hands and fingers. The image pickup computer 110 recognizes the meanings of the gestures from the positions of the user's 160 hands and fingers, executes image editing processing according to the gestures, and projects the animation editing screen again to the screen 150. The above interface solves the problems of usual animation editing systems operated with a mouse or the like. COPYRIGHT: (C)2006,JPO&NCIPI

Animation editing system

An imaging device ( 1 ) includes a camera unit ( 3 ) that captures images of the same object, respectively, using two optical systems, a face part detection unit ( 9 ) that detects a plurality of face parts composing a face included in each of the images captured by the camera unit ( 3 ), a face part luminance calculation unit ( 10 ) that calculates luminance of the detected plurality of face parts, and an exposure control value determination unit ( 12 ) that determines an exposure control value of the camera unit ( 3 ) based on the luminance of the plurality of face parts. A distance measurement unit ( 17 ) in the imaging device ( 1 ) measures distances to the face parts based on the images captured by the camera unit ( 3 ) using the exposure control value. Thus, the imaging device ( 1 ) can measure the distances to the face parts with high accuracy.

Image capturing device, operator monitoring device, method for measuring distance to face, and program

This paper presents an overview of our project on an augmented desk interface system called the EnhancedDesk. The EnhancedDesk is equipped with an infrared camera for tracking a users' hand motions, a color video camera for recognizing objects, and projectors for displaying various kinds of digital information onto the desk. The key technical innovations of the EnhancedDesk include: fast and accurate tracking of multiple fingers for direct manipulation of both real and projected objects with bare hands, interactive object registration and recognition with hand gestures, vision-based tracking of a user's gaze, and addition of interactive functionality overlaid on the desk. This paper presents a brief description of the components, as well as some prototypes of applications of our augmented desk interface system.

Kenji Oka

Papers

Real-time modeling of face deformation for 3d head pose estimation

Two-handed drawing on augmented desk system

Animation editing system

Image capturing device, operator monitoring device, method for measuring distance to face, and program

Video-based tracking of user's motion for augmented desk interface