Architecture for implementing a perceptual user interface. The architecture comprises alternative modalities for controlling computer application programs and manipulating on-screen objects through hand gestures or a combination of hand gestures and verbal commands. The perceptual user interface system includes a tracking component that detects object characteristics of at least one of a plurality of objects within a scene, and tracks the respective object. Detection of object characteristics is based at least in part upon image comparison of a plurality of images relative to a course mapping of the images. A seeding component iteratively seeds the tracking component with object hypotheses based upon the presence of the object characteristics and the image comparison. A filtering component selectively removes the tracked object from the object hypotheses and/or at least one object hypothesis from the set of object hypotheses based upon predetermined removal criteria.

Architecture for controlling a computer using hand gestures

A user inputs digital data to a companion system (80) such as a PDA, a cell telephone, or an appliance device using a virtual input device (50) such as an image of a keyboard. A sensor (20) captures three-dimensional positional information as to the location of the user's fingers (30) in relation to where keys would be on an actual keyboard. This information is processed with respect to finger (30) locations and velocities and shape to determine when virtual keys would have been struck. The processed digital information is output to the companion system (80). The companion system (80) can display an image of a keyboard, including an image of a keyboard showing user fingers, and/or alphanumeric text as such data is input by the user on the virtual input device.

Method and apparatus for entering data using a virtual input device

A depth camera system uses a structured light illuminator and multiple sensors such as infrared light detectors, such as in a system which tracks the motion of a user in a field of view. One sensor can be optimized for shorter range detection while another sensor is optimized for longer range detection. The sensors can have a different baseline distance from the illuminator, as well as a different spatial resolution, exposure time and sensitivity. In one approach, depth values are obtained from each sensor by matching to the structured light pattern, and the depth values are merged to obtain a final depth map which is provided as an input to an application. The merging can involve unweighted averaging, weighted averaging, accuracy measures and/or confidence measures. In another approach, additional depth values which are included in the merging are obtained using stereoscopic matching among pixel data of the sensors.

Depth camera based on structured light and stereo vision

Techniques for facilitating interaction with an application in a motion capture system allow a person to easily begin interacting without manual setup. A depth camera system tracks a person in physical space and evaluates the person's intent to engage with the application. Factors such as location, stance, movement and voice data can be evaluated. Absolute location in a field of view of the depth camera, and location relative to another person, can be evaluated. Stance can include facing a depth camera, indicating a willingness to interact. Movements can include moving toward or away from a central area in the physical space, walking through the field of view, and movements which occur while standing generally in one location, such as moving one's arms around, gesturing, or shifting weight from one foot to another. Voice data can include volume as well as words which are detected by speech recognition.

Recognizing user intent in motion capture system

In a motion capture system having a depth camera, a physical interaction zone of a user is defined based on a size of the user and other factors The zone is a volume in which the user performs hand gestures to provide inputs to an application The shape and location of the zone can be customized for the user The zone is anchored to the user so that the gestures can be performed from any location in the field of view Also, the zone is kept between the user and the depth camera even as the user rotates his or her body so that the user is not facing the camera A display provides feedback based on a mapping from a coordinate system of the zone to a coordinate system of the display The user can move a cursor on the display or control an avatar

Physical interaction zone for gesture-based user interfaces

A method and apparatus for mapping depth of an object (22) in a preferred arrangement uses a projected light pattern to provide a selected texture to the object (22) along the optical axis (24) of observation. An imaging system senses (32, 34) first and second images of the object (22) with the projected light pattern and compares the defocused of the projected pattern in the images to determine relative depth of elemental portions of the object (22).

Apparatus and methods for determining the three-dimensional shape of an object using active illumination and relative blurring in two images due to defocus

Minori Noguchi

Papers

Apparatus and methods for determining the three-dimensional shape of an object using active illumination and relative blurring in two images due to defocus