This paper describes our design and implementation of a computer augmented environment that allows users to smoothly interchange digital information among their portable computers, table and wall displays, and other physical objects. Supported by a camera-based object recognition system, users can easily integrate their portable computers with the pre-installed ones in the environment. Users can use displays projected on tables and walls as a spatially continuous extension of their portable computers. Using an interaction technique called hyperdragging, users can transfer information from one computer to another, by only knowing the physical relationship between them. We also provide a mechanism for attaching digital data to physical objects, such as a videotape or a document folder, to link physical and digital spaces.

/pdf/augmented-surfaces-a-spatially-continuous-work-space-for-2zz5xi71zo.pdf

Augmented surfaces: a spatially continuous work space for hybrid computing environments

Body posture and finger pointing are a natural modality for human-machine interaction, but first the system must know what it's seeing.

Vision-based hand-gesture applications

This paper introduces the Everywhere Displays projector, a device that uses a rotating mirror to steer the light from an LCD/DLP projector onto different surfaces of an environment. Issues of brightness, oblique projection distortion, focus, obstruction, and display resolution are examined. Solutions to some of these problems are described, together with a plan to use a video camera to allow device-free interaction with the projected images. The ED-projector is a practical way to create ubiquitous graphical interfaces to access computational power and networked data. In particular, it is envisioned as an alternative to the carrying of laptops and to the installation of displays in furniture, objects, and walls. In addition, the use of ED-projectors to augment reality without the use of goggles is examined and illustrated with examples.

The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces

We present an end-to-end system that goes from video sequences to high resolution, editable, dynamically controllable face models. The capture system employs synchronized video cameras and structured light projectors to record videos of a moving face from multiple viewpoints. A novel spacetime stereo algorithm is introduced to compute depth maps accurately and overcome over-fitting deficiencies in prior work. A new template fitting and tracking procedure fills in missing data and yields point correspondence across the entire sequence without using markers. We demonstrate a data-driven, interactive method for inverse kinematics that draws on the large set of fitted templates and allows for posing new expressions by dragging surface points directly. Finally, we describe new tools that model the dynamics in the input sequence to enable new animations, created via key-framing or texture-synthesis techniques.

Spacetime faces: high resolution capture for modeling and animation

Blending reality and virtuality, these interfaces let users see each other, along with virtual objects, allowing communication behaviors much more like face-to-face than like screen-based collaboration.

/pdf/collaborative-augmented-reality-43899655jg.pdf

Collaborative augmented reality

We introduce ideas, proposed technologies, and initial results for an office of the future that is based on a unified application of computer vision and computer graphics in a system that combines and builds upon the notions of the CAVE™, tiled display systems, and image-based modeling . The basic idea is to use real-time computer vision techniques to dynamically extract per-pixel depth and reflectance information for the visible surfaces in the office including walls, furniture, objects, and people, and then to either project images on the surfaces, render images of the surfaces , or interpret changes in the surfaces. In the first case, one could designate every-day (potentially irregular) real surfaces in the office to be used as spatially immersive display surfaces, and then project high-resolution graphics and text onto those surfaces. In the second case, one could transmit the dynamic image-based models over a network for display at a remote site. Finally, one could interpret dynamic changes in the surfaces for the purposes of tracking, interaction, or augmented reality applications. To accomplish the simultaneous capture and display we envision an office of the future where the ceiling lights are replaced by computer controlled cameras and “smart” projectors that are used to capture dynamic image-based models with imperceptible structured light techniques, and to display high-resolution images on designated display surfaces. By doing both simultaneously on the designated display surfaces, one can dynamically adjust or autocalibrate for geometric, intensity, and resolution variations resulting from irregular or changing display surfaces, or overlapped projector images. Our current approach to dynamic image-based modeling is to use an optimized structured light scheme that can capture per-pixel depth and reflectance at interactive rates. Our system implementation is not yet imperceptible, but we can demonstrate the approach in the laboratory. Our approach to rendering on the designated (potentially irregular) display surfaces is to employ a two-pass projective texture scheme to generate images that when projected onto the surfaces appear correct to a moving headtracked observer. We present here an initial implementation of the overall vision, in an office-like setting, and preliminary demonstrations of our dynamic modeling and display techniques.

/pdf/the-office-of-the-future-a-unified-approach-to-image-based-1gm8bz8bj6.pdf

Lev Stesin

Papers

The office of the future: a unified approach to image-based modeling and spatially immersive displays