A number of techniques have been proposed for flying through scenes by redisplaying previously rendered or digitized views. Techniques have also been proposed for interpolating between views by warping input images, using depth information or correspondences between multiple images. In this paper, we describe a simple and robust method for generating new views from arbitrary camera positions without depth information or feature matching, simply by combining and resampling the available images. The key to this technique lies in interpreting the input images as 2D slices of a 4D function the light field. This function completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination. We describe a sampled representation for light fields that allows for both efficient creation and display of inward and outward looking views. We hav e created light fields from large arrays of both rendered and digitized images. The latter are acquired using a video camera mounted on a computer-controlled gantry. Once a light field has been created, new views may be constructed in real time by extracting slices in appropriate directions. Since the success of the method depends on having a high sample rate, we describe a compression system that is able to compress the light fields we have generated by more than a factor of 100:1 with very little loss of fidelity. We also address the issues of antialiasing during creation, and resampling during slice extraction. CR Categories: I.3.2 [Computer Graphics]: Picture/Image Generation — Digitizing and scanning, Viewing algorithms; I.4.2 [Computer Graphics]: Compression — Approximate methods Additional keywords: image-based rendering, light field, holographic stereogram, vector quantization, epipolar analysis

/pdf/light-field-rendering-3fds6lbjg4.pdf

Light field rendering

Paul Milgram received the BASc degree from the University of Toronto in 1970, the MSEE degree from the Technion (Israel) in 1973 and the PhD degree from the University of Toronto in 1980 From 1980 to 1982 he was a ZWO Visiting Scientist and a NATO Postdoctoral in the Netherlands, researching automobile driving behaviour From 1982 to 1984 he was a Senior Research Engineer in Human Engineering at the National Aerospace Laboratory (NLR) in Amsterdam, where his work involved the modelling of aircraft flight crew activity, advanced display concepts and control loops with human operators in space teleoperation Since 1986 he has worked at the Industrial Engineering Department of the University of Toronto, where he is currently an Associate Professor and Coordinator of the Human Factors Engineering group He is also cross appointed to the Department of Psychology In 1993-94 he was an invited researcher at the ATR Communication Systems Research Laboratories, in Kyoto, Japan His research interests include display and control issues in telerobotics and virtual environments, stereoscopic video and computer graphics, cognitive engineering, and human factors issues in medicine He is also President of Translucent Technologies, a company which produces "Plato" liquid crystal visual occlusion spectacles (of which he is the inventor), for visual and psychomotor research

/pdf/a-taxonomy-of-mixed-reality-visual-displays-54v3x9pjl3.pdf

A Taxonomy of Mixed Reality Visual Displays

In this paper, I address the question as to why participants tend to respond realistically to situations and events portrayed within an immersive virtual reality system. The idea is put forward, based on the experience of a large number of experimental studies, that there are two orthogonal components that contribute to this realistic response. The first is ‘being there’, often called ‘presence’, the qualia of having a sensation of being in a real place. We call this place illusion (PI). Second, plausibility illusion (Psi) refers to the illusion that the scenario being depicted is actually occurring. In the case of both PI and Psi the participant knows for sure that they are not ‘there’ and that the events are not occurring. PI is constrained by the sensorimotor contingencies afforded by the virtual reality system. Psi is determined by the extent to which the system can produce events that directly relate to the participant, the overall credibility of the scenario being depicted in comparison with expectations. We argue that when both PI and Psi occur, participants will respond realistically to the virtual reality.

/pdf/place-illusion-and-plausibility-can-lead-to-realistic-1hpdya0yai.pdf

Place Illusion and Plausibility Can Lead to Realistic Behaviour in Immersive Virtual Environments

We introduce ideas, proposed technologies, and initial results for an office of the future that is based on a unified application of computer vision and computer graphics in a system that combines and builds upon the notions of the CAVE™, tiled display systems, and image-based modeling . The basic idea is to use real-time computer vision techniques to dynamically extract per-pixel depth and reflectance information for the visible surfaces in the office including walls, furniture, objects, and people, and then to either project images on the surfaces, render images of the surfaces , or interpret changes in the surfaces. In the first case, one could designate every-day (potentially irregular) real surfaces in the office to be used as spatially immersive display surfaces, and then project high-resolution graphics and text onto those surfaces. In the second case, one could transmit the dynamic image-based models over a network for display at a remote site. Finally, one could interpret dynamic changes in the surfaces for the purposes of tracking, interaction, or augmented reality applications. To accomplish the simultaneous capture and display we envision an office of the future where the ceiling lights are replaced by computer controlled cameras and “smart” projectors that are used to capture dynamic image-based models with imperceptible structured light techniques, and to display high-resolution images on designated display surfaces. By doing both simultaneously on the designated display surfaces, one can dynamically adjust or autocalibrate for geometric, intensity, and resolution variations resulting from irregular or changing display surfaces, or overlapped projector images. Our current approach to dynamic image-based modeling is to use an optimized structured light scheme that can capture per-pixel depth and reflectance at interactive rates. Our system implementation is not yet imperceptible, but we can demonstrate the approach in the laboratory. Our approach to rendering on the designated (potentially irregular) display surfaces is to employ a two-pass projective texture scheme to generate images that when projected onto the surfaces appear correct to a moving headtracked observer. We present here an initial implementation of the overall vision, in an office-like setting, and preliminary demonstrations of our dynamic modeling and display techniques.

/pdf/the-office-of-the-future-a-unified-approach-to-image-based-1gm8bz8bj6.pdf

The office of the future: a unified approach to image-based modeling and spatially immersive displays

Understanding Virtual Reality arrives at a time when the technologies behind virtual reality have advanced to the point that it is possible to develop and deploy meaningful, productive virtual reality applications. The aim of this thorough, accessible exploration is to help you take advantage of this moment, equipping you with the understanding needed to identify and prepare for ways VR can be used in your field, whatever your field may be.


By approaching VR as a communications medium, the authors have created a resource that will remain relevant even as the underlying technologies evolve. You get a history of VR, along with a good look at systems currently in use. However, the focus remains squarely on the application of VR and the many issues that arise in the application design and implementation, including hardware requirements, system integration, interaction techniques, and usability. This book also counters both exaggerated claims for VR and the view that would reduce it to entertainment, citing dozens of real-world examples from many different fields and presenting (in a series of appendices) four in-depth application case studies.

* Substantive, illuminating coverage designed for technical and business readers and well-suited to the classroom.
* Examines VR's constituent technologies, drawn from visualization, representation, graphics, human-computer interaction, and other fields, and explains how they are being united in cohesive VR systems.
* Via a companion Web site, provides additional case studies, tutorials, instructional materials, and a link to an open-source VR programming system.

Table of Contents

Foreword 
Preface

Part I - What is Virtual Reality?
Chapter 1: Introduction to Virtual Reality - What it is and Where it Comes From
Chapter 2: VR the Medium

Part II - Virtual Reality Systems 
Chapter 3: Interface to the Virtual World -- Input 
Chapter 4: Interface to the Virtual World -- Output 
Chapter 5: Rendering a Virtual World 
Chapter 6: Interacting with a Virtual World 
Chapter 7: Virtual Reality Experience 
Chapter 8: Experience Design: Applying VR to a Problem
Chapter 9: What Dreams May Come: The Future of VR 

Appendices
A: NICE educational application (EVL) 
B: Crumbs visualization application (NCSA)
C: Aircraft wiring application (Boeing, Inc.) 
D: Placeholder artistic application (Interval Research)

Understanding Virtual Reality: Interface, Application, and Design

A study by Slater, et al., [1995] indicated that naive subjects in an immersive virtual environment experience a higher subjective sense of presence when they locomote by walking-in-place (virtual walking) than when they push-button-fly (along the floor plane). We replicated their study, adding real walking as a third condition. Our study confirmed their findings. We also found that real walking is significantly better than both virtual walking and flying in ease (simplicity, straightforwardness, naturalness) as a mode of locomotion. The greatest difference in subjective presence was between flyers and both kinds of walkers. In addition, subjective presence was higher for real walkers than virtual walkers, but the difference was statistically significant only in some models. Follow-on studies show virtual walking can be substantially improved by detecting footfalls with a head accelerometer. As in the Slater study, subjective presence significantly correlated with subjects' degree of association with their virtual bodies (avatars). This, our strongest statistical result, suggests that substantial potential presence gains can be had from tracking all limbs and customizing avatar appearance. An unexpected by-product was that real walking through our enhanced version of Slater's visual-cliff virtual environment (Figure 1) yielded a strikingly compelling virtual experience - The strongest we and most of our visitors have yet experienced. The most needed system improvement is the substitution of wireless technology for all links to the user. Copyright ACM 1999.

/pdf/walking-walking-in-place-flying-in-virtual-environments-5e18hc0psi.pdf

Walking > walking-in-place > flying, in virtual environments

The defining characteristics of what we call “Fish Tank Virtual Reality” are a stereo image of a three dimensional (3D) scene viewed on a monitor using a perspective projection coupled to the head position of the observer. We discuss some of the relative merits of this mode of viewing as compared to head mounted stereo displays. In addition, we report the experimental investigation of the following variables: 1) whether or not the perspective view is coupled to the actual viewpoint of the observer, 2) whether stereopsis is employed. Experiment 1 involved the subjective comparison of pairs of viewing conditions and the results suggest that head coupling may be more important than stereo in yielding a strong impression of three dimensionality. Experiment 2 involved subjects tracing a path from a leaf of a 3D tree to the correct root (there were two trees intermeshed). The error rates ranged from 22% in the pictorial display, to 1.3% in the head coupled stereo display. The error rates for stereo alone and head coupling alone were 14.7% and 3.2% respectively. We conclude that head coupling is probably more important than stereo in 3D visualization and that head coupling and stereo combined provide an important enhancement to monitor based computer graphics.

http://ccom.unh.edu/vislab/PDFs/FishTankVR.pdf

Fish tank virtual reality

“Fish tank virtual reality” refers to the use of a standard graphics workstation to achieve real-time display of 3D scenes using stereopsis and dynamic head-coupled perspective. Fish tank VR has a number of advantages over head-mounted immersion VR which makes it more practical for many applications. After discussing the characteristics of fish tank VR, we describe a set of three experiments conducted to study the benefits of fish tank VR over a traditional workstation graphics display. These experiments tested user performance under two conditions: (a) whether or not stereoscopic display was used and (b) whether or not the perspective display was coupled dynamically to the positions of a user’s eyes. Subjects using a comparison protocol consistently preferred head coupling without stereo over stereo without head coupling. Error rates in a tree-tracing task similar to one used by Sollenberger and Milgram showed an order of magnitude improvement for head-coupled stereo over a static (nonhead-coupled) display, and the benefits gained by head coupling were more significant than those gained from stereo alone. The final experiment examined two factors that are often associated with human performance in virtual worlds: the lag (or latency) in receiving and processing tracker data arid the rate at which frames are updated. For the tree-tracing task, lag had a larger impact on performance than did frame update rate, with lag having a multiplicative effect on response time. We discuss the relevance of these results for the display of complex 3D data and highlight areas requiring further study,

Evaluating 3D task performance for fish tank virtual worlds

Effects of field of view on performance with head-mounted displays

A new approach to telepresence is presented in which a multitude of stationary cameras are used to acquire both photometric and depth information A virtual environment is constructed by displaying the acquired data from the remote site in accordance with the head position and orientation of a local participant Shown are preliminary results of a depth image of a human subject calculated from closely spaced video camera positions A user wearing a head mounted display walks around this D data that has been inserted into a D model of a simple room Future systems based on this approach may exhibit more natural and intuitive interaction among participants than current D teleconferencing systems

Kevin Arthur

Papers

Walking > walking-in-place > flying, in virtual environments

Fish tank virtual reality

Evaluating 3D task performance for fish tank virtual worlds

Effects of field of view on performance with head-mounted displays

Virtual Space Teleconferencing Using a Sea of Cameras