MITSUBISHI ELECTRIC RESEARCH LABORATORIES
http://www.merl.com
Modern Approaches to Augmented Reality
Oliver Bimber, Ramesh Raskar
TR2006-105 July 2006
Abstract
This tutorial discusses the Spatial Augmented Reality (SAR) concept, its advantages and lim-
itations. It will present examples of state-of-the-art display configurations, appropriate real-
time rendering techniques, details about hardware and software implementations, and current
areas of application. Specifically, it will describe techniques for optical combination using sin-
gle/multiple spatially aligned mirror-beam splitters, image sources, transparent screens and op-
tical holograms. Furthermore, it presents techniques for projector based augmentation of geo-
metrically complex and textured display surfaces, and (along with optical combination) methods
for achieving consistent illumination and occlusion effects. Emerging technologies that have the
potential of enhancing future augmented reality displays will be surveyed.
ACM SIGGRAPH 2006
This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part
without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include
the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of
the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or
republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All
rights reserved.
Copyright
c
Mitsubishi Electric Research Laboratories, Inc., 2006
201 Broadway, Cambridge, Massachusetts 02139
MERLCoverPageSide2
Bimber and Raskar, Spatial Augmented Reality
Siggraph 2005. 1
Modern Approaches to Augmented Reality
Oliver Bimber
1
and Ramesh Raskar
2
1
Bauhaus University, Weimar, Germany
T +49-3643-583724, F +49-3643-583709, Email: bimber@ieee.org, URL: http://www.uni-weimar.de/medien/AR
2
MERL - Mitsubishi Electric Research Lab, Cambridge, USA
T +1-617-621-7533, F +1-617-621-7550, Email: raskar@merl.com, URL : http://www.merl.com/people/raskar/ , http://www.raskar.com
Abstract
This tutorial discusses the Spatial Augmented Reality (SAR) concept, its advantages and limitations. It will
present examples of state-of-the-art display configurations, appropriate real-time rendering techniques,
details about hardware and software implementations, and current areas of application. Specifically, it will
describe techniques for optical combination using single/multiple spatially aligned mirror-beam splitters,
image sources, transparent screens and optical holograms. Furthermore, it presents techniques for projector-
based augmentation of geometrically complex and textured display surfaces, and (along with optical
combination) methods for achieving consistent illumination and occlusion effects. Emerging technologies that
have the potential of enhancing future augmented reality displays will be surveyed.
Categories and Subject Descriptors
(according to ACM CSS): H.5.1 [Multimedia Information Systems]:
Artificial, Augmented and Virtual Realities; I.3.1 [Hardware Architecture]: Three-dimensional Displays;
I.3.2
[Computer Graphics]: Graphics Systems; I.3.3 [Computer Graphics]: Picture/Image Generation – Display
Algorithms, Viewing Algorithms; I.3.7 [Computer Graphics]: Color, Shading, Shadowing, and Texture
1. Introduction and overview
Video see-through and optical see-through head-mounted
displays have been the traditional output technologies for
augmented reality (AR) applications for more than forty
years. However, they still suffer from several technological
and ergonomic drawbacks which prevent them from being
used effectively in all application areas.
Novel approaches have taken augmented reality
beyond traditional eye-worn or hand-held displays -
enabling additional application areas. New display
paradigms exploit large spatially aligned optical elements,
such as mirror beam-splitters, transparent screens or
holograms, as well as video-projectors. Thus, we call this
technological variation “Spatial Augmented Reality
(SAR)”. In many situations, SAR displays are able to
overcome technological and ergonomic limitations of
conventional AR systems. Due to the fall in cost and
availability of projection technology, personal computers
and graphics hardware, there has been a considerable
interest in exploiting SAR systems in universities, research
laboratories, museums, industry and in the art community.
Parallels to the development of virtual environments from
head-attached displays to spatial projection screens can be
clearly drawn. We believe that an analog evolution of
augmented reality has the potential to yield a similar
successful factor in many application domains. Thereby,
SAR and body-attached AR are not competitive, but
complementary.
Chapter 2 gives an overview over different augmented
reality display techniques, from head-attached, over hand-
held to spatial approaches. It will enable readers to identify
parallels between virtual reality and augmented reality
display technology, and stimulate them to think about
alternative display approaches for AR.
Chapter 3 explains interactive rendering techniques that
use fixed function and programmable rendering pipelines
to support spatial optical see-through displays. They aim at
neutralizing optical effects, such as reflection and
refraction on planar or curved spatial optical combiners,
such as transparent screen or mirror beam-splitter
configurations.
Chapter 4 focuses on rendering methods for projector-
based augmentation and illumination. It will be explained
how a correct projection onto geometrically complex and
textured screen surfaces is performed. Furthermore, it
discusses projector-based illumination, and outlines
Bimber and Raskar, Spatial Augmented Reality
Siggraph 2005
2
examples of how it can be used together with optical
combination to create consistent illumination and occlusion
effects.
In chapter 5 we summarize our tutorial and give an
outlook to enabling technologies that might influence
augmented reality technology in the future. The
possibilities and limitations of technologies, such as
autostereoscopy, video projectors, organic light emitting
diodes, light emitting polymers, electronic paper, particular
solid state volumetric and parallax display approaches, and
holography will be outlined.
2. Augmented Reality Displays
Displays are image-forming systems that apply a set of
optical, electronic and mechanical components to generate
images somewhere on the optical path in-between the
observer’s eyes and the physical object to be augmented.
Depending on the optics being used, the image can be
formed on a plane or on a more complex non-planar
surface.
Figure 2.1 illustrates the different possibilities of where
the image can be formed, where the displays are located
with respect to the observer and the real object, and what
type of image is produced (i.e., planar or curved).
Figure 2.1: Image-generation for augmented reality
displays.
Head-attached displays, such as retinal displays, head-
mounted displays, and head-mounted projectors have to be
worn by the observer. While some displays are hand-held,
others are spatially aligned and completely detached from
the users. Retinal displays and several projector-based
approaches form curved images – either on the observer’s
retina or directly on the physical object. Most of the
displays, however, form images on planes – called image-
planes – that can be either head-attached or spatially
aligned. Images behind real objects cannot be formed by a
display that is located in front of real objects. In addition, if
images are formed behind a real object, this object will
occlude the image portion that is required to support
augmentation.
Several pros and cons can be found by comparing the
different types of displays. Most of them will be discussed
within the following sections.
If stereoscopic rendering is used to present mixed (real
and virtual) worlds, two basic fusion technologies are
currently being used: video-mixing and optical
combination.
While video-mixing merges live record video streams
with computer generated graphics and displays the result
on the screen, optical combination generates an optical
image of the real screen (displaying computer graphics)
which appears within the real environment (or within the
viewer’s visual field while observing the real
environment). Both technologies entail a number of
advantages and disadvantages which influence the type of
application they can address.
Today, most of the stereoscopic AR displays require to
wear some sort of goggles to provide stereo separation.
Auto-stereoscopic approaches, however, might play a
dominant role in the future of AR.
In this chapter, we discuss several types of augmented
reality displays. Note that we rather present examples in
each display category, than to provide a complete list of
individual devices.
2.1. Head-Attached Displays
Head-attached displays require the user to wear the display
system on his/her head. Depending on the image generation
technology, three main types exist: Retinal displays that
apply low power lasers to project images directly onto the
retina of the eye, head-mounted displays that use miniature
displays in front of the eyes, and head-mounted projectors
that make use of miniature projectors or miniature LCD
panels with backlighting and project images on the surfaces
of the real environment.
Bimber and Raskar, Spatial Augmented Reality
Siggraph 2005
3
2.1.1. Retinal Displays
Retinal displays [Kol93, Pry98, Lew04] utilize low-power
semiconductor lasers (or –in future– special light-emitting
diods) to scan modulated light directly onto the retina of
the human eye, instead of providing screens in front of the
eyes. This produces a much brighter and higher resolution
image with a potentially wider field of view than a screen-
based display.
Fig 2.2: Simplified diagram of a retinal Display.
Current retinal displays share many shortcomings with
head-munted displays (see section 2.1.2). However, some
additional disadvantages can be identified for existing
versions:
• Only monochrome (red) images are presented since
cheap low-power blue and green lasers do not yet exist;
• The sense of ocular accommodation is not supported due
to the complete bypass of the ocular motor-system by
scanning directly onto the retina. Consequently, the focal
length is fixed;
• Stereoscopic versions do not exist.
The main advantages of retinal displays are the high
brightness and contrast, and low power consumption –
which make them well suited for mobile outdoor
applications. Future generations also hold the potential to
provide dynamic re-focus, full-color stereoscopic images,
and an extremely high resolution and large field-of-view.
2.1.2. Head-Mounted Displays
Head-mounted displays (HMDs) are currently the display
devices which are mainly used for augmented reality
applications.
Two different head-mounted display-technologies exist
to superimpose graphics onto the user's view of the real
world: Video see-through head-mounted displays that make
use of video-mixing and display the merged images within
a closed-view head-mounted display, or optical see-
through head-mounted displays that make use of optical
combiners (essentially half-silvered mirrors or transparent
LCD displays). A comparison between these two general
technologies can be found in [Rol94].
Figure 2.3: Video see-through (left) and optical see-
through (right). Courtesy: Azuma [Azu97].
Several disadvantages can be related to the application of
head-mounted displays as an augmented reality device.
Note that most of these shortcomings are inherited from the
general limitations of head-attached display technology:
• Lack in resolution that is due to limitations of the applied
miniature displays. In the optical see-through case, only
the graphical overlays suffer from a relatively low
resolution, while the real environment can be perceived
in the resolution of the human visual system. For video
see-through devices, both – the real environment and the
graphical overlays – are perceived in the resolution of
the video-source or display;
• Limited field of view that is due to limitations of the
applied optics;
• Imbalanced ratio between heavy optics (that results in
cumbersome and uncomfortable devices) and ergonomic
devices with a low image quality;
• Visual perception issues that is due to the constant image
depth. For optical see-through: Since objects within the
real environment and the image plane that is attached to
the viewer’s head are sensed at different depths, the eyes
are forced to either continuously shift focus between the
different depth levels, or perceive one depth level
![Figure 2.4: Osaka University’s ELMO – An optical seethrough head-mounted display that provides mutual occlusion by using a see-through LCD panel in front of the HMD optics [Kiy00]. Courtesy: Kiyokawa.](/figures/figure-2-4-osaka-universitys-elmo-an-optical-seethrough-head-3ajynf3e.webp)