Augment-able Reality:
Situated Communication through Physical and Digital Spaces
Jun Rekimoto Yuji Ayatsuka Kazuteru Hayashi
Sony Computer Science Laboratory Information Technology Laboratories Dept. of Information Science
3-14-13 Sony Corporation Tokyo Institute of Technology
Higashigotanda, Shinagawa-ku Kitashinagawa, Shinagawa-ku Ookayama Meguro-ku
Tokyo 141-0022 Japan Tokyo 141-0001 Japan Tokyo Japan
rekimoto@csl.sony.co.jp aya@csl.sony.co.jp k-hayasi@is.titech.ac.jp
Abstract
Most existing augmented reality systems only provide
a method for browsing information that is situated in
the real world context. This paper describes a system
that allows users to dynamically attach newly created
digital information such as voice notes or photographs to
the physical environment, through wearable computers as
well as normal computers. Attached data is stored with
contextual tags such as location IDs and object IDs that
are obtained by wearable sensors, so the same or other
wearable users can notice them when they come to the
same context. Similar to the role that Post-it notes play in
community messaging, we expect our proposed method to
be a fundamental communication platform when wearable
computers become commonplace.
1 Introduction
Augmented Reality (AR) systems are designed to
provide an enhanced view of the real world through
see-through head-mounted displays[16] or hand-held
devices[11]. Various kinds of context sensing technolo-
gies, such as position sensors[5, 4], or ID readers[11],
are used to determine digital information according to the
user’s current physical context. Compared with traditional
information retrieval systems, users of AR systems can get
context-aware information without being bothered by cum-
bersomeoperationsthatmightimpedetheuser’sreal-world
tasks.
The concept of wearable computing is pushing this
direction even further. Wearable computers are always
‘‘on’’ always acting, and always sensing the surrounding
environment to offer a better interface to the real world.
Taking into account recent advances in wearable devices,
we can expect that people will soon carry their wearables
at all times, and that our lives will be constantly supported
by context-sensitive information from those wearables.
While most of the existing augmented reality systems
are mainly focusing on context-aware information presen-
tation, information registration interfaces for AR are yet
to be investigated. For example, KARMA[4], which is
a well-known AR system, displays information about the
laser printer based on the current physical position of a
head-worn display. Such information is assumed to be
prepared beforehand. To add new information during a
task, the user has to return to the normal computer envi-
ronment. In that sense, current AR systems are essentially
‘‘context-sensitive browsers’’ for the real world, and in-
formation is limited to a one-way flow. Since two-way
communication tools (e.g, E-mail) have fundamentally dif-
ferent functionality and applications from one-way tools
(e.g., radio), we can imagine several new usages of AR if
it allows bidirectional communications.
This paper presents an environment that supports in-
formation registration in the real world contexts through
wearable and traditional computers. Using this technique,
a user of wearable computers can dynamically create a
data item such as a voice or a photograph and can attach
it to nearby physical objects or locations. To enable this,
the system supports drag-and-drop operations between per-
sonal and context-sensitive spaces. The user of traditional
computers can also add data to the physical environment
through the Web and E-Mail interfaces. By combining
these information flows, we can freely transfer data from
physical space to digital space, or vice versa. We call such
an environment "augment-able reality", because users of
our system are not just browsing information situated in
Sensor
Voice, Photo,
Text, etc.
Sensor
Wearable
Computer
Database
Physical Context
Context-sensitive
Information
Real World
Environment
Wearable
Computer
Database
Physical Context
Context-sensitive
Information
Real World
Environment
Dynamic Creation of
Augmenting Information
Room A
Information Registration and
Browsing through WWW and E-Mail
Interfaces
(a) Traditional Augmented Reality Systems
(
b
)
Au
g
mentable Realit
y
Environments
Commnication
through situated
Digital Information
Figure 1: The Augment-able Reality Concept
the real world, but can also create, attach, or carry the data
with them.
2 Augment-able Environments
Let us start with a scenario when augment-able reality
becomes commonplace.
When you enter the video studio, your eyeglass notifies
you that a new video effector has been installed. You
approach the effector and find a movie that was attached
by your colleague, describing typical usage of equipment.
Meanwhile, you find the VCR is broken and the tape has
been damaged. You create a voice-note with a still picture
of the damaged tape and attach it to the VCR for other
users.
When you go outside for lunch, you find several virtual
messages are floating in front of restaurants. Some are
commercials, but many are messages that were created
and attached by previous visitors, giving information and
ratings on the restaurant. You select a restaurant by
looking at these messages. While you are at lunch, you
remember that you will have visitors at noon. You select an
icon representing your office in the eyeglass, and attach a
voice memo to it ....
As we can see in this example, the concept of augment-
able reality can be summarized in features outlined in the
following sections.
Dynamic creation of augmenting information
People wearing computers and having network facilities
can dynamically create information and attach it to the
user’s surrounding physical environment. From the sys-
tem’s point of view, this is achieved by storing a created
data item with contextual information obtained from sev-
eral wearable sensors. Our current prototype supports
infrared beacons and visual ID markers as contextual infor-
mation. Another possibility is to use geographical locations
based on GPS (global positioning system) or PHS(personal
handy phone) as contextual information.
There are several possibilities about data types that can
be attached to the environment. Our current prototype
supports voice notes and photograph snapshots. Other pos-
sible data types are text (created by a wearable keyboard)
or hand-drawn strokes (created by a miniature touch panel).
Context-sensitive information notification
Similar to other AR systems, the attached data can be
browsed by (the same or other) users who are wearing
computers. The concept of ‘‘wearing’’ is essential; other-
wise users would not notice digitally attached information.
Information sharing among wearable and normal
computer environments
Attached information can also be accessible from users in
normal computing environments. For example, when a
wearable user attaches a voice note to the meeting room, it
becomes visible on the floor map page of the Web browser.
Conversely, normal computer users can attach data to the
physical context through the Web or E-Mail interfaces, and
wearable users will notice the data when they come to the
corresponding situation.
Communication through situated information
The combination of these features creates a new way of
communication. Instead of sending messages to people,
we can place message in "physical contexts" to indirectly
communicate with other people. After having dinner
at a restaurant, for example, people might leave their
impressions or ratings of that restaurant at that location.
Afterwards, other people would be able to check the
Sub-note
PC
CCD Camera
Miniature
Mouse
Microphone
Earphone
IR-sensor
Monocular
See-through
HUD
Interface
Box
Camera Interface
(PCMCIA)
Wireless LAN
(PCMCIA)
Shared
Database
E-mail Interface
Web (Java Applets)
E-Mail Clients
Wearable Unit
Normal Computers
Figure 2: Schematic of the prototype system
messages from the front of the restaurant, or when they are
browsing a map on the Web.
Even when messages are directed to the person, we can
select several contextual attributes to do that. For example,
you may want to attach an e-mail message to a person’s
office door (locally or remotely through the net), to get
their attention when they return to the office.
3 System Design
To achieve the proposed concept described in the previ-
ous section, we are currently developing a set of prototype
systems based on wearable and normal desktop computers.
Figure 2 shows the overall system design of the prototype.
Systems share the same database server on the network
through wired or wireless communication. If one user
attaches a data item to the particular physical context (e.g.,
location), this effect immediately becomes visible from
other computers.
The following subsections present details of each sub-
system as well as their user interfaces.
Figure 3: Two ID systems deployed in the environment
(left: infrared beacons, right: printed 2D matrix codes)
3.1 Environmental Support
To make it easier for wearable computers to capture
surrounding contexts, we have deployed two kinds ID
systems in the environment. One is an infrared (IR) beacon
that periodically emits a unique number to the environment
(Figure 3, left); the other is a printed 2D matrix code that
can be recognized by a head-worn camera (Figure 3, right).
One noticeabledifferencebetweenthesetwoID systems
is emission range. While IR beacons can cover room-size
areas and are relatively robust regarding orientation of the
sensors, printed IDs are more sensitive to distance and
the camera orientation. We are thus using IR beacons to
detect current rooms (e.g., meeting room, office, etc.) and
printed 2D codes for object level identification (e.g., VCR,
bookshelf, etc.).
Since 2D codes are virtually costless and printable as
well, there are some usages that could not be achieved
by other ID systems. For example, we can use small
Post-it cards with a 2D code. This (physical) card can
convey digital data such as voice notes or photographs
with attachment of the ID.
3.2 Wearable System
Hardware
The head-worn part of the wearable system (Figure 4,
above) consists of a monocular see-through head-up dis-
play (based on Sony Glasstron), a CCD camera, an infrared
sensor (based on a remote commander chip for consumer
electronics). These devices are connected to a sub-note
PC (Mitsubishi AMiTY) communicating to the network
through a spread-spectrum wireless LAN (Netwave Air-
See-through
Monocular Display
Infrared
Sensor
CCD Camera
Figure 4: The head-up and hand-held parts of the wearable
system
Surfer). The user controls the system with a miniature
pointing device. A microphone attached to the pointing
device is used to make voice notes (Figure 4, below). The
head-worn camera is primarily used to detect 2D codes
in the environment, but is also used to take still image
Figure 5: Wearable system is used to attach a voice note to
the physical object
Figure 6: Screen layout of the wearable display. The user
looks at this image on their monocular see-through display.
The above two areas (called context-aware panes) are
corresponding to the current physical contexts (location
and object), while the bottom area (called the personal
information tray) stores personal data carried by the user.
pictures. Figure 5 shows a user creating a voicenote with
the wearable unit.
User interface
As a context-aware browser for the real world, this wear-
able system acts mostly the same as our previous AR
system NaviCam[11]. The system recognizes the sur-
rounding environment by recognizing attached visual tags
on physical objects, or by detecting infrared beacons in-
stalled at locations in the environment. For example, when
the system recognizes a visual tag attached to the door of
the meeting room, the meeting schedule would appear on
the see-through head mounted display. In a same way,
when the user is walking into the cafeteria, an infrared
sensor automatically detects the location by receiving an
IR ID from that location. Since the system is equipped with
a wireless LAN, information can dynamically be retrieved
from the (shared) database on the network.
The unique feature of our new system is its ability to
add new data to the physical environment. To enable
this feature, the user switches the system to ‘‘authoring
mode.’’ A window consisting of a personal information
area and context sensitive area then appears on the see-
through display (Figure 6). The personal information area
(also called "the personal information tray") stores the data
carried by the user. The system currently supports voice
notes and still images (captured by a head-worn camera)
as such data.
The other area, called the ‘‘context-aware area,’’ shows
information related to the current physical context. The
area is further split into two panes, the location pane and
the object pane. These panes represent location-level and
object-levelcontexts. Thelocation-level contextis detected
by the IR beacons, while object-level context is determined
by visual tag recognition.
These context-aware panes change when the user moves
to a new situation. For example, when a user enters a
(a)
(b)
(c)
Figure 7: Creating and attaching data to the physical
context. (a) The user first presses the ‘‘CAPTURE’’ button
to take a snapshot and adds a voice note. (b) The newly
created data object then appears on the user’s personal
information tray. (c) The user drags it to the context-aware
pane to attach the data to the current location (the studio).
meeting room, IR beacons from the room tell the wearable
system the meeting room ID, the location pane switches
to the meeting room and icons attached to that location
appear. A gentle sound also rings to announce this context
switch. The user can then open or playback these icons by
pressing one of them with the right button of the hand-held
mouse.
To support intuitive data transfer between personal and
context-aware spaces, the system provides drag and drop
operations between these areas. That is, the user can
create a voice memo, (which will appear on the user’s
personal area as an voice icon), then drag and drop it to the
location pane to attach it to the current recognized location
(Figure 7). Since the operation is bidirectional, the user
can also drag an object from the context-aware panes to
the personal tray to copy information that is attached (by
someone or by the user himself) on the current physical
context.
For example, suppose that you find that the VCR in front
of you has a problem and may damage video tapes. You
create a voice note such as ‘‘This VCR is broken, do not
insert tapes, or they may be damaged,’’ with a microphone
while pressing a record button on the miniature pointing
device. An icon representing this newly created voice note
then appears on the personal tray. You also see that the
object-sensitive pane is showing the VCR name and picture
to indicate the system is recognizing it. You then drag the
voice note icon from the personal tray to the VCR pane;
the voice note is now attached to the VCR. Afterwards,
other users (wearing a computer) who try to use the VCR
will find your warning.
Time Machine Mode
The context-aware panes are not always directly linked to
the current physical situation. When the user taps on the
left side of the window, the system switches to what we
call the ‘‘Time Machine Mode’’ (Figure 8).
In this mode, the user can navigate through previously
visited rooms or objects without physically visiting these
that contexts, by clicking arrow buttons on the context
panes. Thisinterface is similar tothe forward and backward
buttons on web browsers. The user can also visit the future
(past) by clicking buttons around the time indicator.
Combining these navigation techniques allows a user
to attach data to any context at any time. For example,
suppose that you want to attach a meeting agenda to the
conference room but it should not become visible until next
Monday. To do this, you first (virtually) visit the meeting
room in the Time Machine Mode, change the time to the