Dallas, August 18-22 Volume 20, Number 4, 1986
Creating Highly-Interactive and Graphical User Interfaces
by Demonstration
Brad A. Myers
and
William Buxton
Dynamic Graphics Project
Computer Systems Research Institute
University of Toronto
Toronto, Ontario, M5S 1A4
Canada
ABSTRACT
It is very time-consuming and expensive to create the
graphical, highly-interactive styles of user interfaces that
are increasingly common. User Interface Management
Systems (UIMSs) attempt to make the creation of user
interfaces easier, but most existing UIMSs cannot create
the low-level interaction techniques (pop-up, pull-down and
fixed menus, on-screen "light buttons", scroll-bars, ela-
borate feedback mechanisms and animations, etc.) that are
frequently used. This paper describes Peridot, a system
that automatically creates the code for these user inter-
faces while the designer
demonstrates
to the system how
the interface should look and work. Peridot uses rule-
based inferencing so no programming by the designer is
required, and Direct Manipulation techniques are used to
create Direct Manipulation interfaces, which can make full
use of a mouse and other input devices. This allows
extremely rapid protetyping of user interfaces.
CR Categories and Subject Descriptors: D.1.2
[Program-
ming
Techniques]: Automatic Programming; D.2.2
[Software Engineering]: Tools and Techniques -
User
Interfaces;
1.2.2 [Artificial Intelligence]: Automatic Pro-
gramming -
Program Synthesis;
1.3.6 [Computer Graph-
ics]: Methodology and Techniques.
General Terms: Human Factors.
Additional Key Words and Phrases: Programming by
Example, Visual Programming, User Interface Design,
User Interface Management Systems, Graphical User
Interfaces, Direct Manipulation.
Permission to copy without fee all or part of this material is granted
provided that the copies are not made or distributed for direct
commercial advantage, the ACM copyright notice and the title of the
publication and its date appear, and notice is given that copying is by
permission of the Association for Computing Machinery. To copy
otherwise, or to republish, requires a fee and/or specific permission.
© 1986 ACM 0-89791-196-2/86/008/0249 $00.75
1. Introduction
This paper discusses Peridot, a new User Interface
Management System (UIMS) currently under development,
that can create graphical, highly interactive user inter-
faces. Peridot stands for P_rogramming by Example for
Real-time Interface Design Obviating _Typing. It is imple-
mented in Interlisp-D [Xerox 83] on a Xerox DandeTiger
(1109) workstation, and allows the user interface designer
to create user interfaces by
demonstrating
what the user
interface should look like and how the end user will
interact with it. This approach frees designers from hav-
ing to do any programming in the conventional sense, and
allows them to design the user interface in a very natural
manner. The general strategy of Peridot is to allow the
designer to
draw the
screen display that the end user will
see, and to perform actions just as the end user would,
such as moving a mouse, or pressing a mouse button or
keyboard key. The system attempts to guess (or
infer)
the
relationship of that action to existing elements of the user
interface based on context, and asks the designer if the
guess is correct. If so, a piece of cede is generated by the
system that will handle this action for the end user. If
incorrect, other reasonable guesses are tried, or the
designer can explicitly specify the relationship.
The guesses are encoded as simple condition-action
rules, and the generated code is put into small parameter-
ized procedures to help ensure a structured design of the
resulting system. The screen displays and interactions
depend on the values of the parameters to the procedures.
The procedures created by Peridot can be called from appli-
cation programs or used in other user interface procedures
created by demonstration.
Many user interface designers now draw, typically on
paper, scenarios (or "story boards") of how the user inter-
face (UI) will look and act. Unfortunately, it is difficult to
get a feeling for how a system works from the paper
descriptions, and customers of the user interface are not
able to investigate how the system will work. Peridot
enhances the design process by supporting extremely rapid
prototyping with little more effort than drawing the
scenarios on paper. In addition, the user interfaces pro-
duced by Peridot are expected to be efficient enough for use
in the actual end systems.
Another motivation for this style of specifying user
interfaces is that it should be possible to allow non-
programmers to design
and implement
the interfaces. This
will allow professional UI designers (sometimes called
"User Interface Architects" [Foley 84]) and possibly even
end users, to design and modify user interfaces with little
249
S I G G R A P H
'86
I
training and without conventional programming. Virtu-
ally all textual UI specification methods are too compli-
cated and program-like to be used by non-programmers
[Buxton 83].
The
Direct Manipulation
style of user interfaces
[Shneiderman 83][Hutchins 86], where the user typically
uses a mouse to select and manipulate objects on the
screen, has become very popular (and possibly even
predominant) for modern computer systems. Unfor-
tunately, there are virtually no tools available to help
develop the low level interaction techniques that support
these interfaces, so almost all are laboriously programmed
using conventional programming languages. It is well
documented in the literature how expensive this process is
[Williams 83][Smith 82]. This limits the amount of proto-
typing possible, and therefore the quality of the interfaces.
Existing tools to help build user interfaces, called User
Interface Management Systems (UIMSs) [Thomas
83][Olsen 84][Pfaff 85], have not provided a powerful and
flexible way to conveniently generate the interaction tech-
niques for these styles of interfaces. In particular, few sys-
tems have allowed Direct Manipulation techniques to be
used to create the interfaces [Shneiderman 86].
All UIMSs are restricted in the forms of user inter-
faces they can generate [Tanner 85]. Peridot is only aimed
at graphical, Direct Manipulation interfaces. For example,
Peridot should be able to create interfaces like those of the
Apple Macintosh [Williams 84]. Peridot does not help with
textual command interfaces or with the coding of the
semantics of the application. The set of interfaces it will
produce is rich enough, however, to be very interesting and
of practical use for commercial systems.
In summary, the goals of Peridot are that:
1) interaction techniques for Direct Manipulation
interfaces should be supported,
2) the system should be easy to use for the designer
and require little or no training,
3) the designer should not have to write programs,
4) the interface should be visible at all times as it is
developed and changes should be immediately
apparent,
5) the
behavior
of the interface should also be
created in a Direct Manipulation manner and it
should run in real time (points 4 and 5 provide
for extremely rapid prototyping), and
6) the system should create run-time code that is
efficient enough for use in actual application
programs.
This paper presents the design and implementation of
the demonstrational aspects of Peridot. A longer report
providing more detail and covering other aspects is in
preparation [Myers prep]. Throughout this paper, the term
"designer" is used for the person creating user interfaces
(and therefore using Peridot). The term "user" (or "end
user") is reserved for the person using the interface created
by the designer.
2. Background and Related Work
Tanner and Buxton [Tanner 85] present a model of
User Interface Management Systems that identifies a
number of separate parts (see Figure 1). Peridot is aimed
mainly at the "module builder" aspects, but it also covers
the "system glue" and "run-time support" components.
pre-Processor Run-Time ! Post-Processor
LMOdul¢ / /
U[.
// Log
/
ibr~n/,/D~n"'o~/ / rile /
@
Figure 1.
Model for User
Interface Management
Systems (from [Tanner
85]).
The "module builder" creates a library of specific
interaction techniques. Some systems, such as the Macin-
tosh ToolBox [Apple 85] and the routines that come with
most modern window managers [Myers 84][Tesler 81], are
essentially the library portion by itself. Using a library
has the advantage that the final UI will look and act simi-
larly to other UIs created using the same library, but
clearly the styles of interaction available are limited to
those provided. In addition, the libraries themselves
are
often expensive to create: A few UIMSs, such as Syngraph
[O|sen 83] and Squeak [Cardelli 85], are designed to help
with the creation of the interaction techniques that make
up the library, but the indirect and ahstract methods used
by these programs have proved difficult to use. Peridot
attemots to make this process more direct.
Many (probably most) UIMSs concentrate on combin-
ing ("gluing") the modules together after they have been
created, since it is often non-trivial to write the programs
that coordinate the interaction techniques. This is evi-
denced by the need for the MacApp system to help write
programs that use the Macintosh ToolBox. Some, such as
Menulay [Buxton 83] and Trillium [Henderson 86], allow
the designer to see the design as it is created, but most
require that the specification be in a textual language (e.g.
[Hayes 85][Jacob 85]). Although a number of modern
UIMSs allow the layout of the screen to be specified in a
Direct Manipulation manner, virtually all still require the
interaction to be specified in an abstract, indirect way,
such as using state transition networks. Peridot allows
Direct Manipulation to be used for both.
The power in Peridot comes from the use of a new
approach to user interface design. The principles of
Pro-
gramming by Example
and
Visual Programming
have been
adapted to allow the designer to demonstrate the desired
user interface graphically. These principles are defined,
and a comprehensive taxonomy of existing systems that
use them is presented, in [Myers 86]. "Visual Program-
ming" (VP) refers to systems that allow the specification of
programs using graphics. "Programming by Example"
(PBE) systems attempt to infer programs from examples of
the data that the program should process. This inferenc-
ing is either based on examples of input-output pairs
[Shaw 75][Nix 86], or traces of program execution [Bauer
78][Biermann 76b]. Some systems that allow the program-
mer to develop programs using specific examples do not
250
Dallas, August 18-22 Volume 20, Number 4, 1986
use inferencing [Halbert 81 and 84][Lieberman 82][Smith
77]. For example, SmallStar [Halbert 84] allows users to
write programs for the Xerox Star office workstation by
simply performing the normal commands and adding con-
trol flow afterwards. Visual Programming systems, such
as Rehearsal World [Gould 84], have been successful in
making programs more visible and understandable and
therefore easier to create by novices.
Peridot differs from these UIMSs and programming
systems in that it applies Programming by Example and
Visual Programming to the specific domain of graphical
user interface specification. Tinker [Lieberman 82] has
similar aims, but it does not provide inferencing, and code
is specified in a conventional, textual manner in LISP.
Early inferencing systems were rather unsuccessful since
they often guessed the wrong program and it was difficult
for the programmer to check the results without
thoroughly studying the code [Biermann 76a]. In limited
domains, PBE has been more successful, for example, for
editing in the Editing by Example system [Nix 86]. Other
systems that are relevant to the design of Peridot are
those, such as [Pavlidis 85], that try to "beautify" pictures
by inferring relationships among the picture elements
(such as parallel and perpendicular) and modifying the pic-
ture to incorporate them.
3. Sample of Peridot in
Action
The best way to demonstrate how easy it is to create a
user interface with Peridot is to work through an example.
Due to space limitations, we will take a simple interaction:
a menu of strings. The operations discussed in this exam-
ple will be further explained in the following sections.
First, however, we present the Peridot screen.
When using Peridot, the designer sees three windows
and a menu (see Figure 2). The menu, which is on the left,
is used to give commands to Peridot. The window at the
top shows the name of the current procedure, the name of
its arguments, and
examples
of typical values for those
arguments. The window in the center shows what the user
will see as a result of this procedure (the end user inter-
face), and the window at the bottom is used for prompting
the designer and for messages. For debugging Peridot
itself (and for the very few designers that will be
interested), the system can be configured to display the
generated code in a fourth window. Currently this code is
presented in LISP, but creating a more readable form is
possible in the future. The displayed procedure and the
picture are always kept consistent, so if the picture is
edited, the code is changed, and when the code changes,
the picture is also updated. It is not necessary for the
designer to view or use the code to perform any operations
in Peridot.
Figure 3 shows the steps that can be used to create a
procedure that handles a menu with a grey drop shadow.
First, the designer types the name for the procedure,
("MyMenu"),
the name for the parameters
("Items"),
and an
example of a typical value for each parameter (the list:
("Replace", "Move", "Copy", "Delete", "Delete All", "Help", "Abort",
"Undo", "Exit")).
Next, the designer draws a grey box for the
shadow and then a black box for the background slightly
offset from it (see Figure 3a). These commands are given
using the Peridot command menu and a mouse. The sys-
tem guesses that the black box should be the same size as
the grey one and at an offset of 7 in X and Y. The
designer confirms that this is correct. Next, (in Figure 3b)
::,'+'.:;.
,,..:.:,2........,.,'.: .:. • , - ..,..,..::...., • ...... ,..........,..,.,: ,,.:,,,,,,,:,;,,;.,,,4..,:.:+,,:.:,:.:.:,;...,..,:,:,+:+:.:,,:
':~:"~ii~iiii ~iii:i!i!!!i!i!i;i:!i:i~i:;i!;~:!:~::
5trir,~]
[] P~'"'
: oL,~,,~ II
~naao~ ':': ::::i::'F:"
[] un.leer 1 lr~e
a:
i!':iiiii;i!;i.:i !:i!
i!i!;!;!:!;!i!: :i
................. i:iii~
:iiiiiiiiiii:i?ii¢>~:ili ~
,.......,.....
ILor,~<#Jl'~ ~eE(ll~ LO be left flusneo
~t the t.~,
or BiJ,,~se3
,
~pp]y
Ihl~
rule? (Yes
or
NO) no
~#hlt~t~l~4 used to del)end on Uqe posltlOn of Bla¢~ese,#
b.t .aw it doeth t,
S¢,vuhl 51ackege~ be changed t,~ del)end on ~hlEeOJ847 ,ye~
lype tlH~bet lot i'e,:[ COlOr texture ~485e
: "h ":: :';:; :::,
Figure 2.
Figure 3a.
..,,,,,.
,.,,,.,,
::::::::
,:,;,:,;
..,,,,,,
user Interaetiot
Whit. e8264~ ~.eem=- t.o be a :sub-area of
B]ack0263 fitt. ing inside wiLh a border
i iiiili o, o a,, a,-oon .
:.:.:.:.: ..... ~pply this rule? yes
!!!!!!!!!
:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:~:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i:i!i~i!i!i~i;i!i~i!i~i~i~i!i!i~i~i~i~i~i~i:i~i~
Figure 3b.
251
I
S I G G R A P H '86
It.eros
= "Replace ....
~1ove
.... Copy .... De]et.e ....
[
Replace
Move
~_User Intel, actiol
..... Apply this rule? yes
Strir, gFromSelect0266 seems to be centered
FLUSH below StringFro[nSelect8265.
..... Apply this rule? yes
Figure 3c.
terns = "Replace .... Move .... Copy .... Delete .... [
Replace
t4ove
Copy
Delete
Delete All
Help
~bort
Undo
Exit.
..... Apply t.his rule? yes
The lasL t.wo statements seem to be part.
of an iteration over 'IteMs',
..... Replace with an iteration
..... statement? yes
Replace
~love
Copy
Delete
~".~De l ete A1
Help
Abort
~I Undo,
Exit.
.iiiiIThe
size of ~hit.e8264 see,is to--.he
!:::::!l~:ame ,as ~ike sum of the sizes of the
i!i!i!ilStaCked ~trirgs inside it.
!!!i!!l
.....
Apply this rule? ~]es
i:i:i:iIReverse ~hite826~l and BlackB2637 yes
- - , , , ,
i:i:i:i~
Figure 3e.
•i•!•!•!•!•!:!•i•i:i•i••i:i•!:i•i•!•!•i•!•!•!•!•!•!•!•!•i••••i•i••••••••••••••:•••••:•.••••••••••••••••:•••••••••••••••••••••••••••••••••••••••••••••••.••.••.•..•..••.••.
It.eals = "Replace .... hlove .... L-:opy .... Delete" "[
~| Replac:e
i!! 11
Black.0286 seert].s to be tihe same posir_.ion
and heights as the object under t.he mouse
but sortie cipher width,
..... Apply this rule? yes
Black0286 seems t.o have t.he same widt.h
arid LEFT position as ~%it.e@264,
..... Apply t.his rule? yes
Figure 3d. Figure 3f.
A sequence of frames during the definition of a menu interaction technique. (The pictures
for 3b-3f have been expanded to be more readable.) [n 3a, the shadow and background are
drawn (and the system infers that they should be the same size). In 3b, a white area is
nested inside the background, and in 3c the first two elements of the parameter are copied
to the top of the white rectangle. Peridot notices that they are stacked vertically, and that
they are part of an iteration. The rest of the iteration is executed in 3d. The size of the
white rectangle is then changed to be just big enough to
include
all the strings and the sys-
tem changes the black and grey rectangles accordingly 3e. In 3f, the interaction is being
defined using the "simulated mouse."
252
Dallas, August 18-22 Volume 20, Number 4, 1986
a white box inside the black one is drawn, and the system
adjusts it to be a constant 4 pixels all around, after
confirmation from the designer. Next, (in Figure 3c) the
first item in the argument
('"Replace")
is copied to the top of
the white rectangle, and the system asks if it should be
centered at the top. Peridot makes this assumption
because the string was placed approximately centered in
the box, as shown in Figure 3c. If the string had been
placed left-justified in the box instead, then Peridot would
have asked if the string should be left-justified. The sys-
tem asks the designer to confirm every assumption because
sometimes the placing is ambiguous. Next, the second
string, "Move", is copied below "Replace" and the system
guesses that it is also centered. Since the first two ele-
ments of a list have been placed on the screen, the system
guesses that the entire list might be desired, so it asks the
designer if there should be an iteration to display all ele-
ments of the list. After the designer confirms this (in Fig-
ure 3d), the system executes the rest of the iteration and
changes the cede to be a loop. Finally, (in Figure 3e) the
designer adjusts the size of the white rectangle to be
approximately the size of the strings, and the system asks
if the rectangle should be adjusted to fit exactly around all
the strings. The sizes of the black and grey rectangles are
then automatically adjusted to be proportional to the size
of the white rectangle. This completes the presentation
aspects of the menu (Figure 3e). It should be remembered
that the code being generated does not depend on the
specific example values for the parameter; any list of
strings will work correctly.
To specify the
interaction (behavior) of the user inter-
face for the menu, the designer uses an icon that
represents the mouse. First, this "simulated mouse" is
moved over one of the menu items, and then the designer
draws a black rectangle over that item in INVERT draw-
ing mode (see Figure 3f). Peridot infers that the box
should be the same height and Y position as the string,
and the same width and X position as the white box. The
designer then moves the simulated mouse off to the side
and erases the black rectangle. Peridot infers that the box
should be erased when the mouse is no longer over an
object. The designer can perform this action on another
string, or explicitly specify an iteration, and the code that
handles highlighting is completed. Now the designer
"presses" one of the simulated mouse's buttons and
specifies, using a Peridot command, that the object under
the mouse is returned. From this, the system infers that
the procedure should be exited upon button press. The
MyMenu procedure is now complete.
Although the textual description of the designer's
actions is clumsy, only about ten actions had to be per-
formed to create this procedure (plus confirming Peridot's
12 guesses). Once created, the picture or interaction can
be edited, and the menu can used as part of other user
interfaces.
4. General Principles of Peridot
One problem with all demonstrational systems is that
the user's actions are almost always ambiguous. The sys-
tem cannot usually know
why the
person did a particular
action. This is especially true when the system attempts
to infer a general case from a particular example. For
instance, when an item is selected, does the user mean
that particular item, an item with a similar name, an item
at that particular place on the screen, an item with the
same type as the selected one, or an item with some other
property? Early inferencing systems attempted to solve
this problem by guessing and requiring the user to go back
later and check the generated code. Non-inferencing sys-
tems, such as Halbert's system for the Xerox STAR works-
tation [Halbert 81 and 84], require the user to explicitly
specify why objects were chosen. Peridot, on the other
hand, tries to guess what the designer intends by an
action, but, to avoid the problems of earlier systems, it
always asks the designer if each guess is correct. It is
expected that the guesses will usually be correct, which
will save the designer from having to specify a great deal
of extra detail and from having to know a programming
language to express those details. In addition, it is easy to
check for errors since the results of all actions and infer-
ences are always immediately visible on the screen.
Any graphical user interface is composed of two parts:
the
presentation
or layout, which defines what pictures are
on the screen, and the
interaction
or behavior, which deter-
mines how these pictures change with user actions. As
shown in the previous example, these are specified
separately in Peridot. The pictures that Peridot currently
supports are: rectangles filled with various grey shades,
text strings, filled circles, and static pictures drawn with
other programs (e.g. icon@.
Peridot uses inferencing in three different ways.
First, it tries to infer how various objects in the scene are
related graphically. When the designer draws an object, it
usually has some implied relation with other objects that
have already been drawn. For example, a box might be
nested inside another box, or a text string centered at the
top of a box. If the picture was simply a static background
that never changed, it would not be important for the sys-
tem to notice these relationships. In Peridot, however, the
pictures usually depend on the parameters to the pro-
cedure that generate them. For example, the size of the
box around a menu might depend on the number of items
in the menu and the width of the largest item. Peridot
must therefore infer the meaningful relationships among
objects from the drawings that the designer produces. This
object-object inferencing is described in section 5.1.
The second type of inferencing used by Peridot is to
try to guess when control structures are needed. For
example, when the designer displays the first two elements
of a list, Peridot infers that the entire list should be
displayed and will generate an iteration. Conditionals are
also inferred for special cases and exceptions. For exam-
ple, a check-mark might be displayed to show the current
value of a set of choices (as in Figure 2). Iterations and
conditionals are discussed in sections 5.2 and 5.3 respec-
tively.
The final type of inferencing used by Peridot is to try
to guess when actions should happen during the execution
of an interaction. For example, a highlight bar might be
displayed when the left mouse button goes down. This
type of inferencing is described in section 6.
i Straight and curved lines, and individual pixels should be easy to add
in the future, if needed.
253