.)6%23)48/&%.42!,,/2)$! .)6%23)48/&%.42!,,/2)$!
!#5,48)",)/'2!0(83 !#5,48)",)/'2!0(8
%4(/$/&!$*534-%.436%2353-%4(/$/&#/.34!.434)-5,)).4(%%4(/$/&!$*534-%.436%2353-%4(/$/&#/.34!.434)-5,)).4(%
15!.4):#!4)/./&!##52!#8!.$02%#)3)/./&2%.$%2%$$%04().15!.4):#!4)/./&!##52!#8!.$02%#)3)/./&2%.$%2%$$%04().
(%!$-/5.4%$$)30,!83(%!$-/5.4%$$)30,!83
/,,!.$
.)6%23)48/&%.42!,,/2)$!
%8%2
.)6%23)48/&%.42!,,/2)$!
24(52
).!,$5##)
.)6%23)48/&%.42!,,/2)$!
).$3)-),!27/2+3!4(440334!23,)"2!285#&%$5&!#5,48")"
.)6%23)48/&%.42!,,/2)$!)"2!2)%3(440,)"2!285#&%$5
()324)#,%)3"2/5'(44/8/5&/2&2%%!.$/0%.!##%33"84(%!#5,48)",)/'2!0(8!44(!3"%%.!##%04%$&/2
).#,53)/.).!#5,48)",)/'2!0(83"8!.!54(/2)9%$!$-).)342!4/2/&/2-/2%).&/2-!4)/.0,%!3%
#/.4!#45#&%$5
%#/--%.$%$)4!4)/.%#/--%.$%$)4!4)/.
/,,!.$%8%224(52!.$).!,$5##)%4(/$/&!$*534-%.436%2353-%4(/$/&#/.34!.4
34)-5,)).4(%15!.4):#!4)/./&!##52!#8!.$02%#)3)/./&2%.$%2%$$%04().(%!$-/5.4%$$)30,!83
!#5,48)",)/'2!0(83
(440334!23,)"2!285#&%$5&!#5,48")"
J. P. Rolland
jannick@odalab.ucf.edu
C. Meyer
ODA-Lab, School of
Optics/CREOL
4000 Central Florida Blvd.
University of Central Florida
Orlando, FL 32766-2700
K. Arthur
Department of Computer Science
University of North Carolina
Chapel Hill, NC 27599
E. Rinalducci
Department of Psychology
University of Central Florida
Orlando, FL 32766
Presence, Vol. 11, No. 6, December 2002, 610– 625
© 2002 by the Massachusetts Institute of Technology
Method of Adjustments versus
Method of Constant Stimuli in the
Quantification of Accuracy and
Precision of Rendered Depth in
Head-Mounted Displays
Abstract
The utilization of head-mounted displays (HMDs) in high-end applications such as
medical, engineering, and scientific visualization necessitates that the position of ob-
jects be rendered accurately and precisely. Accuracy and precision of rendered
depth for near-field visualization were measured in a custom-designed bench proto-
type HMD. Experimental results were compared to theoretical predictions estab-
lished from a computational model for rendering and presenting virtual images by
Robinett and Rolland (1992). Such a theoretical model provided the necessary
graphics transformations required so that rendered virtual objects be perceived at
the rendered depth in binocular HMDs. Three object shapes of various sizes were
investigated under two methodologies: the method of constant stimuli modified for
random size presentation and the method of adjustments. Results showa2mm
and an 8 mm performance for the accuracy and the precision of rendered depth in
HMDs, respectively. Results of the assessment of rendered depth in HMDs for
near-field visualization support employing the method of adjustments over the
method of constant stimuli whether or not the method of constant stimuli is modi-
fied for random size presentation.
1 Introduction
The basic principle underlying perceiving three dimensions using various
stereoscopes, including head-mounted displays (HMDs), calls for the presenta-
tion of slightly disparate images to the eyes of the user (Wheatstone, 1838).
HMDs differ from common stereoscopes in their ability to update the dis-
played images at interactive speed (Sutherland, 1968). It is well accepted that
most stereoscopic devices provide a striking impression of depth.
With the development of applications such as medical, engineering, and sci-
entific visualization, a striking impression of depth is not sufficient. Users typi-
cally need to perform tasks that include the fine manipulation of objects at arm
length. Such tasks pose challenges for the engineering of the display device
(Rolland & Fuchs, 2000; State et al., 1996). What is important is that depth
can be rendered with accuracy and precision of approximately 1 mm. Ren-
dered depth is especially critical for tasks in which errors may have high cost
610 PRESENCE: VOLUME 11, NUMBER 6
associated with them, such as in some of the medical
applications (Biocca & Rolland, 1998).
We previously reported (Rolland, Ariely, & Gibson,
1995) an investigation to address two fundamental
questions: how accurate is rendered depth in HMDs
and how variable is the depth percept? Results of the
experiments showed various forms of perceptual bias
and variability in the data. In an effort to design systems
that can be employed effectively in high-end applica-
tions, we are investigating engineering issues related to
HMDs and are employing psychophysical methods to
assess the improved technology in a closed loop. In this
paper, we specifically focus on the methodology to
make measurements and we seek to establish new
benchmark performance on an improved HMD device,
given a computational model for rendering and display-
ing the stereoscopic images.
The experimental studies presented in this paper are
based on the computational model of rendered depth
and images presentation by Robinett and Rolland
(1992), in which various possible errors for rendered
depth of virtual objects were discussed. The computa-
tional model includes the specification of all required
graphics transformations (including optical distortion)
from the virtual world to the left and right eyepoints in
a binocular HMD. We applied this model to a custom
HMD technology and predict accuracy and precision of
rendered depth obtainable in the system. We validated
the model by measuring human performance in a set of
psychophysical investigations. The human eye served as
a measuring device to assess the technology under its
naturally working conditions. Thus, perception is simply
used in the assessment of the technology, and visual
pathways and cognitive properties are not subjects of
this investigation.
2 Previous Work and Motivation for the
Research Presented
The need to see depth accurately in 3D visualiza-
tion devices and thus to quantify depth perception in
HMDs has been the focus of multiple studies starting in
the 1960s when HMD technology was used to provide
additional information to pilots flying airplanes and
flight simulators. Pilots conducted studies of size and
depth perception to explain observed misperceptions. In
these experiments, pilots made landings by reference to
panel-mounted periscope screens in airplanes and to
virtual collimated computer graphics images in flight
simulators. (Collimated images refer to virtual monocu-
lar images formed at optical infinity, or distances ⱖ 6
m.) Pilots consistently misjudged the runway as being
smaller and farther away than it was, and consequently
tended to overshoot their landings (Roscoe, Olzak, &
Randle 1976; Palmer & Cronn 1973; Randle, Roscoe,
& Petitt, 1980). Most of these misperceptions have
been associated with the collimation of the virtual images.
Ellis and Bucher (1994), Ellis, Bucher, and Menges
(1995), and Ellis and Merges (1997) have been investi-
gating various types of perceptual bias of real and virtual
objects in a haploscopic display using the method of
adjustments. In such experiments, the virtual stimulus
was a small pyramid pointing downward. Subjects were
asked to locate the top of the pyramid typically with a
light-emitting diode positioned at the end of a stick
physically movable in depth. The combination of real
and virtual objects allows establishing whether the vir-
tual world is registered with respect to the real world
and how objects are perceived in some combination of
objects where they may or may not overlap. Equivalent
studies with only real or virtual objects bring further
control into explaining perceptual bias observed in mix-
ing real and virtual objects.
Utsumi, Milgram, Takemuta, and Kishino (1995)
have investigated the interplay of edge sharpness and
binocular disparity of stereoscopically presented virtual
objects on depth perception. The method of constant
stimuli was adopted. They found large individual differ-
ences in importance given to the blur of the objects’
edges. While for some subjects blur had almost no ef-
fect, for others the effect superseded that of stereopsis.
Identifying factors that induce large individual differ-
ences, either correlated or random, is of primary impor-
tance to designing more effective systems and assessing
their performance.
Surdick et al. (1994) have conducted psychophysical
experiments to investigate the effectiveness of various
Rolland et al. 611
depth cues in a modified Wheatstone virtual display.
They also investigated the effect of the viewing distance
on cue effectiveness. Their stimulus was a square pre-
sented using various cues in isolation or in combination.
The method of constant stimuli was used. The just no-
ticeable difference (JND), which is related to the preci-
sion in measured rendered depth, was estimated from
the collected data and was used as a measure of the ef-
fectiveness of the various cues present in the display.
Understanding which cues are most effective under vari-
ous conditions is essential to the optimization of display
information.
In an earlier paper, we employed a 40 mm cube and a
13 mm dia. cylinder presented relative to each other at
either 0.8 m or 1.2 m (Rolland et al., 1995). Subjects
were asked to judge relative depth of the two objects
whose depth had to be estimated from the location of
their physical centers. The objects could be virtual, real,
or a combination of the two. We had found large vari-
ability in all measurements involving virtual objects. We
established two main possible causes of data variability
in this study: first, conflicts of accommodation and con-
vergence caused by presenting the virtual monocular
images collimated (whereas judgments of relative depth
were performed in the near field) (Robinett & Rolland,
1992; Wann, Rushton, & Mon-Williams, 1995; Marran
& Schor, 1997) and, second, aspects of the methodol-
ogy detailed in section 3. In the research presented
here, we describe how we eliminated possible conflicts
of accommodation and convergence, and we address
issues related to methodology. Specifically, we present
performance measures using two methodologies, and
we demonstrate how the choice of the stimuli affects the
measures of accuracy and precision of rendered depth.
3 Methods
3.1 Apparatus
A third-generation prototype of a conventional
optical see-through, as opposed to projection-based
see-through, HMD served as the experimental setup
(Hua, Girardot, Gao, & Rolland, 2000). In this con-
ventional, yet new instrument shown in figures 1(a) and
1(b), micro adjustments permit precise alignment and
setting of the various components. The overall geome-
try was modified from a previous V-shape geometry to a
horizontal geometry to facilitate alignment (Rolland et
al., 1995). New miniature higher resolution and con-
trast displays were also integrated in the system. The
displays were off-the-shelf, 2.2 in. diagonal Casio active-
matrix LCDs (model TV-7700B). The combination of a
Figure 1. Experimental setup. (a) A chin rest for the human
subject, the optical viewer, two depixelized LCD displays, and a two-
button input device in the form of a ball for easy handling. The two-
button device served in entering responses in the method of constant
stimuli. A continuous dial located to the right also served to locate
objects in depth in the method of adjustments. (b) A schematic of the
optical layout.
612 PRESENCE: VOLUME 11, NUMBER 6
25.5 ⫻ 34 deg. field of view when imaged through the
optics and 429 ⫻ 586 addressable lines yields an effec-
tive resolution of about 3.5 arc-minutes at the eyepoint.
Depixelization screens from Microsharp Technology
were added to the LCDs to blur the boundaries be-
tween pixels while minimizing the induced overall blur
of the image. A depixelization screen, which perceptu-
ally resembles Scotch tape, is made of thin microlenslet
arrays whose lens sizes are selected to match roughly
one color-pixel size. Without the depixelization screens,
individual color pixels could be resolved when the vir-
tual images were located at 0.8 m, the distance at which
the virtual objects in the experiments were displayed.
Results of accuracy and precision of perceived depth,
presented in this paper, may be compared to those ob-
tained with binocular HMDs of at least 3.5 arc-minutes
resolution. This custom-designed display was used in
this set of studies because we had full control of the en-
gineering parameters for the display, and—importantly
for this study—we had the ability to adjust the location
of the optical virtual images for each eye as explained in
subsection 3.4.
3.2 Subjects
Four human subjects were used in each experi-
ment, with further details under each methodology on
the amount of data collected. Human subjects selected
to participate in the experiments were recruited among
the undergraduate and graduate students at the univer-
sity, where all subjects were between eighteen and 35
years of age. All subjects selected had a visual acuity of
20/20 (uncorrected) according to the Snellen acuity
chart and performed at expectation on the Howard-
Dolman test. The Howard-Dolman depth perception
apparatus (Howard, 1919) was obtained from Lafayette
Instrument Company. In this test, human subjects lo-
cated approximately at 5.5 m from the apparatus were
asked to adjust the depth of two vertical rods so that
they appeared equidistant. We ran six blocks of five
measures and calculated the mean and standard devia-
tion of the measures. We set a performance criteria of 3
mm and 6 mm maximum for the mean and the standard
deviation of the response, respectively. Human subjects,
also referred to hereafter as participants, were also fur-
ther tested at 1 m, and the performance measures were
close to an order of magnitude better than at 5.5 m.
3.3 Stimuli
The choice of the stimuli was motivated from a
previous investigation (Rolland et al., 1995) in which
we investigated depth perception in an HMD using a
thin cylinder (13 mm dia.) and a small cube (40 mm on
each side). In the investigation presented here (which
aimed primarily at comparing the method of constant
stimuli with the method of adjustments as detailed in
section 4), we also investigated the potential impact of
the stimuli size and shape on the measured accuracy and
precision of perceived depth in HMDs. Although differ-
ent shapes and sizes of the stimuli were thus considered
as now detailed, it is beyond the scope of this paper to
further investigate various depth cues in HMDs. The
purpose of this investigation is to optimize a methodol-
ogy for measuring accuracy and precision of rendered
depth in HMDs, and the measured performance must
be compared to a theoretical prediction as done in sec-
tion 6 or an equivalent experiment in the real world.
The small cylinder was smooth shaded and had a di-
ameter of 13 mm (equivalent to 1 deg. visual angle) and
a height 235 mm (or 17 deg. visual angle) as used by
Rolland et al. (1995). In addition to the small cylinder,
we created a faceted cylinder (a hexagonal cross section)
of the same height as the small cylinder with approxi-
mately four times the width of the small cylinder. The
cylinder’s maximum diameter was 58 mm and its aver-
age diameter was approximately 52 mm. Such a cylinder
was chosen to create a stimuli with increased 3D struc-
ture, and thus we hypothesized potential improved
depth from shading. Ideally, we would have chosen a 13
mm cylinder with facets. However, given the resolution
of the display of 3.5 arc-minutes and upon adding facets
to the cylinder, its size had to be increased to resolve
the facets. A controlled condition with a cylinder of 52
mm with no facets could have also been added to the
stimuli set. However, because a larger cylinder would
yield less accuracy in perceived depth than a thin cylin-
der, we restrained the cylindrical stimuli to the thin un-
Rolland et al. 613
![Figure 3. [Method of constant stimuli] Left: Average accuracy of measured rendered depth across mirror pairs of virtual stimuli (such as a cube with a cylinder and a cylinder with a cube) when judging the depth of one with respect to the other. Right: Precision of measured rendered depth for the same conditions.](/figures/figure-3-method-of-constant-stimuli-left-average-accuracy-of-30zcd17r.webp)
![Figure 6. [Method of adjustments] Accuracy of measured rendered depth for two human subjects (ABQ and THI) across four pairs of stimuli. Within-subject variability is shown.](/figures/figure-6-method-of-adjustments-accuracy-of-measured-rendered-369kaoy1.webp)
![Figure 5. [Method of adjustments] Left: Average accuracy of measured rendered depth across mirror pairs of virtual stimuli (such as a cube with a cylinder and a cylinder with a cube) when judging the depth of one with respect to the other. Right: Precision of measured rendered depth for the same conditions.](/figures/figure-5-method-of-adjustments-left-average-accuracy-of-1nhqasn8.webp)
![Figure 4. [Method of constant stimuli] Accuracy of measured rendered depth for two human subjects (JE and SU) across four pairs of stimuli. Within-subject variability is shown.](/figures/figure-4-method-of-constant-stimuli-accuracy-of-measured-y1yucann.webp)