Real-time Streaming and Rendering of Terrains
Soumyajit Deb
1,2
, Shiben Bhattacharjee
1
, Suryakant Patidar
1
P. J. Narayanan
1
sdeb@microsoft.com, {shiben@research., skp@research., pjn@}iiit.ac.in
1
Centre for Visual Information Technology
International Institute of Information Technology Hyderabad
2
Microsoft Research India, Banglore
Abstract. Terrains and other geometric models have been traditionally
stored locally. Their remote access presents the characteristics that are a
combination of file serving and realtime streaming like audio-visual me-
dia. This paper presents a terrain streaming system based upon a client
server architecture to handle heterogeneous clients over low-bandwidth
networks. We present an efficient representation for handling terrains
streaming. We design a client-server system that utilizes this represen-
tation to stream virtual environments containing terrains and overlayed
geometry efficiently. We handle dynamic entities in environment and the
synchronization of the same between multiple clients. We also present
a method of sharing and storing terrain annotations for collaboration
between multiple users. We conclude by presenting preliminary perfor-
mance data for the streaming system.
1 Introduction
Traditional graphics applications store all geometry locally in the main memory
itself. Geometry can also be stored remotely and received progressively when
needed and rendered on the fly. Streaming of geometry of large virtual envi-
ronments can be beneficial and difficult if the network bandwidth is low. Such
systems find applications when data cannot be replicated easily. Dynamic envi-
ronments such as those used for battlefield visualization involving real terrains
and multiple players is an example. Different users may read/update parts of
the virtual environment while maintaining a collaborative and consistent system
across heterogeneous users connected from client machines with different capa-
bilities and network bandwidths. A similar situation is presented by massive,
multi-player online games consisting of dynamic persistent worlds.
Geometry cannot be split into frames or chunks unlike media like audio and
video. Some parts of a model cannot be lost unlike video where the loss of a
frame may be acceptable. A complete model is necessary to render geometry.
Each may be used for rendering several frames and hence need to be stored at
the client. In this sense, geometry should be served like files from a file-server.
On the other hand, geometric model should reach in time for real-time rendering
of the final rendered video. Delay can result in the undesirable effects like the
freezing and popping. The real-time constraint makes it possible for us to talk
about streaming geometry. The parallel in geometry to the reduced bit-rate
encoding of audio and video is level of detail (LOD), which is the representation
of the shape at different levels of approximations. The server can send a lower
detail model to the client to reduce various resource requirements. Thus, remote
access of geometry is an interesting mix of data serving and content streaming.
In this paper, we address the issue of streaming terrain data over networks.
Real terrain datasets can span multiple gigabytes in size and are difficult to
render interactively using brute force methods. Level of detail methods must be
used to reduce the amount of detail to be rendered. However, if we are to ren-
der terrains remotely, the biggest bottleneck is the available network bandwidth
between the server and the client. We need to optimize the transmitted terrain
data accordingly depending upon client type and and available bandwidth. This
is a hard problem as the system must achieve performance akin to local ren-
dering at the client end. This is compounded by the fact that the system must
keep track of any dynamic entities that exist in the environment and update
the clients accordingly. This work extends to terrains an earlier work on geom-
etry streaming [1]. We propose an optimized terrain representation based upon
tiles for efficient transmission and rendering in Section 3. In Section 4 we look
at the basic requirements of a geometry streaming system and our design and
implementation of the same. Section 5 presents efficient ways of adapting these
techniques for a terrain rendering system including performance improvements
for the system using the ideas of prefetching and caching. Methods to synchro-
nize and render multiple dynamic entities in the environment are discussed in 6
followed by experimental results in 7 and conclusions 8.
2 Related Work
Media streaming over the web has been popular and several standards exist.
Most media streamers allow a specific bitrate to be chosen based upon available
bandwidth and dynamic changing of bitrate to adjust accurately to client pa-
rameters. Google Earth/Maps streams maps and satellite imagery in real-time
over the internet. However it does not address individual client characteristics
which may lead to lags and freezes.
Djurcilov and Earnshaw added compression of models to VRML and devel-
oped an integrated visualization system, where the basic selection of data is
done by the user [2, 3]. However even after revisions of VRML which included
geometry compression (Li et al.) [4], it is not usable for web-based serving since
data needs to be transmitted before rendering can begin. Commercial products
for remote visualization includes VisServer software from Silicon Graphics which
allows rendering of any OpenGL application on remote clients by transmission of
individual frames. Funkhouser describes a system based upon client server archi-
tecture for multi-user virtual environments [5]. The WireGL/Chromium project
is a system that provides the familiar OpenGL API to each node in a cluster,
virtualizing multiple graphics accelerators into a sort-first parallel renderer with
a parallel interface [6]. [7] uses a crude model at the client for navigation and
streams actual high quality views from the server using viewing parameters to
protect high detail content.
Among geometry based approaches, Schneider and Martin describe a frame-
work which adapts to the client characteristics including network bandwidth and
the client’s graphics capabilities [8]. Martin describes an Adaptive Rendering
and Transmission Environment (ARTE) framework that facilitates the delivery
of 3D models while monitoring the resources available [9]. This uses MPEG4
stream compression which may lead to lag in response to user input. Teler de-
scribes a remote rendering system utilizing path prediction and bandwidth based
level of detail reduction [10]. This system fails to dynamically change/adapt pa-
rameters during the course of the walk-through which may lead to suboptimal
performance. Deb and Narayanan develop a system to stream general polygonal
models between a server and client in [1]. However this approach is suitable for
only tessellated models and may not be the optimal for terrains.
In recent terrain rendering approaches, Lossaso and Hoppe[11] describe how
terrains can be broken into geometric clipmaps of varying metric sizes and that
these clipmaps can be used as Level of Details. This is however not the most opti-
mal representation for streaming. Their method also calculates the blend/morph
factor on a per vertex basis because of inhomoegeneous tile sizes which may slow
down lower end clients. Wagner [12] divides the terrain into regular square tiles
for rendering. However the view frustum culling approach used by [12] fails in
cases when the terrain has large variations in heights. The ground plane is un-
able to include the projection of tiles, which are near to the camera looking
at horizon, since they are out of that projection but inside the view frustum.
Pouderoux and Marvie [13] design an out of core terrain rendering system based
upon a heuristic metric. However they do not address the problem of network
streaming.
In our system, we follow a technique similar to Wagner [12] for dividing the
terrain into square tiles to make it easy for the geometry streaming system to
select regular data to be transmitted to the client. We can calculate blending
factors on a per tile basis because of the use of a regular tile structure thus
reducing the amount of computation. Given the tile indices, their object space
location is easily computable making query systems on the terrain efficient. How-
ever regular tiles become very small at the extremities of the viewing frustum.
We take care of this problem by using very low levels of detail for such tiles in
view. We tweak Wagner’s frustum culling technique by having the projection on
realtime average height of terrain in the view and not simply the ground plane.
3 Terrain Rendering
We describe the various steps involved in first creating our terrain representa-
tion and then rendering it. Before rendering the tiles, we must store them in
a data structure that is suitable for both rendering and also for transmission.
View frustum culling is required to select only the necessary entities in the view
frustum. VFC is very useful for streaming as it allows us to select only a small
portion of the entire terrain for transmission.
3.1 Data Organization
o
o o o o
o
o o o o
o
o o o o
o
o o o o
o
o o o o
o
o o o o
o
o o o
o
o o o o
o
o o o
o
o o o o
o
o o o
o
o o o o
o
o o o
o
o o o o
o
o o o
o
o o o o
o
o o o
o
o o o o
o
o o o
o
o o o o
o
o o o
o
o o o o
o
o o o
o
o o
o
o o
o
o o
o
o
o
o
Sx
Sx
l=0
2 +1 x 2 +1
All heights original
Sx
Sx
33
l=1
2 +1 x 2 +1
Alternative heights
interpolated
33
l=2
2 +1 x 2 +1
Alternative heights
interpolated
22
l=3
2 +1 x 2 +1
Alternative heights
interpolated
11
Fig. 1. Data organization: An e.g. with n = 3 and m = 3, blue circled height values
are original, rest are interpolated. Note that, they occupy the same area on ground
Terrain data consists of a height value for every point x, y on a rectangular
grid. We divide it into tiles of equal size for rendering. By equal we mean they
cover the same rectangular area on the heightmap. To handle levels of detail,
we arrange the data in a specific way. For a tile with size 2
n
× 2
n
height values,
we store m number of LODs, m ≤ n, m is a user defined number based upon
characteristics and size of the terrain. We also keep the distance between adjacent
heights in x, y as s
x
, s
y
Fig 1. For an LOD l we have 2
n−l+1
+1×2
n−l+1
+1 (l > 0)
number of height values and 2
n
+ 1 × 2
n
+ 1 for l = 0. Note the extra heights at
the end corners of the tiles, they are the height values at the starting corners of
the next tile; kept as they help in stitching (see Section 3.5). This means l = 0
holds highest detail and l = m holds lowest detail as illustrated in Fig. 1. For
l > 0 we keep original height values h at (2i, 2j) locations, 0 ≤ i, j ≤ 2
n−l
. We
replace the height values at (2i, 2j + 1) locations with avg(h
2i,2j
, h
2i,2j+2
), at
(2i + 1, 2j) locations with avg(h
2i,2j
, h
2i+2,2j
), at (2i + 1, 2j + 1) locations with
avg(h
2i,2j
, h
2i+2,2j+2
); where i, j vary as bounded. This is done so that while
rendering when LOD l with alternate height values dropped, we don’t see any
change in the structure.
3.2 View Frustum Culling
In each frame, we query the graphics API for view frustum equations and calcu-
late the projection P of the frustum (generally a trapezoid) on the base plain.
This base plain is z = a
h
, a
h
is the approximated average height of the terrain
in view of previous frame. This is because we haven’t accessed the terrain data
yet and thus will be using the data from previous frame assuming that the view
hasn’t changed much. We then calculate orthogonal bounding rectangle of P .
We can directly map the coordinates of the bounding rectangle to tile indices.
Using these tile indices, we find other tiles that are inside P (Fig.2). We keep
the indices that return positive in a tile buffer B
t
for use in rendering. We do not
need to do 3D view frustum culling as terrains are injective functions on x, y,
and thus can be reduced to 2D in turn to reduce number of required calculations.
Bounding rectangle
Projection of view
frustum on the
base plain
Distances separated by
LOD transition distance
Base Line
Tiles rejected out of
projection
Tiles assigned with l=2
Tiles assigned with l=1
Tiles assigned with l=0
Fig. 2. View frustum culling and LOD assignment
3.3 LOD and Blending Factor Calculation
Using the camera parameters we calculate a base line, that is perpendicular to
the view vector and parallel to the ground plane. For each tile in B
t
, we calculate
the perpendicular distance d of its mid point from this line (Fig.2). This distance
d is used to calculate LOD l as bd/l
t
c where l
t
is the LOD transition distance. We
choose this distance d instead of the direct distance of the tile from the camera
because if the field of view of the camera is high, we shall end up rendering tiles
at the corner of screen that are actually close to camera in screen space but far
in object space in very low level of detail. The value frac(d/l
t
) is the blending
factor α. α is used for smooth level of detail changes of tiles as explained in
Section 3.4. We save l and α in B
t
along with the tile indices.
3.4 Rendering
With all data in place, the tiles can be rendered from B
t
. For all tile indices
in B
t
, we load the level l and l + 1 of that tile. The index is clamped to m to
avoid memory exceptions. The distance between adjacent heights for l can be
calculated as (s
x
l
, s
y
l
) = (s
x
, s
y
)2
l
Fig 1. We calculate the heights h for l > 0 as
h = h
(2i,2j)
l
(1 − α) + h
(i,j)
(l+1)
α
l = 0 is a special case: h = h
(i,j)
0
(1 − α) + h
(i,j)
1
α, i, j vary as bounded. We can
now see that when α is 0, h = h
(2i,2j)
l
, and when α is 1, h = h
(i,j)
(l+1)
. Thus
this blending factor is able to smoothly change between the two height values
of 2 different LODs of the same tile as we move the camera. On the fly, we also
calculate the average of the heights at the mid point of these tiles, a
h
, which will
be used in the next frame for view frustum culling (See Section 3.2).
3.5 Tile Stitching
Since every tile is getting assigned l and α independently, we find un-tessellated
areas near the corner of each of the tiles. We assume that a tile on the ground
with LOD l can have a nearby tile whose LOD can be only l − 1 or l + 1. This
makes tile stitching easy and smooth blending of LODs works perfectly. Our
assumption remains true iff l
t
is always more than the maximum distance a tile
can extend on the ground, i.e., the tile is never able to skip an LOD in between.
