Achieving Reliable Parallel Performance in a VoD Storage Server Using
Randomization and Replication
∗
Yung Ryn Choe and Vijay S. Pai
Purdue University
West Lafayette, IN 47907
{yung, vpai}@purdue.edu
Abstract
This paper investigates randomization and replication as
strategies to achieve reliable performance in disk arrays
targeted for video-on-demand (VoD) workloads. A disk ar-
ray can provide high aggregate throughput, but only if the
server can effectively balance the load on the disks. Such
load balance is complicated by two key factors: workload
hotspots caused by differences in popularity among media
streams, and “fail-stutter” faults that arise when the per-
formance of one or more devices drops below expectations
due to manufacturing variations, hardware problems, or
geometry-related variations.
This paper focuses on the random duplicate assignment
(RDA) data allocation policy which places each data block
on two disks chosen at random, independent of other blocks
in the same media stream or other streams. This strategy
is compared to traditional single-disk file allocation, disk
striping (RAID-0), disk mirroring (RAID-1), and random-
ization without duplication. The various allocation schemes
are implemented and tested using a prototype VoD server
with 2 dual-core Opteron processors, 8 SATA disks, and
4 Gigabit Ethernet interfaces running the Linux 2.6 ker-
nel. The results indicate that combining randomization and
replication allows RDA to effectively tolerate both workload
hotspots and fail-stutter faults better than previous schemes.
1 Introduction
Recent years have seen a dramatic increase in computa-
tional capability and network bandwidth available to end-
users in commercial, home, and mobile environments. The
long-running trend of exponential performance growth per
unit cost has now yielded systems ranging from ubiquitous
handheld entertainment devices to high-end desktops con-
nected through Gigabit Ethernet. All levels have seen de-
mand for richer and more targeted content. For example, the
mobile space has seen the recent arrival o f the video iPod as
a tool to enable on-demand playback of saved video clips
such as broadcast TV shows, as well as offerings such as
Verizon V-CAST which have brought video offerings to cell
∗
This work is supported in part by the National Science Foundation
under Grant Nos. CCF-0532448 and CCF-0621457.
1-4244-0910-1/07/$20.00
c
2007 IEEE.
phones. High-end entertainment has seen a rapid increase in
pay-per-view offerings and special-interest (e.g., regional)
satellite channels. Even the educationa l environment has
seen offerings such as iTunes U, which provides video and
audio captures of classroom lectures over the Web. Such
trends motivate investigations into video-on-demand (VoD)
servers. Substantial previous work has analyzed, proto-
typed, and built VoD servers [5, 9, 14, 36]. However, more
recent advances in storage technologies enable such servers
to be built in a cost-effective and storage-efficient manner,
requiring substantially different design strategies.
Rapid improvements in magnetic and electromechani-
cal technologies have facilitated greater information den-
sities and capacities in hard disk drives [13]. Commodity
hard drive capacities have grown as much as 100% per year
since 1997, now reaching capacities as great as 500 GB
with prices under $1 per Gigabyte. The use of point-to-
point links in modern Serial ATA (SATA) interfaces has also
allowed for the construction of cost-effective and resource-
efficient disk arrays, since individual disks no longer need to
contend for shared bus resources as in the o lder Ultra ATA
and SCSI standards. Such disk arrays are ideally-suited as
building blocks for video-on-demand (VoD) servers. Since
a modern 500 GB commodity disk can hold more than
100 DVD-quality (4.7 GB) multimedia streams, a single 2U
video-on-demand server could hold 800 movies. A standard
42U rack of such servers could hold over 16000, dwarfing
the corner video store. Five such racks would yield a greater
selection than Netflix (65,000 movies as of 10/1/2006). Al-
though modern systems can easily accommodate the capac-
ity requirements of VoD storage, it is far more challenging
for the server to actually schedule all the distinct request
streams to the much smaller number of disks while meeting
real-time delivery targets.
While capacity has been growing at an unrivaled clip,
disk internal data rate (sequential access bandwidth from
the m edia to the drive’s own buffers) has been improving at
about 40% per year; disk access latency has only been im-
proving about 15% per year [13]. Thus, a high capacity disk
can only achieve good performance if its workload consists
primarily of large sequential transfers. Although a disk ar-
ray can achieve high aggregate throughput by employing
parallel disks, such parallelism is only possible if the sys-
tem achieves good load balance across the disks. Achiev-
ing such balance is complicated by workload hotspots and
variations, as w ell as by “fail-stutter faults”: performance
degradations caused by manufacturing variations, by hard-
ware glitches such as seek errors and bus timeouts, or by
geometry-related variations (e.g., slower reads toward the
center of the drive platter as a result of lower linear momen-
tum).
One approach to achieving load balance in the pres-
ence of workload hotspots is disk-striping, popularly called
RAID-0. Although disk striping can provide ideal load bal-
ance by parallelizing requests across disks, it m ay also suf-
fer from poor performance. Reddy and Banerjee observed
that disk striping thus increases the effective service time
of a request, reducing aggregate throughput of a disk ar-
ray [25]. In particular, the actual transfer rates of striped
data accesses will be accelerated by the degree of striping,
but the seeks must take place across all the d isks, creat-
ing occupancy at the disk heads. Arpaci-Dusseau further
showed that the performance of a RAID-0 array degrades
to the performance of the slowest disk, crippling perfor-
mance in the case of a single-disk performance degrada-
tion [2]. Although designed primarily for redundancy and
availability, RAID-1 (disk mirroring) may a lso improve per-
formance by sp litting a read-domin ated worklo ad between
two disks [24].
Randomization is a useful alternative to disk striping to
reduce the likelihood of hotspotting. In this strategy, data
blocks are mapped to d isks randomly, with any g iven block
of data equally likely to reside on any disk. Hotspotting
is thus eliminated statistically, and this strategy has been
adopted in video-on-demand systems like RIO [29]. The-
oretical results have nevertheless shown that randomiza-
tion may also suffer from hotspotting. In particular, when
D blocks are randomly selected from D disks, it can be
expected with high probability that one disk has requests
for Θ(
log n
log log n
) blocks, using balls-and-bins analysis [12].
Even with careful randomization, there are known to be
problems such as external sorting which are likely to per-
form poorly [4].
Sanders et al. gave an elegant technique to avoid hotspot-
ting by placing each logical data block on two randomly se-
lected disks so that at least one copy can be read without I/O
bottlenecks [28]. This strategy, called random duplicate al-
location (RDA) trades off some capacity for more reliable
performance, but the current rate of capacity growth may
make this approach a reasonable tradeoff.
This paper explores the performance of various data al-
location policies when applied to a disk array for serv-
ing DVD-quality video-on-dema nd streams. The policies
of naive single-disk allocation, disk striping, disk mirror-
ing, randomization, and random duplicate allocation are
implemented on an actual prototype Vo D storage server,
using application-controlled mapping of blocks to disks.
These strategies are tested under uniform workload distribu-
tion, various workload hotspot models, and single-disk fail-
stutter fault models caused by degraded throughput from
a disk. The results show that random duplicate alloca-
tion is the only strategy studied that performs well under
each of the workload hotspotting models. RDA also suffers
fewer and shorter client starvation incidents than most of
the other schemes when exposed to single-disk fail-stutter
faults. These results from an actual prototype storage server
complement and experimentally confirm the prior theoreti-
cal analysis showing the advantages of RDA [28].
The remainder of this paper proceeds as follows. Sec-
tion 2 gives relevant background information on video-on-
demand servers and fail-stutter faults. Section 3 d escribes
strategies for achieving reliable disk array performance.
Section 4 describes the prototype server hardware and soft-
ware, while Section 5 covers the experimental methodol-
ogy. Section 6 presents the experimental results. Section 7
describes related work, and Section 8 concludes the paper.
2 Background
The network server field in general has achieved great
performance improvements in recent years, with such
strategies as event-driven software architectures and zero-
copy I/O [15, 20, 22, 23, 34]. However, these works have
focused primar ily on workloads with small files and high
disk cache hit ratios (e.g., web servers). The VoD work-
load is different, as each stream may be multip le Giga-
bytes and m ay by itself overwhelm the disk cache. The
web proxy server workload sees substantial disk access,
and some software strategies have focused on minimizing
seeks [17]. However, those file sizes are still small co m-
pared to the VoD workload, and the performance levels re-
ported by proxy benchmarks are less than 300 Mbps [27].
The key issue in the performance of video-on-demand
(VoD) storage servers is managing the disk array. Hard disk
drive capacity has been improving at a 100% annual rate
since the introduction of GMR heads in 1997 [13]. At the
same time, the internal data transfer rate of disks, measured
for sequential access between the magnetic media and the
driver’s internal buffers has been improving at roughly 40%
per year. These improvements have come about primarily
through the greater information d ensity on disk. Disk ac-
cess latency has also improved, but only at about 15% per
year. While the capacity improvements in disks certainly
enable the storage of an ever-greater number of high-bitrate
streams, the performance trends only support substantial
improvements if the workload offered to the disks con-
sists primarily of large sequential transfers. Storage servers
achieve hig h data transfer rates by employing parallel disks,
but effectively utilizing parallel disks for large data streams
is challenging because of the need to multiplex a large num-
ber of distinct request streams onto a smaller number of
disks w hile meeting real-time delivery targets. Addition-
ally, the server must carefully balance the load across the
disks to extract the maximum possible parallelism from the
disk array.
There are two important ways in which disk arrays can
fail to achieve their desired parallelism. One way is due
to realistic workloads in which some media objects are far
more popular than others. Chesire et al. studied client-based
streaming-media workloads and found that the distribution
of client requests to objects is Zipf-like with a Zipf param-
eter of 0.47 [6]. In other words, the number of requests to
the object ranked r is roughly proportional to r
−0.47
.Be-
cause the workload is Zipf-like, ther e will be files that are
requested more frequently than others, causing hotspotting.
If files are not carefully spread across the disks, a workload
hotspot will directly lead to disk load imbalance.
A second way for disk arrays to achieve less-than-
expected parallelism is through the unexpected underper-
formance of a component. For example, Remzi and An-
drea Arpaci-Dusseau cite several documented cases of disks
underperforming when describing fail-stutter faults [3]. In
one case, most disks delivered 5.5 MB/s sequential transfer
bandwidth but one disk delivered only 5.0 MB/s due to bad
block faults that were being remapped elsewhere on disk,
introducing seeks. Van Meter pointed out that performance
can vary by a factor of 2 across the different zones of a sin-
gle disk; consequently, a disk reading from a slow zone will
underperform another one in the same array that happens
to be reading from a faster zone [19]. Other performance
degradations occur for reasons that cannot be easily or di-
rectly analyzed.
3 Achieving Reliable Performance
Reliable performance in a parallel I/O system requires
the system to support a high level of concurrency even in
the presence o f workload hotspots and hardware perfor-
mance variations. Effective mapping of request data blocks
to disks can be critically important in determining perfor-
mance. Temporal variations in the workload and unpre-
dictable hardware performance degradations make it impos-
sible to find an ideal solution for all situations. H owever,
some basic principles can help:
1. The entire contents of a stream should not be on one
disk. Otherwise, severe disk load imbalances could
arise for workloads with requests for multiple streams
that happen to use only the same disk.
2. The granularity of allocating data contiguously should
be at least as large as a single time slice in the stream.
Otherwise, a single time slice from a single stream
could require service from multiple disks, activating
simultaneous seeks on each of those disks and thus re-
ducing the aggregate throughput of the system. The
remaind er of this paper refers to such allocation units
as blocks.
3. The performance of the system should not be dramat-
ically impacted by the underperformance of a single
disk.
Traditional file allocation in an operating system meets
none of the above goals: a file is usually on a single disk,
allocated with data blocks of 4 kB (compared to over 4 MB
of d ata in a 5-second time slice of a DVD-quality video),
and an underperformance on a single disk would impact any
attempt by the system to access files o n that disk. Mod-
ern extent-based file systems typically meet goal 2 above
since they aim to put successive writes to a file in contigu-
ous blocks on disk [18]. This paper refers to schemes that
allocate each stream entirely on one disk as naive.
Disk striping (RAID-0) meets goal 1 by spreading data
across all disks and may meet goal 2 by choosing the stripe
size appropriately [11, 16, 31]. However, goal 3 is not met
File
Block
1
Block
3
Block
2
Figure 1. Depiction of assignment of data
blocks to disks using the random duplicate
assignment strategy.
since prior work has shown that RAID-0 array performance
tracks the slowest disk [2]. Disk mirroring (RAID-1) meets
goal 2. Additionally, it may meet goal 3 by splitting the
workload between the two disks in the mirror-pair intelli-
gently [24]. However, RAID-1 does not meet goal 1 since
hotspotting is on ly spread across a mirror-pair; this partic-
ular mirror still sees m ore load than the other mirrors. The
RIO system stores media blocks on randomly selected disks
to support multiple access patterns and stream popularity
variations [29]. Such randomization can enable goal 1 by
distributing popular content across the disks; goal 2 can be
met by choosing the granularity appropriately. However,
randomization alone may actually exacerbate the problem
of single-disk underperformance since every stream access
will now touch the underperforming disk.
To address limitations in randomization alone, the ran-
dom duplicate allocatio n (RDA) strategy replicates every
content block on two randomly-chosen disks [28]. Figure 1
illustrates how the blocks of a file are replicated and scat-
tered across the disks under RDA. Although replication in
RDA was proposed to help improve load balance and tol-
erate workload variations, it may also be useful to tolerate
single-disk fail-stutter disk faults since the more properly
functioning replica may be chosen preferentially over the
degraded one. I n general, replication requires resolving co-
herence issues on updates; this is not relevant here since the
video-on-demand workload is write-once, read-many.
4 Prototype Implementation
Server hardware. This paper reports the performance
of a prototype streaming storage server that employs mod-
ern multicore processors and a SATA d isk array. The system
specifically includes two 2.2 GHz dual-core AMD Opteron
processors, four Gigabit Ethernet links, 4 GB of DRAM,
two Promise FastTrak TX4200 4-port SATA RAID con-
troller, and 8 Seagate SATA disks with 400 GB capacity
and 7200 RPM rotation speed. The SATA controller is con-
figured in JBOD (Just a Bunch Of Disks) mode for all tests.
Performance tests using lmbench indicate that the disks see
an average access time of 17 ms and a peak sequential trans-
fer rate of 525 Mbps; there is negligible inter-disk con-
tention since each disk has a 3 Gbps point-to-point link to
the controller and the controller itself sits on an 8 Gbps PCI-
Xbus.
Server software. The software platform con sists of a
standard Linux 2.6 kernel. Because the operating system is
unchanged, it may still allocate disk blocks at a minimum
granular ity of 4K; in practice, the filesystem tends to allo-
cate file blocks sequentially. The disk array can hold over
650 4.7 GByte DVD-quality streams. However, these tests
use an even larger number of 1 GByte streams to represent
a broader selection of available conten t. Further, the tests
only access the first 300 MB in order to allow the tests to
complete more quickly. There is no substantial difference
in performance between accesses to 300 MB and 4.7 GB
streams; stream data is extremely unlikely to be reused in
the filesystem cache in any case since each test overflows
the cache in less than 30 seconds.
Stream blocks are mapped to the disk array using the al-
location strategies discussed in Section 3: naive, RAID-0,
RAID-1, random, and RDA. The mapping of stream blocks
to disks is controlled through user-level code for all alloca-
tion strategies tested, allowing full flexibility in allocation
without the complexities o f operating-system coding. Test
results show that user-level data management has less than
1% overhead compared to OS-level implementation of ba-
sic policies (such as naive file allocation). This is possible
because no actual file data is passed to the user-level; the
user-level only records disk numbers and offset positions
for the stream data blocks. All interactions with the file
data are performed using the Linux sendfile system call
which directly dispatches data from a disk file to a n etwork
socket. The system also does not use hardware-controlled
RAID because the controller does not support sufficiently
large disk stripes and does not expose how it balances load
in a disk mirror.
Figure 2 represents the flow of operations in the
application-level server software. The server responds to
a client connection by creating a new thread and then send-
ing a med ia data b lock as an initial buffer. The media data
block size is determined by the target bitrate and the time
slice used for allocation. In this system, the target bitrate
is 6 Mbps for DVD-quality streams and the time slice is
5 seconds. It then enters a repeated pattern of sending a
data block, seeing how long it took to perform that transmis-
sion, and sleeping that connection for the remainder of the
time slice represented by the block. If a transmission takes
longer than that time slice, the server sends the next chunk
at a higher rate (by sleeping for less time) until the client
side catches up. Each time the transmission takes longer
than the time slice, the server adds the excess time to a per-
connection deficit variable and sets the target time for the
Accept
Connection
Server Thread
End of file?
Sendfile()
Adjust time slice as
needed
Time slice
reached?
Sleep
Reduce time
slice
End
Yes
No
No
Yes
Figure 2. Flowchart describing operations
in server thread for processing individual
video-on-demand (VoD) client connection
next data block transmission be the standard time slice less
the deficit. This deficit is reduced (o r reset) if the next trans-
mission co mpletes before the target elapses. If the client
side r epeatedly fails to keep up, its buffer will be exhausted
and it will starve; the server will continue to send at a higher
rate until its target has been m et. The prototype delivers the
content via HTTP over TCP/IP using the application-level
rate-pacing described above; realistic multim edia protocols
would add some network overhead but would not change
the disk access pattern.
5 Experimental Methodology
This paper evaluates the resiliency of various file allo-
cation to workload hotspotting and fail-stutter faults. Each
of these types of tests is conducted by measuring the per-
formance of the prototype VoD storage server described in
Section 4. Four client machines on the same LAN as the
server initiate parallel requests for content from the server.
The results report the maximum number of simultaneous
connections that can be successfully supported by the server
for the given workload. This maximum value is deter-
mined by binary search; any test in which any client con-
nection starves even once after the initial buffering (namely,
in which its data buffer empties because of slow server re-
sponses) is considered to have failed.
This paper explores three workload models. The first is
uniform popularity, in which streams are chosen with equal
likelihood from the collection of available streams and the
workload is exactly balanced across the disks in the naive
allocation scheme.
The second workload model uses Zipf-like request dis-
tributions, with a Zipf popularity parameter of 0 .5 to rep-
resent workload hotspotting akin to that seen by Chesire et
al. [6] and a Zipf parameter of 1.0 to model greater request
hotspotting. Each of these tests also uses a Poisson request
arrival model, with a mean interarrival time of 1 second. In
each of these Zipf popularity models, the naive allocation
strategy is split into “naive expected” and “naive worst”. In
the former, the files are allocated across the disks with no
respect for their popularity ranking. In the latter, the files
are allocated across the disks in such a way that the most
popular files are clustered together on one disk, the second
most popular set of files on the second disk, and so forth
down the disk array. We do not consider “naive best” since
the best possible placement of files on disks would depend
greatly on ever-changing popularity and would thus require
continual reshuffling.
For the third workload model, files known to be allocated
on a specific disk or disk mirror are chosen with an extra
likelihood in addition to a base uniform distribution; this
distribution can be thought of as having between 20–80%
overloading of one of the disks. This workload model only
applies to naive and RAID-1 since RAID-0, random, and
RDA all scatter accesses to a ll disks.
The paper also considers the impact of single-disk fail-
stutter faults in which a disk achieves degraded sequential
transfer rates, as in the examples discussed in Section 2. To
model the impact of a single underperforming disk, each
server access to that disk also invokes an additional transfer
that reads in an amount of useless data proportional to the
amount of requested data. For example, a 1 MB read com-
bined with a 20% additional read will actually read 1.2 MB
for every 1 MB of useful data. The VoD server is then sub-
jected to a uniform request workload at the throughput level
that was achieved by the performance tests. The results re-
port the number of times that client connections starve for
the given fault model, the average duration of each starva-
tion incident, and the maximum duratio n of the starvation
incidents seen for each allocation policy.
6 Experimental Results
6.1 Uniform Workload Distribution
Figure 3 shows the performance of the various stream
allocation strategies when subjected to a uniform workload
distribution. In this distribution, the clients request distinct
streams that are chosen in such a way as to provide a com-
pletely balanced workload even with the naive allocation
strategy. Under this workload, each of naive allocation,
random duplicate allocation, and RAID-1 perform nearly
identically, achieving 332–336 simultaneous 6 Mbps DVD-
quality connections without ever starving the client connec-
tions. It is worth noting that these results, with an aggregate
throughput of over 2 Gbps, exceed any previously reported
numbers from single-machine academic or industrial VoD
servers of which we are aware. (Specific performance cita-
tions are provided in Section 7.)
The random allocation strategy achieves 288 connec-
tions; randomization may actually underperform naive al-
location in a uniform workload since the effects of random-
ness may actually create some load imbalances as some
disks are randomly favored over others. RDA avoids this
problem because it always chooses the less heavily-loaded
replica for a given data block. RAID-0 performs the worst,
at 256 connections. RAID-0 actually sees load imbalance
for this workload because the connections start nearly si-
multaneously and all the streams start their disk striping
on disk 0. Similar problems have b een observed with disk
striping in problems such as external sort [33].
6.2 Workload Hotspotting
Figure 4 presents tests where the clients choose from
the set of streams randomly with a Zipf distribution using
a popularity parameter of 0.5 and 1.0. A parameter of 0.5 is
similar to the actual popularities seen by previous work ana-
lyzing multimedia workloads [6]. The performance of naive
allocation degrades by about 10% in the expected case and
by nearly 50% in the worst case. Recall that naive allocates
each file on a single disk, with the popular files split among
the different disks randomly in the expected case and clus-
tered onto the same disks by order of popularity in the worst
case. The oth er schemes perform similarly to , or slightly
better than, their performance in the uniform distribution.
Skewing may degrade performance by introducing load im-
balance but may also help performance by allowing some
file cache effectiveness; these effects seem to be roughly
in balance for all but the naive allocations. The benefits of
RDA and RAID-1 over the other schemes are slightly more
pronounced here than in the other cases, with a 14% lead
over the nearest competitor (naive expected).
A Zipf distribution parameter of 1.0 represents a still
more skewed distribution. Here the positive effects of skew-
ing in improving cache reuse are much more evident, as all
but the naive worst-case allocation perform better than in
the uniform distribution. RDA outperforms naive-worst by
more than double, outperforms naive-expected by 5%, and
outperforms RAID-1, Random, and RAID-0 by 15–20%.
Figure 5 examines another workload hotspotting model,
where specific disks or disk mirrors are overloaded with a
greater request rate. This hotspot model only applies to
naive and RAID- 1, since the other schemes already dis-
tribute files across all the disks. The number on the X-
axis represents the overload percentage applied to the tar-
get disk or disk mirror above and beyond the base uniform
distribution. For example, an overload of 40% indicates
that the target disk will see 40% more load than the other
disks. Note that the overload rates are not exactly 20, 40,
60, and 80% because the number of streams applied in over-
load must be an integer and may not divide nicely with the
base value. The p erformance of the naive allocation strat-
egy drops nearly 40% when an 82.5% overload is applied to
one disk, while the RAID-1 performance drops 30% when a
75% overload is applied to one disk. This sort of overload-
ing does not apply to RDA, random, or RAID-0, and thus
does not affect their performance.
