$<>@9.;1&@.@2(;6B2>?6@E$<>@9.;1&@.@2(;6B2>?6@E
$+&05<9.>$+&05<9.>
<:=A@2>&062;02.0A9@E$A/960.@6<;?.;1
$>2?2;@.@6<;?
<:=A@2>&062;02
$9.E2>3<>1.=@6B2!$)612<&@>2.:6;4#B2>$9.E2>3<>1.=@6B2!$)612<&@>2.:6;4#B2>
'52;@2>;2@'52;@2>;2@
<;.@5.;*.9=<92
#>24<;>.1A.@2;?@6@A@2<3&062;02'205;<9<4E
%.6;2><?@2>
#>24<;>.1A.@2;?@6@A@2<3&062;02'205;<9<4E
&5.;C262;
#>24<;>.1A.@2;?@6@A@2<3&062;02'205;<9<4E
>6?=6;<C.;
#>24<;>.1A.@2;?@6@A@2<3&062;02'205;<9<4E
.B61!.62>
#>24<;>.1A.@2;?@6@A@2<3&062;02'205;<9<4E
&22;2D@=.423<>.116@6<;.9.A@5<>?
<99<C@56?.;1.116@6<;.9C<>8?.@5@@=?=1D?05<9.>96/>.>E=1D21A0<:=?06-3.0
$.>@<3@52<:=A@2>.;1&E?@2:?>056@20@A>2<::<;?.;1@52646@.9<::A;60.@6<;?.;1
"2@C<>86;4<::<;?
2@A?8;<C5<C.002??@<@56?1<0A:2;@/2;2G@?E<A
6@.@6<;2@.69?6@.@6<;2@.69?
<;.@5.;*.9=<92%.6;2><?@2>&5.;C262;>6?=6;<C.;.B61!.62>E9.;!0".:22.9@<;
$A.B61&@22>2.;1 6A76;,A$9.E2>3<>.1.=@6B2!$B612<?@>2.:6;4<B2>@52;@2>;2@$><0
&$@5$%*<>8?5<=D=9<6@6;4"2C:.42&<A>02?.;1&2;?<>?!.>05
1<65@@=1D1<6<>4<=E>645@&<062@E<3$5<@<#=@60.9
;?@>A:2;@.@6<;;46;22>?#;2=>6;@<>2920@><;600<=E:.E/2:.123<>=2>?<;.9A?2<;9E&E?@2:.@60
>2=><1A0@6<;.;116?@>6/A@6<;1A=960.@6<;<3.;E:.@2>6.96;@56?=.=2>3<>.322<>3<>0<::2>06.9
=A>=<?2?<>:<16G0.@6<;<3@520<;@2;@<3@52=.=2>.>2=><56/6@21
'56?<;32>2;02$><02216;46?/><A45@@<E<A3<>3>22.;1<=2;.002??@5.?/22;.002=@213<>6;09A?6<;6;
<:=A@2>&062;02.0A9@E$A/960.@6<;?.;1$>2?2;@.@6<;?/E.;.A@5<>6F21.1:6;6?@>.@<><3$+&05<9.>$92.?2
0<;@.0@A?63C20.;:.82@56?1<0A:2;@:<>2.002??6/92=1D?05<9.>=1D21A
Downloaded From: http://proceedings.spiedigitallibrary.org/ on 12/19/2013 Terms of Use: http://spiedl.org/terms
270
A player for adaptive MPEG video streaming over the Internet
Jonathan Walpole, Rainer Koster, Shanwei Cen, Crispin Cowan, David Maier, Dylan McNamee, Calton Pu, David Steere
and Liujin
Yu
Department
of
Computer Science and Engineering
Oregon Graduate Institute
of
Science and Technology
P.O.
Box 91000, Portland, Oregon 97006
ABSTRACT
This paper describes the design and implementation
of
a real-time, streaming, Internet video and audio player. The
player has a number
of
advanced features including dynamic adaptation to changes in available bandwidth, latency and
latency variation; a multi-dimensional media scaling capability driven
by
user-specified quality
of
service (QoS) require-
ments; and support for complex content comprising multiple synchronized video and audio streams. The player was
developed as part
of
the QUASAR t project at Oregon Graduate Institute, is freely available, and serves as a testbed for
research in adaptive resource management and QoS control.
Keywords: Internet, video, real-time, MPEG, adaptive, feedback, quality
of
service
1. INTRODUCTION
Digital multimedia systems are becoming Ubiquitous, with nearly all computer platfonns offering support for real-
time datatypes such as audio and video. There has also been a rapid proliferation
of
communications networks, giving
multimedia computing the potential to augment, or even replace, traditional broadcast and print media with more interac-
tive and personalized infonnation services. Compressed digital video
I and the Internet are technologies at the heart
of
this
revolution. This paper addresses the design and implementation
of
a player for interactively streaming video and audio
data, in real-time, across the Internet.
Although the promise
of
ubiquitous multimedia computing and networking
is
exciting, today's real-time multimedia
applications tend to be resource-hungry and inflexible in the presence
of
contention for resources. For example, on shared
computing platforms, when CPU, network or disk bandwidth become short, multimedia applications are often unable to
maintain the real-time playout
of
video and audio data. One proposed solution to this problem is for underlying systems
software to support resource reservations in order to provide QoS guarantees to applications
2
•
3
•
To
solve the problems
of
inflexible multimedia applications, such reservations must be end-ta-end, covering all system components, including net-
work and end-host resources.
Although resource reservation has received significant attention within standards bodies such as the IETF4.\ it is
likely to
take
a long time to achieve wide-spread deployment
of
such end-ta-end, system-level solutions. In the meantime,
multimedia systems researchers are exploring alternative solutions
6
•
One such alternative is to build adaptive multimedia
applications that attempt to preserve presentation quality in one dimension, by allowing quality to vary in other dimen-
sions when resources become scarce
7
.8.9. For example, when available network bandwidth becomes scarce.
an
adaptive
video player could reduce presentation quality in the frame rate dimension (Le., temporal resolution) in order to preserve
real-time play out
at
a given spatial resolution.
The appropriate way to trade quality among various quality dimensions
is
both user and task-dependent. For example.
a user trying to identify license plates from a surveillance video would probably attach a higher value to spatial resolution
than frame rate. In contrast, a user watching a basketball game may attach a higher-value to frame rate than spatial resolu-
tion.
To
support this variety
of
preferences, adaptive multimedia systems should allow higher layers to specify their QoS
requirements,,·lO.1J
.
In this paper, we describe the architecture
of
an adaptive Internet-based video and audio player. The player supports
standard VCR facilities. such as play. fast forward. rewind. pause, etc., for real-time streaming
of
stored MPEG-com-
t.
QUASAR is an abbreviation for QUAlity Specification and Adaptive Resource management.
SPIE Vol.
3240.
0277-786X1981$10.00
Downloaded From: http://proceedings.spiedigitallibrary.org/ on 12/19/2013 Terms of Use: http://spiedl.org/terms
pressed video.
It
also takes advantage
of
the inherent flexibility
of
a software solution to go beyond a simple network
TV.
and offer several advanced features, including complex content comprised
of
multiple synchronized video and audio
streams, simultaneous synchronized access to
distributed servers, pre/etching to hide remote access latency, and adaptive
QoS
control in several quality dimensions. These features allow the player to support advanced applications and make it
robust in heterogeneous environments that have a high variability in the capacity
of
their communication and computation
resources.
The paper is organized as follows. Section 2 outlines the requirements
of
several advanced multimedia applications
that motivate the player's novel features. Section 3 introduces the QoS model and adaptation mechanisms used
in
the
player. Section 4 presents a detailed description
of
the player's architecture, and Section 5 discusses its performance.
Finally, Section 6 concludes and discusses future work.
2. REQUIREMENTS
This section illustrates three advanced multimedia streaming applications with requirements that are not addressed by
current multimedia presentation tools. Such applications need support for complex presentations, composed
of
several
synchronized video or audio clips. and control over several quality dimensions
2.1 Electronic News
Gathering
Television production is moving towards the all-digital studio, in which a single, high-bandwidth data network
replaces the mUltiple, dedicated analog data and control channels
of
conventional studios. Electronic news gathering is
one application
of
the digital studiO
l2
•
In a local television studio, several editors may simultaneously produce broadcast
segments for a daily news show, combining video footage from station archives with network newsfeeds and recently
taken local coverage, along with graphics and voice overs. Simultaneously, video may
be
captured to storage and previ-
ously edited segments played
out
for broadcast. In selecting video clips for inclusion. the news editor might play back
video in forward or reverse modes at up to 100 times the normal speed. However.
in
choosing exactly where
to
start and
end a clip, the editor might want to slow down playback, or even step through frame by frame.
In
determining when to cut
between two camera angles - such as
of
a public official and a questioner in the audience - the editor will want to see the
video from the two angles played back simultaneously, synchronized with each other and with the corresponding audio.
Once the segment is composed from various clips, the news editor will want to review it at normal speed and at near
broadcast quality.
This editing scenario highlights several demands on multimedia streaming systems. First, there can
be
multiple com-
peting users
of
such systems, and
it
must be possible to share resources and prioritize their usage. For example, ptayback
of
a segment for actual broadcast should take priority over editing tasks, but two editors should share resources roughly
equally. Second, user actions, such as changing play speed and direction, have potentially complex impact on presentation
characteristics. For example, speeding up playback to 100 times normal speed shouldn't cause the system to attempt to
display 3000 frames per second. The system should select a subsequence
of
frames to display that provides equivalent
quality to normal-speed playback. Third. the level
of
playback quality is not constant - rough cuts
of
a news story can be
displayed with lower quality than that used during final review
of
the story or during broadcast. Different quality dimen-
sions can also
be
seen in this example - for example, frame rate and synchronization
of
streams.
The editing scenario also has the requirement to deliver multiple streams at once, potentially from different servers.
This requirement arises when editing a news segment that uses both archival and current video footage resident on differ-
ent servers. While it is possible to copy all the clips, in advance, onto a single server, that approach is often impractical.
For example, during the early iterations
of
the editing process, the actual selection and length
of
clips changes frequently.
and hence pre-copying clips then can be prohibitive.
2.2 Sports Video on Demand
Staehli
II
describes a sports video-on-demand application, using professional basketball as an example. Sports events
are often recorded using several cameras and microphones, providing raw footage composed
of
multiple viewpoints and
sound tracks. Interesting actions, such as plays or fouls, may only
be
visible from some .camera angles. and hence access
to the entire raw footage is valuable to support activities such as customizable, interactive replays. Using current multime-
dia presentation technology, however, viewers only have access to a single edited presentation, and there is minimal sup-
port for user interaction.
A sports video on demand system that provides access to all the video and audio recordings
of
an event would create
a new, more interactive way
of
viewing sports. Users could have a small window for each camera and dynamically select
271
Downloaded From: http://proceedings.spiedigitallibrary.org/ on 12/19/2013 Terms of Use: http://spiedl.org/terms
272
what
to
view on the main screen. Replays could
be
selected from specific camera angles, and viewed
in
slow-motion,
while the game continued in additional windows. Commentators and statistics could
be
provided,
as
required, in addi-
tional windows.
To
summarize, the user
of
such a sports video
on
demand system could
have
access
to
all
of
the informa-
tion that is currently available only
to
TV editors.
In
such
an
application, the editor may still
playa
role, perhaps defining
a default view
of
the sporting event that can be further customized
by
the viewer
if
desired.
If
such complex and interactive presentation applications are to
be
extended into the home, across networks such
as
the Internet, adaptive management of scarce resources such
as
network bandwidth will become critical.
In
the example
above, small windows providing additional views may be assigned a lower quality than the main window, hence reducing
their resource consumption. For displaying action in sports events, a high frame rate
is
often needed. This requirement
may be supported in the presence
of
resource contention by lowering the spatial resolution
of
the video. The statistics
board, on the other hand, would be displayed with high spatial resolution to make it readable, but with very low frame
rate. A picture
of
a commentator may not be important, but may
be
nice to have when resources suffice.
In
any
case, com-
mentator audio would likely take precedence over commentator video.
2.3 Intelligence Analysis
Consider an intelligence analyst assembling a multimedia presentation,
to
be
communicated with other analysts via
an internetwork, to back up an assessment
of
the health
of
a foreign leader. Part of the content might be composed
of
video
segments taken from television broadcasts, photos from news agencies, and audio clips from radio addresses. Different
parts of the presentation require emphasizing different aspects
of
quality for optimal interpretation. One part of the presen-
tation is a sequence of video clips and color images arranged
by
date, in order to illustrate changes in complexion and
weight. Here the analyst wants to emphasize individual image quality, both in color fidelity and spatial resolution. Another
part
of
the presentation shows recent video clips
of
the leader walking and greeting visitors, to show stiffness and slow-
ness of movement. The analyst is less concerned with image detail in this segment, but wants to specify that the timing
of
the presentation be very accurate, so as not to introduce artificial jerkiness. Next comes a clip
of
a televised interview,
where the analyst deems audio fidelity most important, to demonstrate slurred speech. A fourth part
of
the presentation
shows two video segments side by side, to demonstrate that what
is
being offered as recent coverage
of
the leader in his
office strongly resembles coverage shown two months ago, but shot from a different angle. Here the most important aspect
of
quality to the analyst is having the two segments stay precisely synchronized.
Under optimal conditions, an analyst viewing the presentation remotely may be able
to
play it back with near-perfect
image quality, timing, audio fidelity and synchronization throughout. More likely, however,
is
that the presentation must
be shown to several clients under a variety of circumstances with different computers, different network connections and
competing processing going on. In these cases, the QoS specifications that the authoring analyst has attached to the vari-
ous parts
of
the presentation should be available to influence resource management decisions. Ideally, the most important
aspects
of
the content should be given the resources they need for accurate rendering. Similarly, the QoS requirements
of
the viewing analysts should also be available to influence resource management, so they get the quality needed
to
support
their task.
2.4
Summary
of
requirements
The examples above have shown a variety
of
requirements for advanced multimedia streaming applications. First,
users should be able to combine data
to
form complex presentations. Operators such
as
concatenation, synchronization
(concurrent presentation), and clipping (selecting particular parts) of streams need to be supported. Modifications
of
clips
such
as
changing the audio gain, scaling a video image, or including a stream
as
slow motion are also useful. Second, the
content
of
a presentation can be distributed, since material from different remote servers
may
be included
by
reference
in
a presentation. During play-back, data must be retrieved from those locations and displayed in real-time. Third, such
applications need to provide
adaptive QoS control. For video, important QoS dimensions include spatial resolution, frame
rate, color accuracy, and temporal jitter.
If
several streams are combined, synchronization
is
another important quality
dimension. Moreover, different parts
of
a presentation may be assigned different priorities. The user should be able
to
trade reduced quality in some dimensions for increased quality in others.
3. BACKGROUND
3.1 QoS
model
Most existing multimedia presentation systems tend either
to
not manage quality explicitly at all, or
try
to control
only a single quality dimension, such as frame rate.
In
addition, quality is often tightly coupled to presentation content and
view in inflexible ways. Examples
of
this excessively tight coupling include binding the frame rate and resolution
of
a
