Early Experience with an Internet Broadcast
System Based on Overlay Multicast
Yang-hua Chu Aditya Ganjam T. S. Eugene Ng
Sanjay G. Rao Kunwadee Sripanidkulchai Jibin Zhan
Hui Zhang
December 2003
CMU-CS-03-214
School of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213
This research was sponsored by DARPA under contract number F30602-99-1-0518, and by NSF under
grant numbers Career Award NCR-9624979 ANI-9730105, ITR Award ANI-0085920, and ANI-9814929.
Additional support was provided by Sloan Research Fellowship and Intel. Views and conclusions contained
in this document are those of the authors and should not be interpreted as representing the official policies,
either expressed or implied, of DARPA, NSF, Intel, or the U.S. government.
Keywords: overlay networks, Internet evaluation, peer-to-peer, multimedia broadcast
Abstract
In this paper, we report on experience in building and deploying an operational Internet
broadcast system based on Overlay Multicast. In over a year, the system has been providing
a cost-effective alternative for Internet broadcast, used by over 3600 users spread across
multiple continents in home, academic and commercial environments. Technical conferences
and special interest groups are the early adopters. Our experience confirms that Overlay
Multicast can be easily deployed and can provide reasonably good application performance.
The experience has led us to identify first-order issues that are guiding our future efforts
and are of importance to any Overlay Multicast protocol or system. Our key contributions
are (i) enabling a real Overlay Multicast application and strengthening the case for overlays
as a viable architecture for enabling group communication applications on the Internet, (ii)
the details in engineering and operating a fully functional streaming system, addressing a
wide range of real-world issues that are not typically considered in protocol design studies,
and (iii) the data, analysis methodology, and experience that we are able to report given our
unique standpoint.
1
1 Introduction
The vision of enabling live video broadcast as a common Internet utility in a manner that
any publisher can broadcast content to any set of receivers has been driving the research
agenda in the networking community for over a decade. The high cost of bandwidth required
for server-based solutions or content delivery networks, and the sparse deployment of IP
Multicast are two main factors that have limited broadcasting to only a subset of Internet
content publishers such as large news organizations. There remains a need for cost-effective
technology for low-budget content publishers such as broadcasters of seminars, workshops
and special interest groups.
Recent work in Overlay Multicast [14, 9, 19, 7, 21, 30, 39, 22, 34, 25, 41, 10, 5] has made
the case that overlay networks are a promising architecture to enable quick deployment of
multicast functionality on the Internet. In such an architecture, application end-points self-
organize into an overlay structure and data is distributed along the links of the overlay.
The responsibilities and cost of providing bandwidth is shared amongst the application
end-points, reducing the burden at the content publisher. The ability for users to receive
content that they would otherwise not have access to provides a natural incentive for them
to contribute resources to the system.
Most of the existing work, including our own earlier work [9, 8], focus on issues related to
“protocol design,” and evaluate their potential using simulation or university-based Internet
test-beds. We believe that an equally important and complementary style of research can be
conducted using an “application-centric” approach. This approach involves the wide-spread
operational use of an application by real users, and letting the experience gained direct the
research process. The more content publishers and receivers rely on the application, the
stronger the case for Overlay Multicast, validating its relevance as a research question. In
addition, the unique experience obtained in the process leads to important insight that can
motivate future research in the area.
In adopting the “application-centric” approach, our primary consideration was to provide
a useful and deployable tool to the general public, and reach operational status as quickly
as possible. Therefore, we identify and address a wide range of issues, some of which are
not typically considered in protocol design studies, but affect the successful deployment of
Overlay Multicast. Our system copes with dynamics in user participation, adapts to appli-
cation performance and Internet dynamics, supports users that have a wide range of network
bandwidth and supports users behind network address translators (NATs) and firewalls. We
have built supporting mechanisms such as logging receiver performance, monitoring of sys-
tem components, and recovering from component failures. In engineering our system, we
have adopted simple or natural solutions, with the provision that the design decisions could
be revisited in the light of future experience. This approach has accelerated the deployment
of the system, and, consequently has led to faster feedback from real deployment.
The challenges involved in obtaining operational experience we report in this paper must
not be underestimated. First, we have invested significant effort in convincing content pub-
lishers and event organizers that it is worth their while to experiment with the new technol-
ogy. Second, while we have made earnest efforts to get our system deployed, the participation
of viewers in our broadcasts depends on a range of factors not under our control, including
2
(ADSL, behind NAT)
(Wireless,
behind firewall)
(Ethernet)
Broadcast Source
(Ethernet)
Encoder
A/V Signal
Media Interface
Monitor
Logger
Media Player
Figure 1: Broadcast system overview.
the content we have access to. Third, unlike conventional research experiments, we have fre-
quently had to work under the pressure to succeed in even our earliest broadcast attempts.
Failures would significantly deter event organizers and limit future adoption of our system.
One consequence is that it is critical to adopt robust, stable and well-tested code – a per-
formance refinement that may seem trivial to incorporate may take months to actually be
deployed.
In over a year, we have been building an operational broadcast system based on Overlay
Multicast and deploying it among more than 3600 real users in real Internet environments
for over 20 events. We view the design and deployment effort as an ongoing process, and
report on the experience accumulated so far. Overall, our experience confirms that Overlay
Multicast is easy to deploy and can provide reasonably good application performance. In
addition, we believe that our unique set of data, analysis methodology, and experience are
useful to the research community.
The rest of this paper is organized as follows. In § 2, we present an overview of the system.
§ 3, 4, and 5 presents the deployment experience, analysis methodology, and performance
analysis of our system. § 6 presents key design lessons learned from the experience that are
guiding the future research directions.
2 System Overview
Figure 1 gives a high-level overview of our broadcast system. The encoder takes the mul-
timedia signal from the camera, converts into audio and video streams, and sends to the
broadcast source. The broadcast source and receivers run an overlay multicast protocol to
disseminate the streams along the overlay. Each receiver gets the broadcast stream, and for-
wards to the media player running on the same machine. In addition, the participating hosts
send performance statistics to the monitor and log server for both on-line and post-mortem
analyses.
The detailed software architecture at the source and the receiver is depicted in Figure 2.
Tracing the data flow, the broadcast source encodes the media signal into audio and multiple
video packet streams (a), marks the packets with priority bits (b), and sends them to the
overlay modules (shaded blocks). Multiple streams and prioritization are discussed in § 2.2.
The overlay modules replicate packets to all of its children (c). Packets are translated from
3
