A Comparison of Mechanisms for Improving TCP Performance over
Wireless Links
Hari Balakrishnan, Venkata N. Padmanabhan, Srinivasan Seshan and Randy H. Katz
1
{hari,padmanab,ss,randy}@cs.berkeley.edu
Computer Science Division, Department of EECS, University of California at Berkeley
Abstract
Reliable transport protocols such as TCP are tuned to per-
form well in traditional networks where packet losses occur
mostly because of congestion. However, networks with
wireless and other lossy links also suffer from significant
losses due to bit errors and handoffs. TCP responds to all
losses by invoking congestion control and avoidance algo-
rithms, resulting in degraded end-to-end performance in
wireless and lossy systems. In this paper, we compare sev-
eral schemes designed to improve the performance of TCP
in such networks. We classify these schemes into three
broad categories: end-to-end protocols, where loss recovery
is performed by the sender; link-layer protocols, that pro-
vide local reliability; and split-connection protocols, that
break the end-to-end connection into two parts at the base
station. We present the results of several experiments per-
formed in both LAN and WAN environments, using
throughput and goodput as the metrics for comparison.
Our results show that a reliable link-layer protocol that is
TCP-aware provides very good performance. Furthermore,
it is possible to achieve good performance without splitting
the end-to-end connection at the base station. We also dem-
onstrate that selective acknowledgments and explicit loss
notifications result in significant performance improve-
ments.
1. Introduction
The increasing popularity of wireless networks indicates
that wireless links will play an important role in future inter-
networks. Reliable transport protocols such as TCP [24, 26]
have been tuned for traditional networks comprising wired
links and stationary hosts. These protocols assume
conges-
tion
in the network to be the primary cause for packet losses
and unusual delays. TCP performs well over such networks
by adapting to end-to-end delays and congestion losses. The
TCP sender uses the cumulative acknowledgments it
receives to determine which packets have reached the
receiver, and provides reliability by retransmitting lost
packets. For this purpose, it maintains a running average of
1. Web page URL http://daedalus.cs.berkeley.edu.
Srinivasan Seshan is now at IBM T.J. Watson Research Center,
Hawthorne, NY (
srini@watson.ibm.com
).
the estimated round-trip delay and the mean linear deviation
from it. The sender identifies the loss of a packet either by
the arrival of several duplicate cumulative acknowledg-
ments or the absence of an acknowledgment for the packet
within a
timeout
interval equal to the sum of the smoothed
round-trip delay and four times its mean deviation. TCP
reacts to packet losses by dropping its transmission (conges-
tion) window size before retransmitting packets, initiating
congestion control or avoidance mechanisms (e.g., slow
start [13]) and backing off its retransmission timer (Karn’s
Algorithm [16]). These measures result in a reduction in the
load on the intermediate links, thereby controlling the con-
gestion in the network.
Unfortunately, when packets are lost in networks for rea-
sons other than congestion, these measures result in an
unnecessary reduction in end-to-end throughput and hence,
sub-optimal performance. Communication over wireless
links is often characterized by sporadic high bit-error rates,
and intermittent connectivity due to handoffs. TCP perfor-
mance in such networks suffers from significant throughput
degradation and very high interactive delays [8].
Recently, several schemes have been proposed to the allevi-
ate the effects of non-congestion-related losses on TCP per-
formance over networks that have wireless or similar high-
loss links [3, 7, 28]. These schemes choose from a variety of
mechanisms, such as local retransmissions, split-TCP con-
nections, and forward error correction, to improve end-to-
end throughput. However, it is unclear to what extent each
of the mechanisms contributes to the improvement in per-
formance. In this paper, we examine and compare the effec-
tiveness of these schemes and their variants, and
experimentally analyze the individual mechanisms and the
degree of performance improvement due to each.
There are two different approaches to improving TCP per-
formance in such lossy systems. The first approach hides
any non-congestion-related losses from the TCP sender and
therefore requires no changes to existing sender implemen-
tations. The intuition behind this approach is that since the
problem is local, it should be solved locally, and that the
transport layer need not be aware of the characteristics of
the individual links. Protocols that adopt this approach
attempt to make the lossy link appear as a higher quality
link with a reduced effective bandwidth. As a result, most of
the losses seen by the TCP sender are caused by congestion.
Appeared in IEEE/ACM Transactions on Networking, Dec.1997. This is a much-extended and revised version of a paper that
appeared at ACM SIGCOMM, 1996.
Examples of this approach include wireless links with reli-
able link-layer protocols such as AIRMAIL [1], split con-
nection approaches such as Indirect-TCP [3], and TCP-
aware link-layer schemes such as the snoop protocol [7].
The second class of techniques attempts to make the sender
aware of the existence of wireless hops and realize that
some packet losses are not due to congestion. The sender
can then avoid invoking congestion control algorithms
when non-congestion-related losses occur — we describe
some of these techniques in Section 3. Finally, it is possible
for a wireless-aware transport protocol to coexist with link-
layer schemes to achieve good performance.
We classify the many schemes into three basic groups,
based on their fundamental philosophy: end-to-end propos-
als, split-connection proposals and link-layer proposals. The
end-to-end protocols attempt to make the TCP sender han-
dle losses through the use of two techniques. First, they use
some form of selective acknowledgments (SACKs) to allow
the sender to recover from multiple packet losses in a win-
dow without resorting to a coarse timeout. Second, they
attempt to have the sender distinguish between congestion
and other forms of losses using an Explicit Loss Notifica-
tion (ELN) mechanism. At the other end of the solution
spectrum, split-connection approaches completely hide the
wireless link from the sender by terminating the TCP con-
nection at the base station. Such schemes use a separate reli-
able connection between the base station and the destination
host. The second connection can use techniques such as
negative or selective acknowledgments, rather than just
standard TCP, to perform well over the wireless link. The
third class of protocols, link-layer solutions, lie between the
other two classes. These protocols attempt to hide link-
related losses from the TCP sender by using local retrans-
missions and perhaps forward error correction [e.g., 18]
over the wireless link. The local retransmissions use tech-
niques that are tuned to the characteristics of the wireless
link to provide a significant increase in performance. Since
the end-to-end TCP connection passes through the lossy
link, the TCP sender may not be fully shielded from wire-
less losses. This can happen either because of timer interac-
tions between the two layers [10], or more likely because of
TCP’s duplicate acknowledgments causing sender fast
retransmissions even for segments that are locally retrans-
mitted. As a result, some proposals to improve TCP perfor-
mance use mechanisms based on the knowledge of TCP
messaging to shield the TCP sender more effectively and
avoid competing and redundant retransmissions [7].
In this paper, we evaluate the performance of several end-
to-end, split-connection and link-layer protocols using end-
to-end throughput and goodput as performance metrics, in
both LAN and WAN configurations. In particular, we seek
to answer the following specific questions:
1. What combination of mechanisms results in best per-
formance for each of the protocol classes?
2. How important is it for link-layer schemes to be aware
of TCP algorithms to achieve high end-to-end through-
put?
3. How useful are selective acknowledgments in dealing
with lossy links, especially in the presence of burst
losses?
4. Is it important for the end-to-end connection to be split
in order to effectively shield the sender from wireless
losses and obtain the best performance?
We answer these questions by implementing and testing the
various protocols in a wireless testbed consisting of Pentium
PC base stations and IBM ThinkPad mobile hosts communi-
cating over a 915 MHz AT&T Wavelan, all running BSD/
OS 2.0. For each protocol, we measure the end-to-end
throughput, and goodputs
for the wired and (one-hop) wire-
less paths. For any path (or link), goodput is defined as the
ratio of the actual transfer size to the total number of bytes
transmitted over that path. In general, the wired and wireless
goodputs differ because of wireless losses, local retransmis-
sions and congestion losses in the wired network. These
metrics allow us to determine the end-to-end performance
as well as the transmission efficiency across the network.
While we used a wireless hop as the lossy link in our exper-
iments, we believe our results are applicable in a wider con-
text to links where significant losses occur for reasons other
than congestion. Examples of such links include high-speed
modems and cable modems.
We show that a reliable link-layer protocol with some
knowledge of TCP results in very good performance. Our
experiments indicate that shielding the TCP sender from
duplicate acknowledgments caused by wireless losses
improves throughput by 10-30%. Furthermore, it is possible
to achieve good performance without splitting the end-to-
end connection at the base station. We also demonstrate that
selective acknowledgments and explicit loss notifications
result in significant performance improvements. For
instance, the simple ELN scheme we evaluated improved
the end-to-end throughput by a factor of more than two
compared to TCP Reno, with comparable goodput values.
The rest of this paper is organized as follows. Section 2
briefly describes some proposed solutions to the problem of
reliable transport protocols over wireless links. Section 3
describes the implementation details of the different proto-
cols in our wireless testbed, and Section 4 presents the
results and analysis of several experiments. Section 5 dis-
cusses some miscellaneous issues related to handoffs, ELN
implementation and selective acknowledgments. We present
our conclusions in Section 6, and mention some future work
in Section 7.
2. Related Work
In this section, we summarize some protocols that have
been proposed to improve the performance of TCP over
wireless links. We also briefly describe some proposed
methods to add SACKs to TCP.
•
Link-layer protocols
: There have been several propos-
als for reliable link-layer protocols. The two main
classes of techniques employed by these protocols are:
error correction, using techniques such as forward error
correction (FEC), and retransmission of lost packets in
response to automatic repeat request (ARQ) messages.
The link-layer protocols for the digital cellular systems
in the U.S. — both CDMA [15] and TDMA [22] — pri-
marily use ARQ techniques. While the TDMA protocol
guarantees reliable, in-order delivery of link-layer
frames, the CDMA protocol only makes a limited
attempt and leaves eventual error recovery to the (reli-
able) transport layer. Other protocols like the AIRMAIL
protocol [1] employ a combination of FEC and ARQ
techniques for loss recovery.
The main advantage of employing a link-layer protocol
for loss recovery is that it fits naturally into the layered
structure of network protocols. The link-layer protocol
operates independently of higher-layer protocols and
does not maintain any per-connection state. The main
concern about link-layer protocols is the possibility of
adverse effect on certain transport-layer protocols such
as TCP, as described in Section 1. We investigate this in
detail in our experiments.
•
Split connection protocols [3, 28]:
Split connection
protocols split each TCP connection between a sender
and receiver into two separate connections at the base
station — one TCP connection between the sender and
the base station, and the other between the base station
and the receiver. Over the wireless hop, a specialized
protocol tuned to the wireless environment may be used.
In [28], the authors propose two protocols — one in
which the wireless hop uses TCP, and another in which
the wireless hop uses a selective repeat protocol (SRP)
on top of UDP. They study the impact of handoffs on
performance and conclude that they obtain no significant
advantage by using SRP instead of TCP over the wire-
less connection in their experiments. However, our
experiments demonstrate benefits in using a simple
selective acknowledgment scheme with TCP over the
wireless connection.
Indirect-TCP [Bakre95] is a split-connection solution
that uses standard TCP for its connection over the wire-
less link. Like other split-connection proposals, it
attempts to separate loss recovery over the wireless link
from that across the wireline network, thereby shielding
the original TCP sender from the wireless link. How-
ever, as our experiments indicate, the choice of TCP
over the wireless link results in several performance
problems. Since TCP is not well-tuned for the lossy link,
the TCP sender of the wireless connection often times
out, causing the original sender to stall. In addition,
every packet incurs the overhead of going through TCP
protocol processing twice at the base station (as com-
pared to zero times for a non-split-connection approach),
although extra copies are avoided by an efficient kernel
implementation. Another disadvantage of split connec-
tions is that the end-to-end semantics of TCP acknowl-
edgments is violated, since acknowledgments to packets
can now reach the source even before the packets actu-
ally reach the mobile host. Also, since split-connection
protocols maintain a significant amount of state at the
base station per TCP connection, handoff procedures
tend to be complicated and slow. Section 5.1 discusses
some issues related to cellular handoffs and TCP perfor-
mance.
•
The Snoop Protocol [7]:
The snoop protocol introduces
a module, called the
snoop agent
, at the base station. The
agent monitors every packet that passes through the TCP
connection in both directions and
maintains a cache of
TCP segments sent across the link that have not yet been
acknowledged by the receiver. A packet loss is detected
by the arrival of a small number of duplicate acknowl-
edgments from the receiver or by a local timeout. The
snoop agent retransmits the lost packet if it has it cached
and suppresses the duplicate acknowledgments. In our
classification of the protocols, the snoop protocol is a
link-layer protocol that takes advantage of the knowl-
edge of the higher-layer transport protocol (TCP).
The main advantage of this approach is that it suppresses
duplicate acknowledgments for TCP segments lost and
retransmitted locally, thereby avoiding unnecessary fast
retransmissions and congestion control invocations by
the sender. The per-connection state maintained by the
snoop agent at the base station is
soft
, and is not essential
for correctness. Like other link-layer solutions, the
snoop approach could also suffer from not being able to
completely shield the sender from wireless losses.
•
Selective Acknowledgments
: Since standard TCP uses
a cumulative acknowledgment scheme, it often does not
provide the sender with sufficient information to recover
quickly from multiple packet losses within a single
transmission window. Several studies [e.g., 11] have
shown that TCP enhanced with selective acknowledg-
ments performs better than standard TCP in such situa-
tions. SACKs were added as an option to TCP by RFC
1072 [14]. However, disagreements over the use of
SACKs prevented the specification from being adopted,
and the SACK option was removed from later TCP
RFCs. Recently, there has been renewed interest in add-
ing SACKs to TCP. Two relevant proposals are the
recent RFC on TCP SACKs [19] and the SMART
scheme [17].
The SACK RFC proposes that each acknowledgment
contain information about up to three non-contiguous
blocks of data that have been received successfully by
the receiver. Each block of data is described by its start-
ing and ending sequence number. Due to the limited
number of blocks, it is best to inform the sender about
the most recent blocks received. The RFC does not spec-
ify the sender behavior, except to require that standard
TCP congestion control actions be performed when
losses occur.
An alternate proposal, SMART, uses acknowledgments
that contain the cumulative acknowledgment and the
sequence number of the packet that caused the receiver
to generate the acknowledgment (this information is a
subset of the three-blocks scheme proposed in the RFC).
The sender uses this information to create a bitmask of
packets that have been delivered successfully to the
receiver. When the sender detects a gap in the bitmask, it
immediately assumes that the missing packets have been
lost without considering the possibility that they simply
may have been reordered. Thus this scheme trades off
some resilience to reordering and lost acknowledgments
in exchange for a reduction in overhead to generate and
transmit acknowledgments.
3. Implementation Details
This section describes the protocols we have implemented
and evaluated. Table 1 summarizes the key ideas in each
scheme and the main differences between them. Figure 1
shows a typical loss situation over the wireless link. Here,
the TCP sender is in the middle of a transfer across a two-
hop network to a mobile host. At the depicted time, the
sender’s congestion window consists of 5 packets. Of the
five packets in the network, the first two packets are lost on
the wireless link. As described in the rest of this section,
each protocol reacts to these losses in different ways and
Name Category Special Mechanisms
E2E end-to-end standard TCP-Reno
E2E-NEWRENO end-to-end TCP-NewReno
E2E-SMART end-to-end SMART-based selective acks
E2E-IETF-SACK end-to-end IETF selective acks
E2E-ELN end-to-end Explicit Loss Notification (ELN)
E2E-ELN-RXMT end-to-end ELN with retransmit on first dupack
LL link-layer none
LL-TCP-AWARE link-layer duplicate ack suppression
LL-SMART link-layer SMART-based selective acks
LL-SMART-TCP-AWARE link-layer SMART and duplicate ack suppression
SPLIT split-connection none
SPLIT-SMART split-connection SMART-based wireless connection
Table 1. Summary of protocols studied in this paper.
1 2 3 4
4
3
2
1
5
5
congestion window = 5
Figure 1. A typical loss situation
TCP Source
Base Station
TCP Receiver
Lossy Link
Packets Stored
at Sender
Packets in Flight
Acknowledgments Returning
generates messages that result in loss recovery. Although
this figure only shows data packets being lost, our experi-
ments have wireless errors in both directions.
3.1 End-To-End Schemes
Although a wide variety of TCP versions are used on the
Internet, the current de facto standard for TCP implementa-
tions is TCP Reno [26]. We call this the E2E protocol, and
use it as the standard basis for performance comparison.
The E2E-NEWRENO protocol improves the performance
of TCP-Reno after multiple packet losses in a window by
remaining in fast recovery mode if the first new acknowl-
edgment received after a fast retransmission is “partial”, i.e,
is less than the value of the last byte transmitted when the
fast retransmission was done. Such partial acknowledge-
ments are indicative of multiple packet losses within the
original window of data. Remaining in fast recovery mode
enables the connection to recover from losses at the rate of
one segment per round trip time, rather than stall until a
coarse timeout as TCP-Reno often would [9, 12].
The E2E-SMART and E2E-IETF-SACK protocols add
SMART-based and IETF selective acknowledgments
respectively to the standard TCP Reno stack. This allows
the sender to handle multiple losses within a window of out-
standing data more efficiently. However, the sender still
assumes that losses are a result of congestion and invokes
congestion control procedures, shrinking its congestion
window size. This allows us to identify what percentage of
the end-to-end performance degradation is associated with
standard TCP’s handling of error detection and retransmis-
sion. We used the SMART-based scheme [17] only for the
LAN experiments. This scheme is well-suited to situations
where there is little reordering of packets, which is true for
one-hop wireless systems such as ours. Unlike the scheme
proposed in [17], we do not use any special techniques to
detect the loss of a retransmission. The sender retransmits a
packet when it receives a SMART acknowledgment only if
the same packet was not retransmitted within the last round-
trip time. If no further SMART acknowledgments arrive, the
sender falls back to the coarse timeout mechanism to
recover from the loss. We used the IETF selective acknowl-
edgement scheme both for the LAN and the WAN experi-
ments. Our implementation is based on the RFC and takes
appropriate congestion control actions upon receiving
SACK information [4].
The E2E-ELN protocol adds an Explicit Loss Notification
(ELN) option to TCP acknowledgments. When a packet is
dropped on the wireless link, future cumulative acknowl-
edgments corresponding to the lost packet are marked to
identify that a non-congestion related loss has occurred.
Upon receiving this information with duplicate acknowl-
edgments, the sender may perform retransmissions without
invoking the associated congestion-control procedures. This
option allows us to identify what percentage of the end-to-
end performance degradation is associated with TCP’s
incorrect invocation of congestion control algorithms when
it does a fast retransmission of a packet lost on the wireless
hop. The E2E-ELN-RXMT protocol is an enhancement of
the previous one, where the sender retransmits the packet on
receiving the first duplicate acknowledgement with the ELN
option set (as opposed to the third duplicate acknowledge-
ment in the case of TCP Reno), in addition to not shrinking
its window size in response to wireless losses.
In practice, it might be difficult to identify which packets
are lost due to errors on a lossy link. However, in our exper-
iments we assume sufficient knowledge at the receiver
about wireless losses to generate ELN information. We
describe some possible implementation policies and strate-
gies for the ELN mechanism in Section 5.2.
3.2 Link-Layer Schemes
Unlike TCP for the transport layer, there is no de facto stan-
dard for link-layer protocols. Existing link-layer protocols
choose from techniques such as Stop-and-Wait, Go-Back-N,
Selective Repeat and Forward Error Correction to provide
reliability. Our base link-layer algorithm, called LL, uses
cumulative acknowledgments to determine lost packets that
are retransmitted locally from the base station to the mobile
host. To minimize overhead, our implementation of LL
leverages off TCP acknowledgments instead of generating
its own. Timeout-based retransmissions are done by main-
taining a smoothed round-trip time estimate, with a mini-
mum timeout granularity of 200 ms to limit the overhead of
processing timer events. This still allows the LL scheme to
retransmit packets several times before a typical TCP Reno
transmitter would time out. LL is equivalent to the snoop
agent that does not suppress any duplicate acknowledg-
ments, and does not attempt in-order delivery of packets
across the link (unlike protocols proposed in [15], [22]).
While the use of TCP acknowledgments by our LL protocol
renders it atypical of traditional ARQ protocols, we believe
that it still preserves the key feature of such protocols: the
ability to retransmit packets locally, independently of and
on a much faster time scale than TCP. Therefore, we expect
the qualitative aspects of our results to be applicable to gen-
eral link-layer protocols.
We also investigated a more sophisticated link-layer proto-
col (LL-SMART) that uses selective retransmissions to
improve performance. The LL-SMART protocol performs
this by applying a SMART-based acknowledgment scheme
at the link layer. Like the LL protocol, LL-SMART uses
TCP acknowledgments instead of generating its own and
limits its minimum timeout to 200 ms. LL-SMART is
equivalent to the snoop agent performing retransmissions
based on selective acknowledgements but not suppressing
duplicate acknowledgments at the base station.
