Foundations and Trends
R
in
Theoretical Computer Science
Vol. 1, No 2 (2005) 117–236
c
2005 S. Muthukrishnan
Data Streams: Algorithms and Applications
S. Muthukrishnan
Rutgers University, New Brunswick, NJ, USA, muthu@cs.rutgers.edu
Abstract
In the data stream scenario, input arrives very rapidly and there is lim-
ited memory to store the input. Algorithms have to work with one or
few passes over the data, space less than linear in the input size or time
significantly less than the input size. In the past few years, a new the-
ory has emerged for reasoning about algorithms that work within these
constraints on space, time, and number of passes. Some of the meth-
ods rely on metric embeddings, pseudo-random computations, sparse
approximation theory and communication complexity. The applications
for this scenario include IP network traffic analysis, mining text mes-
sage streams and processing massive data sets in general. Researchers
in Theoretical Computer Science, Databases, IP Networking and Com-
puter Systems are working on the data stream challenges. This article is
an overview and survey of data stream algorithmics and is an updated
version of [175].
1
Introduction
We study the emerging area of algorithms for processing data streams
and associated applications, as an applied algorithms research agenda.
We begin with three puzzles.
1.1 Puzzle 1: Finding Missing Numbers
Let π be a permutation of {1,...,n}. Further, let π
−1
be π with one
element missing. Paul shows Carole π
−1
[i] in increasing order i. Carole’s
task is to determine the missing integer. It is trivial to do the task if
Carole can memorize all the numbers she has seen thus far (formally,
she has an n-bit vector), but if n is large, this is impractical. Let us
assume she has only a few – say O(logn) – bits of memory. Nevertheless,
Carole must determine the missing integer.
This starter puzzle has a simple solution: Carole stores
s =
n(n +1)
2
−
j≤i
π
−1
[j],
which is the missing integer in the end. Each input integer entails one
subtraction. The total number of bits stored is no more than 2logn.
This is nearly optimal because Carole needs at least log n bits in the
118
1.1. Puzzle 1: Finding Missing Numbers 119
worst case since she needs to output the missing integer. (In fact, there
exists the following optimal algorithm for Carole using log n bits. For
each i, store the parity sum of the ith bits of all numbers seen thus
far. The final parity sum bits are the bits of the missing number.) A
similar solution will work even if n is unknown, for example by letting
n = max
j≤i
π
−1
[j] each time.
Paul and Carole have a history. It started with the “twenty ques-
tions” problem solved in [201]. Paul, which stood for Paul Erdos, was
the one who asked questions. Carole is an anagram for Oracle. Aptly,
she was the one who answered questions. Joel Spencer and Peter Win-
kler used Paul and Carole to coincide with Pusher and Chooser respec-
tively in studying certain chip games in which Carole chose which
groups the chips falls into and Paul determined which group of chips
to push. In the puzzle above, Paul permutes and Carole cumulates.
Generalizing the puzzle a little further, let π
−2
be π with two
elements missing. The natural solution would be for Carole to store
s =
n(n+1)
2
−
j≤i
π
−2
[j] and p = n! −
j≤i
π
−2
[j], giving two equa-
tions with two unknown numbers, but this will result in storing large
number of bits since n! is large. Instead, Carole can use far fewer bits
tracking
s =
n(n +1)
2
−
j≤i
π
−2
[j] and ss =
n(n + 1)(2n +1)
6
−
j≤i
(π
−2
[j])
2
In general, what is the smallest number of bits needed to identify the
k missing numbers in π
−k
? Following the approach above, the solution
is to maintain power sums
s
p
(x
1
,...,x
k
)=
k
i=1
(x
i
)
p
,
for p =1,...,k and solving for x
i
’s. A different method uses elementary
symmetric polynomials [169]. The ith such polynomial σ
i
(x
1
,...,x
k
)is
the sum of all possible i term products of the parameters, i.e.,
σ
i
(x
1
,...,x
k
)=
j
1
<···<j
i
x
j
1
···x
j
i
.
Carole continuously maintains σ
i
’s for the missing k items in field F
q
for some prime n ≤ q ≤ 2n, as Paul presents the numbers one after the
120 Introduction
other (the details are in [169]). Since
i=1,...,k
(z − x
i
)=
k
i=0
(−1)
i
σ
i
(x
1
,...,x
k
)z
k−i
,
Carole needs to factor this polynomial in F
q
to determine the missing
numbers. No deterministic algorithms are known for the factoring prob-
lem, but there are randomized algorithms take roughly O(k
2
logn) bits
and time [214]. The elementary symmetric polynomial approach above
comes from [169] where the authors solve the set reconciliation prob-
lem in the communication complexity model. The subset reconciliation
problem is related to our puzzle.
Generalizing the puzzle, Paul may present a multiset of elements in
{1,...,n} with a single missing integer, i.e., he is allowed to re-present
integers he showed before; Paul may present updates showing which
integers to insert and which to delete, and Carole’s task is to find
the integers that are no longer present; etc. All of these problems are
no longer (mere) puzzles; they are derived from motivating data stream
applications.
1.2 Puzzle 2: Fishing
Say Paul goes fishing. There are many different fish species U =
{1,...,u}. Paul catches one fish at a time, a
t
∈ U being the fish species
he catches at time t. c
t
[j]=|{a
i
| a
i
= j,i ≤ t}| is the number of times
he catches the species j up to time t. Species j is rare at time t if it
appears precisely once in his catch up to time t. The rarity ρ[t]ofhis
catch at time t is the ratio of the number of rare j’s to u:
ρ[t]=
|{j | c
t
[j]=1}|
u
.
Paul can calculate ρ[t] precisely with a 2u-bit vector and a counter for
the current number of rare species, updating the data structure in O(1)
operations per fish caught. However, Paul wants to store only as many
bits as will fit his tiny suitcase, i.e., o(u), preferably O(1) bits.
Suppose Paul has a deterministic algorithm to compute ρ[t] pre-
cisely. Feed Paul any set S ⊂ U of fish species, and say Paul’s algorithm
1.2. Puzzle 2: Fishing 121
stores only o(u) bits in his suitcase. Now we can check if any i ∈ S by
simply feeding Paul i and checking ρ[t + 1]: the number of rare items
decreases by one if and only if i ∈ S. This way we can recover entire S
from his suitcase by feeding different i’s one at a time, which is impos-
sible in general if Paul had only stored o(|S|) bits. Therefore, if Paul
wishes to work out of his one suitcase, he can not compute ρ[t] exactly.
This argument has elements of lower bound proofs found in the area of
data streams.
However, proceeding to the task at hand, Paul can approximate ρ[t].
Paul picks k random fish species each independently, randomly with
probability 1/u at the beginning and maintains the number of times
each of these fish types appear in his bounty, as he catches fish one
after another. Say X
1
[t],...,X
k
[t] are these counts after time t. Paul
outputs ˆρ[t]=
|{i|X
i
[t]=1}|
k
as an estimator for ρ.Wehave,
Pr(X
i
[t]=1)=
|{j | c
t
[j]=1}|
u
= ρ[t],
for any fixed i and the probability is over the fish type X
i
.Ifρ[t]is
large, say at least 1/k,ˆρ[t] is a good estimator for ρ[t] with arbitrarily
small ε and significant probability.
However, typically, ρ is unlikely to be large because presumably u
is much larger than the species found at any spot Paul fishes. Choosing
a random species from {1,...,u} and waiting for it to be caught is
ineffective. We can make it more realistic by redefining rarity with
respect to the species Paul in fact sees in his catch. Let
γ[t]=
|{j | c
t
[j]=1}|
|{j | c
t
[j] =0}|
.
As before, Paul would have to approximate γ[t] because he can not
compute it exactly using a small number of bits. Following [28], define
a family of hash functions H⊂[n] → [n] (where [n]={1,...,n})tobe
min-wise independent if for any X ⊂ [n] and x ∈ X,wehave
Pr
h∈H
[h(x) = min h(X)] =
1
|X|
,
where, h(X)={h(x): x ∈ X}. Paul chooses k min-wise independent
hash functions h
1
,h
2
,...,h
k
for some parameter k to be determined
