1
Theory and Practice of Bloom Filters for
Distributed Systems
Sasu Tarkoma, Christian Esteve Rothenberg, and Eemil Lagerspetz
Abstract—Many network solutions and overlay networks uti-
lize probabilistic techniques to reduce information processing
and networking costs. This survey article presents a number of
frequently used and useful probabilistic techniques. Bloom filters
and their variants are of prime importance, and they are heavily
used in various distributed systems. This has been reflected in
recent research and many new algorithms have been proposed for
distributed systems that are either directly or indirectly based on
Bloom filters. In this survey, we give an overview of the basic and
advanced techniques, reviewing over 20 variants and discussing
their application in distributed systems, in particular for caching,
peer-to-peer systems, routing and forwarding, and measurement
data summarization.
Index Terms—Bloom filters, probabilistic structures, dis-
tributed systems
I. INTRODUCTION
Many network solutions and overlay networks utilize prob-
abilistic techniques to reduce information processing and net-
working costs. This survey presents a number of frequently
used and useful probabilistic techniques. Bloom filters (BF)
and their variants are of prime importance, and they are heavily
used in various distributed systems. This has been reflected in
recent research and many new algorithms have been proposed
for distributed systems that are either directly or indirectly
based on Bloom filters.
Fast matching of arbitrary identifiers to values is a basic
requirement for a large number of applications. Data objects
are typically referenced using locally or globally unique identi-
fiers. Recently, many distributed systems have been developed
using probabilistic globally unique random bit strings as node
identifiers. For example, a node tracks a large number of peers
that advertise files or parts of files. Fast mapping from host
identifiers to object identifiers and vice versa are needed. The
number of these identifiers in memory may be great, which
motivates the development of fast and compact matching
algorithms.
Given that there are millions or even billions of data
elements, developing efficient solutions for storing, updating,
and querying them becomes increasingly important. The key
idea behind the data structures discussed in this survey is that
by allowing the representation of the set of elements to lose
some information, in other words to become lossy, the storage
requirements can be significantly reduced.
The data structures presented in this survey for probabilistic
representation of sets are based on the seminal work by Burton
S. Tarkoma and E. Lagerspetz are with University of Helsinki, Department
of Computer Science
C. E. Rothenberg is with the University of Campinas (Unicamp), Depart-
ment of Computer Engineering and Industrial Automation
Bloom in 1970. Bloom first described a compact probabilistic
data structure that was used to represent words in a dictionary.
There was little interest in using Bloom filters for networking
until 1995, after which this area has gained widespread interest
both in academia and in the industry. This survey provides
an up-to-date view to this emerging area of research and
development that was first surveyed in the work of Broder
and Mitzenmacher [1].
Section II introduces the functionality and parameters of the
Bloom filter as a hash-based, probabilistic data structure. The
theoretical analysis is complemented with practical examples
and common practices in the underpinning hashing techniques.
Section III surveys as many as twenty-three Bloom filter
variants discussing their key features and their differential be-
haviour. Section IV covers a number of recent applications in
distributed systems, such as caches, database servers, routers,
security, and packet forwarding relying on packet header size
Bloom filters. Finally, Section V concludes the survey with a
brief summary on the rationale behind the widespread use of
the polymorphic Bloom filter data structure.
II. BLOOM FILTERS
The Bloom filter is a space-efficient probabilistic data struc-
ture that supports set membership queries. The data structure
was conceived by Burton H. Bloom in 1970 [2]. The structure
offers a compact probabilistic way to represent a set that can
result in false positives (claiming an element to be part of
the set when it was not inserted), but never in false negatives
(reporting an inserted element to be absent from the set). This
makes Bloom filters useful for many different kinds of tasks
that involve lists and sets. The basic operations involve adding
elements to the set and querying for element membership in
the probabilistic set representation.
The basic Bloom filter does not support the removal of ele-
ments; however, a number of extensions have been developed
that also support removals. The accuracy of a Bloom filter
depends on the size of the filter, the number of hash functions
used in the filter, and the number of elements added to the set.
The more elements are added to a Bloom filter, the higher the
probability that the query operation reports false positives.
Broder and Mitzenmacher have coined the Bloom filter
principle [1]:
Whenever a list or set is used, and space is at a
premium, consider using a Bloom filter if the effect
of false positives can be mitigated.
A Bloom filter is an array of m bits for representing a set
S = {x
1
, x
2
, . . . , x
n
} of n elements. Initially all the bits in the
2
filter are set to zero. The key idea is to use k hash functions,
h
i
(x), 1 ≤ i ≤ k to map items x ∈ S to random numbers
uniform in the range 1, . . . m. The hash functions are assumed
to be uniform. The MD5 hash algorithm is a popular choice
for the hash functions.
An element x ∈ S is inserted into the filter by setting the
bits h
i
(x) to one for 1 ≤ i ≤ k. Conversely, y is assumed a
member of S if the bits h
i
(y) are set, and guaranteed not to
be a member if any bit h
i
(y) is not set. Algorithm 1 presents
the pseudocode for the insertion operation. Algorithm 2 gives
the pseudocode for the membership test of a given element x
in the filter. The weak point of Bloom filters is the possibility
for a false positive. False positives are elements that are not
part of S but are reported being in the set by the filter.
Data: x is the object key to insert into the Bloom filter.
Function: insert(x)
for j : 1 . . . k do
/
*
Loop all hash functions k
*
/
i ← h
j
(x);
if B
i
== 0 then
/
*
Bloom filter had zero bit at
position i
*
/
B
i
← 1;
end
end
Algorithm 1: Pseudocode for Bloom filter insertion
Data: x is the object key for which membership is tested.
Function: ismember(x) returns true or false to the
membership test
m ← 1;
j ← 1;
while m == 1 and j ≤ k do
i ← h
j
(x);
if B
i
== 0 then
m ← 0;
end
j ← j + 1;
end
return m;
Algorithm 2: Pseudocode for Bloom member test
Figure 1 presents an overview of a Bloom filter. The Bloom
filter consists of a bitstring of length 32. Three elements have
been inserted, namely x, y, and z. Each of the elements have
been hashed using k = 3 hash functions to bit positions in
the bitstring. The corresponding bits have been set to 1. Now,
when an element not in the set, w, is looked up, it will be
hashed using the same three hash functions into bit positions.
In this case, one of the positions is zero and hence the Bloom
filter reports correctly that the element is not in the set. It may
happen that all the bit positions of an element report that the
corresponding bits have been set. When this occurs, the Bloom
filter will erroneously report that the element is a member of
the set. These erroneous reports are called false positives. We
observe that for the inserted elements, the hashed positions
correctly report that the bit is set in the bitstring.
Figure 2 illustrates a practical example of a Bloom filter
through adding and querying elements. In this example, the
Fig. 1. Overview of a Bloom filter
Fig. 2. Addition and query example using a Bloom filter
Bloom filter is a bitstring of length 16. The bit positions are
numbered 0 to 15, from right to left. Three hash functions
are used: h
1
, h
2
, and h
3
, being MD5, SHA1 and CRC32,
respectively. The elements added are text strings containing
only a single letter. The Bloom filter starts out empty, with
all bits unset, or zero. When adding an element, the values
of h
1
through h
3
(modulo 16) are calculated for the element,
and corresponding bit positions are set to one. After adding
a and b, the Bloom filter has positions 15, 9, 8, 3 and 1 set.
In this case, a and b have one common bit position (8). We
further add elements y and l. After this, positions 15, 14, 13,
10, 9, 8, 7, 5, 3 and 1 are set. When we query for q and z, the
same hash functions are used. Bit positions that correspond
to q and z are examined. If the three bits for an element
are set, that element is assumed to be present. In the case
of q, position 0 is not set, and therefore q is guaranteed not to
be present in the Bloom filter. However, z is assumed to be
present, since the corresponding bits have been set. We know
that z is a false positive: it is reported present though it is not
actually contained in the set of added elements. The bits that
correspond to z (positions 15, 10 and 7) were set through the
addition of elements b, y and l.
For optimal performance, each of the k hash functions
should be a member of the class of universal hash functions,
which means that the hash functions map each item in the
universe to a random number uniform over the range. The
development of uniform hashing techniques has been an
active area of research. An almost ideal solution for uniform
hashing is presented in [3]. In practice, hash functions yielding
sufficiently uniformly distributed outputs, such as MD5 or
CRC32, are useful for most probabilistic filter purposes. For
candidate implementations, see the empirical evaluation of 25
hash functions by Henke et al. [4]. Later in Section II-C we
discuss relevant hashing techniques further.
A Bloom filter constructed based on S requires space O(n)
and can answer membership queries in O(1) time. Given x ∈
3
TABLE I
KEY BLOOM FILTER PARAMETERS
Parameters Increase
Number of hash functions (k) More computation, lower false positive rate as k → k
opt
Size of filter (m) More space is needed, lower false positive rate
Number of elements in the set (n) Higher false positive rate
S, the Bloom filter will always report that x belongs to S, but
given y 6∈ S the Bloom filter may report that y ∈ S.
Table I examines the behaviour of three key parameters
when their value is either decreased or increased. Increasing
or decreasing the number of hash functions towards k
opt
can
lower false positive ratio while increasing computation in
insertions and lookups. The cost is directly proportional to the
number of hash functions. The size of the filter can be used to
tune the space requirements and the false positive rate (fpr).
A larger filter will result in fewer false positives. Finally, the
size of the set that is inserted into the filter determines the
false positive rate. We note that although no false negatives
(fn) occur with regular BFs, some variants will be presented
later in the article that may result in false negatives.
A. False Positive Probability
We now derive the false positive probability rate of a Bloom
filter and the optimal number of hash functions for a given
false positive probability rate. We start with the assumption
that a hash function selects each array position with equal
probability. Let m denote the number of bits in the Bloom
filter. When inserting an element into the filter, the probability
that a certain bit is not set to one by a hash function is
1 −
1
m
. (1)
Now, there are k hash functions, and the probability of any
of them not having set a specific bit to one is given by
1 −
1
m
k
. (2)
After inserting n elements to the filter, the probability that
a given bit is still zero is
1 −
1
m
kn
. (3)
And consequently the probability that the bit is one is
1 −
1 −
1
m
kn
. (4)
For an element membership test, if all of the k array
positions in the filter computed by the hash functions are set
to one, the Bloom filter claims that the element belongs to the
set. The probability of this happening when the element is not
part of the set is given by
1 −
1 −
1
m
kn
!
k
≈
1 − e
−kn/m
k
. (5)
1e-009
1e-008
1e-007
1e-006
1e-005
0.0001
0.001
0.01
0.1
1
1 10 100 1000 10000 100000
False positive probability (p)
Number of inserted elements (n)
False positive rate of Bloom filters
m=64
m=512
m=1024
m=2048
m=4096
Fig. 3. False positive probability rate for Bloom filters.
We note that e
−kn/m
is a very close approximation of (1 −
1
m
)
kn
[1]. The false positive probability decreases as the size
of the Bloom filter, m, increases. The probability increases
with n as more elements are added. Now, we want to minimize
the probability of false positives, by minimizing (1 −e
−kn/m
)
k
with respect to k. This is accomplished by taking the derivative
and equaling to zero, which gives the optimal value of k
k
opt
=
m
n
ln 2 ≈
9m
13n
. (6)
This results in the false positive probability of
1
2
k
≈ 0.6185
m/n
. (7)
Using the optimal number of hashes k
opt
, the false positive
probability can be rewritten and bounded
m
n
≥
1
ln 2
. (8)
This means that in order to maintain a fixed false positive
probability, the length of a Bloom filter must grow linearly
with the number of elements inserted in the filter. The number
of bits m for the desired number of elements n and false
positive rate p, is given by
m = −
n ln p
(ln 2)
2
. (9)
Figure 3 presents the false positive probability rate p as a
function of the number of elements n in the filter and the filter
size m. An optimal number of hash functions k = (m/n) ln 2
has been assumed.
There is a factor of log
2
e ≈ 1.44 between the amount of
4
space used by a Bloom filter and the optimal amount of space
that can be used. There are other data structures that use space
closer to the lower bound, but they are more complicated (cf.
[5], [6], [7]).
Recently, Bose et al. [8] have shown that the false positive
analysis originally given by Bloom and repeated in many sub-
sequent articles is optimistic and only a good approximation
for large Bloom filters. The revisited analysis proves that the
commonly used estimate (Eq. 5) is actually a lower bound and
the real false positive rate is larger than expected by theory,
especially for small values of m.
B. Operations
Standard Bloom filters do not support the removal of
elements. Removal of an element can be implemented by
using a second Bloom filter that contains elements that have
been removed. The problem of this approach is that the false
positives of the second filter result in false negatives in the
composite filter, which is undesirable. Therefore a number of
dedicated structures have been proposed that support deletions.
These are examined later in this survey.
A number of operations involving Bloom filters can be
implemented easily, for example the union and halving of a
Bloom filter. The bit-vector nature of the Bloom filter allows
the union of two or more Bloom filters simply by performing
bitwise OR on the bit-vectors. Given two sets S
1
and S
2
, a
Bloom filter B that represents the union S = S
1
∪ S
2
can
be created by taking the OR of the original Bloom filters
B = B
1
∨ B
2
assuming that m and the hash functions are the
same. The merged filter B will report any element belonging
to S
1
or S
2
as belonging to set S. The following theorem
gives a lower bound for the false positive rate of the union of
Bloom filters [9]:
Theorem 1: The false positive probability of BF (A∪ B) is
not less than that of BF (A) and BF (B). At the same time,
the false positive probability of BF (A) ∪ BF (B) is also not
less than that of BF (A) and BF (B).
If the BF size m is divisible by 2, halving can be easily
done by bitwise ORing the first and second halves together.
Now, the range of the hash functions needs to be accordingly
constrained, for instance, by applying the mod(m/2) to the
hash outputs.
Bloom filters can be used to approximate set intersection;
however, this is more complicated than the union operation.
One straightforward approach is to assume the same m and
hash functions and to take the logical AND operation between
the two bit-vectors. The following theorem gives the proba-
bility for this to hold [9]:
Theorem 2: If BF (A ∩ B), BF (A), and BF (B) use the
same m and hash functions, then BF (A ∩ B) = BF (A) ∩
BF (B) with probability (1 − 1/m)
k
2
|A−A∩B||B−A∩B|
.
The inner product of the bit-vectors is an indicator of
the size of the intersection [1]. The idea of a bloomjoin
was presented by Mackert and Lohman in 1986 [10]. In a
bloomjoin, two hosts, A and B, compute the intersection of
two sets S
1
and S
2
, when A has the first set and B the second.
It is not feasible to send all the elements from A to B, and vice
versa. In a bloomjoin, S
1
is represented using a Bloom filter
and sent from A to B. B can then compute the intersection
and send back this set. Host A can then check false positives
with B in a final round.
C. Hashing techniques
Hash functions are the key building block of probabilistic
filters. There is a large literature on hash functions spanning
from randomness analysis to security evaluation over many
networking and computing applications. We focus on the best
practices and recent developments in hashing techniques which
are relevant to the performance and practicality of Bloom filter
constructs. For further details, deeper theoretical foundations
and system-specific applications we refer to related work, such
as [4], [11], [12], [13].
One noteworthy property of Bloom filters is that the false
positive performance depends only on the bit-per-element ratio
(m/n) and not on the form or size of the hashed elements.
As long as the size of the elements can be bounded, hashing
time can be assumed to be a constant factor. Considering the
trend in computational power versus memory access time, the
practical bottleneck is the amount of (slow) memory accesses
rather than the hash computation time. Nevertheless, whenever
a filter application needs to run at line speed, hardware-
amenable per-packet operations are critical [13].
In the following subsections, we briefly present hashing
techniques that are the basis for good Bloom filter implemen-
tations. We start with perfect hashing, which is an alternative
to Bloom filters when the set is known beforehand and it is
static. Double hashing allows reducing the number of true hash
computations. Partitioned hashing and multiple hashing deal
with how bits are allocated in a Bloom filter. Finally, the use
of simple hash functions is considered.
1) Perfect Hashing Scheme: A simple technique called
perfect hashing (or explicit hashing) can be used to store a
static set S of values in an optimal manner using a perfect hash
function. A perfect hash function is a computable bijection
from S to an array of |S| = n hash buckets. The n-size
array can be used to store the information associated with
each element x ∈ S [5].
Bloom filter like functionality can be obtained by, given
a set of elements S, first finding a perfect hash function P
and then storing at each location an f = 1/ǫ bit fingerprint,
computed using some (pseudo-)random hash function H.
Figure 4 illustrates this perfect hashing scheme.
Lookup of x simply consists of computing P (x) and check-
ing whether the stored hash function value matches H(x).
When x ∈ S, the correct value is always returned, and when
x /∈ S a false positive (claiming the element being in S) occurs
with probability at most ǫ. This follows from the definition of
2-universal hashing by Carter and Wengman [14], that any
element y not in S has probability at most ǫ of having the
same hash function value h(y) as the element in S that maps
to the same entry of the array.
While space efficient, this approach is disconsidered for
dynamic environments, because the perfect hash function
needs to be recomputed when the set S changes.
5
Element 1 Element 2 Element 3 Element 4 Element 5
Fingerprint(4) Fingerprint(5) Fingerprint(2) Fingerprint(1) Fingerprint(3)
Fig. 4. Example of explicit hashing
Another technique for minimal perfect hashing was intro-
duced by Antichi et al. [15]. It relies on Bloom filters and
Blooming Trees to turn the imperfect hashing of a Bloom
filter into a perfect hashing. The technique gives space and
time savings. This technique also requires a static set S, but
can handle a huge number of elements.
2) Double Hashing: The improvement of the double hash-
ing technique over basic hashing is being able to generate
k hash values based on only two universal hash functions
as base generators (or “seed” hashes). As a practical conse-
quence, Bloom filters can be built with less hashing operations
without sacrificing performance. Kirsch and Mitzenmacher
have shown [16] that it requires only two independent hash
functions, h
1
(x) and h
2
(x), to generate additional “pseudo”
hashes defined as:
h
i
(x) = h
1
(x) + f(i) ∗ h
2
(x) (10)
where i is the hash value index, f(i) can be any arbitrary
function of i (e.g., i
2
), and x is the element being hashed. For
Bloom filter operations, the double hashing scheme reduces the
number of true hash computations from k down to two without
any increase in the asymptotic false positive probability [16].
3) Partitioned Hashing: In this hashing technique, the k
hash functions are allocated disjoint ranges of m/k consec-
utive bits instead of the full m-bit array space. Following
the same false positive probability analysis of Sec. II-A, the
probability of a specific bit being 0 in a partitioned Bloom
filter can be approximated to:
(1 − k/m)
n
≈ e
−kn/m
(11)
While the asymptotic performance remains the same, in
practice, partitioned Bloom filters exhibit a poorer false posi-
tive performance as they tend to have larger fill factors (more
1s) due to the m/k bit range restriction. This can be explained
by the observation that:
(1 − 1/m)
k∗n
> (1 − k/m)
n
(12)
4) Multiple Hashing: Multiple hashing is a popular tech-
nique that exploits the notion of having multiple hash choices
and having the power to choose the most convenient candidate.
When applied for hash table constructions, multiple hashing
provides a probabilistic method to limit the effects of collisions
by allocating elements more-or-less evenly distributed. The
original idea was proposed by Azar et al. in his seminal work
on balanced allocations [17]. Formulating hashing as a balls
into bins problem, the authors show that if n balls are placed
sequentially into m for m = O(n) with each ball being
placed in one of a constant d = 2 randomly chosen bins,
then, after all balls are inserted, the maximal load in a bin is,
with high probability, (ln ln n)/ln d + O(1). V
¨
ocking et al.
[18] elaborate on this observation and propose the always-go-
left algorithm (or d-left hashing scheme) to break ties when
inserting (chained) elements to the least loaded one among the
d partitioned candidates.
As a result this hashing technique provides an almost
optimal (up to an additive constant) load-balancing scheme.
In addition to the balancing improvement, partitioning the
hash buckets (i.e., bins) into groups makes d-left hashing
more hardware friendly as it allows the parallelized look-
up of the d hash locations. Thus, hash partitioning and tie-
breaking have elevated d-left hashing as an optimal technique
for building high performance (negligible overflow probabil-
ities) data structures such as the multiple level hash tables
(MHT) [19] or counting Bloom filters [20]. A breakthrough
Bloom filter design was recently proposed using an open-
addressed multiple choice hash table based on d-left hashing,
element fingerprints (a smaller representation like the last f
bits of the element hash) and dynamic bit reassignment [21].
After all optimizations, the authors show that the performance
is comparable to plain Bloom filter constructs, outperforms
traditional counting Bloom filter constructs (see d-left CBF
in Sec. III-B), and easily extensible to support practical
networking applications (e.g., flow tracking in Sec. IV-D).
The power of (two) choices has been exploited by Lumetta
and Mitzenmacher to improve the false positive performance
of Bloom filters [22]. The key idea consists of considering not
one but two groups of k hash functions. On element insertion,
the selection criteria is based on the group of k hash functions
that sets fewer bits to 1. The caveat is that when checking for
elements, both groups of k hash functions need to be checked
since there is no information on which group was initially used
and false positives can potentially be claimed for either group.
Although it may appear counter-intuitive, under some settings
(high m/n ratios), setting fewer ones in the filter actually pays
off the double checking operations.
Fundamentally similar in exploiting the power of choices
in producing less dense (improved) Bloom filters, the method
proposed by Hao et al. [23] is based on a partitioned hashing
technique which results in a choice of hash functions that set
fewer bits. Experimental results show that this improvement
can be as much as a ten-fold increase in performance over
standard constructs. However, the choice of hash functions
cannot be done on an element basis as in [22], and its
applicability is constrained to non-dynamic environments.
5) Simple hash functions: A common assumption is to
consider output hash values as truly random, that is, each
hashed element is independently mapped to a uniform location.
While this is a great aid to theoretical analyses, hash function
implementations are known to behave far worse than truly ran-
dom ones. On the other hand, empirical works using standard
universal hashing have been reporting negligible differences in
practical performance compared to predictions assuming ideal
hashing (see [24] for the case of Bloom filters).
Mitzenmacher and Vadhany [25] provide the seeds to for-
mally explaining this gap between the theory and practice