scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

A relay-assisted distributed source coding problem

15 Aug 2008-pp 136-141
TL;DR: A relay-assisted distributed source coding problem with three statistically correlated sources is formulated and studied and a single unified coding strategy which subsumes all four special classes is derived.
Abstract: A relay-assisted distributed source coding problem with three statistically correlated sources is formulated and studied. Natural lower bounds for the rates are presented. The lower bounds are shown to be achievable for three special classes of joint source distributions. The achievable coding strategies for the three special classes are observed to have markedly different characteristics. A special class for which the lower bounds are not achievable is presented. A single unified coding strategy which subsumes all four special classes is derived.

Summary (1 min read)

Remark:

  • For any arbitrarily small positive real number and all n sufficiently large, X n can be encoded at location-a at the rate H(X|Y) bits per sample, and decoded at location-b with a probability which can be made smaller than .
  • The converse to the Slepian-Wolf coding theorem proves that the lower bound R a ≥ H(X|Y) cannot be improved.
  • The cutset bound (3.5) in Section III-A proves that the lower bound R b ≥ H(X|Z) cannot be improved.

B. Special cases in which the cutset bounds are tight

  • Thus for any arbitrarily small positive real number and all n sufficiently large, X n can be encoded at location-a at the rate H(X|Y, Z) bits per sample, and decoded at location-b with a probability which can be made smaller than .
  • The authors shall call this relay-assisted distributed source coding strategy as relay decode and re-encode.
  • The operation at the intermediate location-b is to simply forward the message from location-a to location-c without any processing.
  • The authors shall call this relay-assisted distributed source coding strategy as relay forward.
  • The operation at the intermediate location-b is to (i) encode its observations to be reproduced at location-c and (ii) forward the message from location-a to location-c without any processing.

is, the auxiliary random variables U, V satisfy the following two Markov chains: U-X-(Y, Z) and V-(U, Y)-(X, Z). Then all rate-pairs

  • EQUATION are operationally admissible for relay-assisted distributed source coding and cover all the four previously discussed special cases.
  • The authors first show that all the four previously discussed special cases are covered by Theorem 1.
  • The authors now sketch the proof that all rate-pairs (R a , R b ) satisfying (4.7) and (4.8) are operationally admissible for relayassisted distributed source coding.
  • The encoder also randomly partitions the tuple of codewords C u into nonoverlapping bins.

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

A
Relay-Assisted Distributed Source Coding
Problem
Prakash Ishwar
Dept. Electrical and Computer Engineering
Boston University
Boston, MA 02215, USA
Email: pi@bu.edu
S. Sandeep Pradhan
Dept. Electrical Engineering and Computer Science
University of Michigan
Ann Arbor, MI 48109, USA
Email: pradhanv@eecs.umich.edu
Abstract A relay-assisted distributed source coding problem
with three statistically correlated sources is formulated and
studied. Natural lower bounds for the rates are presented. The
lower bounds are shown to be achievable for three special classes
of joint source distributions. The achievable coding strategies for
the three special classes are observed to have markedly dierent
characteristics. A special class for which the lower bounds are not
achievable is presented. A single unified coding strategy which
subsumes all four special classes is derived.
I. I
Consider the following distributed source coding problem
which is illustrated in Fig. 1. n samples X
n
:= (X
1
, . . . , X
n
)
of an information source available at location-a need to be
reproduced at location-c with high probability. The message
from location-a, of bit rate R
a
bits per source sample, is
relayed through an intermediate location-b which has access
to n samples Y
n
of a second information source which are
statistically correlated to X
n
. The message from location-b to
location-c is of bit rate R
b
bits per source sample. Location-c
has access to n samples Z
n
of a third information source which
are correlated to X
n
and Y
n
. The reproduction at location-c is
b
X
n
. The goal is to characterize the set of minimal rate pairs
(R
a
, R
b
) for which P(X
n
,
b
X
n
) 0 as n (the rate region).
The problem just described belongs to the general class of
source network problems [1, Chapter 3] (also called distributed
source coding problems) which have been studied in the Infor-
mation Theory literature in the 70’s and 80’s. Beginning the
late 90’s, the construction of practical distributed source codes
for emerging wireless sensor network applications [2] received
much attention and renewed interest in the study of general
source network problems. For instance, the three sources in
Fig. 1 can represent the correlated observation samples of
three physically separated, radio-equipped, sensing devices
which are measuring some physical quantities of interest in the
vicinity of their respective locations. In some situations, the
device at location-c may need to learn about the observations
at location-a but is uninterested in the observations of an
intermediate location-b whose connectivity with both location-
a and location-c is orders of magnitude stronger than that
between location-a and location-c. Depending on the relative
strengths of the connections between location-a and location-b
and between location-b and location-c, it may be necessary to
use codes of dierent rates on these links to achieve the desired
objective of reproducing X
n
at location-c. This motivates the
characterization of the entire rate region of minimal rate pairs
(R
a
, R
b
) for which P(X
n
,
b
X
n
) can be driven down to zero as
n . More recently, the study of general source networks
has also benefited from insights which are emerging from the
related area of network coding [3].
In the terminology of [1, p. 246], the source network of
Fig. 1 has three input vertices corresponding to the three
sources, one output vertex corresponding to the decoder at
location-c, and two intermediate vertices corresponding to the
encoders at locations a and b. The depth of this network which
is the length of the longest path from an input to an output is
3. The structure of this network is significantly dierent from
that of several related source network problems of depth 2
studied in the 70’s and 80’s whose rate regions are fairly well
understood (see [4], [1, Chapter 3], and references therein). In
more recent terminology, the specific source network of Fig. 1
would be the simplest example of the so-called single-demand
line source networks (see for example [5], [6]).
Despite their apparent simplicity, the general rate regions of
line source networks are not well understood except in very
special cases when the joint probability distribution of the
sources and/or the fidelity criteria for source reproductions
have some special properties. For instance, inner and (non-
matching) outer bounds for the rate-distortion region for
a slightly more general class of tree source networks are
derived in [5]. However, it is assumed that there exists an
underlying hidden source conditioned on which all the source
observations become independent. Moreover, all locations
need to reproduce the hidden source to within some fidelity
as measured by a single-letter distortion function. A recent
closely related work [7] studies the relay-assisted distributed
source coding problem of Fig. 1 but when X
n
Z
n
Y
n
forms
a Markov chain (degraded side information) and when X
n
needs to be reproduced at location-b and location-c only to
within expected distortion levels as measured by associated
single-letter fidelity criteria. Inner and (non matching) outer
bounds for the rate-distortion region are derived and some
special cases are discussed. In the context of asymptotically
lossless reconstruction of X
n
at (only) location-c, the condition
X
n
Z
n
Y
n
corresponds to the special case 2 in Section III-B.

R
b
a b c
X
n
Y
n
Z
n
b
X
n
R
a
Fig.
1. Relay-assisted distributed source coding.
Other related works include [8], [9].
As discussed in what follows, the study of even the sim-
plest line source network with three sources (Fig. 1) oers
new insights for ecient information processing and coding
in source networks. The presence of an intermediate relay
with correlated side information Y
n
makes the problem both
interesting and challenging in a number of ways. Perhaps the
two most striking features of this problem are the following.
(i) It is not necessary for the relay to reproduce the source
X
n
.
1
Hence, the relay need not “fully decode” the message
from location-a. (ii) The presence of correlated Y
n
at the relay
should be helpful but it is not necessary that Y
n
be reproduced
at location-c for X
n
to be reproduced at location-c. Thus we see
that there is a very interesting set of interactions and tradeos
which come into play in this problem. These considerations
also give rise to a number of distinct coding strategies for the
relay. For instance, the relay can simply forward the message
from location-a to location-c without any processing, it can
encode some of its own observations and convey it to location-
c, or it can process the message from location-a using Y
n
to
decode an intermediate description of X
n
and then jointly re-
encode the intermediate description together with a part of Y
n
and send it to location-c, etc.
The problem is mathematically formulated in Section II.
Section III derives a natural outer bound for the rate region
and discusses three special source correlation structures for
which the outer bound is tight. The study of these three special
cases is rewarding because it exposes three coding strategies
of significantly dierent flavors. An interesting special case
for which the outer bound is not tight is also presented. The
distinctiveness of the three coding strategies motivates the
search for a single unified coding strategy which subsumes
all the previously discussed strategies as special cases. This
is accomplished by Theorem 1 in Section IV. Concluding
remarks are made Section V.
Notation: For n N, a
n
:= (a
1
, . . . , a
n
). Kronecker delta
δ
uv
: δ
uv
= 1 if u = v and is zero otherwise.
II. S   P
Let ((X
i
, Y
i
, Z
i
))
n
i=1
, n N, be a three-component discrete-
time, memoryless, stationary source, that is, for i = 1, 2, 3, . . . ,
(X
i
, Y
i
, Z
i
) i.i.d. p
X,Y,Z
(x, y, z) with (x, y, z) X × Y × Z.
It is assumed that max{|X|, |Y|, |Z|} < . X
n
is observed at
1
If X
n
is also required to be reproduced at location-b then the problem
is much easier and the rate region is immediate (cf. remark at the end of
Section II). Thus the interesting problem is when X
n
must be reproduced at
only location-c.
location-a, Y
n
at location-b, and Z
n
at location-c. The goal
is to reproduce X
n
at location-c. The constraint is that the
message from location-a has to pass through the intermediate
location-b. The problem is illustrated and summarized in Fig. 1
Definition 1: A relay-assisted distributed source code with
parameters (n, m
a
, m
b
) is the triple (e
a
, e
b
, g
c
):
e
a
: X
n
{1, . . . , m
a
}, e
b
: {1, . . . , m
a
}×Y
n
{1, . . . , m
b
},
g
c
: {1, . . . , m
b
} × Z
n
X
n
.
Here, n is the blocklength, e
a
will be called the source
encoder, e
b
will be called the relay encoder, and g
c
will
be called the source decoder. The quantities (1/n) log
2
(m
a
)
and (1/n) log
2
(m
b
) will be called the source and relay
block-coding rates (in bits per sample) respectively.
Definition 2: A rate-pair (R
a
, R
b
) is said to be operationally
admissible for relay-assisted distributed source coding if, for
every > 0, there exists an N
N, such that, for all n
N
, there exists a relay-assisted distributed source code with
parameters (n, m
a
, m
b
) satisfying
1
n
log
2
(m
a
) R
a
+
, (2.1)
1
n
log
2
(m
b
) R
b
+
, (2.2)
P(g
c
(e
b
(e
a
(X
n
), Y
n
), Z
n
) , X
n
) . (2.3)
Let W
a
:= e
a
(X
n
), W
b
:= e
b
(W
a
, Y
n
),
b
X
n
:= g
c
(W
b
, Z
n
), and
P
e
:= P(
b
X
n
, X
n
).
Definition 3: The operationally admissible rate region for
relay-assisted distributed source coding, R
op
, is the set of
all operationally admissible rate-pairs for relay-assisted
distributed source coding.
It can be verified from the above definitions that R
op
is a
closed convex subset of [0, )
2
which contains [log
2
|X|, )
2
.
Problem statement: Characterize R
op
in terms of computable
single-letter [1] information quantities.
Remark: The problem of characterizing the rate region be-
comes considerably easier if X
n
has to be reproduced at both
location-b and location-c with an error probability which tends
to zero as n tends to infinity. The rate region in this case
is given by R
a
H(X|Y), R
b
H(X|Z): For any arbitrarily

small
positive real number and all n suciently large, X
n
can be encoded at location-a at the rate H(X|Y) bits per
sample, and decoded at location-b with a probability which
can be made smaller than . This follows from the coding
theorem due to Slepian and Wolf [10] with X
n
as the source
of information to be encoded at location-a and Y
n
as the side-
information available to a decoder at location-b. The Slepian-
Wolf coding theorem applied a second time with X
n
as the
source of information available at location-b and Z
n
as the
side-information available to a decoder at location-c shows
that, for any arbitrarily small positive real number and all
n suciently large, X
n
can be re-encoded at location-b at the
rate H(X|Z) bits per sample, and decoded at location-c with
a probability which can be made smaller . The converse to
the Slepian-Wolf coding theorem proves that the lower bound
R
a
H(X|Y) cannot be improved. The cutset bound (3.5) in
Section III-A proves that the lower bound R
b
H(X|Z) cannot
be improved. Thus the rate-region is fully characterized.
III. S C
A. The cutset bounds
Proposition 1: If a rate-pair (R
a
, R
b
) is operationally admis-
sible for relay-assisted distributed source coding then
R
a
H(X|Y, Z), (3.4)
R
b
H(X|Z) (3.5)
where (X, Y, Z) p
X,Y,Z
(x, y, z) and H(X|Z), H(X|Y, Z), are
conditional entropies [10] in bits.
Proof: (R
a
, R
b
) R
op
> 0, N
N : n N
, there
exists a relay-assisted distributed source code with parameters
(n, m
a
, m
b
) satisfying conditions (2.1), (2.2), and (2.3). By
Fano’s inequality [10],
H(X
n
|
b
X
n
) 1 + nP
e
log
2
|X| 1 + n log
2
|X|. (3.6)
For all > 0 and all n N
, we have the following information
inequalities
R
a
(a)
1
n
log
2
m
a
(b)
1
n
H(W
a
)
(c)
1
n
H(W
a
|Y
n
, Z
n
)
(d)
=
1
n
I(W
a
; X
n
|Y
n
, Z
n
)
=
1
n
(
H(X
n
|Y
n
, Z
n
) H(X
n
|Y
n
, Z
n
, W
a
)
)
(e)
= H(X|Y, Z)
1
n
H(X
n
|Y
n
, Z
n
, W
a
, W
b
,
b
X
n
)
( f )
H(X|Y, Z)
1
n
H(X
n
|
b
X
n
)
(g)
H(X|Y, Z)
1
n
log
2
|X| + 1
where
(a) is due to (2.1), (b) is because W
a
takes no more than
m
a
distinct values, (c) because conditioning does not increase
entropy, (d) because W
a
is a deterministic function of X
n
, (e)
is because (X
i
, Y
i
, Z
i
) i.i.d. p
X,Y,Z
(x, y, z) and because W
b
and
b
X
n
are respectively functions of (W
a
, Y
n
) and (W
b
, Z
n
), ( f ) is
because unconditioning does not decrease entropy, and (g) is
due to Fano’s inequality (3.6). Since the last inequality holds
for all > 0 and all n N
,
R
a
H(X|Y, Z).
In a similar manner, for all > 0 and all n N
, we have the
following information inequalities
R
b
(a)
1
n
log
2
m
b
(b)
1
n
H(W
b
)
(c)
1
n
H(W
b
|Z
n
)
1
n
I(W
b
; X
n
|Z
n
)
=
1
n
(
H(X
n
|Z
n
) H(X
n
|Z
n
, W
b
)
)
(d)
= H(X|Z)
1
n
H(X
n
|Z
n
, W
b
,
b
X
n
)
(e)
H(X|Z)
1
n
H(X
n
|
b
X
n
)
( f )
H(X|Z)
1
n
log
2
|X| + 1
where
(a) is due to (2.2), (b) is because W
b
takes no more than
m
b
distinct values, (c) because conditioning does not increase
entropy, (d) is because (X
i
, Z
i
) i.i.d. p
X,Z
(x, z) and because
b
X
n
is a deterministic function of (W
b
, Z
n
), (e) is because
unconditioning does not decrease entropy, and ( f ) is due to
Fano’s inequality (3.6). Since the last inequality holds for all
> 0 and all n N
,
R
b
H(X|Z).
B.
Special cases in which the cutset bounds are tight
Case-1: X–Y–Z is a Markov chain. In this case,
H(X|Y, Z) = H(X|Y). Thus for any arbitrarily small
positive real number and all n suciently large, X
n
can be encoded at location-a at the rate H(X|Y, Z) bits
per sample, and decoded at location-b with a probability
which can be made smaller than . This follows from
the Slepian-Wolf coding theorem with X
n
as the source
of information to be encoded at location-a and Y
n
as
the side-information available to a decoder at location-b.
The Slepian-Wolf coding theorem applied a second
time with X
n
as the source of information available at
location-b and Z
n
as the side-information available to
a decoder at location-c shows that, for any arbitrarily
small positive real number and all n suciently

lar
ge, X
n
can be re-encoded at location-b at the rate
H(X|Z) bits per sample, and decoded at location-c with
a probability which can be made smaller . We shall call
this relay-assisted distributed source coding strategy as
relay decode and re-encode.
Case-2: X–Z–Y is a Markov chain. In this case,
H(X|Y, Z) = H(X|Z). The Slepian-Wolf coding theorem
with X
n
as the source of information available at
location-a and Z
n
as the side-information available to
a decoder at location-c shows that, for any arbitrarily
small positive real number and all n suciently large,
X
n
can be encoded at location-a at the rate H(X|Y, Z) bits
per sample, and decoded at location-c with a probability
which can be made smaller than . The operation at the
intermediate location-b is to simply forward the message
from location-a to location-c without any processing. We
shall call this relay-assisted distributed source coding
strategy as relay forward.
Case-3: H(Y|X, Z) = 0, that is, for all tuples (x, z) X×Z
for which p
X,Z
(x, z) > 0, y is a deterministic function
of (x, z). In this case, H(X|Z) = H(Y|Z) + H(X|Y, Z)
which is seen be expanding H(X, Y|Z) in two dierent
ways by the chain rule for conditional entropy [10]. The
Slepian-Wolf coding theorem with Y
n
as the source of
information available at location-b and Z
n
as the side-
information available to a decoder at location-c shows
that, for any arbitrarily small positive real number and
all n suciently large, Y
n
can be encoded at location-b at
the rate H(Y|Z) bits per sample, and decoded at location-
c with a probability which can be made smaller than
. The Slepian-Wolf coding theorem applied a second
time with X
n
as the source of information available at
location-a and (Y
n
, Z
n
) as the side-information available
to a decoder at location-c shows that, for any arbitrarily
small positive real number and all n suciently large,
X
n
can be encoded at location-a at the rate H(X|Y, Z) bits
per sample, and decoded at location-c with a probability
which can be made smaller than . The operation at the
intermediate location-b is to (i) encode its observations to
be reproduced at location-c and (ii) forward the message
from location-a to location-c without any processing. We
shall call this relay-assisted distributed source coding
strategy as relay encode and forward.
C. A case in which the cutset bounds are not tight
Case-4: X = Y = Z = {0, 1}, p
Y,Z
(y, z) = 0.5(1
p)δ
yz
+ 0.5p(1 δ
yz
), p (0, 1), and X = Y Z where
Y Z is the Boolean AND function. According to the
cutset bounds, R
a
H(X|Y, Z) = 0 and R
b
H(X|Z) =
0.5H(Y Z|Z = 1) = 0.5H(Y|Z = 1) = 0.5h
2
(p) where
h
2
(α) := α log
2
(α) (1 α) log
2
(1 α), α [0, 1] is
the binary entropy function [10]. If R
a
= 0 then it can
be argued that the problem is equivalent to a depth-2
network source coding problem with sources Y
n
and Z
n
available at location-b and location-c respectively and the
goal being to reproduce the sample-wise Boolean AND
function of Y
n
and Z
n
at location-c with a probability
which tends to one as n tends to infinity. This latter
problem has been studied by Yamamoto [11] and by
Han and Kobayashi [12]. Lemma 1 in [12] applied to
the current problem proves that R
b
H(Y|Z) = h
2
(p) >
0.5h
2
(p). Thus the cutset bounds are not tight for general
p
X,Y,Z
.
IV. A G I R R
Theorem 1: (Single coding strategy covering all cases) Let
(X, Y, Z, U, V) be random variables taking values in X × Y ×
Z × U × V, where U and V are finite alphabets. For all
(x, y, z, u, v) X × Y × Z × U × V, let
p
X,Y,Z,U,V
(x, y, z, u, v) = p
X,Y,Z
(x, y, z) · p
U|X
(u|x) · p
V|U,Y
(v|u, y),
where p
U|X
(·|·) and p
V|U,Y
(·|·) are conditional pmfs, that is,
the auxiliary random variables U, V satisfy the following two
Markov chains: UX(Y, Z) and V(U, Y)(X, Z). Then all
rate-pairs (R
a
, R
b
) satisfying
R
a
I(X; U|Y) + H(X|V, Z) (4.7)
R
b
I(Y, U; V|Z) + H(X|V, Z) (4.8)
are operationally admissible for relay-assisted distributed
source coding and cover all the four previously discussed
special cases.
Proof-sketch: We first show that all the four previously dis-
cussed special cases are covered by Theorem 1. For Case-1,
set U = V = X and U = V = X, for Case-2, set |U| = |V| = 1,
for Cases 3 and 4, set |U| = 1, V = Y, and V = Y.
We now sketch the proof that all rate-pairs (R
a
, R
b
) satis-
fying (4.7) and (4.8) are operationally admissible for relay-
assisted distributed source coding. The proof sketch follows
well known random coding and random binning arguments in
[1], [10], [13].
Let (X, Y, Z, U, V) be random variables taking values in X ×
Y × Z × U × V with the joint pmf p
X,Y,Z,U,V
(x, y, z, u, v) =
p
X,Y,Z
(x, y, z)·p
U|X
(u|x)·p
V|U,Y
(v|u, y) as stated in Theorem 1. Let
X := X
n
, Y := Y
n
, Z := Z
n
, > 0, and C
x
:= (x(1), . . . , x(|X|
n
))
be an ordering of the set of all sourcewords X
n
.
Random codebook generation for location-a: Generate m
0
u
i.i.d. codewords U( j) := U
n
( j), j = 1, . . . , m
0
u
, of block-
length n, according to the product pmf p
U
(u) :=
Q
n
i=1
p
U
(u
i
),
independent of (X, Y, Z). The tuple of codewords C
u
:=
(U(1), . . . , U(m
0
u
)) is available at locations a and b. Generate
m
0
u
i.i.d. bin-indices A
j
, j = 1, . . . , m
0
u
, according to the
uniform pmf over the set {1, . . . , m
u
}, independent of (X, Y, Z)
and {U( j), j = 1, . . . , m
0
u
}. The tuple of bin-indices A :=
(A
1
, . . . , A
m
0
u
) is available at locations a and b.
The interpretation is that for j = 1, . . . , m
0
u
, codeword U( j)
is assigned to a bin of codewords which has a bin-index A
j
.
Also generate a second set of |X|
n
i.i.d. bin-indices
˜
A
j
, j = 1, . . . , |X|
n
, according to the uniform pmf over the

set {1,
. . . , m
x
}, independent of (X, Y, Z) and {U( j), A
j
, j =
1, . . . , m
0
u
}. The tuple of bin-indices
e
A := (
˜
A
1
, . . . ,
˜
A
|X|
n
) is
available at locations a and c.
The interpretation is that for j = 1, . . . , |X|
n
, the sourceword
x( j) C
x
is assigned to a bin of sourcewords which has a
bin-index
˜
A
j
.
Encoding at location-a: Let
J
u
:= min{ j = 1, . . . , m
0
u
: (X, U( j)) T
(n)
(p
X,U
)}
with the convention that min
j
{} = 1, where {} denotes the
empty set, and where T
(n)
(p
X,U
) denotes the strongly p
X,U
-
typical subset of X
n
× U
n
[1]. Also let J
x
be the index of
X in the tuple of sourcewords C
x
, that is, X = x(J
x
). Set
W
a
:= (A
J
u
,
˜
A
J
x
).
The interpretation is that the encoder at location-a first
“quantizes” the sourceword X to the codeword U(J
u
). This
“quantization” is accomplished by searching through the tuple
of codewords C
u
and finding the first codeword U(J
u
) which
is strongly p
X,U
typical with X. The encoder also randomly
partitions the tuple of codewords C
u
into nonoverlapping bins.
The “quantized” codeword U(J
u
) is assigned to a bin with
bin-index A
J
u
in this random partitioning. In addition, the
encoder also randomly partitions the tuple of all sourcewords
C
x
into nonoverlapping bins. The observed sourceword X,
which occurs at position J
x
in the tuple C
x
, is assigned to a bin
with bin-index
˜
A
J
x
in this random partitioning. The bin-index
A
J
u
of U(J
u
) together with the bin-index
˜
A
J
x
of X = x(J
x
)
comprise the message W
a
sent from location-a to location-b.
Random codebook generation for location-b: Generate m
0
v
i.i.d. relay-codewords V( j) := V
n
( j), j = 1, . . . , m
0
v
, of block-
length n, according to the product pmf p
V
(v) :=
Q
n
i=1
p
V
(v
i
),
independent of (X, Y, Z), {U( j), A
j
, j = 1, . . . , m
0
u
}, and
{
˜
A
j
, j = 1, . . . , |X|
n
}. The tuple of relay-codewords C
v
:=
(V(1), . . . , V(m
0
v
)) is available at locations b and c. Generate
m
0
v
i.i.d. bin-indices B
j
, j = 1, . . . , m
0
v
, according to the
uniform pmf over the set {1, . . . , m
v
}, independent of (X, Y, Z),
{U( j), A
j
, j = 1, . . . , m
0
u
}, {
˜
A
j
, j = 1, . . . , |X|
n
}, and {V( j), j =
1, . . . , m
0
v
}. The tuple of bin-indices B := (B
1
, . . . , B
m
0
v
) is
available at locations b and c.
The interpretation is that for j = 1, . . . , m
0
v
, codeword V( j)
is assigned to a bin of codewords which has a bin-index B
j
.
Encoding at location-b: Noting that W
a
= (A
J
u
,
˜
A
J
x
) and Y
are available at location-b, let
ˆ
J
u
:= min{ j = 1, . . . , m
0
u
: A
j
= A
J
u
, (U( j), Y) T
(n)
(p
U,Y
)}
and
J
v
:= min{ j = 1, . . . , m
0
v
: (U(
ˆ
J
u
), Y, V( j)) T
(n)
(p
U,Y,V
)}.
Set W
b
:= (B
J
v
,
˜
A
J
x
).
The interpretation is that the encoder at location-b first esti-
mates the codeword U(J
u
) as U(
ˆ
J
u
). This is accomplished by
searching through the bin of codewords in C
u
whose bin-index
is A
J
u
and finding the first codeword U(
ˆ
J
u
) which is strongly
p
U,Y
typical with Y. The encoder next “jointly-quantizes”
the pair (U(
ˆ
J
u
), Y) to the relay-codeword V(J
v
). This “joint-
quantization” is accomplished by searching through the tuple
of relay-codewords C
v
and finding the first relay-codeword
V(J
v
) which is strongly p
U,Y,V
typical with U(
ˆ
J
u
) and Y. The
encoder also randomly partitions the tuple of relay-codewords
C
v
into nonoverlapping bins. The “quantized” relay-codeword
V(J
v
) is assigned to a bin with bin-index B
J
v
in this random
partitioning. The bin-index B
J
v
of V(J
v
) together with the bin-
index
˜
A
J
x
of X = x(J
x
) comprise the message W
b
sent from
location-b to location-c.
Decoding at location-c: Noting that W
b
= (B
J
v
,
˜
A
J
x
) and Z
are available at location-c, let
ˆ
J
v
:= min{ j = 1, . . . , m
0
v
: B
j
= B
J
v
, (V( j), Z) T
(n)
(p
V,Z
)}
and
ˆ
J
x
:= min
n
j = 1, . . . , |X|
n
:
˜
A
j
=
˜
A
J
x
, (X( j), V(
ˆ
J
v
), Z)
T
(n)
(p
X,V,Z
)
o
.
Set
b
X := x(
ˆ
J
x
).
The interpretation is that the decoder at location-c first esti-
mates the relay-codeword V(J
v
) as V(
ˆ
J
v
). This is accomplished
by searching through the bin of relay-codewords in C
v
whose
bin-index is B
J
v
and finding the first codeword V(
ˆ
J
v
) which
is strongly p
V,Z
typical with Z. The decoder next estimates
the sourceword X = x(J
x
) as
b
X = x(
ˆ
J
x
). This is accomplished
by searching through the set of all sourcewords in C
x
whose
bin-index is
˜
A
J
x
and finding the first sourceword X(
ˆ
J
x
) which
is strongly p
X,V,Z
typical with V(
ˆ
J
v
) and Z.
Error events:
E
1
: {X < T
(n)
(p
X
)}.
As n , by the strong law of larger numbers, P(E
1
) 0.
E
2
: {(X, U(J
u
)) < T
(n)
(p
X,U
)}.
There exists an
1
(n, ) > 0 such that (i)
1
(n, ) 0 as n
and
0 and (ii)
P
(
E
2
| E
c
1
)
0 if
(1/n) log
2
m
0
u
= I(X; U) +
1
(n, ). (4.9)
E
3
: {(X, U(J
u
), Y) < T
(n)
(p
X,U,Y
)}.
P(E
3
| E
c
2
) 0 as n by the Markov lemma [10] because
UXY is a Markov chain.
E
4
: {∃ j , J
u
: A
j
= A
J
u
, (U( j), Y) T
(n)
(p
U,Y
)},
There exists an
2
(n, ) > 0 such that (i)
2
(n, ) 0 as n
and 0 and (ii) P(E
4
) 0 if
(1/n)(log
2
m
0
u
log
2
m
u
) = I(U; Y)
2
(n, ). (4.10)
Conditions (4.9) and (4.10) and the Markov chain UX–(Y, Z)
imply the following condition on (1/n) log
2
m
u
:
1
n
log
2
m
u
1
(n,
)
2
(n, ) = I(X; U) I(U; Y)
= I(X; U|Y). (4.11)
Hence under condition (4.11), P(
ˆ
J
u
, J
u
) 0 as n .
E
5
: {(U(J
u
), Y, V(J
v
)) < T
(n)
(p
X,U,Y,V
)},
There exists an
3
(n, ) > 0 such that (i)
3
(n, ) 0 as n
and 0 and (ii) P(E
5
| (E
3
E
4
)
c
) 0 if
(1/n) log
2
m
0
v
= I(U, Y; V) +
3
(n, ). (4.12)

Citations
More filters
Journal ArticleDOI
TL;DR: In this article, the authors considered the two-way cascade and triangular rate-distortion problem, where the side information available at user 2 is a degraded version of the source side information at the source node and user 1.
Abstract: In this paper, we consider the cascade and triangular rate-distortion problems where the same side information is available at the source node and user 1, and the side information available at user 2 is a degraded version of the side information at the source node and user 1. We characterize the rate-distortion region for these problems. For the cascade setup, we show that, at user 1, decoding and rebinning the codeword sent by the source node for user 2 is optimum. We then extend our results to the two-way cascade and triangular setting, where the source node is interested in lossy reconstruction of the side information at user 2 via a rate limited link from user 2 to the source node. We characterize the rate-distortion regions for these settings. Complete explicit characterizations for all settings are given in the quadratic Gaussian case. We conclude with two further extensions: a triangular source coding problem with a helper, and an extension of our two-way cascade setting in the quadratic Gaussian case.

23 citations

Proceedings ArticleDOI
13 Jun 2010
TL;DR: A correspondence between the problem of computing functions of data streams while employing limited memory in a standard information-theoretic framework and a functional source coding problem in cascade/line networks is established.
Abstract: We consider the problem of computing functions of data streams while employing limited memory in a standard information-theoretic framework. A streaming system with memory constraint has to observe a collection of sources X 1 ,X 2 ,…,X m sequentially, store synopses of the sources in memory, and compute a function of the sources based on the synopses. We establish a correspondence between this problem and a functional source coding problem in cascade/line networks. For the general functional source coding problem in cascade networks, we derive inner and outer bounds, and for distributions satisfying certain properties, we characterize the achievable rate-region exactly for the computation of any function. As a result of the correspondence we established, this result also characterizes the minimum amount of memory required to compute the function in a streaming system. We briefly discuss the implications of this result for the problem of distinct value computation.

12 citations

Proceedings ArticleDOI
01 Sep 2012
TL;DR: The main result is an inner bound to the rate region of this problem which is tight when X - Y - Z forms a Markov chain.
Abstract: A transmitter has access to X, a relay has access to Y, and a receiver has access to Z and wants to compute a given function ƒ(X, Y, Z). How many bits must be transmitted from the transmitter to the relay and from the relay to the receiver so that the latter can reliably recover ƒ(X, Y, Z)? The main result is an inner bound to the rate region of this problem which is tight when X - Y - Z forms a Markov chain.

8 citations

Journal ArticleDOI
TL;DR: The average number of bits that need to be conveyed to that end by each sender to the relay and by the relay to the receiver is studied, in the limit of multiple instances.
Abstract: Two remote senders observe $X$ and $Y$ , respectively, and can noiselessly send information via a common relay node to a receiver that observes $Z$ . The receiver wants to compute a function $f(X,Y,Z)$ of these possibly related observations, without error. We study the average number of bits that need to be conveyed to that end by each sender to the relay and by the relay to the receiver, in the limit of multiple instances. We relate these quantities to the entropy region of a probabilistic graph with respect to a Cartesian representation of its vertex set, which we define as a natural extension of graph entropy. General properties and bounds for the graph entropy region are derived, and mapped back to special cases of the distributed computing setup.

7 citations

Proceedings ArticleDOI
01 Sep 2010
TL;DR: This paper extends the class of distributions for which the outer bounds are achievable for all functions and a certain classes of distributions on the sources and characterizes the memory requirement for the associated streaming computation problem.
Abstract: We consider the problem of determining the memory required to compute functions of data streams. A streaming system with memory constraint has to observe a collection of sources X 1 ;X 2 ;…;X m sequentially, store synopses of the sources in memory, and compute a function of the sources based on the synopses. We are interested in the memory requirement, the number of bits of memory required to compute the function. In an earlier work, we established a correspondence between this problem and a functional source coding problem in cascade/line networks, and for the latter we derived inner and outer bounds on the rate region. In particular we showed that the outer bounds are achievable for all functions and a certain classes of distributions on the sources. In this paper we extend the class of distributions for which the outer bounds are achieved. By virtue of the correspondence between the two problems, this also characterizes the memory requirement for the associated streaming computation problem.

7 citations


Cites background from "A relay-assisted distributed source..."

  • ...F (X1, X2, X3) = X1 was considered in [7]....

    [...]

References
More filters
Book
01 Jan 1991
TL;DR: The author examines the role of entropy, inequality, and randomness in the design of codes and the construction of codes in the rapidly changing environment.
Abstract: Preface to the Second Edition. Preface to the First Edition. Acknowledgments for the Second Edition. Acknowledgments for the First Edition. 1. Introduction and Preview. 1.1 Preview of the Book. 2. Entropy, Relative Entropy, and Mutual Information. 2.1 Entropy. 2.2 Joint Entropy and Conditional Entropy. 2.3 Relative Entropy and Mutual Information. 2.4 Relationship Between Entropy and Mutual Information. 2.5 Chain Rules for Entropy, Relative Entropy, and Mutual Information. 2.6 Jensen's Inequality and Its Consequences. 2.7 Log Sum Inequality and Its Applications. 2.8 Data-Processing Inequality. 2.9 Sufficient Statistics. 2.10 Fano's Inequality. Summary. Problems. Historical Notes. 3. Asymptotic Equipartition Property. 3.1 Asymptotic Equipartition Property Theorem. 3.2 Consequences of the AEP: Data Compression. 3.3 High-Probability Sets and the Typical Set. Summary. Problems. Historical Notes. 4. Entropy Rates of a Stochastic Process. 4.1 Markov Chains. 4.2 Entropy Rate. 4.3 Example: Entropy Rate of a Random Walk on a Weighted Graph. 4.4 Second Law of Thermodynamics. 4.5 Functions of Markov Chains. Summary. Problems. Historical Notes. 5. Data Compression. 5.1 Examples of Codes. 5.2 Kraft Inequality. 5.3 Optimal Codes. 5.4 Bounds on the Optimal Code Length. 5.5 Kraft Inequality for Uniquely Decodable Codes. 5.6 Huffman Codes. 5.7 Some Comments on Huffman Codes. 5.8 Optimality of Huffman Codes. 5.9 Shannon-Fano-Elias Coding. 5.10 Competitive Optimality of the Shannon Code. 5.11 Generation of Discrete Distributions from Fair Coins. Summary. Problems. Historical Notes. 6. Gambling and Data Compression. 6.1 The Horse Race. 6.2 Gambling and Side Information. 6.3 Dependent Horse Races and Entropy Rate. 6.4 The Entropy of English. 6.5 Data Compression and Gambling. 6.6 Gambling Estimate of the Entropy of English. Summary. Problems. Historical Notes. 7. Channel Capacity. 7.1 Examples of Channel Capacity. 7.2 Symmetric Channels. 7.3 Properties of Channel Capacity. 7.4 Preview of the Channel Coding Theorem. 7.5 Definitions. 7.6 Jointly Typical Sequences. 7.7 Channel Coding Theorem. 7.8 Zero-Error Codes. 7.9 Fano's Inequality and the Converse to the Coding Theorem. 7.10 Equality in the Converse to the Channel Coding Theorem. 7.11 Hamming Codes. 7.12 Feedback Capacity. 7.13 Source-Channel Separation Theorem. Summary. Problems. Historical Notes. 8. Differential Entropy. 8.1 Definitions. 8.2 AEP for Continuous Random Variables. 8.3 Relation of Differential Entropy to Discrete Entropy. 8.4 Joint and Conditional Differential Entropy. 8.5 Relative Entropy and Mutual Information. 8.6 Properties of Differential Entropy, Relative Entropy, and Mutual Information. Summary. Problems. Historical Notes. 9. Gaussian Channel. 9.1 Gaussian Channel: Definitions. 9.2 Converse to the Coding Theorem for Gaussian Channels. 9.3 Bandlimited Channels. 9.4 Parallel Gaussian Channels. 9.5 Channels with Colored Gaussian Noise. 9.6 Gaussian Channels with Feedback. Summary. Problems. Historical Notes. 10. Rate Distortion Theory. 10.1 Quantization. 10.2 Definitions. 10.3 Calculation of the Rate Distortion Function. 10.4 Converse to the Rate Distortion Theorem. 10.5 Achievability of the Rate Distortion Function. 10.6 Strongly Typical Sequences and Rate Distortion. 10.7 Characterization of the Rate Distortion Function. 10.8 Computation of Channel Capacity and the Rate Distortion Function. Summary. Problems. Historical Notes. 11. Information Theory and Statistics. 11.1 Method of Types. 11.2 Law of Large Numbers. 11.3 Universal Source Coding. 11.4 Large Deviation Theory. 11.5 Examples of Sanov's Theorem. 11.6 Conditional Limit Theorem. 11.7 Hypothesis Testing. 11.8 Chernoff-Stein Lemma. 11.9 Chernoff Information. 11.10 Fisher Information and the Cram-er-Rao Inequality. Summary. Problems. Historical Notes. 12. Maximum Entropy. 12.1 Maximum Entropy Distributions. 12.2 Examples. 12.3 Anomalous Maximum Entropy Problem. 12.4 Spectrum Estimation. 12.5 Entropy Rates of a Gaussian Process. 12.6 Burg's Maximum Entropy Theorem. Summary. Problems. Historical Notes. 13. Universal Source Coding. 13.1 Universal Codes and Channel Capacity. 13.2 Universal Coding for Binary Sequences. 13.3 Arithmetic Coding. 13.4 Lempel-Ziv Coding. 13.5 Optimality of Lempel-Ziv Algorithms. Compression. Summary. Problems. Historical Notes. 14. Kolmogorov Complexity. 14.1 Models of Computation. 14.2 Kolmogorov Complexity: Definitions and Examples. 14.3 Kolmogorov Complexity and Entropy. 14.4 Kolmogorov Complexity of Integers. 14.5 Algorithmically Random and Incompressible Sequences. 14.6 Universal Probability. 14.7 Kolmogorov complexity. 14.9 Universal Gambling. 14.10 Occam's Razor. 14.11 Kolmogorov Complexity and Universal Probability. 14.12 Kolmogorov Sufficient Statistic. 14.13 Minimum Description Length Principle. Summary. Problems. Historical Notes. 15. Network Information Theory. 15.1 Gaussian Multiple-User Channels. 15.2 Jointly Typical Sequences. 15.3 Multiple-Access Channel. 15.4 Encoding of Correlated Sources. 15.5 Duality Between Slepian-Wolf Encoding and Multiple-Access Channels. 15.6 Broadcast Channel. 15.7 Relay Channel. 15.8 Source Coding with Side Information. 15.9 Rate Distortion with Side Information. 15.10 General Multiterminal Networks. Summary. Problems. Historical Notes. 16. Information Theory and Portfolio Theory. 16.1 The Stock Market: Some Definitions. 16.2 Kuhn-Tucker Characterization of the Log-Optimal Portfolio. 16.3 Asymptotic Optimality of the Log-Optimal Portfolio. 16.4 Side Information and the Growth Rate. 16.5 Investment in Stationary Markets. 16.6 Competitive Optimality of the Log-Optimal Portfolio. 16.7 Universal Portfolios. 16.8 Shannon-McMillan-Breiman Theorem (General AEP). Summary. Problems. Historical Notes. 17. Inequalities in Information Theory. 17.1 Basic Inequalities of Information Theory. 17.2 Differential Entropy. 17.3 Bounds on Entropy and Relative Entropy. 17.4 Inequalities for Types. 17.5 Combinatorial Bounds on Entropy. 17.6 Entropy Rates of Subsets. 17.7 Entropy and Fisher Information. 17.8 Entropy Power Inequality and Brunn-Minkowski Inequality. 17.9 Inequalities for Determinants. 17.10 Inequalities for Ratios of Determinants. Summary. Problems. Historical Notes. Bibliography. List of Symbols. Index.

45,034 citations


"A relay-assisted distributed source..." refers background or methods in this paper

  • ...P(E3 | E2) ↓ 0 as n ↑ ∞ by the Markov lemma [10] because U–X–Y is a Markov chain....

    [...]

  • ...This follows from the coding theorem due to Slepian and Wolf [10] with Xn as the source of information to be encoded at location-a and Yn as the sideinformation available to a decoder at location-b....

    [...]

  • ...5h2(p) where h2(α) := −α log2(α) − (1 − α) log2(1 − α), α ∈ [0, 1] is the binary entropy function [10]....

    [...]

  • ...The Slepian-Wolf coding theorem with Yn as the source of information available at location-b and Zn as the sideinformation available to a decoder at location-c shows that, for any arbitrarily small positive real number and all n sufficiently large, Yn can be encoded at location-b at the rate H(Y |Z) bits per sample, and decoded at locationc with a probability which can be made smaller than ....

    [...]

  • ...The Slepian-Wolf coding theorem applied a second time with Xn as the source of information available at location-a and (Yn,Zn) as the side-information available to a decoder at location-c shows that, for any arbitrarily small positive real number and all n sufficiently large, Xn can be encoded at location-a at the rate H(X|Y,Z) bits per sample, and decoded at location-c with a probability which can be made smaller than ....

    [...]

Book
26 Sep 2014
TL;DR: This new edition presents unique discussions of information theoretic secrecy and of zero-error information theory, including the deep connections of the latter with extremal combinatorics.
Abstract: Csiszr and Krner's book is widely regarded as a classic in the field of information theory, providing deep insights and expert treatment of the key theoretical issues. It includes in-depth coverage of the mathematics of reliable information transmission, both in two-terminal and multi-terminal network scenarios. Updated and considerably expanded, this new edition presents unique discussions of information theoretic secrecy and of zero-error information theory, including the deep connections of the latter with extremal combinatorics. The presentations of all core subjects are self contained, even the advanced topics, which helps readers to understand the important connections between seemingly different problems. Finally, 320 end-of-chapter problems, together with helpful solving hints, allow readers to develop a full command of the mathematical techniques. It is an ideal resource for graduate students and researchers in electrical and electronic engineering, computer science and applied mathematics.

3,404 citations

Journal ArticleDOI
TL;DR: A new domain of collaborative information communication and processing through the framework on distributed source coding using syndromes, which enables highly effective and efficient compression across a sensor network without the need to establish inter-node communication, using well-studied and fast error-correcting coding algorithms.
Abstract: Distributed nature of the sensor network architecture introduces unique challenges and opportunities for collaborative networked signal processing techniques that can potentially lead to significant performance gains. Many evolving low-power sensor network scenarios need to have high spatial density to enable reliable operation in the face of component node failures as well as to facilitate high spatial localization of events of interest. This induces a high level of network data redundancy, where spatially proximal sensor readings are highly correlated. We propose a new way of removing this redundancy in a completely distributed manner, i.e., without the sensors needing to talk, to one another. Our constructive framework for this problem is dubbed DISCUS (distributed source coding using syndromes) and is inspired by fundamental concepts from information theory. We review the main ideas, provide illustrations, and give the intuition behind the theory that enables this framework.We present a new domain of collaborative information communication and processing through the framework on distributed source coding. This framework enables highly effective and efficient compression across a sensor network without the need to establish inter-node communication, using well-studied and fast error-correcting coding algorithms.

563 citations


"A relay-assisted distributed source..." refers background in this paper

  • ...Beginning the late 90’s, the construction of practical distributed source codes for emerging wireless sensor network applications [2] received...

    [...]

01 Jan 1978

558 citations

Journal ArticleDOI
TL;DR: There does not exist a finite value of R for which even infinitely many agents can make D arbitrarily small, and in this isolated-agents case the asymptotic behavior of the minimal error frequency in the limit as L and then R tend to infinity is determined.
Abstract: We consider a new problem in multiterminal source coding motivated by the following decentralized communication/estimation task. A firm's Chief Executive Officer (CEO) is interested in the data sequence {X(t)}/sub t=1//sup /spl infin// which cannot be observed directly, perhaps because it represents tactical decisions by a competing firm. The CEO deploys a team of L agents who observe independently corrupted versions of {X(t)}/sub t=1//sup /spl infin//. Because {X(t)} is only one among many pressing matters to which the CEO must attend, the combined data rate at which the agents may communicate information about their observations to the CEO is limited to, say, R bits per second. If the agents were permitted to confer and pool their data, then in the limit as L/spl rarr//spl infin/ they usually would be able to smooth out their independent observation noises entirely. Then they could use their R bits per second to provide the CEO with a representation of {X(t)} with fidelity D(R), where D(/spl middot/) is the distortion-rate function of {X(t)}. In particular, with such data pooling D can be made arbitrarily small if R exceeds the entropy rate H of {X(t)}. Suppose, however, that the agents are not permitted to convene, Agent i having to send data based solely on his own noisy observations {Y/sub i/(t)}. We show that then there does not exist a finite value of R for which even infinitely many agents can make D arbitrarily small. Furthermore, in this isolated-agents case we determine the asymptotic behavior of the minimal error frequency in the limit as L and then R tend to infinity.

468 citations