scispace - formally typeset
Search or ask a question
Proceedings ArticleDOI

Cascade multiterminal source coding

28 Jun 2009-pp 1199-1203
TL;DR: The general contribution toward understanding the limits of the cascade multiterminal source coding network is in the form of inner and outer bounds on the achievable rate region for satisfying a distortion constraint for an arbitrary distortion function d(x, y, z).
Abstract: We investigate distributed source coding of two correlated sources X and Y where messages are passed to a decoder in a cascade fashion. The encoder of X sends a message at rate R1 to the encoder of Y. The encoder of Y then sends a message to the decoder at rate R 2 based both on Y and on the message it received about X. The decoder's task is to estimate a function of X and Y. For example, we consider the minimum mean squared-error distortion when encoding the sum of jointly Gaussian random variables under these constraints. We also characterize the rates needed to reconstruct a function of X and Y losslessly. Our general contribution toward understanding the limits of the cascade multiterminal source coding network is in the form of inner and outer bounds on the achievable rate region for satisfying a distortion constraint for an arbitrary distortion function d(x, y, z). The inner bound makes use of a balance between two encoding tactics—relaying the information about X and recompressing the information about X jointly with Y. In the Gaussian case, a threshold is discovered for identifying which of the two extreme strategies optimizes the inner bound. Relaying outperforms recompressing the sum at the relay for some rate pairs if the variance of X is greater than the variance of Y.

Summary (2 min read)

I. INTRODUCTION

  • Distributed data collection, such as aggregating measurements in a sensor network, has been investigated from many angles [1] .
  • Computing functions of observations in a network has been considered in various other settings, such as the two-node back-and-forth setting of [2] and the multiple access channel setting in [3] .
  • In the cascade multiterminal network, the answer breaks down quite intuitively.

B. Rate-Distortion Region

  • The goal is for X", Y", and Z" to satisfy an average letterby-letter distortion constraint D with high probability.
  • A finite distortion function d(x, y, z) specifies the penalty incurred for any triple (x, y, z).
  • The rate-distortion region R for a particular source joint distribution Po(x, y) and distortion function d is the closure of achievable rate-distortion triples, given as, EQUATION.

III. GENERAL INNER BOUND

  • The cascade multiterminal source coding network presents an interesting dilemma.
  • The second encoder could jointly compress the source sequence yn along with the auxiliary sequence, treating it as if it was also a random source sequence.
  • A lot is known about the auxiliary sequence, such as the codebook it came from, allowing it to be summarized more easily than this approach would allow.

II. PROBLEM SPECIFICS

  • Channel setting that is solved in its full generality.
  • Berger and Tung [11] first considered the multiterminal source coding problem, where correlated sources are encoded separately with loss.
  • The difference is that communication between the source encoders in this network replaces one of the direct channels to the decoder.
  • In their setting, the decoder has side information, and the relay has access to a physically degraded version of it.
  • Then the authors consider specific cases, such as encoding the sum of jointly Gaussian random variables, computing functions, and even coordinating actions.

IV. GENERAL OUTER BOUND

  • Finally, Encoder 1 sends the bin numbers bu(i) and bv(j, i) to Encoder 2. Encoder 2 considers all codewords in Cu with bin number bu(i) and finds that only u" (i) is e-jointly typical with yn with respect to p(y, u).
  • Finally, E can be chosen small enough to satisfy the rate and distortion inequalities.
  • The forwarded message keeps its sparse codebook in tact, while the decoded and recompressed message enjoys the efficiency that comes with being bundled with Y. Theorem 3.1 (Inner bound):.

2 y

  • Only has one significant free parameter due to Markovity, All remammg quantities that define the region Rin are conditioned on U, including the final estimate at the decoder since U is available to the decoder.
  • Therefore, after fixing p(x, y, u) the authors can remove U entirely from the optimization problem by exploiting the idiosyncracies of the jointly Gaussian distribution.
  • This greatly reduces the dimensionality of the problem.

2) Outer bound:

  • The outer bound Rout is optimized with Gaussian auxiliary random variables.
  • The result is the following lower bound on distortion.
  • This puts us in the recompress regime of the inner bound.
  • From this the authors find a piece-wise upper bound on the sum-rate-distortion function.

I ' ffici E(XY)

  • The two encoding strategies employed are to either forward the message from Encoder 1",to the Decoder, or to use the message to construct an",estimate X" at Encoder 2 and then compress the vector sum X" + yn and send it to the Decoder, but not both.
  • The determining factor for deciding which method to use is a comparison of the rate R 1 with the quantity ~log2 ~. Case 1: 2The optimal rate region for computing functions of data in the standard multiterminal source coding network is currently an open problem [17] .
  • This optimization is carefully investigated in [6] and equated to a graph entropy problem.

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

ISIT 2009, Seoul, Korea, June 28 - July 3, 2009
Cascade Multiterminal Source Coding
Paul Cuff, Han-I Su, and Abbas EI Gamal
Department
of
Electrical Engineering
Stanford University
E-mail: {cuff, hanisu, abbas}@stanford.edu
Abstract-We
investigate distributed source coding of two
correlated sources X
and
Y where messages
are
passed to a
decoder in a cascade fashion. The encoder of X sends a message
at
rate
R
1
to the encoder of Y. The encoder of Y then sends
a message to the decoder at
rate
R
2
based both on Y and
on the message it received
about
X.
The decoder's
task
is to
estimate a function of X
and
Y.
For
example, we consider the
minimum mean
squared-error
distortion when encoding the sum
of jointly Gaussian
random
variables
under
these constraints. We
also characterize the rates needed to reconstruct a function of
X and Y losslessly.
Our
general contribution toward
understanding
the limits
of the cascade multiterminal source coding network is in the
form of
inner
and
outer
bounds on the achievable
rate
region
for satisfying a distortion
constraint
for an
arbitrary
distortion
function
d(x, y, z). The
inner
bound makes use of a balance
between two encoding
tactics-relaying
the information
about
X
and recompressing
the
information
about
X jointly with Y. In
the Gaussian case, a threshold is discovered for identifying which
of
the
two extreme strategies optimizes the
inner
bound. Relaying
outperforms recompressing the sum at the relay for some
rate
pairs if the variance of X is
greater
than
the variance of Y.
I.
INTRODUCTION
Distributed data collection, such as aggregating measure-
ments in a sensor network, has been investigated from many
angles [1]. Various algorithms exist for passing messages to
neighbors in order to collect information or compute functions
of
data. Here we join in the investigation
of
the minimum
descriptions needed to quantize and collect data in a network,
and we do so by studying a particular small network. These
results provide insight for optimal communication strategies
in larger networks.
In the network considered here, two sources
of
information
are to be described by separate encoders and passed to a
single decoder in a cascade fashion. That is, after receiving a
message from the first encoder, the second encoder creates a
final message that summarizes the information available about
both sources and sends it to the decoder. We refer to this setup
as the
cascade multiterminal source coding network, shown
in Figure 1. Discrete i.i.d. sources
Xi
E X and
Yi
E
Yare
jointly distributed according to the probability mass function
Po(x, y). Encoder 1 summarizes a block
of
n symbols
X"
with a message I E
{l,
..., 2
n R 1
}
and sends it to Encoder
2. After receiving the message, Encoder 2 sends an index
J E
{l,
..., 2
n R 2
}
to describe what it knows about both sources
to the decoder, based on the message
I and on the observations
Y",
The decoder then uses the index J to construct a sequence
Z",
where each Zi is an estimate
of
a desired function
of
Xi
and
Yi.
X
n
y
n
~
~
Encoder 1
1 E [2
nR1
].
Encoder 2
J E [2
nR2
].
Decoder
i(X
n
)
j(1,
yn)
zn(J)
~
zn
Fig. 1. Cascade Multiterminal Source Coding. The i.i.d. source sequences
Xl,
... , X
nand
Yl, ... ,Y
n
are jointly distributed according to Po(x, y).
Encoder 1 sends a message 1 about the sequence
Xl,
... , X
n
at rate
Rl
to Encoder 2. The second encoder then sends a message J about both
source sequences at rate
R2 to the decoder. We investigate the rates required
to produce a sequence
Z
1,
...
, Zn with various goals in mind, such as
reconstructing estimates
of
X"
or
yn
or a function
of
the two.
For example, consider the lossless case. Suppose we wish to
compute a function
of
X and Y in the cascade multiterminal
source coding network. What rates are needed to reliably
calculate
Zi
==
!(X
i,
Yi)
at the decoder? Computing functions
of
observations in a network has been considered in various
other settings, such as the two-node back-and-forth setting
of
[2] and the multiple access channel setting in [3]. In the
cascade multiterminal network, the answer breaks down quite
intuitively. For the message from Encoder 1 to Encoder 2, use
Wyner-Ziv encoding [4] to communicate the function values.
Then apply lossless compression to the function values at
Encoder 2. Computing functions
of
data in a Wyner-Ziv setting
was introduced by Yamamoto [5], and the optimal rate for
lossless computation was shown by Orlitsky and Roche [6] to
be the conditional graph entropy on an appropriate graph.
A particular function for which the optimal rates are easy to
identify is the encoding
of
binary sums
of
binary symmetric
X and Y that are equal with probability p, as proposed by
Komer and Marton [7]. For this computation, the required
rates are
R
1
2::
h(p) and R
2
2::
h(p), where h is the binary
entropy function. Curiously, the same rates are required in the
standard multiterminal source coding setting.
Encoding
of
information sources at separate encoders has
attracted a lot
of
attention in the information theory community
over the years. The results
of
Slepian-Wolf encoding and com-
munication through the Multiple Access Channel (MAC) are
surprising and encouraging. Slepian and Wolf [8] showed that
separate encoders can compress correlated sources losslessly
at the same rate as a single encoder. Ahlswede [9] and Liao
[10] fully characterized the capacity region for the general
memoryless MAC, making it the only multi-user memoryless
978-1-4244-4313-0/09/$25.00 ©2009 IEEE
1199
Authorized licensed use limited to: Stanford University. Downloaded on March 02,2010 at 16:58:14 EST from IEEE Xplore. Restrictions apply.

ISIT 2009, Seoul, Korea, June 28 - July 3, 2009
where
(1)
{I,
2, ...},
X
n
---+
{I, , 2
n R 1
} ,
yn
x {I, , 2
n R 1
}
---+
{I, ..., 2
n R 2
} ,
{I, ..., 2
n R 2
}
---+
Z";
3j
3z
n
such that
Due to the flexibility in defining the distortion function
d, the decoded sequence
Z"
can play a number
of
different
roles.
If
the goal is to estimate both sources X and Y with
a distortion constraint, then
Z
==
(X, Y) encompasses both
estimates, and
d can be defined accordingly. Alternatively, the
decoder may only need to estimate
X,
in which case Y acts
as side information at a relay, and
Z
==
X. In general, the
decoder could produce estimates
of
any function
of
X and Y.
B. Rate-Distortion Region
The triple
(R
1,
R
2
,
D)
is an achievable rate-distortion triple
for the distortion function
d and source distribution Po(x, y)
if the following holds:
For
\IE
> 0,
3n E
3i
A. Objective
The goal is for
X",
Y",
and
Z"
to satisfy an average letter-
by-letter distortion constraint
D with high probability. A finite
distortion function
d(x, y, z) specifies the penalty incurred
for any triple
(x, y, z). Therefore, the objective is to reliably
produce a sequence
Z"
that satisfies
1 n
- L
acx;
Yi,
Zi) < D.
n
i=l
The rate-distortion region R for a particular source joint
distribution
Po(x, y) and distortion function d is the closure
of
achievable rate-distortion triples, given as,
R
!!:.-
el{
achievable (R
1
,
R
2
, D) triples}. (2)
III.
GENERAL
INNER
BOUND
The cascade multiterminal source coding network presents
an interesting dilemma. Encoder 2 has to summarize both
the source sequence
yn
and the message I that describes
X":
Intuition from related information theory problems, like
Wyner-Ziv coding, suggests that for efficient communication
the message
I from Encoder 1 to Encoder 2 will result
in a phantom sequence
of
auxiliary random variables that
are jointly typical with
X"
and
yn
according to a selected
joint distribution. The second encoder could jointly compress
the source sequence
yn
along with the auxiliary sequence,
treating it as if it was also a random source sequence. But this
is too crude. A lot is known about the auxiliary sequence, such
as the codebook it came from, allowing it to be summarized
more easily than this approach would allow. In some situations
II.
PROBLEM
SPECIFICS
The encoding
of
source symbols into messages is described
in detail in the introduction and is depicted in Figure 1.
channel setting that is solved in its full generality. Thus, the
feasibility
of
describing two independent data sources without
loss through a noisy channel with interference to a single
decoder is solved.
Beyond the two cases mentioned, slight variations to the
scenario result in a multitude
of
open problems in distributed
source coding. For example, the feasibility
of
describing two
correlated data sources through a noisy MAC is not solved.
Furthermore, allowing the source coding to be done with
loss raises even more uncertainty. Berger and Tung [11] first
considered the multiterminal source coding problem, where
correlated sources are encoded separately with loss. Even
when no noisy channel is involved, the optimal rate region
is not known, but ongoing progress continues [12] [13].
The cascade multiterminal source coding setting is similar to
multiterminal source coding considered by Berger and Tung in
that two sources
of
information are encoded in a distributed
fashion with loss. The difference is that communication be-
tween the source encoders in this network replaces one
of
the direct channels to the decoder. Thus, joint encoding is
enabled to a degree, but the down side is that any message
from Encoder 1 to the Decoder must now cross two links.
The general cascade multiterminal source coding problem
includes many interesting variations. The decoder may need
to estimate both X and Y, X only, Y only, or some other
function
of
both, such as the sum
of
two jointly Gaussian
random variables, considered in Section V-A. Vasudevan, Tian,
and Diggavi [14] looked at a similar cascade communication
system with a relay. In their setting, the decoder has side
information, and the relay has access to a physically degraded
version
of
it. Because
of
the degradation, the decoder knows
everything it needs about the relay's side information, so the
relay does not face the dilemma
of
mixing in some
of
the
side information into its outgoing message. In the cascade
multiterminal source coding setting
of
this paper, the decoder
does not have side information. Thus, the relay is faced with
coalescing the two pieces
of
information into a single message.
Other research involving similar network settings can be found
in [15], where Gu and Effros consider a more general network
but with the restriction that the information Y is a function
of
the information
X,
and [16], where Bakshi et. al. identify
the optimal rate region for lossless encoding
of
independent
sources in a longer cascade (line) network.
In this paper we present inner and outer bounds on the
general rate-distortion region for the cascade multiterminal
source coding problem. The inner bound addresses the chal-
lenge
of
compressing a sequence that is itself the result
of
a
lossy compression. Then we consider specific cases, such as
encoding the sum
of
jointly Gaussian random variables, com-
puting functions, and even coordinating actions. The bounds
are tight for computing functions and achieving some types
of
coordinated actions.
1200
Authorized licensed use limited to: Stanford University. Downloaded on March 02,2010 at 16:58:14 EST from IEEE Xplore. Restrictions apply.

ISIT 2009, Seoul, Korea, June 28 - July 3, 2009
IV.
GENERAL
OUTER
BOUND
Proof: Identify the message I from Encoder 1 along
with the past and future variables in the sequence
yn
as the
auxiliary random variable
U.
Theorem 4.1 (Outer bound): The rate-distortion region R
for the cascade multiterminal source coding network
of
Figure
1 is contained in the region
Rout
defined in (4). Rate-distortion
triples outside
of
Rout
are
not
achievable.
That
is,
(6)
R
cRout.
conditioned on
un(i)
E C
u
symbol-by-symbol according to
p(vlu). Similarly, for each i and k, independently generate
the sequences
zn(k, i) E CZ,i conditioned on
un(i)
E Cu
symbol-by-symbol according to p(zlu).
Finally, assign bin numbers. For every sequence
un(i)
E Cu
assign a
random
bin
bu(i)
E {I, ...,2
n
(I (X;U
1
Y)+2E)} . Also,
for each
i
and
each
vn(j,
i) E
CV,i
assign a random bin
bv(j,
i) E {I, ...,2
n
(I (X;VIY,U)+2E)} .
Successful encoding and decoding is as follows. Encoder
1 first finds a sequence
u"
(i) E Cu that is e-jointly typical
with
X"
with respect to p(x, u). Then Encoder 1 finds a
sequence
u"
(j,
i) E
CV,i
that is e-jointly typical with the
pair
(X";
un(i))
with respect to p(x, u, v). Finally, Encoder 1
sends the bin numbers
bu(i)
and
bv(j,
i) to
Encoder
2.
Encoder 2 considers all codewords in
Cu with bin
number
bu(i) and finds that only
u"
(i) is e-jointly typical with
yn
with respect to p(y, u). Then Encoder 2 considers all
codewords in
CV,i
with bin
number
bv(j,
i) and finds that
only
u"
(j, i) is e-jointly typical with the
pair
(Y",
u"
(i))
with respect to p(y, u, v). Finally, Encoder 2 finds a se-
quence
zn(k, i) E CZ,i that is e-jointly typical with the triple
(yn,un(i),vn(j,i))
with respect to
p(y,u,v,z)
and sends
both
i and k to the Decoder.
The decoder produces
Z"
==
zn(k, i). Due to the Markov
Lemma
[11] and the structure
of
p(x, y, z, u, v), the triple
(X",
Y",
zn)
will be e-jointly typical with high probability.
Finally,
E can be chosen small enough to satisfy the rate
and
distortion inequalities.
(5)
it proves
more
efficient to simply pass the description from
Encoder 1 straight to the
Decoder
rather than to treat it as a
random source
and
recompress at the second encoder.
While still allowing the message
I from Encoder 1 to be
associated with a codebook
of
auxiliary sequences, we
would
like to take advantage
of
the sparsity
of
the codebook as we
form a description at Encoder 2. One way to accommodate
this is to split the message from Encoder 1 into two parts.
One
part
is forwarded by Encoder 2, and the other
part
is
decoded by Encoder 2 into a sequence
of
auxiliary variables
and
compressed with
yn
as
if
it were a random source
sequence. The forwarded message keeps its sparse codebook
in tact, while the decoded and recompressed message enjoys
the efficiency that comes with being bundled with Y. This
results in an inner
bound
R
in
for the rate-distortion region
R.
The definition
of
R
in
is found in (3) at the bottom
of
this
page. The region
R
in
is already convex (for fixed Po(x, y) and
d), so there is no
need
to convexify using time-sharing.
Theorem 3.1 (Inner bound): The rate-distortion region R
for the cascade multiterminal source coding network
of
Figure
1 contains the region
R
in
.
Every rate-distortion triple in R
in
is achievable. That is,
Proof: For lack
of
space, we give only a description
of
the encoding and decoding strategies involved in the
proof
and
skip the probability
of
error analysis. We use familiar tech-
niques
of
randomized codebook construction, jointly typical
encoding, and binning.
For any rate-distortion triple in
R
in
there is an associated
joint
distribution p(x, y, z, u, v) that satisfies the inequalities
in (3). Construct three sets
of
codebooks, Cu,
CV,i,
and CZ,i,
for i
==
1,2,
..., ICul, where
Co
{un(i)}:\,
CV,i
{vn(j,
i)
}j~l'
CZ,i
{zn(k,
i)};:==l.
Let
m.;
==
2
n
(I (X;U)+E),
m2
==
2
n
(I (X;VIU)+E),
and
m3
2
n
(I (Y,V;ZIU)+E).
Randomly generate the sequences
un(i)
E Cu i.i.d. ac-
cording to
p(u), independent for each i. Then for each i
and j, independently generate the sequences vn (j, i) E CV,i
3p(x,y,z,u,v)
==Po(x,y)p(u,vlx)p(zly,u,v)
such
that}
D > E(d(X,
y,
Z)), .
R
1
>
I(X;
u,
VIY),
R
2
>
I(X;
U) + ttv,V; ZIU).
(3)
3p(x,y,z,u)
==Po(x,y)p(ulx)p(zly,u) such
that}
D
~
E(d(X, y, Z)), .
R
1
~
I(X;
UIY),
R
2
~
I(X,
y;
Z).
(4)
1201
Authorized licensed use limited to: Stanford University. Downloaded on March 02,2010 at 16:58:14 EST from IEEE Xplore. Restrictions apply.

ISIT 2009, Seoul, Korea, June 28 - July 3, 2009
1 P
x
R
1
< - log2
-p
.
2
y
If
the variance
of
X is larger than Y and the rate R1 is
small, then the optimal encoding
method
is to forward the
message
I from Encoder 1 to the
Decoder
without changing
it. By rearranging the inequality, we see that
2-
2R1
Px
> P
y.
1
Px
R
1
> - log2
-p
.
2 y
If
the rate R1 is large enough, then the optimal encoding
method
is to recompress at Encoder 2. This will allow for
a
more
efficient encoding
of
the sum in the second message
J rather than encoding two components
of
the estimate
separately.
The distortion in this case is
1To perform this optimization, first note that the marginal distribution
p(x, y, u) determines the quantities
I(X;
U) and
I(X;
UI:-),
and p(x, 'lI'.u)
only has one significant free parameter due to Markovity, All remammg
quantities that define the region
Rin
are conditioned on U, including the final
estimate at the decoder since
U is available to the decoder. Therefore, after
fixing
p(x, y, u) we can remove U entirely from the optimization problem
by exploiting the idiosyncracies
of
the jointly Gaussian distribution. Namely,
reduce the rates
R;
and R2 appropriately and solve the problem without U
with X replacing X and Y replacing Y, where X is the error in estimating
X with
U, and Y is the error in estimating Y with U. This greatly reduces
the dimensionality
of
the problem.
<
R(D)
then
From rate-distortion theory we
know
that
2-
2R1
P
x
is the
mean
squared-error that results from compressing X at rate
R 1. The fact that the variance
of
the error introduced by the
compression at Encoder 1 is larger than the variance
of
Y
subtly indicates that the description
of
X was
more
efficiently
compressed by Encoder 1 than it
would
be
if
mixed
with Y
and recompressed.
The estimate
of
X from Encoder 1, represented by U, which
is forwarded by Encoder 2,
might
be limited by either R
1
or R
2
.
In the case that R
2
is completely saturated with the
description
of
U at rate I (X; U), there is no use trying to use
any excess rate
R
1
-
I(X;
UIY) from Encoder 1 to Encoder
2 because it will have no way
of
reaching the decoder. On
the other hand, in the case that
R 1 is the limiting factor for
the description
of
U at rate
I(X;
UIY), then the excess rate
R
2
-
I (X; U) can be
used
to describe Y to the decoder. We
state the distortion separately for each
of
these cases.
If R
2
<
~
log2
(22;~;/)
then,
D
2-
2R2
(Px+y
+ (1 - p2) (2
2R2
- 1)
Py)
.
1
(2
2 R 1
p2)
h
If
R
2
> 21og2
1-;2
ten,
D ((1 -
p2)2-
2R1
-
(1
- p
22-
2R1)
2-
2R2)
Px
+
2-
2R2
(Px+y
+ (2
2R1
-
1)
Py)
.
Again,
Px
+y is the variance
of
the sum X + Y.
2) Outer bound: The outer
bound
Rout
is optimized with
Gaussian auxiliary random variables. However, for simplicity,
we optimize an even looser
bound
by minimizing R
1
and R
2
separately (cut-set
bound)
for a given distortion constraint. The
result is the following lower
bound
on distortion.
D >
max{2-
2R 1
(1 -
p2)P
X,
2-
2R2
Px+y}.
(8)
3) Sum-Rate: Consider the sum-rate R
1
+ R
2
required to
achieve a given distortion level
D. We can compare the sum-
rate-distortion function
R(D) for the inner and outer bounds.
Let
Px
~
P
y.
This puts us in the recompress regime
of
the inner bound. By optimizing (7) subject to R
1
+ R
2
==
R,
we find that the optimal values
Ri
and R
2
satisfy
R;
-
u;
~
log2
((1
~:~)PX
),
as long as R is greater than the right-hand side. Notice that
R
2
is
more
useful than R
1
,
as we
might
expect. From this
we find a piece-wise
upper
bound
on the sum-rate-distortion
function. Similarly we find a piece-wise lower
bound
based
on (8).
Sum-rate upper bound. Low distortion region:
D < (1 _
p2)PX
(2
_(1 -
p2)PX)
,
Px+y
(7)
D
(1 - p2) (1 -
2-
2R2)
2-
2R1
Px
+
2-
2R2
Px+y,
where
Px
+y is the variance
of
the sum X + Y.
Case 2: (Forward)
V.
SPECIAL
CASES
A. Sum
of
Jointly Gaussian
Suppose we wish to encode two jointly Gaussian data
sources at Encoder
1
and
Encoder
2 in order to produce an
estimate
of
the sum at the decoder with small
mean
squared-
error distortion. Let
X and Y be zero-mean jointly Gaussian
random variables, where
X has variance P
x
,
Y has variance
.
I'
ffici
E(XY)
P
y
,
and
their corre ation coe cient IS p
==
a X
ay
.
1) Inner bound: We can explore the region R
in
byoptimiz-
ing overjointly Gaussian random variables U, V,
and
Z to find
achievable rate-distortion triples
(R
1,
R
2
,
D).
This restricted
search
might
not
find all extremal rate-distortion points in R
in
;
still it provides an inner
bound
on the rate-distortion region. 1
The optimization
of
R
in
with the restriction
of
only consid-
ering jointly Gaussian distributions
p(x, y, z, u, v) leads to two
contrasting strategies depending on the variances
Px
and
Py
of
the sources
and
the rate R 1. The two encoding strategies
employed are to either forward the message from Encoder
1
",to
the Decoder, or to use the message to construct an",estimate
X"
at Encoder 2
and
then compress the vector sum
X"
+
yn
and
send it to the Decoder, but
not
both. In other words, either
let
V
==
0 (forward only) or let U
==
0 (recompress only).
The determining factor for deciding which
method
to use is a
comparison
of
the rate R
1
with the quantity
~
log2
~
.
Case 1: (Recompress)
1202
Authorized licensed use limited to: Stanford University. Downloaded on March 02,2010 at 16:58:14 EST from IEEE Xplore. Restrictions apply.

ISIT 2009, Seoul, Korea, June 28 - July 3, 2009
2The optimal rate region for computing functions
of
data in the standard
multiterminal source coding network is currently an open problem [17].
High distortion region: (up to D ::;
Px+y)
R(D) <
~
10
(PX+y
- (1 - p2)P
X)
2 g2 D - (1 - p2)PX .
Sum-rate lower bound. Low distortion region:
due to the Markovity constraint X -
(Y,V) - Z and the fact that
Z is a function
of
X and Y. Therefore, for this distribution
p,
all
of
the inequalities in R
in
are satisfied for the point
(R
1
,
R
2
,
D).
The outer bound
Rout
makes it clear that optimal encoding
is achieved by using Wyner-Ziv encoding from Encoder 1
to compute the value
of
the function Z at Encoder 2. This
optimization is carefully investigated in [6] and equated to a
graph entropy problem. Then Encoder 2 compresses
Z to the
entropy limit.
C. Markov Coordination
It is possible to talk about achieving a joint distribution
of
coordinated actions p(x, y, z)
==
Po(x, y)p(zlx, y) without
referring to a distortion function, as in [18]. Under some
conditions
of
the joint distribution, the bounds R
in
and
Rout
are tight. One obvious condition is when
X,
Y, and Z form
(9)
(10)
R
1
>
I(X;
ZIY),
R
2
>
I(X;
Z),
REFERENCES
VI.
ACKNOWLEDGMENT
The authors would like to recognize Haim Permuter's vari-
ous contributions and curiosity toward this work. This work is
supported by the National Science Foundation through grants
CCF-0515303 and CCF-0635318.
are achievable. And all rate pairs in
Rout
satisfy these inequal-
ities.
[1] A. Giridhar and P.R. Kumar. Toward a theory
of
in-network compu-
tation in wireless sensor networks. Communications Magazine, IEEE,
44(4):98-107, April 2006.
[2] Nan Ma and P. Ishwar. Two-terminal distributed source coding with
alternating messages for function computation. In IEEE International
Symposium on Information Theory,
Toronto, 2008.
[3] B. Nazer and M. Gastpar. Computation over multiple-access channels.
IEEE Trans. on Info. Theory, 53(10):3498-3516, Oct. 2007.
[4] A. Wyner and J. Ziv. The rate-distortion function for source coding
with side information at the decoder. IEEE Trans. on Info. Theory,
22(1):1-10, Jan 1976.
[5] H. Yamamoto. Wyner - ziv theory for a general function
of
the correlated
sources (corresp.). IEEE Trans. on Info. Theory, 28(5): 803-807, Sep
1982.
[6] A. Orlitsky and 1.R. Roche. Coding for computing. IEEE Trans. on
Info. Theory,
47(3):903-917, Mar 2001.
[7] J. Komer and K. Marton. How to encode the modulo-two sum
of
binary
sources. IEEE Trans. on Info. Theory, 25(2):219-221, 1979.
[8] D. Slepian and 1. Wolf. Noiseless coding
of
correlated information
sources. IEEE Trans. on Info. Theory, 19:471-480, 1973.
[9] R. Ahlswede. Multi-way communication channels. In Proceedings
of
2nd International Symposium on Information Theory (Thakadsor,
Armenian SSR, Sept.
1971), pages
23-52,
1973.
[10] H. Liao. Multiple access channels. In Ph.D. thesis, Department
of
Electrical Engineering, University
of
Hawaii, Honolulu, 1972.
[11] T. Berger. Multiterminal source coding. In G. Longo, editor, Information
Theory Approach to Communications,
pages 171-231. CISM Cource and
Lecture, 1978.
[12] A.B. Wagner, S. Tavildar, and P. Viswanath. Rate region
of
the quadratic
gaussian two-encoder source-coding problem. IEEE Trans. on Info.
Theory,
54(5):1938-1961, May 2008.
[13] K. Krithivasan and S. Pradhan. An achievable rate region for distributed
source coding with reconstruction
of
an arbitrary function
of
the sources.
In IEEE International Symposium on Information Theory, Toronto, 2008.
[14] D. Vasudevan, C. Tian, and S N. Diggavi. Lossy source coding for a
cascade communication system with side-informations. 2006.
[15] W. H. Gu and M. Effros. On multi-resolution coding and a two-hop
network. In Data Compression Conference, 2006.
[16] M. Bakshi, M. Effros, W. H. Gu, and R. Koetter. On network coding
of
independent and dependent sources in line networks. In IEEE
International Symposium on Information Theory,
Nice, 2007.
[17] T. Han and K. Kobayashi. A dichotomy
of
functions f(x,y)
of
correlated
soursed (x,y). IEEE Trans. on Info. Theory, 33:69-76, 1987.
[18] T. Cover and H. Permuter. Capacity
of
coordinated actions. In IEEE
International Symposium on Information Theory,
Nice, 2007.
the Markov chain X - Y - Z. In this case, there is no need to
send a message
I from Encoder 1, and the only requirement
for achievability is that
R
2
2:
I (Y; Z).
Another class
of
joint distributions Po(x, y)p(zlx, y) for
which the rate bounds are provably tight is all distributions
forming the Markov chain Y - X -
Z. This encompasses the
case where Y is a function
of
X,
as in [15]. To prove that the
bounds are tight, choose
U
==
Z and V
==
0 for R
in
. We find
that rate pairs satisfying
H(Z)
- H(ZIY, V)
H(Z)
- H(ZIY, V, X)
H(Z)
I(X,
Y; Z),
I(Y,
V; Z)
then
R(D) > 11
(PX+Y)
11
((1
-
p2)PX)
2"
og2
-V
+
2"
og2 D .
High distortion region: (up to D ::; P
x
+
y
)
R(D) >
~
log2 (
p~y
) .
Lemma 5.1: The gap between the upper and lower bounds
on the optimal sum-rate (derived from
R
in
and
Rout)
needed
to encode the sum
of
jointly Gaussian sources in the cas-
cade multiterminal network with a squared-error distortion
constraint
D is no more than 1 bit, shrinking as D increases,
for any jointly Gaussian sources satisfying
Px
::; P
y.
B. Computing a Function
Instead
of
estimating a function
of
X and Y, we might
want to compute a function exactly. Here we show that the
bounds
R
in
and
Rout
are tight for this lossless case." To do
so, we consider an arbitrary point
(R
1,
R
2
,
D)
E
Rout
and its
associated distribution
p(x, y, z, u). For the inner bound R
in
we use the same distribution
p;
however, let U
==
0 and V take
the role
of
U from the outer bound. Notice that the Markovity
constraints are satisfied. Now consider,
1203
Authorized licensed use limited to: Stanford University. Downloaded on March 02,2010 at 16:58:14 EST from IEEE Xplore. Restrictions apply.
Citations
More filters
Book
16 Jan 2012
TL;DR: In this article, a comprehensive treatment of network information theory and its applications is provided, which provides the first unified coverage of both classical and recent results, including successive cancellation and superposition coding, MIMO wireless communication, network coding and cooperative relaying.
Abstract: This comprehensive treatment of network information theory and its applications provides the first unified coverage of both classical and recent results. With an approach that balances the introduction of new models and new coding techniques, readers are guided through Shannon's point-to-point information theory, single-hop networks, multihop networks, and extensions to distributed computing, secrecy, wireless communication, and networking. Elementary mathematical tools and techniques are used throughout, requiring only basic knowledge of probability, whilst unified proofs of coding theorems are based on a few simple lemmas, making the text accessible to newcomers. Key topics covered include successive cancellation and superposition coding, MIMO wireless communication, network coding, and cooperative relaying. Also covered are feedback and interactive communication, capacity approximations and scaling laws, and asynchronous and random access channels. This book is ideal for use in the classroom, for self-study, and as a reference for researchers and engineers in industry and academia.

2,442 citations

Journal ArticleDOI
TL;DR: This work asks what dependence can be established among the nodes of a communication network given the communication constraints, and develops elements of a theory of cooperation and coordination in networks.
Abstract: We develop elements of a theory of cooperation and coordination in networks. Rather than considering a communication network as a means of distributing information, or of reconstructing random processes at remote nodes, we ask what dependence can be established among the nodes given the communication constraints. Specifically, in a network with communication rates {Ri,j} between the nodes, we ask what is the set of all achievable joint distributions p(x1, ..., xm) of actions at the nodes of the network. Several networks are solved, including arbitrarily large cascade networks. Distributed cooperation can be the solution to many problems such as distributed games, distributed control, and establishing mutual information bounds on the influence of one part of a physical system on another.

289 citations

Journal ArticleDOI
TL;DR: In this paper, the authors considered the network coding problem for a single-receiver network and gave a lower bound on the computing capacity in terms of the Steiner tree packing number and a different bound for symmetric functions.
Abstract: The following network computing problem is considered. Source nodes in a directed acyclic network generate independent messages and a single receiver node computes a target function f of the messages. The objective is to maximize the average number of times f can be computed per network usage, i.e., the “computing capacity”. The network coding problem for a single-receiver network is a special case of the network computing problem in which all of the source messages must be reproduced at the receiver. For network coding with a single receiver, routing is known to achieve the capacity by achieving the network min-cut upper bound. We extend the definition of min-cut to the network computing problem and show that the min-cut is still an upper bound on the maximum achievable rate and is tight for computing (using coding) any target function in multi-edge tree networks. It is also tight for computing linear target functions in any network. We also study the bound's tightness for different classes of target functions. In particular, we give a lower bound on the computing capacity in terms of the Steiner tree packing number and a different bound for symmetric functions. We also show that for certain networks and target functions, the computing capacity can be less than an arbitrarily small fraction of the min-cut bound.

130 citations

01 Jan 2009
TL;DR: This work develops elements of a theory of coordination in networks using tools from information theory and asks for the set of all possible joint distributions p(x1, ..., x m) of actions at the nodes of a network when rate-limited communication is allowed between the nodes.
Abstract: In this work, we develop elements of a theory of coordination in networks using tools from information theory. We ask questions of this nature: If three different tasks are to be performed in a shared effort between three people, but one of them is randomly assigned his responsibility, how much must he tell the others about his assignment? If two players of a multiplayer game wish to collaborate, how should they best use communication to generate their actions? More generally, we ask for the set of all possible joint distributions p(x1, ..., x m) of actions at the nodes of a network when rate-limited communication is allowed between the nodes. Several networks are solved, including arbitrarily large cascade networks. Distributed coordination can be the solution to many problems such as distributed games, distributed control, and establishing mutual information bounds on the physical influence of one part of a system on another.

75 citations


Cites background from "Cascade multiterminal source coding..."

  • ...This dissertation draws on the author’s collaborative work from [32], [24], [33], and [34]....

    [...]

  • ...Therefore, we find as special cases that the bounds in Theorem 8 are tight if X is a function of Y , if Y is a function of X, or if the reconstruction Z is a function of X and Y [24]....

    [...]

Proceedings ArticleDOI
13 Jun 2010
TL;DR: In this article, the authors consider the optimality of source-channel separation in networks, and show that such a separation approach is optimal or approximately optimal for a large class of scenarios, namely, when the sources are mutually independent, and each source is needed only at one destination (or at multiple destinations at the same distortion level).
Abstract: We consider the optimality of source-channel separation in networks, and show that such a separation approach is optimal or approximately optimal for a large class of scenarios. More precisely, for lossy coding of memoryless sources in a network, when the sources are mutually independent, and each source is needed only at one destination (or at multiple destinations at the same distortion level), the separation approach is optimal; for the same setting but each source is needed at multiple destinations under a restricted class of distortion measures, the separation approach is approximately optimal, in the sense that the loss from optimum can be upper-bounded. The communication channels in the network are general, including various multiuser channels with finite memory and feedback, the sources and channels can have different bandwidths, and the sources can be present at multiple nodes.

60 citations

References
More filters
Journal ArticleDOI
David Slepian1, Jack K. Wolf
TL;DR: The minimum number of bits per character R_X and R_Y needed to encode these sequences so that they can be faithfully reproduced under a variety of assumptions regarding the encoders and decoders is determined.
Abstract: Correlated information sequences \cdots ,X_{-1},X_0,X_1, \cdots and \cdots,Y_{-1},Y_0,Y_1, \cdots are generated by repeated independent drawings of a pair of discrete random variables X, Y from a given bivariate distribution P_{XY} (x,y) . We determine the minimum number of bits per character R_X and R_Y needed to encode these sequences so that they can be faithfully reproduced under a variety of assumptions regarding the encoders and decoders. The results, some of which are not at all obvious, are presented as an admissible rate region \mathcal{R} in the R_X - R_Y plane. They generalize a similar and well-known result for a single information sequence, namely R_X \geq H (X) for faithful reproduction.

4,165 citations


"Cascade multiterminal source coding..." refers background in this paper

  • ...Encoding of information sources at separate encoders has attracted a lot of attention in the information theory community over the years....

    [...]

Journal ArticleDOI
TL;DR: The quantity R \ast (d) is determined, defined as the infimum ofrates R such that communication is possible in the above setting at an average distortion level not exceeding d + \varepsilon .
Abstract: Let \{(X_{k}, Y_{k}) \}^{ \infty}_{k=1} be a sequence of independent drawings of a pair of dependent random variables X, Y . Let us say that X takes values in the finite set \cal X . It is desired to encode the sequence \{X_{k}\} in blocks of length n into a binary stream of rate R , which can in turn be decoded as a sequence \{ \hat{X}_{k} \} , where \hat{X}_{k} \in \hat{ \cal X} , the reproduction alphabet. The average distortion level is (1/n) \sum^{n}_{k=1} E[D(X_{k},\hat{X}_{k})] , where D(x,\hat{x}) \geq 0, x \in {\cal X}, \hat{x} \in \hat{ \cal X} , is a preassigned distortion measure. The special assumption made here is that the decoder has access to the side information \{Y_{k}\} . In this paper we determine the quantity R \ast (d) , defined as the infimum ofrates R such that (with \varepsilon > 0 arbitrarily small and with suitably large n )communication is possible in the above setting at an average distortion level (as defined above) not exceeding d + \varepsilon . The main result is that R \ast (d) = \inf [I(X;Z) - I(Y;Z)] , where the infimum is with respect to all auxiliary random variables Z (which take values in a finite set \cal Z ) that satisfy: i) Y,Z conditionally independent given X ; ii) there exists a function f: {\cal Y} \times {\cal Z} \rightarrow \hat{ \cal X} , such that E[D(X,f(Y,Z))] \leq d . Let R_{X | Y}(d) be the rate-distortion function which results when the encoder as well as the decoder has access to the side information \{ Y_{k} \} . In nearly all cases it is shown that when d > 0 then R \ast(d) > R_{X|Y} (d) , so that knowledge of the side information at the encoder permits transmission of the \{X_{k}\} at a given distortion level using a smaller transmission rate. This is in contrast to the situation treated by Slepian and Wolf [5] where, for arbitrarily accurate reproduction of \{X_{k}\} , i.e., d = \varepsilon for any \varepsilon >0 , knowledge of the side information at the encoder does not allow a reduction of the transmission rate.

3,288 citations

Journal ArticleDOI
01 Oct 2007
TL;DR: It is shown that there is no source-channel separation theorem even when the individual sources are independent, and joint source- channel strategies are developed that are optimal when the structure of the channel probability transition matrix and the function are appropriately matched.
Abstract: The problem of reliably reconstructing a function of sources over a multiple-access channel (MAC) is considered. It is shown that there is no source-channel separation theorem even when the individual sources are independent. Joint source-channel strategies are developed that are optimal when the structure of the channel probability transition matrix and the function are appropriately matched. Even when the channel and function are mismatched, these computation codes often outperform separation-based strategies. Achievable distortions are given for the distributed refinement of the sum of Gaussian sources over a Gaussian multiple-access channel with a joint source-channel lattice code. Finally, computation codes are used to determine the multicast capacity of finite-field multiple-access networks, thus linking them to network coding.

758 citations

Proceedings Article
01 Jan 1973

757 citations


"Cascade multiterminal source coding..." refers background in this paper

  • ...The results of Slepian-Wolf encoding and communication through the Multiple Access Channel (MAC) are urprising and encouraging....

    [...]

01 Jan 1978

558 citations