UC San Diego
UC San Diego Previously Published Works
Title
Joint Source/Channel Coding and MAP Decoding of Arithmetic Codes
Permalink
https://escholarship.org/uc/item/4zc3f7nv
Journal
IEEE Transactions on Communications, 53(6)
ISSN
0090-6778
Authors
Grangetto, M.
Cosman, P.
Olmo, G.
Publication Date
2005-06-01
DOI
10.1109/TCOMM.2005.849690
Peer reviewed
eScholarship.org Powered by the California Digital Library
University of California
IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 53, NO. 6, JUNE 2005 1007
Joint Source/Channel Coding and MAP Decoding
of Arithmetic Codes
Marco Grangetto, Member, IEEE, Pamela Cosman, Senior Member, IEEE, and Gabriella Olmo, Member, IEEE
Abstract—In this paper, a novel maximum a posteriori (MAP)
estimation approach is employed for error correction of arith-
metic codes with a forbidden symbol. The system is founded on the
principle of joint source channel coding, which allows one to unify
the arithmetic decoding and error correction tasks into a single
process, with superior performance compared to traditional sepa-
rated techniques. The proposed system improves the performance
in terms of error correction with respect to a separated source
and channel coding approach based on convolutional codes, with
the additional great advantage of allowing complete flexibility in
adjusting the coding rate. The proposed MAP decoder is tested in
the case of image transmission across the additive white Gaussian
noise channel and compared against standard forward error
correction techniques in terms of performance and complexity.
Both hard and soft decoding are taken into account, and excellent
results in terms of packet error rate and decoded image quality
are obtained.
Index Terms—Arithmetic coding, image transmission, joint
source/channel coding (JSCC), maximum
(MAP)
estimation.
I. INTRODUCTION
T
HE FUTURE of telecommunications is being driven by
two impressive recent phenomena, namely the widespread
diffusion of the Internet and the development of personal mo-
bile communications. The migration to the wireless channel of
Internet-based services such as multimedia communications,
video conferencing, and digital image and music sharing is
colliding with the bandwidth and power limitations imposed by
the mobile environment. As a consequence, the research com-
munity is moving toward more interdisciplinary approaches
involving networking, digital communications and multimedia
expertise in order to improve the overall quality of service
yielded by the system.
In this light,
joint source/channel coding (JSCC) techniques
are emerging as a natural integration of the multimedia and
digital communication worlds. In fact, the wireless bandwidth
limitation and the high data rates and latency constraints im-
posed by multimedia services are emphasizing the practical
shortcomings of Shannon’s source-channel separation theorem
[1]. JSCC techniques are based on the fact that in practical
Paper approved by A. K. Khandani, the Editor for Coding and Signal of
the IEEE Communications Society. Manuscript received May 1, 2003; revised
September 9, 2004 and December 1, 2004.
M. Grangetto and G. Olmo are with the Center for Multimedia Radio Com-
munications (CERCOM), Politecnico di Torino, 10129 Torino, Italy (e-mail:
marco.grangetto@polito.it; gabriella.olmo@polito.it).
P. Cosman is with the Department of Electrical and Computer Engineering,
University of California at San Diego, La Jolla, CA 92093-0407 USA (e-mail:
pcosman@code.ucsd.edu).
Digital Object Identifier 10.1109/TCOMM.2005.849690
cases the source encoder is not able to ideally decorrelate the
input sequence; some implicit redundancy is still present in
the compressed stream and can be properly exploited by the
decoder for error control. As a consequence, it is possible
to improve the decoder performance considering source and
channel coding jointly.
In the JSCC field, considerable attention has been devoted in
the past to the error resilience of variable length codes (VLCs)
[2]–[7]. VLCs represent the final entropy coding stage for many
coding standards such as JPEG, MPEG-4 and H.263, and their
robustness to transmission errors is clearly a major issue. In
[2], [3] the residual redundancy in the source encoder output
is represented by a Markov model, and is used as a form of im-
plicit channel protection at the decoder side; exact and approx-
imate maximum a posteriori (MAP) sequence estimators are
proposed. Results are provided in the case of image transmission
across the binary symmetric channel (BSC); the source coder
implements Huffman coding of neighboring pixel differences.
Another MAP decoding technique for VLCs is proposed in [4],
and tested in the case of transmission of a first order Markov
source. In [5] soft decoding is used, and results for MPEG-4
reversible VLCs are reported. In [6], a low complexity soft de-
coding of VLCs is proposed; the results also include joint source
channel decoding when turbo codes are used for error correc-
tion. In [7] the turbo principle is applied to joint source-channel
decoding of VLCs when cascaded with a channel code.
On the other hand, emerging coding standards such as JPEG
2000 [8] and JBIG2 [9] for still pictures and H.264 [10] for
video sequences use arithmetic coding (AC) as the final entropy
coding stage. AC can allocate fractional numbers of bits to input
symbols, thus improving the compression efficiency [11]. More-
over, it easily encompasses efficient adaptive coding solutions.
However, AC is very sensitive to transmission error; the arith-
metic decoder has poor resynchronization properties and the
high compression efficiency prevents MAP decoding based on
residual redundancy. Hence there is a great interest in resilient
AC, and in JSCC techniques based on AC [12]–[23]. In [12]
an error recovery technique based on automatic repeat request
(ARQ) is implemented by means of AC with proper insertion
of markers. In [13], [14] an AC that embeds channel coding is
presented; the idea is to enforce a minimum Hamming distance
constraint among encoded sequences, allowing the implemen-
tation of a MAP estimator at the decoder side. AC with error
detection capability is proposed in [15], where variable to fixed
length AC is designed. Another way to obtain error detection,
based on the insertion of a forbidden symbol in the input al-
phabet, was initially proposed in [16], and further studied in [17]
and [18]. The forbidden symbol allows one to adjust the amount
of coding redundancy to be embedded in the coded stream. At
0090-6778/$20.00 © 2005 IEEE
1008 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 53, NO. 6, JUNE 2005
the expense of compression efficiency, it permits error detection
at the decoder side. In [17], an ARQ strategy is validated in the
case of lossless image transmission across the BSC. The coding
redundancy conveyed by the forbidden symbol can be recog-
nized by the decoder in order to attempt not only error detection
but also error correction. In [19], error correction is performed
in the case of transmission across the additive white Gaussian
noise (AWGN) channel; binary signalling with null zone soft de-
coding is employed. The performance is evaluated in terms of
packet recovery rate for differentially encoded images. In [20],
sequential decoding of arithmetic coded data is discussed and
a very simple arithmetic encoder is shown to achieve excellent
performance in terms of error correction. In [21], a JSCC con-
catenated scheme based on AC and trellis coded modulation is
applied to image transmission. In [22] and [23], some prelimi-
nary work on MAP decoding of AC with a forbidden symbol is
presented in the case of image transmission across the BSC.
In this paper, the MAP decoding approach is employed for
error correction of binary AC with a forbidden symbol; the MAP
approach is employed for both hard and soft decoding in the case
of transmission across the AWGN channel. Two different se-
quential search techniques are applied to the estimation problem
and compared in terms of complexity and performance. AC with
a forbidden symbol and MAP estimation allow us to design a
novel joint source/channel coding and decoding scheme with at-
tractive features in terms of error correction, adaptivity and rate
flexibility. The original contributions of our work mainly rely on
the use of a MAP decoding metric, where the
a priori knowledge
of the source is taken into account in the correction algorithm,
along with the forbidden symbol error detection and the channel
transition probability. Some of this work has appeared in [22],
[23], where transmission across the binary symmetric channel
was considered. In [19] another JSCC approach based on AC
with a forbidden symbol is proposed. In [19] a 256-symbol AC
is employed to encode pixel prediction errors and the error cor-
rection task is based on a maximum-likelihood (ML) criterion,
i.e., minimum distance. Results for AWGN channel transmis-
sion with soft decoding are given. In the present paper, the vi-
ability of the proposed approach is demonstrated in the case
of image transmission in two typical cases: lossless predictive
coding and lossy progressive coding. The first application al-
lows us to validate the proposed techniques and to compare the
MAP estimation algorithms with the results in [19], based on
ML decoding. The second set of experiments is particularly in-
novative, since it couples the proposed error resilient sequen-
tial entropy coding and the popular SPIHT image coder. The
ability to sequentially decode the SPIHT embedded bitstream in
the presence of transmission errors is particularly attractive and
it exhibits a better performance than powerful error protection
techniques based on FEC [24]. During the review of the present
paper, an independent work on sequential arithmetic decoding
for reliable image transmission has been published [25]. In this
case synchronization markers, instead of the forbidden symbol,
are used for error detection purpose. A preliminary comparison
with this alternative technique is reported in Section VI.
The paper is organized as follows. In Section II, AC with a
forbidden symbol is briefly reviewed. In Section III, the error
correction task is formulated in terms of a MAP estimation
problem, and the employed sequential decoding algorithms are
Fig. 1. Binary arithmetic encoder with forbidden symbol
.
described in Section IV. In Sections V and VI, we evaluate the
performance of the proposed technique. Finally, Section VII
shows our conclusions and offers directions for future work.
II. A
RITHMETIC
CODING (AC)
In this paper we select a simple, yet significant, case study
where the input sequence
,
, is constituted by binary outcomes of a binary memo-
ryless source with symbol probabilities
and
, , .Itis
worth noticing that the concepts introduced here can be general-
ized to more sophisticated source models as well as to adaptive
AC.
Binary arithmetic encoding is an iterative task, whose objec-
tive is to map the input sequence
, onto a variable
length binary string
, , representing its probability.
This task is performed by progressively refining the probability
interval corresponding to
. The probability interval is initial-
ized to (0,1) and then the interval portion corresponding to the
encoded symbol is iteratively selected. After
iterations, an in-
terval
, whose size corresponds to the input sequence prob-
ability, is obtained and encoded by means of the shortest bi-
nary sequence
belonging to it; the expected number of bits
required by this operation is
, where
is the memoryless source entropy rate.
The decoding task, in the error free case, simply follows the
dual process and consists of iteratively finding the interval to
which the encoded codeword belongs. It is worth pointing out
that both encoding and decoding can be accomplished sequen-
tially, avoiding excessive delay [11].
The decoding process is very fragile with respect to transmis-
sion errors; in fact, a single flipped bit can shift the correct code-
word to the adjacent interval, causing irreversible desynchro-
nization. This behavior has motivated the design of a number
of error detection tools for AC, briefly described in Section I.
In this paper, we use an error detecting AC based on the intro-
duction of a forbidden symbol into the coding alphabet [16]. A
forbidden symbol
, which is never encoded and whose prob-
ability is fixed to an arbitrary value
, provides a form
of coding redundancy. The introduction of
clearly implies a
modification of the symbol probabilities, that become less ac-
curate with respect to the source model as
increases. This cor-
responds to an amount of coding redundancy per encoded bit
[17], that is forced in the encoded binary
sequence at the expense of compression efficiency. In Fig. 1, we
GRANGETTO et al.: JSCC AND MAP DECODING OF ARITHMETIC CODES 1009
Fig. 2. Block diagram of the proposed transmission system.
show three iterations of the modified encoder with a forbidden
symbol, which is based on the ternary alphabet “0”, “1” and
“
” with probabilities , and , respectively.
The presence of
causes an interval shrinkage by a factor
at each iteration, which corresponds to a codeword , whose
expected length
is increased with re-
spect to the case without the forbidden symbol. At the decoder
side, the presence of
can be used for error detection; if the
decoder detects the forbidden symbol, it means that transmis-
sion errors have occurred. Countermeasures can then be taken,
such as retransmission. In [17], the reliability of this error detec-
tion mechanism is evaluated. In particular, if an error occurs in a
certain position, the probability that the error detection delay is
greater than
symbols is . Therefore, a large value of
assures fast error detection, but it greatly reduces the compres-
sion efficiency. The performance of an error correcting decoder
based on the forbidden symbol detection is described in the fol-
lowing sections. Moreover, the use of the forbidden symbol ex-
hibits additional advantages if compared with other error detec-
tion techniques, such as the use of periodic cyclic redundancy
check (CRC) or synchronization markers. First of all, the coding
redundancy can be flexibly controlled by means of a single pa-
rameter, i.e., the value of the probability
, which takes on con-
tinuous values and allows one to achieve any coding rate. Fi-
nally, the forbidden symbol guarantees the so called continuous
error detection [18], since errors can be revealed sequentially
during AC decoding; on the contrary, the use of periodic CRC
or synchronization markers permits error detection only after a
whole block of data has been completely decoded.
III. MAP D
ECODING METRIC
The coding redundancy associated with the forbidden symbol
can be used by the decoder to select the best estimate of the
encoded sequence, received through an error prone channel.
In the following, we consider the transmission scheme whose
block diagram is represented in Fig. 2. The variable length code-
word
, corresponding to the input
sequence
, is transmitted across
the channel with transition probability
. Note that in
the channel block we include modulation, channel transmission
and demodulation. The receiver observes the demodulated se-
quence
. It is worth noticing that, if one
models sequences
, and as random variables, they consti-
tute a Markov chain, where AC introduces a variable amount of
memory between source and coded symbols.
The objective of the MAP decoder is to find the most probable
input sequence
(1)
Therefore, the estimation is based on the following decoding
metric:
(2)
In principle the decoding task consists of evaluating the metric
(2) for all possible pairs
, such that the length of the
encoded sequence
is equal to ; in the following we will
denote the subset of codewords of length
as . The
metric (2) includes the channel transition probability, the a
priori source probability
and the term . The first
two terms are known and can be evaluated on the basis of the
channel model and the a priori source model, respectively. The
last term requires more attention. It can be written as
(3)
It can be easily understood that the evaluation of this term is as
complex as the whole decoding metric and requires knowledge
of the subset
, which is infeasible for practical values of .
Therefore, proper approximations will be adopted, as detailed in
Sections III-A and III-B.
Finally, in the case of memoryless channels, it is useful to
express the decoding metric in the additive form
(4)
where
(5)
The term
represents the a priori probability of the
source symbols output by the arithmetic decoder associated
with the
th bit of codeword ; it is worth noticing that, due to
the variable length nature of AC, the number of decoded source
symbols is variable and depends on both the codeword
, and
the bit position
. The additive metric can be employed
as the branch metric by the sequential techniques used to find
the best estimate. In the following sections we will analyze the
branch metric for transmission across the AWGN channel with
hard and soft decoding.
1010 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 53, NO. 6, JUNE 2005
A. BPSK With Hard Decoding
For transmission across an AWGN channel using binary
phase-shift keying (BPSK) modulation with signal to noise
ratio
and hard decoding, the channel transition proba-
bility is
if
if
(6)
with
.
As already discussed, the evaluation of
is not straight-
forward. Hence we adopt the approximation
,
which amounts to assuming that there are
equally likely
possible codewords
of length . Because of the variable
length of AC, this assumption does not generally hold true; in
other words, not all the possible sequences of length
are valid
AC codewords. Nevertheless, the adopted assumption has pro-
vided satisfactory results for MAP decoding of VLC in [2], [3]
and can be justified by the concept of typical sequence [26]. In
fact, for large values of
, the typical sequences ,defined as
those ones with
“0”s and “1”s, are equally likely and
are mapped onto AC codewords of
bits. The number
of such sequences is
, and can be approximated to
by means of Stirling’s formula. In conclusion, by means of this
latter approximation, the branch decoding metric (5) turns out
to be
(7)
B. BPSK With Soft Decoding
For BPSK modulation with soft decoding, the MAP estimator
observes the values
, , where
is the modulated level and is a
Gaussian noise sample with zero mean and variance
. This
channel model yields
(8)
Using conditional probabilities, we can write
, where is the th bit of the
transmitted codeword, and is assumed to take values “0” and
“1” with equal probability. As in the hard decoding case, the
equiprobability hypothesis greatly simplifies the evaluation.
Thus it is straightforward to obtain
(9)
Finally, in the soft decoding case, the branch decoding metric
becomes
(10)
IV. S
EQUENTIAL SEARCH
TECHNIQUES
This section is devoted to the description of the algorithms
used to evaluate the MAP decoding metrics (7) and (10). Di-
rect evaluation of the MAP metric over the subset
is infea-
sible for two reasons. The first is that, for typical values of
,
it is impractical to store all
codewords , so as to isolate
those belonging to
. The second is that the large cardinality
of
would anyhow prevent direct evaluation of the MAP
metric. Therefore it is essential to resort to suboptimal search
techniques, able to sequentially travel along the subset
in
order to pick up the most likely sequence.
The first obstacle can be solved by exploiting the forbidden
symbol
: the search for the best estimate is enlarged to all
binary sequences of length
, ; those
sequences that are not admissible codewords
can be dis-
carded, upon
detection. The proposed decoder is implemented
by means of a search algorithm along the branches of the bi-
nary tree representing
. During tree exploration, the branch
metric (7) or (10) can be accumulated, and non admissible code-
words can be pruned upon error detection. It is worth recalling
that error detection occurs with a delay whose probability de-
pends on the value of
. The problem of the large cardinality
of the search space can be tackled by means of the sequential
search strategies described in the following sections.
A. Stack Algorithm (SA)
The stack algorithm (SA) [27] is known as a metric first tech-
nique. The best path selection is based on a greedy approach,
extending at each iteration the best stored path, i.e., the one with
the best accumulated metric (4). This is accomplished by storing
all the visited paths in an ordered list, with predefined maximum
length
. Each element of the list contains the accumulated
metric and the state information for sequential arithmetic de-
coding. At each iteration, the best stored path is extended one
branch forward. The extended path is dropped if the forbidden
symbol is revealed or if the number of decoded symbols ex-
ceeds
. Moreover, a branch is pruned if its metric falls below
a threshold, so as to avoid the extension of extremely unlikely
paths. The branching goes on until the stopping criterion is ful-
filled; in our implementation, the algorithm terminates as soon
as the best path in storage corresponds to a valid input sequence
, i.e., a path in the tree that corresponds to the decoding of
source bits, when all the codeword bits have been consumed.
Similarly to the Viterbi algorithm, decoding can be per-
formed sequentially, due to the merging of all paths after a
certain delay of
processed bits [27], [26]. In fact, as the
least likely branches in the tree are progressively pruned, the
surviving paths tend to merge into a single one, while moving
back toward the tree root. This property can be exploited to
progressively output decoded bits as soon as the decoding tree
has collapsed into one single path.
B. M-Algorithm (MA)
The M-algorithm (MA) limits the search space to a number
of paths at each depth in the tree; for this reason, it can
![Fig. 9. Average PSNR as a function of E =N obtained with the proposed MAP decoder and soft decoding technique [25].](/figures/fig-9-average-psnr-as-a-function-of-e-n-obtained-with-the-3vk44cco.webp)
