Let (X,Y) be a pair of discrete random variables with X taking one of M possible values, Suppose the value of X is to be determined, given the value of Y, by asking questions of the form "Is X equal to x?" until the answer is "Yes". Let G(x|y) denote the number of guesses in any such guessing scheme when X=x, Y=y. We prove that E[G(X|Y)/sup /spl rho//]/spl ges/(1+lnM)/sup -/spl rho///spl Sigma//sub y/[/spl Sigma//sub x/P/sub X,Y/(x,y)/sup 1/1+/spl rho//]/sup 1+/spl rho// for any /spl rho//spl ges/0. This provides an operational characterization of Renyi's entropy. Next we apply this inequality to the estimation of the computational complexity of sequential decoding. For this, we regard X as the input, Y as the output of a communication channel. Given Y, the sequential decoding algorithm works essentially by guessing X, one value at a time, until the guess is correct. Thus the computational complexity of sequential decoding, which is a random variable, is given by a guessing function G(X|Y) that is defined by the order in which nodes in the tree code are hypothesized by the decoder. This observation, combined with the above lower bound on moments of G(X|Y), yields lower bounds on moments of computation in sequential decoding. The present approach enables the determination of the (previously known) cutoff rate of sequential decoding in a simple manner; it also yields the (previously unknown) cutoff rate region of sequential decoding for multiaccess channels. These results hold for memoryless channels with finite input alphabets.

/pdf/an-inequality-on-guessing-and-its-application-to-sequential-32we3a219s.pdf

An inequality on guessing and its application to sequential decoding

The purpose of this paper is to organize and clarify the work of the past decade on burst-correcting codes. Our method is, first, to define an idealized model, called the classic bursty channel, toward which most burst-correcting schemes are explicitly or implicitly aimed; next, to bound the best possible performance on this channel; and, finally, to exhibit classes of schemes which are asymptotically optimum and serve as archetypes of the burstcorrecting codes actually in use. In this light we survey and categorize previous work on burst-correcting codes. Finally, we discuss qualitatively the ways in which real channels fail to satisfy the assumptions of the classic bursty channel, and the effects of such failures on the various types of burst-correcting schemes. We conclude by comparing forward-error-correction to the popular alternative of automatic repeat-request (ARQ).

Burst-Correcting Codes for the Classic Bursty Channel

We introduce a new algorithm for realizing maximum likelihood (ML) decoding for arbitrary codebooks in discrete channels with or without memory, in which the receiver rank-orders noise sequences from most likely to least likely. Subtracting noise from the received signal in that order, the first instance that results in a member of the codebook is the ML decoding. We name this algorithm GRAND for Guessing Random Additive Noise Decoding. We establish that GRAND is capacity-achieving when used with random codebooks. For rates below capacity, we identify error exponents, and for rates beyond capacity, we identify success exponents. We determine the scheme’s complexity in terms of the number of computations that the receiver performs. For rates beyond capacity, this reveals thresholds for the number of guesses by which, if a member of the codebook is identified, that it is likely to be the transmitted code word. We introduce an approximate ML decoding scheme where the receiver abandons the search after a fixed number of queries, an approach we dub GRANDAB, for GRAND with ABandonment. While not an ML decoder, we establish that the algorithm GRANDAB is also capacity-achieving for an appropriate choice of abandonment threshold, and characterize its complexity, error, and success exponents. Worked examples are presented for Markovian noise that indicate these decoding schemes substantially outperform the brute force decoding approach.

Capacity-Achieving Guessing Random Additive Noise Decoding

Sequential decoding is characterized as a sequential search for the shortest path through a trellis. An easily analyzed algorithm closely related to the stack and Fano algorithms is described. Martingale techniques are used to find the distribution of computation on totally symmetric channels. For general channels, our universal bounding technique yields the well-known Pareto distribution of computation, as well as a bound on error probability that is asymptotically optimum in the high-rate range. The performance-complexity relation is shown to be asymptotically Pareto for both sequential and maximum-likelihood (Viterbi) decoding, with the same exponent in either case. A semisequential list-of-L Viterbi algorithm is introduced to extend the analogies below Rcomp.

Convolutional codes III. Sequential decoding

The cutoff rate R0(W) of a discrete memoryless channel (DMC) W is often used as a figure of merit alongside the channel capacity C(W). If a channel W is split into two possibly correlated subchannels W1, W2, the capacity function always satisfies C(W1)+C(W2)lesC(W), while there are examples for which R0(W1)+R0(W2 )>R0(W). The fact that cutoff rate can be "created" by channel splitting was noticed by Massey in his study of an optical modulation system. This paper gives a general framework for achieving similar gains in the cutoff rate of arbitrary DMCs by methods of channel combining and splitting. The emphasis is on simple schemes that can be implemented in practice. We give several examples that achieve significant gains in cutoff rate at little extra system complexity. Theoretically, as the complexity grows without bound, the proposed framework is capable of boosting the cutoff rate of a channel to arbitrarily close to its capacity in a sense made precise in the paper. Apart from Massey's work, the methods studied here have elements in common with Forney's concatenated coding idea, a method by Pinsker for cutoff rate improvement, and certain coded-modulation techniques, namely, Ungerboeck's set-partitioning idea and Imai-Hirakawa multilevel coding; these connections are discussed in the paper

/pdf/channel-combining-and-splitting-for-cutoff-rate-improvement-23k9jz53bv.pdf

Channel combining and splitting for cutoff rate improvement

We consider a coding-decoding scheme which can permit reliable data communication at rates up to the capacity of a discrete memoryless channel, and which offers a reasonable trade off between performance and complexity. The new scheme embodies algebraic and sequential coding-decoding stages. Data is initially coded by an algebraic (Reed-Solomon) encoder into blocks of N symbols, each symbol represented by n binary digits. The N n-bit symbols in a block are transmitted separately and independently through N parallel subsystems, each consisting of a sequential coder, an independent discrete memoryless channel, and a sequential decoder in tandem. Those coded n-bit symbols which would require the most sequential decoding computations are treated as erasures and decoded by a Reed-Solomon decoder. We show that the hybrid technique reduces the variability of the amount of sequential decoding computation. We also derive asymptotic results for the probabilities of error and buffer overflow as functions of the system complexity.

D. D. Falconer

Papers

A hybrid coding scheme for discrete memoryless channels