Showing papers presented at "Information Theory Workshop in 2002"

Network coding and error correction

Abstract: We introduce network error-correcting codes for error correction when a source message is transmitted to a set of receiving nodes on a network. The usual approach in existing networks, namely link-by-link error correction, is a special case of network error correction. The network generalizations of the Hamming bound and the Gilbert-Varshamov bound are derived.

On phase noise channels at high SNR

Abstract: Lapidoth and Moser (2001) have recently proposed a general technique for obtaining upper bounds on channel capacity via a dual expression in which the maximization over probability distributions on the channel input alphabet is replaced with a minimization over probability distributions on the channel output alphabet. They have also introduced the notion of "capacity achieving input distributions that escape to infinity" in order to study channel capacity at high signal-to-noise (SNR) ratios. In this partly tutorial paper we demonstrate the use of these ideas by applying them to the study of communication over discrete-time channels impaired by additive white Gaussian noise and phase noise.

A simple, unified approach to nearly optimal multiuser detection and space-time coding

Abstract: Techniques using interleaving as the basic means for signal separation are introduced for both multiple access systems and multiple transmit antenna systems A very low-cost chip-by-chip iterative detection algorithm is presented The proposed schemes can achieve nearly optimal performance for system with a large numbers of users or transmit antennas

Source broadcasting with unknown amount of receiver side information

Abstract: The Slepian-Wolf scheme for source coding with side information at the receiver, assures that the sender can send the source X at a rate of only the conditional entropy H(X|Y-) bits per source symbol, which is the minimal possible rate even if the sender knew the side information Y. However, the Slepian-Wolf result requires knowledge of the optimal required rate. In this paper we consider a situation where this rate is not known, possibly since the source is broadcasted to many heterogeneous receivers. The approach is based on recent results regarding sending a common information over a broadcast channel.

Information geometry of statistical inference - an overview

Abstract: The present paper gives a short introduction to information geometry, by using a simple model of an exponential family which is a dually flat Riemannian space The paper then overviews some of the applications of information geometry: 1) the higher-order asymptotic theory of estimation; 2) semiparametric estimation of the parameter of interest; 3) learning neural networks under the Riemannian structure; and 4) analysis of turbo codes, low density parity check codes and belief propagation algorithm

Low density parity check codes over groups and rings

Abstract: The role of low density parity check principles in the design of group codes for coded modulation is examined. In this context, the structure of linear codes over certain rings /spl Zopf//sub m/ and G/sub m/ is discussed, and LDPC codes over these ring structures are designed.

Game theory, maximum generalized entropy, minimum discrepancy, robust Bayes and Pythagoras

Abstract: Suppose that, for purposes of inductive inference or choosing an optimal decision, we wish to select a single distribution P* to act as representative of a class /spl Gamma/ of such distributions The maximum entropy principle ("MaxeEnt") (Jaynes 1989; Csiszar 1991) is widely applied for this purpose, but its rationale has often been controversial (Shimony 1985; Seidenfeld 1986) Here we emphasize and generalize a reinterpretation of the maximum entropy principle (Topsoe (1979); Walley (1991); Grunwald (1998)): that the distribution P* that maximizes the entropy over /spl Gamma/ also minimizes the worst-case expected logarithmic score (log loss) In the terminology of decision theory (Berger 1985), P* is a robust Bayes, or /spl Gamma/-minimax, act, when loss is measured by the log loss This gives a decision-theoretic justification for maximum entropy

A new construction for low density parity check convolutional codes

Abstract: Low density parity check (LDPC) block codes have been shown to achieve near capacity performance for binary transmission over noisy channels. Block codes, however, require splitting the data to be transmitted into frames, which can be a disadvantage in some applications. Convolutional codes, on the other hand, have no such requirement, and are well suited for continuous transmission. Felstrom and Zigangirov (1999) proposed the construction of periodic time-varying convolutional codes with LDPC matrices. A set of time-invariant LDPC convolutional codes was described by Sridharan et al. (2002). The codes of Felstrom and Zigangirov were obtained by random construction techniques whereas those of Sridharan et al. were essentially algebraic constructions. The new LDPC convolutional codes described here are obtained by introducing a degree of randomness into the latter construction.

On the performance of space-time codes

Abstract: This paper provides an overview of the results in a recent journal submission by the same authors. The first part of that paper studies the pairwise error probability of codewords (PEP) of a space-time code over a quasistatic channel, using an approach that allows both known and unknown channel cases to be considered simultaneously. A closed-form expression for the PEP is provided, and given a constraint on the sum of the squares of the singular values of the difference signal matrix, it is shown that the PEP is minimized by choosing signals with equal singular values. A useful sequence of simple upper and lower bounds that converge to the PEP is also provided. An example space-time code is introduced and shown using this sequence of bounds to outperform the corresponding orthogonal-design-based space-time (ODST) code at all values of SNR. Exact expressions, based on the PEP, are given for the asymptotic coding and diversity gain. It is shown that the diversity gain remains unchanged if the PEP is replaced by either the codeword error probability (CEP) or else the message symbol error probability (SEP). Signal-design implications of the above results are also discussed. The second part deals with ODST codes. It is shown that ODST codes represent an instance of orthogonal signaling. This observation is used to derive a closed-form expression for the pairwise error probability of message symbols (PEP-ms) of these codes, as well as an expression for coding gain based on PEP-ms, that is exact in the case of BPSK signaling.

Cryptography with information theoretic security

Abstract: Summary form only given. We discuss information-theoretic methods to prove the security of cryptosystems. We study what is called, unconditionally secure (or information-theoretically secure) cryptographic schemes in search for a system that can provide long-term security and that does not impose limits on the adversary's computational power.

Asymptotic design and analysis of multistage detectors with unequal powers

Abstract: In this work we provide equations to precisely calculate the asymptotic weighting of multistage detectors satisfying the individually and jointly LMMSE criteria in the projection subspace for scenarios with unequal powers Additionally, a general expression of the SINR achievable at the filter output as system size grows large is derived Such an equation can be applied to any multistage detector We specialize this result to both the individually and jointly LMMSE multistage detector with asymptotic weighting We show that the individually LMMSE detector outperforms the other detector in the case of unequal received powers while both the detectors are equivalent in the case of equal received powers

Universal discrete denoising

Abstract: We propose a discrete denoising algorithm, that, based on the observation of the output of a known discrete memoryless channel (DMC), estimates the input sequence to minimize a given fidelity criterion The algorithm is universal in the sense that it requires no knowledge of the input sequence or its statistical properties Yet, asymptotically it performs as well as the optimum denoiser that knows the input sequence distribution The proposed denoising algorithm is practical, and can be implemented in O(n log n) time and O(n/sup 2/3/ log n) storage complexity Extensions to the case of delay-constrained denoising, and to the case of channel uncertainty, are briefly discussed

Multi-covering radii of codes with rank metric

Abstract: Properties of multi-covering radii for codes with rank distance (RD) are investigated and certain bounds are established

On the capacity loss due to separation of detection and decoding in large CDMA systems

Abstract: The performance loss due to separation of detection and decoding on the binary-input Gaussian CDMA channel is calculated in the large system limit. It is shown that a previous result found for the Gaussian input alphabet holds also for the binary input alphabet.

The secret key capacity for multiple terminals

Abstract: We consider the problem of characterizing the secret key (SK)-capacity for an arbitrary number of terminals, each of which observes a distinct component of a discrete memoryless multiple source, with unconstrained public communication allowed between these terminals. Our main contribution is the determination of SK-capacity for an arbitrary subset of terminals with the remaining terminals serving as "helpers," when an eavesdropper observes the communication between the terminals but does not have access to any other information. We also determine the private key (PK)-capacity when the eavesdropper additionally wiretaps some of the helper terminals from which too the key must then be concealed.

The throughput of an LDPC-based incremental-redundancy scheme over block-fading channels

Abstract: INcremental-Redundancy (INR) is a form of hybrid ARQ where the receiver asks the transmitter for additional parity bits when decoding is not successful. This technique is particularly useful in time-selective fading channels, since it implements variable-rate adaptive transmission with a very simple feedback binary channel, where the feedback messages are positive or negative acknowledgments (ACK and NACK, respectively). In this work an information-theoretic analysis of the achievable throughput and delay of INR over block-fading channels for random binary codes and for low-density parity-check (LDPC) binary linear codes.

LDPC code design for turbo equalization

Abstract: We discuss techniques to characterize the probability density function of the extrinsic information at the output of a soft interference canceler based equalizer when used in a turbo equalizer Then, we show how to use this to compute thresholds for low density parity check (LDPC) codes and to design good LDPC code ensembles for static and time-varying intersymbol interference channels For other types of equalizers, we propose to design LDPC codes whose extrinsic information transfer (EXIT) diagram is matched to that of the equalizer

Full-rank, full-rate STBCs from division algebras

Abstract: Construction of rate-optimal full-diversity space-time block codes (STBC) over symmetric PSK signal sets using cyclotomic field extensions of the field of rational /spl Qopf/ was reported previously, and for a variety of other signal sets using non-cyclotomic field extensions. Fields are commutative division algebras. Construction of full-rate STBCs with full-diversity using a class of non-commutative division algebras (cyclic division algebras) was also reported previously and the Alamouti code was shown to be a special case with an algebraic uniqueness property. In this paper, we present the basic principle behind these constructions and also obtain full-diversity, full-rate STBCs using a different class of non-commutaive division algebra constructed by Brauer. The interrelationship between codes constructed from division algebras, linear dispersion codes and codes from orthogonal designs is discussed.

Optimal multiresolution quantization for scalable multimedia coding

Abstract: We have investigated the problem of designing an optimal entropy-constrained multiresolution scalar quantizer under the criterion of minimizing the expected distortion weighted by the probability of transmission rate and for arbitrary probability mass function of signal amplitude An O(LN/sup 3/) algorithm is proposed to solve the optimization problem, where L is the number of refinement stages, and N is the size of input symbol alphabet The proposed algorithm is globally optimal, and furthermore, it is more general than its locally optimal predecessors in terms of quantizer structures

T-Direct codes: an application to T-user BAC

Abstract: A T-Direct code is defined as the set of T F-ary linear codes /spl Gamma//sub 1/, /spl Gamma//sub 2/, , /spl Gamma//sub T/ such that /spl Gamma//sub i/ /spl cap/ /spl Gamma//sub i//sup /spl perp//={0} for each i, where /spl Gamma//sub i//sup /spl perp//= /spl Gamma//sub 1/ /spl oplus/ /spl Gamma//sub 2/ /spl oplus//spl middot//spl middot//spl middot//spl oplus/ /spl Gamma//sub i-1/ /spl oplus/ /spl Gamma//sub i+1/ /spl oplus//spl middot//spl middot//spl middot//spl oplus/ /spl Gamma//sub T/ is the dual of /spl Gamma//sub i/ with respect to the direct sum /spl Lambda/= /spl Gamma//sub 1/ /spl oplus/ /spl Gamma//sub 2/ /spl oplus//spl middot//spl middot//spl middot//spl oplus/ /spl Gamma//sub T/ An algebraic characterization to this class of codes is given An application of T-Direct codes to the noiseless T-user binary adder channel is also presented

4Approximate realization of hidden Markov chains

Abstract: In this paper we consider the approximate realization problem for finite valued hidden Markov models ie stochastic processes Y=f(X) where X is a finite state Markov chain and f a many-to-one function Given the laws p/sub Y/(/spl middot/) of Y the weak realization problem consists in finding a Markov chain X and a function f such that, at least distributionally, Y/spl sim/f(X) The approximate realization problem consists in finding X and f such that Y and f (X) are close The approximation criterion we use is the informational divergence between properly defined nonnegative (componentwise) matrices related to the processes To construct the realization we apply recent results on the approximate factorization of nonnegative matrices

On the correlations between a combining function and functions of fewer variables

Abstract: The Hamming distance of a Boolean function to the functions having many linear structures is an important cryptographic parameter. Most notably, the accuracy of the approximation of the combining function by a function of fewer variables is a major issue in most attacks against combination generators. Here, we show that the distance of a function to the functions having a k-dimensional linear space is highly related to its nonlinearity. In particular, we prove that there is no accurate approximation of any highly nonlinear function by a function depending on a small subset of its input variables.

Asymptotics of quickest change detection procedures under a Bayesian criterion

Abstract: The optimal detection procedure for detecting changes in independent and identically distributed sequences (i.i.d.) in a Bayesian setting was derived by Shiryaev in the nineteen sixties. However, the analysis of the performance of this procedure in terms of the average detection delay and false alarm probability has been an open problem. In this paper, we investigate the performance of Shiryaev's procedure in an asymptotic setting where the false alarm probability goes to zero. The asymptotic study is performed not only in. the i.d.d. case where the Shiryaev's procedure is optimal but also in a general, non-i.i.d. case. In the latter case, we show that Shiryaev's procedure is asymptotically optimum under mild conditions. We also show that the two popular non-Bayesian detection procedures, namely the Page and Shiryaev-Roberts-Pollak procedures, are not optimal (even asymptotically) under the Bayesian criterion. The results of this study are shown to be especially important in studying the asymptotics of decentralized quickest change detection procedures.

Capacity of multi-antenna channels in the low-power regime

Abstract: In emerging mobile systems users must operate very often in the low-power regime Specifically, almost 40 % of geographical locations experience signal-to-noise ratios (SNR) below 0 dB Despite its relevance, the multi-antenna low-power regime had not been analyzed in depth until the paper by S Verdu (see IEEE Trans on Inform Theory, p1319-43, June 2002), where the figure of merit is not the SNR, but rather the normalized energy per information bit, E/sub b//N/sub 0/ This paper expands these findings using a channel model that realistically describes the conditions found in typical wireless systems The focus is on channels that are known to the receiver, but unknown to the transmitter

Jointly optimal paging and registration for a symmetric random walk

Abstract: Jointly optimal paging and registration policies are identified for a cellular network composed of a linear array of cells Motion is modeled as a random walk with a symmetric, unimodal step size distribution Minimization of the discounted, infinite-horizon average cost is addressed The jointly optimal pair of paging and registration policies is found The optimal registration policy is a distance threshold type: the mobile station registers whenever its distance from the previous reporting point exceeds a-threshold The paging policy is ping-pong type: cells are searched in an order of increasing distance from the cell in which the previous report occurred

The half a bit loss of robust source/channel codebooks

Abstract: We consider the problem of choosing robust codebooks for rate-distortion relative to difference distortion measures, and for capacity-cost relative to additive noise channels. We define an information theoretic formula for the min-max redundancy associated with robust random source codes, and similarly for the loss of robust random channel codes. We then show that this min-max performance can be approached by structured, dithered lattice codebooks. Applications to coding with side information are discussed.

Core capacity region of energy-limited wireless ad hoc networks

Abstract: We define the core capacity region of energy-limited wireless ad hoc networks and show that it is non-empty under the linear utility model. The main motivation for this paper is the question: "Can wireless ad hoc networks grow into a single grand wireless network in the future?" We answer this question by developing a framework in which we incorporate the incentives of users into the definition of network capacity.

Watermarking identification for private and public users: the broadcast channel approach

Abstract: In this work we study the problem of identification via the degraded broadcast channel with random parameters. Inner and outer bounds are derived on the identification capacity region when the random parameters are known non-causally to the encoder and to the non-degraded decoder. Applications to watermarking are suggested. In particular, this model describes the situation when watermarks are supposed to be identified by both private and public users.

Distributed encoding of sensor data

Abstract: In this paper, we ignore transmission issues and focus on the total number of bits to transmit to the collector to form a reconstruction of the field with a given MSE. We assume that all sensors can transmit bits to the collector without error. With this assumption, with total number of bits as the cost measure, and with the style of coding, it can be argued that sensor-to-sensor relaying offers no advantages. This problem is similar to image coding and transmission, except that the quantization, encoding and transmission are constrained to take place separately at each sensor (pixel location), in contrast to traditional image coding and transmission, wherein the entire image is available for quantization, encoding, and transmission. Due to the need to separately encode values from separate sensors, we pursue a Slepian-Wolf style coding approach.

Dynamical systems, curves and coding for continuous alphabet sources

Abstract: Good codes for transmitting a continuous-alphabet source over an AWGN channel can be constructed using simple dynamical systems The trajectories of the dynamical systems that we consider are curves in /spl Ropf//sup N/, and we use these curves as signal sets for a modulation system In this paper we consider the problem of choosing the parameters of the dynamical system such that the length of its trajectory is maximized subject to a constraint on the minimum distance between its "folds" We provide some general results on the construction of such curves and show how to select the parameters optimally in the case N=6 This is done by reducing the problem to one of choosing a vector (1, a, b) in /spl Zopf//sup 3/ for which a high packing density is obtained for the lattice /spl Lambda//sub p/ obtained by projecting /spl Zopf//sup 3/ into the plane orthogonal to (1, a, b) Two approaches are used to prove the central result of the paper