Showing papers in "IEEE Transactions on Information Theory in 1987"

The capacity of the Hopfield associative memory

Abstract: Techniques from coding theory are applied to study rigorously the capacity of the Hopfield associative memory. Such a memory stores n -tuple of \pm 1 's. The components change depending on a hard-limited version of linear functions of all other components. With symmetric connections between components, a stable state is ultimately reached. By building up the connection matrix as a sum-of-outer products of m fundamental memories, one hopes to be able to recover a certain one of the m memories by using an initial n -tuple probe vector less than a Hamming distance n/2 away from the fundamental memory. If m fundamental memories are chosen at random, the maximum asympotic value of m in order that most of the m original memories are exactly recoverable is n/(2 \log n) . With the added restriction that every one of the m fundamental memories be recoverable exactly, m can be no more than n/(4 \log n) asymptotically as n approaches infinity. Extensions are also considered, in particular to capacity under quantization of the outer-product connection matrix. This quantized memory capacity problem is closely related to the capacity of the quantized Gaussian channel.

Trellis-coded modulation with multidimensional constellations

Abstract: Trellis-coded modulation schemes using four-, eight-, or 16 -dimensional constellations have a number of potential advantages over the usual two-dimensional schemes: a smaller constituent two-dimensional constellation, easier tolerance to phase ambiguities, and a better trade-off between complexity and coding gain. A number of such schemes are presented and evaluated. Starting with a variety of multidimensional lattices, we show how to select multidimensional constellations, how to partition them into subsets, how to construct trellis codes using those subsets, and how to map hits to constellation points. Simplifications of the Viterbi decoding algorithm are presented. We conclude that there are multidimensional trellis-coded modulation schemes that perform better for the same complexity than do two-dimensional schemes.

Optimal quantizer design for noisy channels: An approach to combined source - channel coding

Abstract: We present an analysis of the zero-memory quantization of memoryless sources when the quantizer output is to be encoded and transmitted across a noisy channel. Necessary conditions for the joint optimization of the quantizer and the encoder/decoder pair are presented, and an iterative algorithm for obtaining a locally optimum system is developed. The performance of this locally optimal system, obtained for the class of generalized Gaussian distributions and the binary symmetric channel, is compared against the optimum performance theoretically attainable (using rate-distortion theoretic arguments), as well as against the performance of Lloyd-Max quantizers encoded using the natural binary code and the folded binary code. It is shown that this optimal design could result in substantial performance improvements. The performance improvements are more noticeable at high bit rates and for broad-tailed densities.

An updated table of minimum-distance bounds for binary linear codes

Abstract: In 1973 Helgert and Sfinaff published a table of upper and lower bounds on the maximum minimum-distance for binary linear error-correcting codes up to length 127 . This article presents an updated table incorporating numerous improvements that have appeared since then. To simplify the updating task the author has developed a computer program that systematically investigates the consequences of each improvement by applying several well-known general code-construction techniques. This program also made it possible to check the original table. Furthermore, it offers a quick and reliable update service for future improvements.

The capacity region of the discrete memoryless interference channel with strong interference (Corresp.)

Abstract: The capacity region of the discrete memoryless interference channel with strong interference is established.

Network control by Bayesian broadcast

Abstract: A transmission control strategy is described for slotted-ALOHA-type broadcast channels with ternary feedback. At each time slot, each station estimates the probability that n stations are ready to transmit a packet for each n , using Bayes' rule and the observed history of collisions, successful transmissions, and holes (empty slots). A station transmits a packet in a probabilistic manner based on these estimates. This strategy is called Bayesian broadcast. An elegant and very practical strategy--pseudo-Bayesian broadcast--is then derived by approximating the probability estimates with a Poisson distribution with mean u and further simplifying. Each station keeps a copy of u , transmits a packet with probability 1/ u , and then updates u in two steps: For collisions, increment u by (e-2)^{-l}=1.39221 \cdots . For successes and holes, decrement u by 1 . Set u to \max ( u + \hat{\lambda}, 1) , where \hat{\lambda} is an estimate of the arrival rate \lambda of new packets into the system. Simulation results are presented showing that pseudo-Bayesian broadcast performs well in practice, and methods that can be used to prove that certain versions of pseudo-Bayesian broadcast are stable for \lambda are discussed.

New trellis codes based on lattices and cosets

Abstract: A new technique is proposed for constructing trellis codes. which provides an alternative to Ungerboeck's method of "set partitioning." The new codes use a signal constellation consisting of points from an n -dimensional lattice \Lambda , with an equal number of points from each coset of a sublattice \Lambda ' . One part of the input stream drives a generalized convolutional code whose outputs are cosets of \Lambda ' , while the other part selects points from these cosets. Several of the new codes are better than those previously known.

Identification of linear stochastic systems via second- and fourth-order cumulant matching

Abstract: The identification problem for time-invariant single-input single-output linear stochastic systems driven by non-Gaussian white noise is considered The system is not restricted to be minimum phase, and it is allowed to contain all-pass components A least-squares criterion that involves matching the second- and the fourth-order cumulant functions of the noisy observations is proposed Knowledge of the probability distribution of the driving noise is not required An order determination criterion that is a modification of the Akaike information criterion is also proposed Strong consistency of the proposed estimator is proved under certain sufficient conditions Simulation results are presented to illustrate the method

New results in binary multiple descriptions

Abstract: An encoder whose input is a binary equiprobable memoryless source produces one output of rate R_{1} and another of rate R_{2} . Let D_{1}, D_{2}, and D_{0} , respectively, denote the average error frequencies with which the source data can be reproduced on the basis of the encoder output of rate R_{l} only, the encoder output of rate R_{2} only, and both encoder outputs. The two-descriptions problem is to determine the region R of all quintuples (R_{1}, R_{2}, D_{1}, D_{2}, D_{0}) that are achievable in thc usual Shannon sense. Let R(D)=1+D \log_{2} D+(1-D) \log_{2}(1-D) denote the error frequency rate-distortion function of the source. The "no excess rate case" prevails when R_{1} + R_{2} = R(D_{0}) , and the "excess rate case" when R_{1} + R_{2} > R(D_{0}) . Denote the section of R at (R_{1}, R_{2}, D_{0}) by D(R_{1} R_{2}, D_{0}) =\{(D_{1},D_{2}): (R_{1}, R_{2}, D_{1},D_{2},D_{0}) \in R} . In the no excess rate case we show that a portion of the boundary of D(R_{1}, R_{2}, D_{0}) coincides with the curve (\frac{1}{2} + D_{1}-2D_{0})(\frac_{1}_{2} + D_{2}-2D_{0})= \frac{1}{2}(1-2D_{0})^{2} . This curve is an extension of Witsenhausen's hyperbola bound to the case D_{0} > 0 . It follows that the projection of R onto the (D_{1}, D_{2}) -plane at fixed D_{0} consists of all D_{1} \geq D_{0} and D_{2} \geq D_{0} that lie on or above this hyperbola. In the excess rate case we show by counterexample that the achievable region of El Gamal and Cover is not tight.

Using simulated annealing to design good codes

Abstract: Simulated annealing is a computational heuristic for obtaining approximate solutions to combinatorial optimization problems. It is used to construct good source codes, error-correcting codes, and spherical codes. For certain sets of parameters codes that are better than any other known in the literature are found.

Asymptotically convergent modified recursive least-squares with data-dependent updating and forgetting factor for systems with bounded noise

Abstract: Continual updating of estimates required by most recursive estimation schemes often involves redundant usage of information and may result in system instabilities in the presence of bounded output disturbances. An algorithm which eliminates these difficulties is investigated. Based on a set theoretic assumption, the algorithm yields modified least-squares estimates with a forgetting factor. It updates the estimates selectively depending on whether the observed data contain sufficient information. The information evaluation required at each step involves very simple computations. In addition, the parameter estimates are shown to converge asymptotically, at an exponential rate, to a region around the true parameter.

On the performance evaluation of trellis codes

Abstract: Generating function techniques for analyzing the error event and the bit-error probabilities for trellis codes are considered. The conventional state diagram approach for linear codes where the number of states is equal to the number of trellis states cannot be applied directly to arbitrary trellis codes, and instead, a state diagram where the number of states is equal to the square of the number of trellis states must be used. It is shown that for an interesting class of trellis codes a modified generating function can be defined for which the number of states {\em is equal to} the number of trellis states. The class of codes considered includes trellis codes of rate R=(n-1)/n based upon set partitioning whenever the first partition breaks the signal constellation into two subsets which have the same "configuration matrix," i.e., the same ordered set of mutual distances. The complexity of calculating this modified generating function is the same as for the ordinary generating function of a convolutional code with the same number of trellis states. Bounds on the performance of some interesting codes are given based upon this method.

Encoding of correlated observations

Abstract: An important class of engineering problems involves sensing an environment and making estimates based on the phenomena sensed. In the traditional model of this problem, the sensors' observations are available to the estimator without alteration. There is .growing interest in {\em distributed} sensing systems in which several observations are communicated to the estimator over channels of limited capacity. The observations must be separately encoded so that the target can be estimated with minimum distortion. Two questions are addressed for a special case of this problem wherein there are two sensors which observe noisy data and communicate with a single estimator: 1) if the encoder is unlimited in complexity, what communication rates and distortions can be achieved, 2) if the encoder must be a quantizer (a mapping of a single observation sample into a digital output), how can it be designed for good performance? The first question is treated by the techniques of information theory. It is proved that a given combination of rates and distortion is achievable if there exist degraded versions of the observations that satisfy certain formulas. The second question is treated by two approaches. In the first, the outputs of the quantizers undergo a second stage of encoding which exploits their correlation to reduce the output rate. Algorithms which design the second stage are presented and tested. The second approach is based on the {\em distributional distance}, a measure of dissimilarity between two probability distributions. An algorithm to modify a quantizer for increased distributional distance is derived and tested.

The design of joint source and channel trellis waveform coders

Abstract: The generalized Lloyd algorithm is applied to the design of joint source and channel trellis waveform coders to encode discrete-time continuous-amplitude stationary and ergodic sources operating over discrete memoryless noisy channels. Experimental results are provided for independent and autoregressive Gaussian sources, binary symmetric channels, and absolute error and squared error distortion measures. Performance of the joint codes is compared with the tandem combination of a trellis source code and a trellis channel code on the independent Gaussian source using the squared error distortion measure operating over an additive white Gaussian noise channel. It is observed that the jointly optimized codes achieve performance close to or better than that of separately optimized tandem codes of the same constraint length. Performance improvement via a predictive joint source and channel trellis code is demonstrated for the autoregressive Gaussian source using the squared error distortion measure.

Hypothesis testing with multiterminal data compression

Abstract: The multiterminal hypothesis testing H: XY against H: XY is considered where X^{n} (X^{n}) and Y^{n} (Y^{n}) are separately encoded at rates R_{1} and R_{2} , respectively. The problem is to determine the minimum \beta_{n} of the second kind of error probability, under the condition that the first kind of error probability \alpha_{n} \leq \epsilon for a prescribed 0 . A good lower bound \theta_{L}(R_{1}, R_{2}) on the power exponent \theta (R_{1}, R_{2},\epsilon)= \lim \inf_{n \rightarrow \infty}(-1/n \log \beta_{n}) is given and several interesting properties are revealed. The lower bound is tighter than that of Ahlswede and Csiszar. Furthermore, in the special case of testing against independence, this bound turns out to coincide with that given by them. The main arguments are devoted to the special case with R_{2} = \infty corresponding to full side information for Y^{n}(Y^{n}) . In particular, the compact solution is established to the complete data compression cases, which are useful in statistics from the practical point of view.

Ultimate instability of exponential back-off protocol for acknowledgment-based transmission control of random access communication channels

Abstract: When several users simultaneously transmit over a shared communication channel, the messages are lost and must be retransmitted later. Various protocols specifying when to retransmit have been proposed and studied in recent years. One protocol is "binary exponential back-off," used in the local area network Ethernet. A mathematical model with several idealizations (discrete time slots, infinite users, no deletions) is shown to be unstable in that the asymptotic rate of successful transmissions is zero, however small the arrival rate.

Feedback can at most double Gaussian multiple access channel capacity (Corresp.)

Abstract: The converse for the discrete memoryless multiple access channel is generalized and is used to derive strong bounds on the total capacity (sum of the rates of all the senders) of an m -user Gaussian multiple access channel in terms of the input covariance matrix. These bounds are used to show that the total capacity of the channel with feedback is less than twice the total capacity without feedback. The converse for the general multiple access channel is also used to show that for any m -user multiple access channel, feedback cannot increase the total capacity by more than a factor of m .

A list-type reduced-constraint generalization of the Viterbi algorithm

Abstract: The Viterbi algorithm (VA), an optimum decoding rule for a Q -ary trellis code of constraint length K , operates by taking the best survivor from each of Q^{K-1} lists of candidates at each decoding step. A generalized VA (GVA) is proposed that makes comparisons on the basis of a label of length L(L\leq K) . It selects, incorporating the notion of list decoding, the S best survivors from each of Q^{L-1} lists of candidates at each decoding step. Coding theorems for a discrete memoryless channel are proved for GVA decoding and shown to be natural generalizations of those for VA decoding. An example of intersymbol interference removal is given to illustrate the practical benefits that the GVA can provide.

Robust transmission of unbounded strings using Fibonacci representations

Abstract: Families of Fibonacci codes and Fibonacci representations are defined. Their main attributes are robustness, manifesting itself by the local containment of errors; and simple encoding and decoding. The main application explored is the transmission of binary strings in which the length is in an unknown range, using robust Fibonacci representations instead of the conventional error-sensitive logarithmic ramp representation. Though the former is asymptotically longer than the latter, the former is actually shorter for very large initial segments of integers.

Interval and recency rank source coding: Two on-line adaptive variable-length schemes

Abstract: In the schemes presented the encoder maps each message into a codeword in a prefix-free codeword set. In interval encoding the codeword is indexed by the interval since the last previous occurrence of that message, and the codeword set must be countably infinite. In recency rank encoding the codeword is indexed by the number of distinct messages in that interval, and there must be no fewer codewords than messages. The decoder decodes each codeword on receipt. Users need not know message probabilities, but must agree on indexings, of the codeword set in an order of increasing length and of the message set in some arbitrary order. The average codeword length over a communications bout is never much larger than the value for an off-line scheme which maps the j th most frequent message in the bout into the j th shortest codeword in the given set, and is never too much larger than the value for off-line Huffman encoding of messages into the best codeword set for the bout message frequencies. Both schemes can do much better than Huffman coding when successive selections of each message type cluster much more than in the independent case.

Algebraic decoding of the (23,12,7) Golay code (Corresp.)

Abstract: It is shown that the algebraic method for decoding three-error-correcting BCH codes is also applicable to complete decoding of the (23,12,7) Golay code.

On the equivalence between Berlekamp's and Euclid's algorithms (Corresp.)

Abstract: It is shown that Berlekamp's iterative algorithm can be derived from a normalized version of Euclid's extended algorithm. Simple proofs of the results given recently by Cheng are also presented.

A new distributed algorithm to find breadth first search trees

Abstract: A new distributed algorithm is presented for constructing breadth first search (BFS) trees. A BFS tree is a tree of shortest paths from a given root node to all other nodes of a network under the assumption of unit edge weights; such trees provide useful building blocks for a number of routing and control functions in communication networks. The order of communication complexity for the new algorithm is O(V^{1.6} + E) where V is the number of nodes and E the number of edges. For dense networks with E \geq V^{1.6} this order of complexity is optimum.

A dichotomy of functions F(X, Y) of correlated sources (X, Y)

Abstract: Consider separate encoding of correlated sources X^{n}=(X_{l}, \cdots ,X_{n}), Y^{n} = (Y_{l}, \cdots ,Y_{n}) for the decoder to reliably reproduce a function \{F(X_{i}, Y_{i})\}^{n}_{i=1} . We establish the necessary and sufficient condition for the set of all achievable rates to coincide with the Slepian-Wolf region whenever the probability density p(x,y) is positive for all (x,y) .

The forwarding index of communication networks

Abstract: A network is defined as an undirected graph and a routing which consists of a collection of simple paths connecting every pair of vertices in the graph. The forwarding index of a network is the maximum number of paths passing through any vertex in the graph. Thus it corresponds to the maximum amount of forwarding done by any node in a communication network with a fixed routing. For a given number of vertices, each having a given degree constraint, we consider the problem of finding networks that minimize the forwarding index. Forwarding indexes are calculated' for cube networks and generalized de Bruijn networks. General bounds are derived which show that de Bruijn networks are asymptotically optimal. Finally, efficient techniques for building large networks with small forwarding indexes out of given component networks are presented and analyzed.

Remarks on codes from Hermitian curves (Corresp.)

Abstract: Parameters and generator matrices are given for the codes obtained by applying Goppa's algebraic-geometric construction method to Hermitian curves in PG (2,q) , where q = 2^{2s} for some s\in {\bf N} . Automorphisms of these codes are also discussed, and some results on self-duality and weak self-duality are given.

Conditional limit theorems under Markov conditioning

Abstract: Let X_{1},X_{2},\cdots be independent identically distributed random variables taking values in a finite set X and consider the conditional joint distribution of the first m elements of the sample X_{1},\cdots , X_{n} on the condition that X_{1}=x_{1} and the sliding block sample average of a function h(\cdot , \cdot) defined on X^{2} exceeds a threshold \alpha > Eh(X_{1}, X_{2}) . For m fixed and n \rightarrow \infty , this conditional joint distribution is shown to converge m the m -step joint distribution of a Markov chain started in x_{1} which is closest to X_{l}, X_{2}, \cdots in Kullback-Leibler information divergence among all Markov chains whose two-dimensional stationary distribution P(\cdot , \cdot) satisfies \sum P(x, y)h(x, y)\geq \alpha , provided some distribution P on X_{2} having equal marginals does satisfy this constraint with strict inequality. Similar conditional limit theorems are obtained when X_{1}, X_{2},\cdots is an arbitrary finite-order Markov chain and more general conditioning is allowed.

Gaussian arbitrarily varying channels

Abstract: The {\em arbitrarily varying channel} (AVC) can be interpreted as a model of a channel jammed by an intelligent and unpredictable adversary. We investigate the asymptotic reliability of optimal random block codes on Gaussian arbitrarily varying channels (GAVC's). A GAVC is a discrete-time memoryless Gaussian channel with input power constraint P_{T} and noise power N_{e} , which is further corrupted by an additive "jamming signal." The statistics of this signal are unknown and may be arbitrary, except that they are subject to a power constraint P_{J} . We distinguish between two types of power constraints: {\em peak} and {\em average.} For peak constraints on the input power and the jamming power we show that the GAVC has a random coding capacity. For the remaining cases in which either the transmitter or the jammer or both are subject to average power constraints, no capacities exist and only \lambda -capacities are found. The asymptotic error probability suffered by optimal random codes in these cases is determined. Our results suggest that if the jammer is subject only to an average power constraint, reliable communication is impossible at any positive code rate.

Products of linear recurring sequences with maximum complexity

Abstract: Conditions are derived which guarantee that products of linear recurring sequences attain maximum linear complexity. It is shown that the product of any number of maximum-length GF (q) sequences has maximum linear complexity, provided only the degrees of the corresponding minimal polynomials are distinct and greater than two. It is also shown that if the roots of any number of (not necessarily irreducible) minimal polynomials are simple and lie in extension fields of pairwise relatively prime degrees, then the product of the corresponding GF (q) sequences attains maximum linear complexity, provided only that no two roots of any minimal polynomial are linearly dependent over the groundfield GF (q) (which is automatically satisfied when q = 2) . The results obtained for products are extended to arbitrary linear combinations of product sequences.

Broadcast channels with arbitrarily correlated sources

Abstract: A new coding theorem for the broadcast channel with arbitrarily correlated sources is presented. The result covers all the previously established coding techniques. In particular, it includes Marton's coding theorem as a properly special case.