Showing papers in "IEEE Transactions on Information Theory in 1981"
TL;DR: It is shown that choosing a transmission order for the digits that is appropriate for the graph and the subcodes can give the code excellent burst-error correction abilities.
Abstract: A method is described for constructing long error-correcting codes from one or more shorter error-correcting codes, referred to as subcodes, and a bipartite graph. A graph is shown which specifies carefully chosen subsets of the digits of the new codes that must be codewords in one of the shorter subcodes. Lower bounds to the rate and the minimum distance of the new code are derived in terms of the parameters of the graph and the subeodes. Both the encoders and decoders proposed are shown to take advantage of the code's explicit decomposition into subcodes to decompose and simplify the associated computational processes. Bounds on the performance of two specific decoding algorithms are established, and the asymptotic growth of the complexity of decoding for two types of codes and decoders is analyzed. The proposed decoders are able to make effective use of probabilistic information supplied by the channel receiver, e.g., reliability information, without greatly increasing the number of computations required. It is shown that choosing a transmission order for the digits that is appropriate for the graph and the subcodes can give the code excellent burst-error correction abilities. The construction principles
3,246 citations
TL;DR: A new achievable rate region for the general interference channel which extends previous results is presented and evaluated and the capacity of a class of Gaussian interference channels is established.
Abstract: A new achievable rate region for the general interference channel which extends previous results is presented and evaluated. The technique used is a generalization of superposition coding to the multivariable case. A detailed computation for the Gaussian channel case clarifies to what extent the new region improves previous ones. The capacity of a class of Gaussian interference channels is also established.
1,872 citations
TL;DR: The capacity region of a Gaussian interference channel with two separate messages is obtained for the case of moderately strong interference and it is shown that the region coincides with the one where both messages are required in both receiving terminals.
Abstract: The capacity region of a Gaussian interference channel with two separate messages is obtained for the case of moderately strong interference. It is shown that the region coincides with the one where both messages are required in both receiving terminals.
769 citations
TL;DR: A simple and efficient method of simulation is discussed for point processes that are specified by their conditional intensities, based on the thinning algorithm which was introduced recently by Lewis and Shedler for the simulation of nonhomogeneous Poisson processes.
Abstract: A simple and efficient method of simulation is discussed for point processes that are specified by their conditional intensities. The method is based on the thinning algorithm which was introduced recently by Lewis and Shedler for the simulation of nonhomogeneous Poisson processes. Algorithms are given for past dependent point processes containing multivariate processes. The simulations are performed for some parametric conditional intensity functions, and the accuracy of the simulated data is demonstrated by the likelihood ratio test and the minimum Akaike information criterion (AIC) procedure.
750 citations
TL;DR: Universal coding theory is surveyed from the viewpoint of the interplay between delay and redundancy and the price for universality turns out to be acceptably small.
Abstract: Universal coding theory is surveyed from the viewpoint of the interplay between delay and redundancy. The price for universality turns out to be acceptably small.
655 citations
IBM1
TL;DR: A general class of so-called first-in first-out (FIFO) arithmetic codes is described which require no alphabet extension devices and which therefore can be used in conjunction with the best models.
Abstract: The problems arising in the modeling and coding of strings for compression purposes are discussed. The notion of an information source that simplifies and sharpens the traditional one is axiomatized, and adaptive and nonadaptive models are defined. With a measure of complexity assigned to the models, a fundamental theorem is proved which states that models that use any kind of alphabet extension are inferior to the best models using no alphabet extensions at all. A general class of so-called first-in first-out (FIFO) arithmetic codes is described which require no alphabet extension devices and which therefore can be used in conjunction with the best models. Because the coding parameters are the probabilities that define the model, their design is easy, and the application of the code is straightforward even with adaptively changing source models.
519 citations
TL;DR: The principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information) is a general method of inference about an unknown probability density when there exists a prior estimate of the density and new information in the form of constraints on expected values.
Abstract: The principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information) is a general method of inference about an unknown probability density when there exists a prior estimate of the density and new information in the form of constraints on expected values. Various fundamental properties of cross-entropy minimization are proven and collected in one place. Cross-entropy's well-known properties as an information measure are extended and strengthened when one of the densities involved is the result of cross-entropy minimization. The interplay between properties of cross-entropy minimization as an inference procedure and properties of cross-entropy as an information measure is pointed out. Examples are included and general analytic and computational methods of finding minimum cross-entropy probability densities are discussed.
384 citations
TL;DR: This region generally exceeds the achievable rate region without feedback and exceeds the rate point found by Gaarder and Wolf for the binary erasure multiple-access channel with feedback.
Abstract: An achievable rate region R_{1} \leq I(X_{1};Y|X_{2},U), R_{2} \leq I(X_{2}; Y|X_{1},U), R_{1}+R_{2} \leq I(X_{1}, X_{2};Y) , where p(u,x_{l},x_{2},y)= p(u)p(x_{l}|u)p(x_{2}|u)p(y|x_{l},x_{2}) , is established for the multiple-access channel with feedback. Time sharing of these achievable rates yields the rate region of this paper. This region generally exceeds the achievable rate region without feedback and exceeds the rate point found by Gaarder and Wolf for the binary erasure multiple-access channel with feedback. The presence of feedback allows the independent transmitters to understand each other's intended transmissions before the receiver has sufficient information to achieve the desired decoding. This allows the transmitters to cooperate in the transmission of information that resolves the residual uncertainty of the receiver. At the same time, independent information from the transmitters is superimposed on the cooperative correction information. The proof involves list codes and block Markov encoding.
309 citations
TL;DR: The stochastic model representing the closed boundary is invariant to transformations like sealing, translation, choice of starting point, and rotation over angles that are multiples of 2\pi/N, where N is the number of observations.
Abstract: The analysis of closed boundaries of arbitrary shapes on a plane is discussed. Specifically, the problems of representation and reconstruction are considered. A one-to-one correspondence between the given closed boundary and a univariate or multivariate sequence of real numbers is set up. Univariate or multivariate circular autoregressive models are suggested for the representation of the sequence of numbers derived from the closed boundary. The stochastic model representing the closed boundary is invariant to transformations like sealing, translation, choice of starting point, and rotation over angles that are multiples of 2\pi/N , where N is the number of observations. Methods for estimating the unknown parameters of the model are given and a decision rule for choosing the appropriate number of coefficients is included. Constraints on the estimates are derived so that the estimates are invariant to the transformations of the boundaries. The stochastic model enables the reconstruction of a dosed boundary using FFT algorithms. Results of simulations are included and the application to contour coding is discussed.
254 citations
TL;DR: A local distance measure is shown to optimize the performance of the nearest neighbor two-class classifier for a finite number of samples using the difference between the finite sample error and the asymptotic error as the criterion of improvement.
Abstract: A local distance measure is shown to optimize the performance of the nearest neighbor two-class classifier for a finite number of samples. The difference between the finite sample error and the asymptotic error is used as the criterion of improvement. This new distance measure is compared to the well-known Euclidean distance. An algorithm for practical implementation is introduced. This algorithm is shown to be computationally competitive with the present nearest neighbor procedures and is illustrated experimentally. A closed form for the corresponding second-order moment of this criterion is found. Finally, the above results are extended to
251 citations
TL;DR: A new and simple method is proposed for finding good encoders both for channels and for sources with side information, which relies on the continuous version of a graph decomposition result of Lovasz.
Abstract: A new and simple method is proposed for finding good encoders both for channels and for sources with side information. This method relies on the continuous version of a graph decomposition result of Lovasz. The presently known best exponential error bounds for both problems follow in a unified manner with an improvement on the source coding bound. The previous bounds for universal codes of the authors and Marton are also improved.
TL;DR: The following theorem is offered, which states that the conditional distribution of a given random variable X is the (normalized) product of the maximum entropy distribution and the initial distribution.
Abstract: It is well-known that maximum entropy distributions, subject to appropriate moment constraints, arise in physics and mathematics. In an attempt to find a physical reason for the appearance of maximum entropy distributions, the following theorem is offered. The conditional distribution of X_{l} given the empirical observation (1/n)\sum^{n}_{i}=_{l}h(X_{i})=\alpha , where X_{1},X_{2}, \cdots are independent identically distributed random variables with common density g converges to f_{\lambda}(x)=e^{\lambda^{t}h(X)}g(x) (Suitably normalized), where \lambda is chosen to satisfy \int f_{lambda}(x)h(x)dx= \alpha . Thus the conditional distribution of a given random variable X is the (normalized) product of the maximum entropy distribution and the initial distribution. This distribution is the maximum entropy distribution when g is uniform. The proof of this and related results relies heavily on the work of Zabell and Lanford.
TL;DR: An information theory approach to the theory and practice of linear predictive coded speech compression systems is developed and it is shown that a traditional LPC system can be viewed as a minimum distortion or nearest-neighbor system where the distortion measure is a minimum discrimination information between a speech process model and an observed frame of actual speech.
Abstract: An information theory approach to the theory and practice of linear predictive coded (LPC) speech compression systems is developed. It is shown that a traditional LPC system can be viewed as a minimum distortion or nearest-neighbor system where the distortion measure is a minimum discrimination information between a speech process model and an observed frame of actual speech. This distortion measure is used in an algorithm for computer-aided design of block source codes subject to a fidelity criterion to obtain a 750-bits/s speech compression system that resembles an LPC system but has a much lower rate, a larger memory requirement, and requires no on-line LPC analysis. Quantitative and informal subjective comparisons are made among our system and LPC systems.
TL;DR: A simple proof using random partitions and typicality is given for Marton's coding theorem for broadcast channels.
Abstract: A simple proof using random partitions and typicality is given for Marton's coding theorem for broadcast channels.
TL;DR: The properties of linear codes over GF (q) that provide unequal error protection (UEP) of information digits are discussed and a design is proposed for optimal binary systematic linear UEP codes.
Abstract: The properties of linear codes over GF (q) that provide unequal error protection (UEP) of information digits are discussed. A design is proposed for optimal binary systematic linear UEP codes. Broad classes of iterative and concatenated UEP codes are constructed. Majority decoding algorithms for linear iterative UEP codes are described.
TL;DR: Even assuming a noiseless model for photon communication for which capacity is infinite, it is unlikely that a signaling efficiency of even 10 nats/photon could be achieved practically, and it is shown that pulse-position modulation plus Reed-Solomon coding yields practical results in the range of 2 to 3 nats-photon.
Abstract: In a recent paper, Pierce studied the problems of communicating at optical frequencies using photon-counting techniques, and concluded that "at low temperatures we encounter insuperable problems of encoding long before we approach [channel capacity]." In this paper it is shown that even assuming a noiseless model for photon communication for which capacity (measured in nats/photon) is infinite, it is unlikely that a signaling efficiency of even 10 nats/photon could be achieved practically. On the positive side, it is shown that pulse-position modulation plus Reed-Solomon coding yields practical results in the range of 2 to 3 nats/photon.
TL;DR: The theta functions of a number of important packings and tables of the first fifty or so of their coefficients are given in this paper.
Abstract: The theta function of a sphere packing gives the number of centers at each distance from the origin. The theta functions of a number of important packings ( A_{n},D_{n},E_{n} , the Leech lattice, and others) and tables of the first fifty or so of their coefficients are given in this paper.
TL;DR: An exact and approximate realization theory for estimation and model filters of second-order stationary stochastic sequences is presented, and how the techniques presented constitute a generalization of many aspects of the Levinson-Szego theory of partial realizations is shown.
Abstract: An exact and approximate realization theory for estimation and model filters of second-order stationary stochastic sequences is presented. The properties of J -lossless matrices as a unifying framework are used. Necessary and sufficient conditions for the exact realization of an estimation filter and a model filter as a submatrix of a J -lossless system are deduced. An extension of the so-called Schur algorithm yields an approximate J -lossless realization based on partial past information about the process. The geometric properties of such partial realizations and their convergence are studied. Finally, connections with the Nevanlinna-Pick problem are made, and how the techniques presented constitute a generalization of many aspects of the Levinson-Szego theory of partial realizations is shown. As a consequence generalized recursive formulas for reproducing kernels and Christoffel-Darboux formulas are obtained. In this paper the scalar case is considered. The matrix case will be considered in a separate publication.
TL;DR: The Elias source coding scheme is modified to permit a source sequence of practically unlimited length to be coded as a single codeword using arithmetic of only limited precision, showing the result to be a nonblock arithmetic code of the first in, first out (FIFO) type.
Abstract: The Elias source coding scheme is modified to permit a source sequence of practically unlimited length to be coded as a single codeword using arithmetic of only limited precision. The result is shown to be a nonblock arithmetic code of the first in, first out (FIFO) type-- source symbols are decoded in the same order as they were encoded. Codeword lengths which are near optimum for the specified statistical properties of the source can be achieved. Explicit encoding and decoding algorithms are Provided which effectively implement the coding scheme. Applications to data compression and cryptography are suggested.
TL;DR: Although the problem is formulated in terms of frequencies, the results are applicable to any signaling scheme where M orthogonal signals are used in each signaling interval including time partitioning of the interval.
Abstract: Two specific noiseless multiple-access channel models, the T -user M -frequency multiple-access channel with and without intensity information, are studied in this paper. Information-theoretic bounds on the transmission rate for both models are presented. Constructive coding schemes are given for both channels which achieve zero error probability and whose rate sum is close to the information-theoretic bounds. Although the problem is formulated in terms of frequencies, the results are applicable to any signaling scheme where M orthogonal signals are used in each signaling interval including time partitioning of the interval.
IBM1
TL;DR: An information-theoretic method is used to show that if only finitely many types of conflicts can be distinguished by the protocol, utilization of the channel at rates approaching capacity is impossible.
Abstract: A general model is presented for synchronous protocols that resolve conflicts among message transmissions to a multiple-access broadcast channel An information-theoretic method is used now to show that if only finitely many types of conflicts can be distinguished by the protocol, utilization of the channel at rates approaching capacity is impossible A random-coding argument is used to show that if the number of conflicting transmissions can be determined (which requires distinguishing infinitely many types of conflicts) then utilization of the channel at rates arbitrarily close to capacity can be achieved
TL;DR: The capacity region for the discrete memoryless multiple-access channel without time synchronization at the transmitters and receivers is shown to be the same as the known capacity regionfor the ordinary multiple- access channel.
Abstract: The capacity region for the discrete memoryless multiple-access channel without time synchronization at the transmitters and receivers is shown to be the same as the known capacity region for the ordinary multiple-access channel. The proof utilizes time sharing of two optimal codes for the ordinary multiple-access channel and uses maximum likelihood decoding over shifts of the hypothesized transmitter words.
TL;DR: A new approach to the classical fixed-length noiseless source coding problem is proposed for the case of finite ergodic Markov sources, based on simple counting arguments and the central notion of "Markov type" is introduced.
Abstract: A new approach to the classical fixed-length noiseless source coding problem is proposed for the case of finite ergodic Markov sources. This approach is based on simple counting arguments. The central notion of "Markov type" (a set containing all the source sequences having the same transition counts from letter to letter) is introduced and the cardinality of such a set is evaluated via graph theoretical tools. The error exponent is shown to be a weighted average of informational divergences, and the universal character of the result is stressed. As a corollary, the classical source coding theorem (determining the achievable rates) is rederived.
TL;DR: The capacity of the single-mode photon-counting channel in the region in which quantum effects are most pronounced is high enough to suggest that photon counting be seriously considered for applications such as a deep space to near Earth link.
Abstract: Previous work has shown that the capacity of the single-mode photon-counting channel in the region in which quantum effects are most pronounced is 1/(kT \ln 2) bit/J. Here T is the quantum noise temperature, and k is Boltzmann's constant. However, to achieve this, the number of photons per second must be very low, so this does not determine the capacity of the average power-limited photon channel in bits per second. Here the previous work is extended and the capacity is determined in bits per second. This includes a determination of the optimum counting time. The capacity in bits per second is for low T asymptotic to P_{s}/(kT \ln 2) , where P_{s} is the average signal power. This is high enough to suggest that photon counting be seriously considered for applications such as a deep space to near Earth link. Some related problems are also considered including a peak power limitation.
TL;DR: The outcome of the calculations on decimal sequences of several primes is presented, and the results indicate that the autocorrelation function approximates that obtained for sequences of independent, equally likely random digits.
Abstract: The properties of decimal sequences of rational numbers are investigated with the idea of using them as random sequences and error-correcting codes, and in other applications in communications. Several structural properties of decimal sequences are presented with special attention being given to binary decimal sequences and reciprocals of primes. For a certain class of decimal sequences of l/q, q prime, it is shown that the digits spaced half a period apart add up to r - 1 , where r is the base in which the sequence is expressed. Also for the same class all subsequences of length m , where r^{m} > q , are distinct. These underlying structural properties have made it possible to establish a lower bound on the Hamming distance between a given sequence and its cyclic shifts and also to obtain an upper bound on the autocorrelation function. A condition for the cross correlation of two decimal sequences being zero has also been obtained. The outcome of the calculations on decimal sequences of several primes is presented, and the results indicate that the autocorrelation function approximates that obtained for sequences of independent, equally likely random digits.
TL;DR: A coding theorem is established for a trellis encoding of a stationary and ergodic source over a discrete memoryless noisy channel which shows that such communication systems can perform arbitrarily close to the source distortion-rate function evaluated at the channel capacity.
Abstract: In a trellis encoding communication system the decoder is a time-invariant nonlinear filter with finite memory (sliding-block code), and the encoder is a trellis search algorithm matched to the decoder. A coding theorem is established for a trellis encoding of a stationary and ergodic source over a discrete memoryless noisy channel which shows that such communication systems can perform arbitrarily close to the source distortion-rate function evaluated at the channel capacity.
TL;DR: An instructive counterexample is presented showing that the coding strategy of Cover, El Gamal, and Salehi for multiple access channels with arbitrarily correlated sources is not optimal.
Abstract: An instructive counterexample is presented showing that the coding strategy of Cover, El Gamal, and Salehi for multiple access channels with arbitrarily correlated sources is not optimal. Some new insight into the very difficult problem of determining the true capacity regions is given by this example.
TL;DR: The asymptotic detection performance of the locally optimum detector under non-Gaussian conditions is derived and compared with that for the corresponding detector optimized for operations in Gaussian noise.
Abstract: A locally optimum detector structure is derived for the detection of weak signals in non-Gaussian environments. Optimum performance is obtained by employing a zero-memory nonlinearity prior to the matched filter that would be optimum for detecting the signal were the noise Gaussian. The asymptotic detection performance of the locally optimum detector under non-Gaussian conditions is derived and compared with that for the corresponding detector optimized for operations in Gaussian noise. Numerical results for the asymptotic detection performance are shown for signal detection in noise environments of practical interest.
TL;DR: The capacity region for the arbitrarily varying multiple-access channel is determined, and a region of achievable rate triplets for the general broadcast channel is given including as special cases all capacity regions known until now for broadcast channels.
Abstract: The capacity region for the arbitrarily varying multiple-access channel is determined. A region of achievable rate triplets for the arbitrarily varying general broadcast channel is then given. Finally a region of achievable rate triplets for the general broadcast channel is given including as special cases all capacity regions known until now for broadcast channels.
TL;DR: An algorithm is given to find a prefix condition code that minimizes the value of the moment generating function of the codeword length distribution for a given positive argument.
Abstract: An algorithm is given to find a prefix condition code that minimizes the value of the moment generating function of the codeword length distribution for a given positive argument. This algorithm is used in an iterative way to yield a code that maximizes the rate of decay of the probability of buffer overflow as the buffer length increases.