scispace - formally typeset
Search or ask a question

Showing papers in "IEEE Transactions on Information Theory in 1991"


Journal ArticleDOI
J. Lin1
TL;DR: A novel class of information-theoretic divergence measures based on the Shannon entropy is introduced, which do not require the condition of absolute continuity to be satisfied by the probability distributions involved and are established in terms of bounds.
Abstract: A novel class of information-theoretic divergence measures based on the Shannon entropy is introduced. Unlike the well-known Kullback divergences, the new measures do not require the condition of absolute continuity to be satisfied by the probability distributions involved. More importantly, their close relationship with the variational distance and the probability of misclassification error are established in terms of bounds. These bounds are crucial in many applications of divergence measures. The measures are also well characterized by the properties of nonnegativity, finiteness, semiboundedness, and boundedness. >

4,113 citations


Journal ArticleDOI
TL;DR: A calculus is developed for obtaining bounds on delay and buffering requirements in a communication network operating in a packet switched mode under a fixed routing strategy, and burstiness constraints satisfied by the traffic that exits the element are derived.
Abstract: A calculus is developed for obtaining bounds on delay and buffering requirements in a communication network operating in a packet switched mode under a fixed routing strategy. The theory developed is different from traditional approaches to analyzing delay because the model used to describe the entry of data into the network is nonprobabilistic. It is supposed that the data stream entered into the network by any given user satisfies burstiness constraints. A data stream is said to satisfy a burstiness constraint if the quantity of data from the stream contained in any interval of time is less than a value that depends on the length of the interval. Several network elements are defined that can be used as building blocks to model a wide variety of communication networks. Each type of network element is analyzed by assuming that the traffic entering it satisfies bursting constraints. Under this assumption, bounds are obtained on delay and buffering requirements for the network element; burstiness constraints satisfied by the traffic that exits the element are derived. >

2,049 citations


Journal ArticleDOI
TL;DR: A method to analyze the flow of data in a network consisting of the interconnection of network elements is presented and it is shown how regulator elements connected in series can be used to enforce general burstiness constraints.
Abstract: For pt.I see ibid., vol.37, no.1, p.114-31 (1991). A method to analyze the flow of data in a network consisting of the interconnection of network elements is presented. Assuming the data that enters the network satisfies burstiness constraints, burstiness constraints are derived for traffic flowing between network elements. These derived constraints imply bounds on network delay and buffering requirements. By example, it is shown that the use of regulator elements within the network can reduce maximum network delay. It is also found that such a use of regulator elements can enlarge the throughput region where finite bounds for delay are found. Finally, it is shown how regulator elements connected in series can be used to enforce general burstiness constraints. >

1,007 citations


Journal ArticleDOI
TL;DR: The authors propose the application of a Poisson process model of novelty, which ability to predict novel tokens is evaluated, and it consistently outperforms existing methods and offers a small improvement in the coding efficiency of text compression over the best method previously known.
Abstract: Approaches to the zero-frequency problem in adaptive text compression are discussed. This problem relates to the estimation of the likelihood of a novel event occurring. Although several methods have been used, their suitability has been on empirical evaluation rather than a well-founded model. The authors propose the application of a Poisson process model of novelty. Its ability to predict novel tokens is evaluated, and it consistently outperforms existing methods. It is applied to a practical statistical coding scheme, where a slight modification is required to avoid divergence. The result is a well-founded zero-frequency model that explains observed differences in the performance of existing methods, and offers a small improvement in the coding efficiency of text compression over the best method previously known. >

835 citations


Journal ArticleDOI
TL;DR: The authors focus on the entropy power inequality (including the related Brunn-Minkowski, Young's, and Fisher information inequalities) and address various uncertainty principles and their interrelations.
Abstract: The role of inequalities in information theory is reviewed, and the relationship of these inequalities to inequalities in other branches of mathematics is developed. The simple inequalities for differential entropy are applied to the standard multivariate normal to furnish new and simpler proofs of the major determinant inequalities in classical mathematics. The authors discuss differential entropy inequalities for random subsets of samples. These inequalities when specialized to multivariate normal variables provide the determinant inequalities that are presented. The authors focus on the entropy power inequality (including the related Brunn-Minkowski, Young's, and Fisher information inequalities) and address various uncertainty principles and their interrelations. >

797 citations


Journal ArticleDOI
TL;DR: An alternative projection algorithm is described that reconstructs a signal from azero-crossing representation, which is stabilized by keeping the value of the wavelet transform integral between each pair of consecutive zero-crossings.
Abstract: The completeness, stability, and application to pattern recognition of a multiscale representation based on zero-crossings is discussed. An alternative projection algorithm is described that reconstructs a signal from a zero-crossing representation, which is stabilized by keeping the value of the wavelet transform integral between each pair of consecutive zero-crossings. The reconstruction algorithm has a fast convergence and each iteration requires O(N log/sup 2/ (N)) computation for a signal of N samples. The zero-crossings of a wavelet transform define a representation which is particularly well adapted for solving pattern recognition problems. As an example, the implementation and results of a coarse-to-fine stereo-matching algorithm are described. >

743 citations


Journal ArticleDOI
TL;DR: By viewing the minimum Hamming weight as a certain minimum property of one-dimensional subcodes, a generalized notion of higher-dimensional Hamming weights is obtained, which characterize the code performance on the wire-tap channel of type II.
Abstract: Motivated by cryptographical applications, the algebraic structure, of linear codes from a new perspective is studied. By viewing the minimum Hamming weight as a certain minimum property of one-dimensional subcodes, a generalized notion of higher-dimensional Hamming weights is obtained. These weights characterize the code performance on the wire-tap channel of type II. Basic properties of generalized weights are derived, the values of these weights for well-known classes of codes are determined, and lower bounds on code parameters are obtained. Several open problems are also listed. >

709 citations


Journal ArticleDOI
TL;DR: It is shown that in order to achieve optimal successive refinement the necessary and sufficient conditions are that the solutions of the rate distortion problem can be written as a Markov chain and all finite alphabet signals with Hamming distortion satisfy these requirements.
Abstract: The successive refinement of information consists of first approximating data using a few bits of information, then iteratively improving the approximation as more and more information is supplied. The goal is to achieve an optimal description at each stage. In general, an ongoing description which is rate-distortion optimal whenever it is interrupted is sought. It is shown that in order to achieve optimal successive refinement the necessary and sufficient conditions are that the solutions of the rate distortion problem can be written as a Markov chain. In particular, all finite alphabet signals with Hamming distortion satisfy these requirements. It is also shown that the same is true for Gaussian signals with squared error distortion and for Laplacian signals with absolute error distortion. A simple counterexample with absolute error distortion and a symmetric source distribution which shows that successive refinement is not always achievable is presented. >

600 citations


Journal ArticleDOI
TL;DR: An index of resolvability is proved to bound the rate of convergence of minimum complexity density estimators as well as the information-theoretic redundancy of the corresponding total description length to demonstrate the statistical effectiveness of the minimum description-length principle as a method of inference.
Abstract: The authors introduce an index of resolvability that is proved to bound the rate of convergence of minimum complexity density estimators as well as the information-theoretic redundancy of the corresponding total description length. The results on the index of resolvability demonstrate the statistical effectiveness of the minimum description-length principle as a method of inference. The minimum complexity estimator converges to true density nearly as fast as an estimator based on prior knowledge of the true subclass of densities. Interpretations and basic properties of minimum complexity estimators are discussed. Some regression and classification problems that can be examined from the minimum description-length framework are considered. >

549 citations


Journal ArticleDOI
TL;DR: A mu -(n*n,k) array code C over a field F is a k-dimensional linear space of n*n matrices over F such that every nonzero matrix in C has rank >or= mu.
Abstract: A mu -(n*n,k) array code C over a field F is a k-dimensional linear space of n*n matrices over F such that every nonzero matrix in C has rank >or= mu . It is first shown that the dimension of such array codes must satisfy the Singleton-like bound k >

507 citations


Journal ArticleDOI
TL;DR: Most known good classes of signal space codes are shown to be generalized coset codes, and therefore geometrically uniform, including lattice-type trellis codes based on lattice partitions Lambda / Lambda ' such that Z/sup N// Lambda/ Lambda '/4Z/Sup N/ is a lattice partition chain.
Abstract: A signal space code C is defined as geometrically uniform if, for any two code sequences in C, there exists an isometry that maps one sequence into the other while leaving the code C invariant. Geometrical uniformity, a strong kind of symmetry, implies such properties as a) the distance profiles from code sequences in C to all other code sequences are all the same, and b) all Voronoi regions of code sequences in C have the same shape. It is stronger than Ungerboeck Zehavi-Wolf symmetry or Calderbank-Sloane regularity. Nonetheless, most known good classes of signal space codes are shown to be generalized coset codes, and therefore geometrically uniform, including (a) lattice-type trellis codes based on lattice partitions Lambda / Lambda ' such that Z/sup N// Lambda / Lambda '/4Z/sup N/ is a lattice partition chain, and (b) phase-shift-keying (PSK)-type trellis codes based on up to four-way partitions of a 2/sup n/-PSK signal set. >

Journal ArticleDOI
TL;DR: A novel new integral transform that is adapted for signals of this type is introduced and used to derive estimation and classification algorithms that are simple to implement and that exhibit good performance.
Abstract: The measurement of the parameters of complex signals with constant amplitude and polynomial phase, measured in additive noise, is considered. A novel new integral transform that is adapted for signals of this type is introduced. This transform is used to derive estimation and classification algorithms that are simple to implement and that exhibit good performance. The algorithms are extended to constant amplitude and continuous nonpolynomial phase signals. >

Journal ArticleDOI
TL;DR: It is demonstrated that for very noisy channels and a heavily correlated source, when the code book size is large, the number of encoding regions is considerably smaller than the codebook size-implying a reduction in encoding complexity.
Abstract: The performance and complexity of channel-optimized vector quantizers are studied for the Gauss-Markov source. Observations on the geometric structure of these quantizers are made, which have an important implication on the encoding complexity. For the squared-error distortion measure, it is shown that an operation equivalent to a Euclidean distance measurement with respect to an appropriately defined set of points (used to identify the encoding regions) can be used to perform the encoding. This implies that the encoding complexity is proportional to the number of encoding regions. It is then demonstrated that for very noisy channels and a heavily correlated source, when the codebook size is large, the number of encoding regions is considerably smaller than the codebook size-implying a reduction in encoding complexity. >

Journal ArticleDOI
TL;DR: The well-known Baum-Eagon inequality provides an effective iterative scheme for finding a local maximum for homogeneous polynomials with positive coefficients over a domain of probability values.
Abstract: The well-known Baum-Eagon inequality (1967) provides an effective iterative scheme for finding a local maximum for homogeneous polynomials with positive coefficients over a domain of probability values. However, in many applications the goal is to maximize a general rational function. In view of this, the Baum-Eagon inequality is extended to rational functions. Some of the applications of this inequality to statistical estimation problems are briefly described. >

Journal ArticleDOI
Raymond W. Yeung1
TL;DR: The author presents a new approach to understanding the underlying mathematical structure of Shannon's information measures, which provides answers to the following two questions: for any finite number of random variables and for any information-theoretic identity via the formal substitution of symbols.
Abstract: The author presents a new approach to understanding the underlying mathematical structure of Shannon's information measures, which provides answers to the following two questions for any finite number of random variables. (1) For any information-theoretic identity, is there a corresponding set-theoretic identity via the formal substitution of symbols? (2) For any set-theoretic identity, is there a corresponding information-theoretic identity and, if so, in what sense? The author establishes the analogy between information theory and set theory. Therefore, each information-theoretic operation can formally be viewed as a set-theoretic operation and vice versa. This point of view, which the author believes is of fundamental importance has apparently been overlooked in the past by information theorists. As a consequence the I-diagram, which is a geometrical representation of the relationship among the information measures, is introduced. The I-diagram is analogous to the Venn diagram in set theory. The use of the I-diagram is discussed. >

Journal ArticleDOI
TL;DR: This algorithm is restated using the generalized Breiman, Friedman, Olshen, and Stone (BFOS) algorithm for both cases of convex and nonconvex quantizer functions (QFs), and its complexity is analyzed.
Abstract: P.H. Westerink et al. (1988) developed an optimal bit allocation algorithm that is simplified when all operational distortion-rate functions (which are referred to as quantizer functions) are convex. This algorithm is restated using the generalized Breiman, Friedman, Olshen, and Stone (BFOS) algorithm (a recently developed technique for variable rate vector quantizer design) for both cases of convex and nonconvex quantizer functions (QFs), and its complexity is analyzed. The use of the generalized BFOS algorithm for optimal bit allocation is analysed. It is shown that if each source has a convex quantizer function then the complexity of the algorithm is low. >

Journal ArticleDOI
TL;DR: The relative minimum distance d/sub min//n of q-ary repeated-root cyclic codes of rate r>or=R is proven to tend to zero as the largest multiplicity of a root of the generator g(x) increases to infinity.
Abstract: A parity-check matrix for a q-ary repeated-root cyclic code is derived using the Hasse derivative. Then the minimum distance of a q-ary repeated-root cyclic code is expressed in terms of the minimum distance of a certain simple-root cyclic code. With the help of this result, several binary repeated-root cyclic codes of lengths up to n=62 are shown to contain the largest known number of codewords for their given length and minimum distance. The relative minimum distance d/sub min//n of q-ary repeated-root cyclic codes of rate r>or=R is proven to tend to zero as the largest multiplicity of a root of the generator g(x) increases to infinity. It is further shown that repeated-root cycle codes cannot be asymptotically better than simple-root cyclic codes. >

Journal ArticleDOI
TL;DR: A generalization of the Berlekamp-Massey algorithm is presented for synthesizing minimum length linear feedback shift registers for generating prescribed multiple sequences and conditions for guaranteeing that the connection polynomial of the shortestlinear feedback shift register obtained by the algorithm will be the error-locator polynometric are determined.
Abstract: A generalization of the Berlekamp-Massey algorithm is presented for synthesizing minimum length linear feedback shift registers for generating prescribed multiple sequences. A more general problem is first considered, that of finding the smallest initial set of linearly dependent columns in a matrix over an arbitrary field, which includes the multisequence problem as a special case. A simple iterative algorithm, the fundamental iterative algorithm (FIA), is presented for solving this problem. The generalized algorithm is then derived through a refinement of the FIA. Application of this generalized algorithm to decoding cyclic codes up to the Hartmann-Tzeng (HT) bound and Roos bound making use of multiple syndrome sequences is considered. Conditions for guaranteeing that the connection polynomial of the shortest linear feedback shift register obtained by the algorithm will be the error-locator polynomial are determined with respect to decoding up to the HT bound and special cases of the Roos bound. >

Journal ArticleDOI
TL;DR: The authors show that a jammer who can change a fixed fraction p > indicates that the maximum rate of (n,e,L) codes, which correct all sets of e or fewer errors in a block of n bits under list-of-L decoding, is limited.
Abstract: In the list-of-L decoding of a block code the receiver of a noisy sequence lists L possible transmitted messages, and is in error only if the correct message is not on the list. Consideration is given to (n,e,L) codes, which correct all sets of e or fewer errors in a block of n bits under list-of-L decoding. New geometric relations between the number of errors corrected under list-of-1 decoding and the (larger) number corrected under list-of-L decoding of the same code lead to new lower bounds on the maximum rate of (n,e,L) codes. They show that a jammer who can change a fixed fraction p >

Journal ArticleDOI
A.B. Sorensen1
TL;DR: A class of codes in the Reed-muller family, the projective Reed-Muller codes (PRM codes), is studied and the duals are characterized, and the cyclic properties are studied.
Abstract: A class of codes in the Reed-Muller family, the projective Reed-Muller codes (PRM codes), is studied. The author defines the PRM codes of all orders and discusses their relation to polynomial codes. The exact parameters of PRM codes are given. The duals are characterized, and, in parallel to the classical works on generalized Reed-Muller codes, the cyclic properties are studied. Tables over parameters of the codes are given. >

Journal ArticleDOI
TL;DR: The present design of a family of p/sup n/ p-phase sequences is asymptotically optimum with respect to its correlation properties, and, in comparison with many previous nonbinary designs, the present design has the additional advantage of not requiring an alphabet of size larger than three.
Abstract: For the case where p is an odd prime, n>or=2 is an integer, and omega is a complex primitive pth root of unity, a construction is presented for a family of p/sup n/ p-phase sequences (symbols of the form omega /sup i/), where each sequence has length p/sup n/-1, and where the maximum nontrivial correlation value C/sub max/ does not exceed 1+ square root p/sup n/. A complete distribution of correlation values is provided. As a special case of this construction, a previous construction due to Sidelnikov (1971) is obtained. The family of sequences is asymptotically optimum with respect to its correlation properties, and, in comparison with many previous nonbinary designs, the present design has the additional advantage of not requiring an alphabet of size larger than three. The new sequences are suitable for achieving code-division multiple access and are easily implemented using shift registers. They wee discovered through an application of Deligne's bound (1974) on exponential sums of the Weil type in, several variables. The sequences are also shown to have strong identification with certain bent functions. >

Journal ArticleDOI
J.H. van Lint1
TL;DR: It is demonstrated that a binary cyclic code of length 2n (n odd) can be obtained from two cyclic codes of length n by the well-known mod u mod u+v mod construction, and the structure theorem generalizes to other characteristics and to other lengths.
Abstract: In the theory of cyclic codes, it is common practice to require that (n,q)=1, where n is the word length and F/sub q/ is the alphabet. It is shown that the even weight subcodes of the shortened binary Hamming codes form a sequence of repeated-root cyclic codes that are optimal. In nearly all other cases, one does not find good cyclic codes by dropping the usual restriction that n and q must be relatively prime. This statement is based on an analysis for lengths up to 100. A theorem shows why this was to be expected, but it also leads to low-complexity decoding methods. This is an advantage, especially for the codes that are not much worse than corresponding codes of odd length. It is demonstrated that a binary cyclic code of length 2n (n odd) can be obtained from two cyclic codes of length n by the well-known mod u mod u+v mod construction. This leads to an infinite sequence of optimal cyclic codes with distance 4. Furthermore, it is shown that low-complexity decoding methods can be used for these codes. The structure theorem generalizes to other characteristics and to other lengths. Some comparisons of the methods using earlier examples are given. >

Journal ArticleDOI
TL;DR: It is shown that any signal set in N-dimensional Euclidean space that is matched to an abstract group is essentially what D. Slepian (1968) called a group code for the Gaussian channel and that any such signal set is equivalent to coded phase modulation with linear codes over Z/sub M/.
Abstract: Recently, linear codes over Z/sub M/ (the ring of integers mod M) have been presented that are matched to M-ary phase modulation. The general problem of matching signal sets to generalized linear algebraic codes is addressed based on these codes. A definition is given for the notion of matching. It is shown that any signal set in N-dimensional Euclidean space that is matched to an abstract group is essentially what D. Slepian (1968) called a group code for the Gaussian channel. If the group is commutative, this further implies that any such signal set is equivalent to coded phase modulation with linear codes over Z/sub M/. Some further results on such signal sets are presented, and the signal sets matched to noncommutative groups and the linear codes over such groups are discussed. >

Journal ArticleDOI
TL;DR: A version of the discrete-time slotted ALOHA protocol operating with finitely many buffered terminals is considered, and it is proven that the closure of the stability region of the protocol is the same as theclosure of the Shannon capacity region ofThe collision channel without feedback.
Abstract: A version of the discrete-time slotted ALOHA protocol operating with finitely many buffered terminals is considered. The stability region is defined to be the set of vectors of arrival rates lambda =( lambda /sub 1/,. . ., lambda /sub M/) for which there exists a vector of transmission probabilities such that the system is stable. It is assumed that arrivals are independent from slot to slot, and the following model for the arrival distribution in a slot is assumed: the total number of arrivals in any slots is geometrically distributed, with the probability that such an arrival is at node i being lambda /sub i/ times ( Sigma /sub k/ lambda /sub k/)/sup -1/, independent of the others. With this arrival model, it is proven that the closure of the stability region of the protocol is the same as the closure of the Shannon capacity region of the collision channel without feedback, as determined by J.L. Massey and P. Mathys (1985). At present it is not clear if this result depends on the choice of arrival distribution. The basic probabilistic observation is that the stationary distribution and certain conditional distributions derived from it have positive correlations for bounded increasing functions. >

Journal ArticleDOI
TL;DR: This study presents the proposed estimator, a two-step iterative estimation technique that is ideally suited for the Class A estimation problem since the observations can be readily treated as incomplete data, and shows that the sequence of estimates obtained via the EM algorithm converges.
Abstract: The Class A Middleton noise model is a commonly used statistical-physical, parametric model for non-Gaussian interference superimposed on a Gaussian background In this study, the problem of efficient estimation of the Class A parameters for small sample sizes is considered The proposed estimator is based on the EM algorithm, a two-step iterative estimation technique that is ideally suited for the Class A estimation problem since the observations can be readily treated as incomplete data For the single-parameter estimation problem, a closed-form expression for the estimator is obtained Furthermore, for the single-parameter estimation problem, it is shown that the sequence of estimates obtained via the EM algorithm converges, and a characterization of the point to which the sequence converges is given In particular, it is shown that if the limit point of this convergent sequence is an interior point of the parameter set of interest, then it must be a stationary point of the traditional likelihood function In addition, for both the single-parameter and two-parameter estimation problems, the small-sample-size performance of the proposed EM algorithm is examined via an extensive simulation study >

Journal ArticleDOI
TL;DR: Trellis-coded quantization is generalized to allow a vector reproduction alphabet and it is shown that for a stationary ergodic vector source, the quantization noise is zero-mean and of a variance equal to the difference between the source variance and the variance of the reproduction sequence.
Abstract: Trellis-coded quantization is generalized to allow a vector reproduction alphabet. Three encoding structures are described, several encoder design rules are presented, and two design algorithms are developed. It is shown that for a stationary ergodic vector source, if the optimized trellis-coded vector quantization reproduction process is jointly stationary and ergodic with the source, then the quantization noise is zero-mean and of a variance equal to the difference between the source variance and the variance of the reproduction sequence. Several examples illustrate the encoder design procedure and performance. >

Journal ArticleDOI
P. Piret1
TL;DR: It is conjectured that the critical ranges of the two-dimensional problems are also different, and the author presents results to substantiate this conjecture.
Abstract: The connectivity of a two-dimensional radio network as a function of the range of the transmitters is still an open problem. It has bee conjectured that the critical range of this problem is the same as the one for the covering problem. The analysis of the one-dimensional problem makes it clear that situations can exist where these two ranges are different. It is conjectured that the critical ranges of the two-dimensional problems are also different. The author presents results to substantiate this conjecture. >

Journal ArticleDOI
TL;DR: The Gaussian arbitrarily varying channel with input constraint Gamma and state constraint Lambda admits input sequences x=(x/sub 1/,---,X/sub n/) of real numbers with Sigma x/sub i//sup 2/ > as discussed by the authors.
Abstract: The Gaussian arbitrarily varying channel with input constraint Gamma and state constraint Lambda admits input sequences x=(x/sub 1/,---,X/sub n/) of real numbers with Sigma x/sub i//sup 2/ >

Journal ArticleDOI
TL;DR: The performance of generalized minimum distance (GMD) decoding is better if the new criterion is used, and the weights used in GMD decoding are generalized to permit each of the possible M symbol values to have a different weight.
Abstract: A novel acceptance criterion that is less stringent than previous criteria is developed. The criterion accepts the codeword that is closest to the received vector for many cases where previous criteria fail to accept any codeword. As a result, the performance of generalized minimum distance (GMD) decoding is better if the new criterion is used. For M-ary signaling, the weights used in GMD decoding are generalized to permit each of the possible M symbol values to have a different weight. >

Journal ArticleDOI
TL;DR: The multilevel technique for combining block coding and modulation is investigated, and a technique is presented for analyzing the error performance of block modulation codes for an additive white Gaussian noise channel based on soft-decision maximum likelihood decoding.
Abstract: The multilevel technique for combining block coding and modulation is investigated. A general formulation is presented for multilevel modulation codes in terms of component codes with appropriate distance measures. A specific method for constructing multilevel block modulation codes with interdependency among component codes is proposed. Given a multilevel block modulation code C with no interdependency among the binary component codes, the proposed method gives a multilevel block modulation code C' that has the same rate as C, a minimum squared Euclidean distance not less than that of C, a trellis diagram with the same number of states as that of C, and a smaller number of nearest neighbor codewords than that of C. Finally, a technique is presented for analyzing the error performance of block modulation codes for an additive white Gaussian noise (AWGN) channel based on soft-decision maximum likelihood decoding. Error probabilities of some specific codes are evaluated by simulation and upper bounds based on their Euclidean weight distributions. >