Showing papers in "IEEE Transactions on Information Theory in 1973"
TL;DR: The minimum number of bits per character R_X and R_Y needed to encode these sequences so that they can be faithfully reproduced under a variety of assumptions regarding the encoders and decoders is determined.
Abstract: Correlated information sequences \cdots ,X_{-1},X_0,X_1, \cdots and \cdots,Y_{-1},Y_0,Y_1, \cdots are generated by repeated independent drawings of a pair of discrete random variables X, Y from a given bivariate distribution P_{XY} (x,y) . We determine the minimum number of bits per character R_X and R_Y needed to encode these sequences so that they can be faithfully reproduced under a variety of assumptions regarding the encoders and decoders. The results, some of which are not at all obvious, are presented as an admissible rate region \mathcal{R} in the R_X - R_Y plane. They generalize a similar and well-known result for a single information sequence, namely R_X \geq H (X) for faithful reproduction.
4,165 citations
TL;DR: This paper generalizes Cover's results on broadcast channels with two binary symmetric channels (BSC) to the class of degraded channels with N components, and shows a procedure to expurgate a good random broadcast code leading to a bound on the maximum probability of error.
Abstract: This paper generalizes Cover's results on broadcast channels with two binary symmetric channels (BSC) to the class of degraded channels with N components. A random code, and its associated decoding scheme, is shown to have expected probability of error going to zero for all components simultaneously as the codeword length goes to infinity, if the point representing the rates to the various receivers falls in the set of achievable rates described by this paper. A procedure to expurgate a good random broadcast code is given, leading to a bound on the maximum probability of error. Binary symmetric broadcast channels always fall in the class of degraded broadcast channels. The results of the paper are applied to this class of channels of potential practical importance.
570 citations
TL;DR: This work provides an explicit scheme for calculating the index of any sequence in S according to its position in the lexicographic ordering of S, thus resulting in a data compression of (log\midS\mid)/n.
Abstract: Let S be a given subset of binary n-sequences. We provide an explicit scheme for calculating the index of any sequence in S according to its position in the lexicographic ordering of S . A simple inverse algorithm is also given. Particularly nice formulas arise when S is the set of all n -sequences of weight k and also when S is the set of all sequences having a given empirical Markov property. Schalkwijk and Lynch have investigated the former case. The envisioned use of this indexing scheme is to transmit or store the index rather than the sequence, thus resulting in a data compression of (\log\midS\mid)/n .
565 citations
TL;DR: This paper considers noiseless coding for sources with unknown parameters, primarily in terms of variable-length coding, with performance measured as a function of the coding redundancy relative to the per-letter conditional source entropy given the unknown parameter.
Abstract: Universal coding is any asymptotically optimum method of block-to-block memoryless source coding for sources with unknown parameters. This paper considers noiseless coding for such sources, primarily in terms of variable-length coding, with performance measured as a function of the coding redundancy relative to the per-letter conditional source entropy given the unknown parameter. It is found that universal (i.e., zero redundancy) coding in a weighted sense is possible if and only if the per-letter average mutual information between the parameter space and the message space is zero. Universal coding is possible in a maximin sense if and only if the channel capacity between the two spaces is zero. Universal coding is possible in a minimax sense if and only if a probability mass function exists, independent of the unknown parameter, for which the relative entropy of the known conditional-probability mass-function is zero. Several examples are given to illustrate the ideas. Particular attention is given to sources that are stationary and ergodic for any fixed parameter although the whole ensemble is not. For such sources, weighted universal codes always exist if the alphabet is finite, or more generally if the entropy is finite. Minimax universal codes result if an additional entropy stability constraint is applied. A discussion of fixed-rate universal coding is also given briefly with performance measured by a probability of error.
399 citations
344 citations
TL;DR: The form of the unit threshold likelihood ratio receiver in the detection of a known deterministic signal in additive sirp noise is shown to be a correlation receiver or a matched filter.
Abstract: The n th-order characteristic functions (cf) of spherically-invariant random processes (sirp) with zero means are defined as cf, which are functions of n th-order quadratic forms of arbitrary positive definite matrices p . Every n th-order spherically-invariant characteristic function (sicf) is represented as a weighted Lebesgue-Stieltjes integral transform of an arbitrary univariate probability distribution function F(\cdot) on [0,\infty) . Furthermore, every n th-order sicf has a corresponding spherically-invariant probability density (sipd). Then we show that every n th-order sicf (or sipd) is a random mixture of a n th-order Gaussian cf [or probability density]. The randomization is performed on
u^2 \rho , where
u is a random variable (tv) specified by the F(\cdot) function. Examples of sirp are given. Relations to previously known results are discussed. Various expectation properties of Gaussian random processes are valid for sirp. Related conditional expectation, mean-square estimation, semMndependence, martingale, and closure properties are given. Finally, the form of the unit threshold likelihood ratio receiver in the detection of a known deterministic signal in additive sirp noise is shown to be a correlation receiver or a matched filter. The associated false-alarm and detection probabilities are expressed in closed forms.
339 citations
TL;DR: A theorem concerning the entropy of a certain sequence of binary random variables is established and this result is applied to the solution of three problems in multi-user communication.
Abstract: In this, the first part of a two-part paper, we establish a theorem concerning the entropy of a certain sequence of binary random variables. In the sequel we will apply this result to the solution of three problems in multi-user communication, two of which have been open for some time. Specifically we show the following. Let X and Y be binary random n -vectors, which are the input and output, respectively, of a binary symmetric channel with "crossover" probability p_0 . Let H\{X\} and H\{ Y\} be the entropies of X and Y , respectively. Then \begin{equation} \begin{split} \frac{1}{n} H\{X\} \geq h(\alpha_0), \qquad 0 \leq \alpha_0 &\leq 1, \Rightarrow \\ \qquad \qquad \&\qquad \frac{1}{n}H\{Y\} \geq h(\alpha_0(1 - p_0) + (1 - \alpha_0)p_0) \end{split} \end{equation} where h(\lambda) = -\lambda \log \lambda - (1 - \lambda) \log(l - \lambda), 0 \leq \lambda \leq 1 .
309 citations
TL;DR: A survey of known results on certain aspects of the level-crossing properties of random processes is presented and provides a basis for further study in the area.
Abstract: In a variety of practical problems involving random processes, it is necessary to have statistical information on their level-crossing properties. This paper presents a survey of known results on certain aspects of this problem and provides a basis for further study in the area. The goal has been to give a broad view of the problems considered in the literature and a brief indication of the techniques used in their solution. Much material of a more or less historical nature has been included since, to the authors' knowledge, no other survey of this nature exists.
302 citations
TL;DR: The Cramer-Rao matrix bound for the vector delay estimate is derived, and used to show that either properly filtered beamformers or properly filtered systems of multiplier-correlators can be used to provide efficient estimates.
Abstract: For the purpose of localizing a distant noisy target, or, conversely, calibrating a receiving array, the time delays defined by the propagation across the array of the target-generated signal wavefronts are estimated in the presence of sensor-to-sensor-independent array self-noise. The Cramer-Rao matrix bound for the vector delay estimate is derived, and used to show that either properly filtered beamformers or properly filtered systems of multiplier-correlators can be used to provide efficient estimates. The effect of suboptimally filtering the array outputs is discussed.
261 citations
TL;DR: It is shown that the optimal receiver measures the photon annihilation operator, which corresponds to optical heterodyning, demonstrating the possible optimality of nonselfadjoint operators and clearly indicates the importance of considering more general quantum measurements in quantum signal detection.
Abstract: A quantum mechanical form of the Cramer-Rao inequality and a minimum-mcan-square-error quantum estimator for multiple parameters are derived, allowing all possible quantum measurements of the received field. The role of nonselfadjoint operators is emphasized in the formulation. Relations of our results to previous work on quantum estimation are discussed. For the estimation of complex mode amplitudes of coherent signals in Gaussian noise, it is shown that the optimal receiver measures the photon annihilation operator, which corresponds to optical heterodyning. This demonstrates the possible optimality of nonselfadjoint operators and clearly indicates the importance of considering more general quantum measurements in quantum signal detection.
248 citations
TL;DR: A new lower bound on the probability of decoding error for the case of rates above capacity is presented, which forms a natural companion to Gallager's random coding bound for rates below capacity.
Abstract: A new lower bound on the probability of decoding error for the case of rates above capacity is presented. It forms a natural companion to Gallager's random coding bound for rates below capacity. The strong converse to the coding theorem follows immediately from the proposed lower bound.
TL;DR: It is shown that the polynomials (x - c)^i, i = 0,1,2,\cdots, have the "weight-retaining" property that any linear combination of these polynmials with coefficients in GF(q) has Hamming weight at least as great as that of the minimum degree polynomial included.
Abstract: For any nonzero element c of a general finite field GF(q) , it is shown that the polynomials (x - c)^i, i = 0,1,2,\cdots , have the "weight-retaining" property that any linear combination of these polynomials with coefficients in GF(q) has Hamming weight at least as great as that of the minimum degree polynomial included. This fundamental property is then used as the key to a variety of code constructions including 1) a simplified derivation of the binary Reed-Muller codes and, for any prime p greater than 2, a new extensive class of p -ary "Reed-Muller codes," 2) a new class of "repeated-root" cyclic codes that are subcodes of the binary Reed-Muller codes and can be very simply instrumented, 3) a new class of constacyclic codes that are subcodes of the p -ary "Reed-Muller codes," 4) two new classes of binary convolutional codes with large "free distance" derived from known binary cyclic codes, 5) two new classes of long constraint length binary convolutional codes derived from 2^r -ary Reed-Solomon codes, and 6) a new class of q -ary "repeated-root" constacyclic codes with an algebraic decoding algorithm.
TL;DR: This work presents some new algorithms that yield the gain matrix for the Kalman filter directly without having to solve separately for the error-covariance matrix and potentially have other computational benefits.
Abstract: Recursive least-squares estimates for processes that can be generated from finite-dimensional linear systems are usually obtained via an n \times n matrix Riccati differential equation, where n is the dimension of the state space. In general, this requires the solution of n(n + 1)/2 simultaneous nonlinear differential equations. For constant parameter systems, we present some new algorithms that in several cases require only the solution of less than 2np or n(m + p) simultaneous nonlinear differential equations, where m and p are the dimensions of the input and observation processes, respectively. These differential equations are said to be of Chandrasekhar type, because they are similar to certain equations introduced in 1948 by the astrophysicist S. Chandrasekhar, to solve finite-interval Wiener-Hopf equations arising in radiative transfer. Our algorithms yield the gain matrix for the Kalman filter directly without having to solve separately for the error-covariance matrix and potentially have other computational benefits. The simple method used to derive them also suggests various extensions, for example, to the solution of nonsymmetric Riccati equations.
TL;DR: An adaptive maximum-likelihood sequence estimator for a digital pulse-amplitude-modulated sequence in the presence of finite-duration unknown slowly time-varying intersymbol interference and additive white Gaussian noise is developed.
Abstract: An adaptive maximum-likelihood sequence estimator for a digital pulse-amplitude-modulated sequence in the presence of finite-duration unknown slowly time-varying intersymbol interference and additive white Gaussian noise is developed. Predicted performance and simulation results for specific channels are given.
TL;DR: In this paper, the authors gave a tabulation of binary convolutional codes with maximum free distance for rates of 1/2, 1/3, and 1/4 for all constraint lengths up to and including nu = 14.
Abstract: This paper gives a tabulation of binary convolutional codes with maximum free distance for rates \frac{1}{2}, \frac{1}{3} , and \frac{1}{4} for all constraint lengths (measured in information digits)
u up to and including nu = 14 . These codes should be of practical interest in connection with Viterbi decoders.
TL;DR: Nonparametric density estimation using the k -nearest-neighbor approach is discussed and a functional form for the optimum k in terms of the sample size, the dimensionality of the observation space, and the underlying probability distribution is obtained.
Abstract: Nonparametric density estimation using the k -nearest-neighbor approach is discussed. By developing a relation between the volume and the coverage of a region, a functional form for the optimum k in terms of the sample size, the dimensionality of the observation space, and the underlying probability distribution is obtained. Within the class of density functions that can be made circularly symmetric by a linear transformation, the optimum matrix for use in a quadratic form metric is obtained. For Gaussian densities this becomes the inverse covariance matrix that is often used without proof of optimality. The close relationship of this approach to that of Parzen estimators is then investigated.
TL;DR: A suboptimum but simple and practical receiver is proposed for digital signaling over channels with a relatively long impulse response that consists of a linear adaptive equalizer in cascade with a maximum-likelihood sequence estimator.
Abstract: Optimum nonlinear receivers for digital signaling over channels with a relatively long impulse response become too complex to be practical. In this paper a suboptimum but simple and practical receiver is proposed for such channels. The receiver consists of a linear adaptive equalizer in cascade with a maximum-likelihood sequence estimator. A general method of finding the error probability of a maximum-likelihood sequence estimator (MLSE) in the presence of correlated noise is developed. An upper bound on the performance of an MLSE is also presented for the case when thc actual channel is different from the channel estimate known to the MLSE. The performance of the proposed receiver is analyzed and results of the analysis and computer simulation are shown for a typical telephone channel.
TL;DR: A new mean-square-error automatic equalizer for synchronous data transmission is developed that utilizes Rosen's gradient-projection method to optimize parameters in the discrete frequency domain.
Abstract: A new mean-square-error automatic equalizer for synchronous data transmission is developed. It utilizes Rosen's gradient-projection method to optimize parameters in the discrete frequency domain. The algorithm converges (in the mean) for any channel even in the presence of noise. It is shown that for the channels considered, convergence is faster (in a bounded sense) than for comparable time-domain equalizers.
TL;DR: A linear correspondence is developed between theStates of a rate- k/n convolutional encoder G and the states of a corresponding syndrome former H^T, where H is an encoder of the code dual to the code generated by G .
Abstract: A linear correspondence is developed between the states of a rate- k/n convolutional encoder G and the states of a corresponding syndrome former H^T , where H is an encoder of the code dual to the code generated by G . This correspondence is used to find an expression for the number of all-zero paths of length \tau in the code trellis; the answer depends only on the constraint lengths of the dual code. A partial answer to the resynchronization problem also falls out of this development.
TL;DR: The standard performance measures of information theory, including the channel capacity C, the rate-distortion function R(D), and the reliability-rate function E(R), are developed from a new and unifying point of view.
Abstract: The standard performance measures of information theory, including the channel capacity C, the rate-distortion function R(D), and the reliability-rate function E(R), are developed from a new and unifying point of view. New such measures are also introduced. The approach originates in a treatment of the simple hypothesis-testing problem by use of the discrimination, a function of two probability distributions q1 and q2 def ined by
TL;DR: It is shown that in this class of window pairs, the window w_o(t) yields spectral estimates with minimum bias, and the variance of the resulting estimate is smaller than that obtained with the known windows of the same size.
Abstract: The class of window pairs w(t) \leftrightarrowW(\\omega) is considered such that w(t) = 0 for \mid t \mid > M and W(\omega) \geq 0 . It is shown that in this class, the window \begin{equation} w_o(t) = \frac{1}{\pi} \biggl| \sin \frac{\pi t}{M} \biggr| + \biggl( 1- \frac{\midt \mid}{M} \biggr) \cos \frac{\pi t}{M}, \qquad \mid t \mid \leq M \end{equation} yields spectral estimates with minimum bias. Furthermore, the variance of the resulting estimate is smaller than that obtained with the known windows of the same size.
TL;DR: A new class of lower bounds to rate-distortion functions of stationary processes with memory and single-letter vector-valued distortion measures is derived, and this class is shown to include or imply all other well-knownLower bounds to rates of such sources and distortion measures.
Abstract: A new class of lower bounds to rate-distortion functions of stationary processes with memory and single-letter vector-valued distortion measures is derived. This class is shown to include or imply all other well-known lower bounds to rates of such sources and distortion measures. The derivation is based on the definition and properties of the conditional rate-distortion function. In addition to providing a unified and intuitive approach to lower bounds, this approach yields several interesting relations among conditional, joint, and marginal rates that are similar to and sometimes identical with the analogous relations among the corresponding entropies.
[...]
TL;DR: This paper proves the four most important properties of Goppa codes, which are known to have actual distances asymptotically equal to their designed distances and unlike primitive BCH codes, have actual minimum distances much greater than twice the number of errors.
Abstract: Goppa described a new class of linear noncyclic error-correcting codes in [1] and [2]. This paper is a summary of Goppa's work, which is not yet available in English. ^1 We prove the four most important properties of Goppa codes. 1) There exist q -ary Goppa codes with lengths and redundancies comparable to BCH codes. For the same redundancy, the Goppa code is typically one digit longer. 2) All Goppa codes have an algebraic decoding algorithm which will correct up to a certain number of errors, comparable to half the designed distance of BCH codes. 3) For binary Goppa codes, the algebraic decoding algorithm assumes a special form. 4) Unlike primitive BCH codes, which are known to have actual distances asymptotically equal to their designed distances, long Goppa codes have actual minimum distances much greater than twice the number of errors, which are guaranteed to be correctable by the algebraic decoding algorithm. In fact, long irreducible Goppa codes asymptotically meet the Gilbert bound.
TL;DR: A method of constructing codes capable of correcting an arbitrary number of asymmetrical errors is given, related to a problem in the additive theory of numbers.
Abstract: A method of constructing codes capable of correcting an arbitrary number of asymmetrical errors is given. The method is related to a problem in the additive theory of numbers.
Journal Article•
TL;DR: The random coding bound of information theory provides a well-known upper bound to the probability of decoding error for the best code of a given rate and block length, which is constructed by upper bounding the average error probability over an ensemble of codes as mentioned in this paper.
Abstract: The random coding bound of information theory provides a well-known upper bound to the probability of decoding error for the best code of a given rate and block length. The bound is constructed by upperbounding the average error probability over an ensemble of codes. The bound is known to give the correct exponential dependence of error probability on block length for transmission rates above the critical rate, but it gives an incorrect exponential dependence at rates below a second lower critical rate. Here we derive an asymptotic expression for the average error probability over the ensemble of codes used in the random coding bound. The result shows that the weakness of the random coding bound at rates below the second critical rate is due not to upperbounding the ensemble average, but rather to the fact that the best codes are much better than the average at low rates.
TL;DR: It is shown that when x has finite range, a distribution maximizing the entropy exists and is unique and a table is given which enables the maximum-entropy distribution to be computed.
Abstract: The entropy H of an absolutely continuous distribution with probability density function p(x) is defined as H = - \int p(x) \log p(x) dx . The formal maximization of H , subject to the moment constraints \int x^r p(x) dx = \mu_r, r = 0,1,\cdots,m , leads to p(x) = \exp (- \sum_{r=0}^m \lamnbda_r x^r) , where the \lambda_r have to be chosen so as to satisfy the moment constraints. Only the case m = 2 is considered. It is shown that when x has finite range, a distribution maximizing the entropy exists and is unique. When the range is [0,\infty) , the maximum-entropy distribution exists if, and only if, \mu_2 \leq 2 \mu_1^2 , and a table is given which enables the maximum-entropy distribution to be computed. The case \mu_2 > 2 \mu_1^2 is discussed in some detail.
TL;DR: This correspondence derives the maximum-entropy spectrum by obtaining a spectrum that is forced to maximize the entropy of a stationary random process by choosing the spectrum that maximizes the entropy for the process.
Abstract: Maximum-entropy processing is a method for computing the power density spectrum from the first N lags of the autocorrelation function. Unlike the discrete Fourier transform, maximum-entropy processing does not assume that the other lag values are zero. Instead, one mathematically ensures that the fewest possible assumptions about unmeasured data are made by choosing the spectrum that maximizes the entropy for the process. The use of the maximum entropy approach to spectral analysis was introduced by Burg [1]. In this correspondence, the authors derive the maximum-entropy spectrum by obtaining a spectrum that is forced to maximize the entropy of a stationary random process.
TL;DR: It is shown that the rate-distortion bound (R(d) \leq C) remains true when -\log x in the definition of mutual information is replaced by an arbitrary concave nonincreasing function satisfying some technical conditions.
Abstract: It is shown that the rate-distortion bound (R(d) \leq C) remains true when -\log x in the definition of mutual information is replaced by an arbitrary concave (\cup) nonincreasing function satisfying some technical conditions. Examples are given showing that for certain choices of the concave functions, the bounds obtained are better than the classical rate-distortion bounds.
TL;DR: The table is obtained by combining the best of the existing bounds on d_{max} (n,k) with the minimum distances of known codes and a variety of code-construction techniques.
Abstract: This paper presents a table of upper and lower bounds on d_{max} (n,k) , the maximum minimum distance over all binary, linear (n,k) error-correcting codes. The table is obtained by combining the best of the existing bounds on d_{max} (n,k) with the minimum distances of known codes and a variety of code-construction techniques.