Showing papers in "IEEE Transactions on Information Theory in 1971"

Alternative interpretation of maximum entropy spectral analysis (Corresp.)

Abstract: Maximum entropy spectral analysis is a method for the estimation of power spectra with a higher resolution than can be obtained with conventional techniques. This is achieved by extrapolation of the autocorrelation function in such a way that the entropy of the corresponding probability density function is maximized in each step of the extrapolation. This correspondence also gives a simple interpretation of the method without entropy considerations.

Feedback equalization for fading dispersive channels

Abstract: Data transmission through a slowly fading dispersive channel is considered. A receiver that linearly operates on both the received signal and reconstructed data is postulated. Assuming an absence of decision errors, the receiver is optimized for a minimum-mean-square-error criterion. Transfer functions are determined and superiority over nonfeedback receivers is indicated. The feedback receiver can be realized in a slowly varying unknown environment by means of an adaptive technique that requires neither test signals nor statistical estimation. The receiver will eliminate timing jitter and Doppler shifts. In addition, the receiver provides a time-diversity effect, as the receiver probability of error averaged over the fading statistics is lower in the presence of dispersion than in its absence.

Instrumentable tree encoding of information sources (Corresp.)

Abstract: We study here the use of tree codes to encode time-discrete memoryless sources with respect to a fidelity criterion. An easily instrumented scheme is proposed for use with binary sources and the Hamming distortion metric. Results of simulation with random and convolutional codes are given.

Mutual information of the white Gaussian channel with and without feedback

Abstract: The following model for the white Gaussian channel with or without feedback is considered: \begin{equation} Y(t) = \int_o ^{t} \phi (s, Y_o ^{s} ,m) ds + W(t) \end{equation} where m denotes the message, Y(t) denotes the channel output at time t , Y_o ^ {t} denotes the sample path Y(\theta), 0 \leq \theta \leq t. W(t) is the Brownian motion representing noise, and \phi(s, y_o ^ {s} ,m) is the channel input (modulator output). It is shown that, under some general assumptions, the amount of mutual information I(Y_o ^{T} ,m) between the message m and the output path Y_o ^ {T} is directly related to the mean-square causal filtering error of estimating \phi (t, Y_o ^{t} ,m) from the received data Y_o ^{T} , 0 \leq t \leq T . It follows, as a corollary to the result for I(Y_o ^ {T} ,m) , that feedback can not increase the capacity of the nonband-limited additive white Gaussian noise channel.

Barankin Bounds on Parameter Estimation

Abstract: The Schwarz inequality is used to derive the Barankin lower bounds on the covariance matrix of unbiased estimates of a vector parameter. The bound is applied to communications and radar problems in which the unknown parameter is embedded in a signal of known form and observed in the presence of additive white Gaussian noise. Within this context it is shown that the Barankin bound reduces to the Cramer-Rao bound when the signal-to-noise ratio (SNR) is large. However, as the SNR is reduced beyond a critical value, the Barankin bound deviates radically from the Cramer-Rao bound, exhibiting the so-called threshold effect. The bounds were applied to the linear FM waveform, and within the resulting class of bounds it was possible to select one that led to a closed-form expression for the lower bound on the variance of an unbiased range estimate. This expression clearly demonstrates the threshold behavior one must expect when using a nonlinear modulation system. Tighter bounds were easily obtained, but these had to be evaluated numerically. The sidelobe structure of the linear FM compressed pulse leads to a significant increase in the variance of the estimate. For a practical linear FM pulse of 1- \mu s duration and 40-MHz bandwidth, the radar must operate at an SNR greater than 10 dB if meaningful unbiased range estimates are to be obtained.

RKHS approach to detection and estimation problems--I: Deterministic signals in Gaussian noise

Abstract: First it is shown how the Karhunen-Loeve approach to the detection of a deterministic signal can be given a coordinate-free and geometric interpretation in a particular Hilbert space of functions that is uniquely determined by the covariance function of the additive Gaussian noise. This Hilbert space, which is called a reproducing-kernel Hilbert space (RKHS), has many special properties that appear to make it a natural space of functions to associate with a second-order random process. A mapping between the RKHS and the linear Hilbert space of random variables generated by the random process is studied in some detail. This mapping enables one to give a geometric treatment of the detection problem. The relations to the usual integral-equation approach to this problem are also discussed. Some of the special properties of the RKHS are developed and then used to study the singularity and stability of the detection problem and also to suggest simple means of approximating the detectability of the signal. The RKHS for several multidimensional and multivariable processes is presented; by going to the RKHS of functionals rather than functions it is also shown how generalized random processes, including white noise and stationary processes whose spectra grow at infinity, are treated.

Correlative level coding and maximum-likelihood decoding

Abstract: Modems for digital communication often adopt the so-called correlative level coding or the partial-response signaling, which attains a desired spectral shaping by introducing controlled intersymbol interference terms. In this paper, a correlative level encoder is treated as a linear finite-state machine and an application of the maximum-likelihood decoding (MLD) algorithm, which was originally proposed by Viterbi in decoding convolutional codes, is discussed. Asymptotic expressions for the probability of decoding error are obtained for a class of correlative level coding systems, and the results are confirmed by computer simulations. It is shown that a substantial performance gain is attainable by this probabilistic decoding method.

Robust detection of a known signal in nearly Gaussian noise

Abstract: A detector that is not nonparametric, but that nevertheless performs well over a broad class of noise distributions is termed a robust detector. One possible way to obtain a certain degree of robustness or stability is to look for a min-max solution. For the problem of detecting a signal of known form in additive, nearly Gaussian noise, the solution to the min-max problem is obtained when the signal amplitude is known and the nearly Gaussian noise is specified by a mixture model. The solution takes the form of a correlator-limiter detector. For a constant signal, the correlator-limiter detector reduces to a limiter detector, which is shown to be robust in terms of power and false alarm. By adding a symmetry constraint to the nearly normal noise and formulating the problem as one of local detection, the limiter-correlator is obtained as the local min-max solution. The limiter-correlator is shown to be robust in terms of asymptotic relative efficiency (ARE). For a pulse train of unknown phase, a limiter-envelope sum detector is also shown to be robust in terms of ARE.

Detectors, bandpass nonlinearities, and their optimization: Inversion of the Chebyshev transform

Abstract: From the voltage-response characteristic of a memoryless nonlinearity the output amplitude in any harmonic zone is easily found as a function of the amplitude of a narrow-band input, but no general method has been known for inverting this (Chebyshev) transformation. The inversion is of interest because the best detector, bandpass non-linearity, or harmonic generator for various purposes, e.g., maximization of the output signal-to-noise ratio, is most readily described in terms of its amplitude-response characteristic for the desired harmonic zone. A simple integration over the amplitude-response characteristic (or describing function) is found to yield the required voltage-response curve.

Signal detection in Gaussian noise of unknown level: An invariance application

Abstract: The concept of invariance in hypothesis testing is brought to bear on the problem of detecting signals of known form and unknown energy in Gaussian noise of unknown level. The noise covariance function is assumed to be K(t,u) = \sigma^2 \pho(t,u) where \rho(t,u) is the known form of the covariance function and \sigma^2 is the unknown level. Classical approaches to signal detection depend on the assumption that K(t,u) is known completely. Then, a correlation-type receiver that is the uniformly most powerful (UMP) test of H_o (signal absent) versus H_1 (signal present) can be derived. When \sigma^2 is unknown, there exists no UMP test. However, it is shown in this paper that there exists a test of H_o versus H_1 that is UMP-invariant for a very natural group of transformations on the space of observations. The derived test is found to be independent of knowledge about the noise level \sigma^2 , since the derived test (receiver) contains an error-free estimate of \sigma^2 . This utopian conclusion is reconciled by noting that the derived receiver can never be physically realized. It is shown that any physically realizable version of the receiver has a t -distributed test statistic. This permits choice of operating receiver thresholds and evaluation of performance characteristics.

Inversionless decoding of binary BCH codes

Abstract: The iterative algorithm for decoding binary BCH codes presented by Berlekamp and, in an alternative form, by Massey is modified to eliminate inversion. Because inversion in a finite field is time consuming and requires relatively complex circuitry, this new algorithm should he useful in practical applications of multiple-error-correcting binary BCH codes.

Convergence of the nearest neighbor rule

Abstract: If the nearest neighbor rule (NNR) is used to classify unknown samples, then Cover and Hart [1] have shown that the average probability of error using n known samples (denoted by R_n ) converges to a number R as n tends to infinity, where R^ {\ast} \leq R \leq 2R^ {\ast} (1 - R^ {\ast}) , and R^ {\ast} is the Bayes probability of error. Here it is shown that when the samples lie in n -dimensional Euclidean space, the probability of error for the NNR conditioned on the n known samples (denoted by L_n . so that EL_n = R_n) converges to R with probability 1 for mild continuity and moment assumptions on the class densities. Two estimates of R from the n known samples are shown to be consistent. Rates of convergence of L_n to R are also given.

Codes based on inaccurate source probabilities

Abstract: Information theory obtains efficient codes by encoding messages in large blocks. The code design requires block probabilities that are often hard to measure accurately. This paper studies the effect of inaccuracies in the block probabilities and gives coding procedures that anticipate some of the worst errors. For an efficient code, the mean number d of digits per letter must be kept small. In some cases the expected value of d can be related to the size of the sample on which probability estimates are based. To underestimate badly the probability of a common letter or block is usually a serious error. To ensure against this possibility, some coding procedures are given that avoid extremely long codewords. These codes provide a worthwhile insurance but are still very efficient if the probability estimates happen to be correct.

Note on optimal selection of independent binary-valued features for pattern recognition (Corresp.)

Abstract: Given a set of conditionally independent binary-valued features, a counter example is given to a possible claim that the best subset of features must contain the best single feature.

Generation of binary sequences with controllable complexity

Abstract: Complexity of a binary sequence is measured by the amount of the sequence required to define the remainder. It is shown that, while maximum length (L = 2^r - 1) sequences from r -stage linear logic feedback generators have minimum complexity, it is a simple matter to use such sequences as bases for deriving other more complex sequences of the same length. The complexity is controllable up to maximum complexity, which means that no fractional part of a sequence will define the remainder. It is demonstrated that, from the 2^L /L cyclically distinct sequences of length L , most of which are highly complex, it is possible to select a priori those with acceptable noiselike statistics. Practical schemes based on the Langford problem are given for implementing large quantities of such sequences.

Spectral representation of a periodic nonstationary random process

Abstract: This paper deals with the periodic nonstationary process, the mean value and the correlation function of which are invariant under shift by a multiple of a certain period. The spectral representation is derived by making use of Loeve's harmonizability theorem on a second-order nonstationary process. The process is represented as a sum of infinite stationary processes among which covariances exist. Each stationary process has a nonoverlapping frequency band of equal width, the center of which corresponds to a harmonic of the fundamental frequency determined by the period. The correlation function, dependent on two points, is represented in terms of a matrix-valued spectral density that is hermitian and nonnegative definite. The representations in other possible forms are also given. Finally some properties, special processes, and examples produced by a certain stationary random sequence are discussed.

Application of quantum estimation theory to analog communication over quantum channels

Abstract: Recent inquiries into optical communication have raised questions as to the validity of classical detection and estimation theory for weak light fields. Helstrom [1] proposed that the axioms of quantum mechanics be incorporated into a quantum approach to optical estimation and detection. In this paper, we discuss two important results, the quantum equivalent of the minimum-mean-square-error (MMSE) estimator and the quantum Cramer-Rao bound for estimation of the parameters of an electromagnetic field. The first result, a new one, is applied to linear modulation. We show that homodyning is the optimal demodulation scheme in that case. Parallels to the classical MMSE estimator are drawn. The Cramer-Rao bound, first derived in the quantum ease by Helstrom [1], is applied to specific estimation problems. Details are left to the references, but interesting results are presented.

Recursive Bayesian estimation with uncertain observation (Corresp.)

Abstract: Two different problems of estimating a discrete stochastic process, in the face of Markov dependent uncertainty regarding the presence of the process at each stage of the observation sequence, are considered. Recursive Bayes optimal estimator algorithms are derived for the two cases considered, and the differences between them brought out explicitly.

Bounds on the rate-distortion function for stationary sources with memory

Abstract: In this paper, we study discrete-time stationary sources S with memory. The rate R(\beta) of the source relative to a distortion measure is compared with R^ \ast (\beta) , the rate of the memoryless source S^ /ast with the same marginal statistics as S . We show that R^ \ast (\beta) - \Delta \leq R(\beta) \leq R^ \ast (\beta) , where \Delta is a measure of the memory of the source. A number of interesting applications of these bounds are given.

Rate distortion functions for finite-state finite-alphabet Markov sources

Abstract: A lower bound to the rate-distortion function R(D) of finite-alphabet sources with memory is derived for the class of balanced distortion measures. For finite-state finite-alphabet Markov sources, sufficient conditions are given for the existence of a strictly positive average distortion D_c such that R(D) equals its lower bound for 0 \leqq D \leqq D_c . The bound is evaluated for the Hamming and Lee distortion measures and is identical to the corresponding bound for memoryless sources having the same entropy and alphabet. These results are applied to yield a simple proof of the converse of the noisy-channel coding theorem for sources satisfying the sufficient conditions for equality with the lower bound and channels with memory. D_c is evaluated explicitly for the special case of the binary asymmetric Markov source.

A class of simple and optimal strategies for block coding on the binary symmetric channel with noiseless feedback

Abstract: A class of simple feedback strategies is developed. The fixed-length codewords can be described by an interesting sequence of trees. The decoder scans the received block in the reverse direction, starting at the most recent bit. As this scan progresses, a certain pair of characteristic bit patterns are replaced by the digits 0 or 1, respectively. For certain values of the information rate R , this class of strategies corrects the largest error fraction possible.

Quantizing for maximum output entropy (Corresp.)

Abstract: The entropy at the output of a quantizer is equal to the average mutual information between unquantized and quantized random variables. Thus, for a fixed number of quantization levels, output entropy is a reasonable information-theoretic criterion of quantizer fidelity. It is shown that, for a class of signal distributions, which includes the Gaussian, the quantizers with maximum output entropy (MOE) and minimum average error (MAE) are approximately the same within a multiplicative constant.

Rate 1/2 convolutional codes with complementary generators

Abstract: The structure of a class of rate \frac{1}{2} convolutional codes called complementary codes is investigated. Of special interest are properties that permit a simplified evaluation of free distance. Methods for finding the codes with largest free distance in this class are obtained. A synthesis procedure and a search procedure that result in good codes up to constraint lengths of 24 are described. The free and minimum distances of the best complementary codes are compared with the best known bounds and with the distances of other known codes.

On the construction of a class of majority-logic decodable codes

Abstract: The attractiveness of majority-logic decoding is its simple implementation Several classes of majority-logic decodable block codes have been discovered for the past two decades In this paper, a method of constructing a new class of majority-logic decodable block codes is presented Each code in this class is formed by combining majority-logic decodable codes of shorter lengths A procedure for orthogonalizing codes of this class is formulated For each code, a lower bound on the number of correctable errors with majority-logic decoding is obtained An upper bound on the number of orthogonalization steps for decoding each code is derived Several majority-logic decodable codes that have more information digits than the Reed-Muller codes of the same length and the same minimum distance are found Some results presented in this paper are extensions of the results of Lin and Weldon [11] and Gore [12] on the majority-logic decoding of direct product codes

On decoding of Reed-Solomon codes

Abstract: It is shown how nonsystematic Reed-Solomon (RS) codes encoded by means of the Chinese remainder theorem can be decoded using the Berlekamp algorithm. The Chien search and calculation of error values are not needed but are replaced by a polynomial division and added calculation in determining the syndrome. It is shown that for certain cases of low-rate RS codes, the total decoding computation may be less than the usual method used with cyclic codes. Encoding and decoding for shorter length codes is presented.

Optimal mismatched filter design for radar ranging, detection, and resolution

Abstract: In a multiple-target environment a radar signal processor often uses weighting filters that are not matched to the transmitted waveform. In this paper the mean-square range-estimation error, the detection Signal-to-noise ratio (SNR), and the effects of sidelobes are expressed in terms of the impulse response of an arbitrary mismatched filter. It is desired to find that impulse response that results in the minimum range-estimate variance subject to preassigned constraints on the side-lobes and the detection SNR. It is shown that for detecting the radar target and estimating its range, the form of the optimum filter is a modified transversal equalizer. If only detection is required, the optimum filter reduces to the transversal equalizer. Certain parameters upon which the solution depends can be found by solving a nonlinear programming problem. Numerical results are given for an interesting class of transmitted waveforms.

Comparison of line-by-line and two-dimensional encoding of random images

Abstract: Most methods of encoding images require complicated implementation. Thus it is of interest to compare the transmission rates that can be achieved by classes of encoding methods of different complexity. We consider two classes of encoding operations. The first class allows any possible operation on the two-dimensional image source output. The second class allows only certain restricted operations on the image. In acquisition of images by electronic means, the image intensity is in general scanned line by line, resulting in data that appear as a sequence of ordinary time series. In encoding, a simpler implementation results if one accepts the time series from a single scan line and encodes it independently of adjacent scan lines. This limits storage requirements to a single scan line and limits processing to operations on a one-parameter time series instead of operations on a two-dimensional field. This is the second class of encoding operations that we consider. We choose a distortion measure and discuss the rate-distortion function, which represents the minimum rate required by any encoding method in the first (arbitrary complex encoding). We then derive the minimum rate that can be achieved by any encoder from the second class. These rates are compared for a specific example.

Cyclic and multiresidue codes for arithmetic operations

Abstract: In this paper, the cyclic nature of AN codes is defined after a brief summary of previous work in this area is given. New results are shown in the determination of the range for single-error-correcting AN codes when A is the product of two odd primes p_1 and p_2 , given the orders of 2 modulo p_1 and modulo p_2 . The second part of the paper treats a more practical class of arithmetic codes known as separate codes. A generalized separate code, called a multiresidue code, is one in which a number N is represented as \begin{equation} [N, \mid N \mid _ {m1}, \mid N \mid _{m2}, \cdots , \mid N \mid _{mk}] \end{equation} where m_i are pairwise relatively prime integers. For each AN code, where A is composite, a multiresidue code can be derived having error-correction properties analogous to those of the AN code. Under certain natural constraints, multiresidue codes of large distance and large range (i.e., large values of N ) can be implemented. This leads to possible realization of practical single and/or multiple-error-correcting arithmetic units.

Capacity of a continuous memoryless channel with feedback

Abstract: Shannon showed that the capacity of a discrete memoryless channel can not be increased by noiseless feedback. It has been conjectured that this should be true for a continuous memoryless channel, provided such a channel is appropriately defined. We precisely define such a channel from two mathematically different points of view and rigorously prove that its capacity can not be increased by feedback.

Decoding binary block codes on Q -ary output channels

Abstract: This paper presents an algebraic technique for decoding binary block codes in situations where the demodulator quantizes the received signal space into Q > 2 regions. The method, referred to as weighted erasure decoding (WED), is applicable in principle to any block code for which a binary decoding procedure is known.