Showing papers in "IEEE Transactions on Information Theory in 1969"

Shift-register synthesis and BCH decoding

Abstract: It is shown in this paper that the iterative algorithm introduced by Berlekamp for decoding BCH codes actually provides a general solution to the problem of synthesizing the shortest linear feedback shift register capable of generating a prescribed finite sequence of digits. The shift-register approach leads to a simple proof of the validity of the algorithm as well as providing additional insight into its properties. The equivalence of the decoding problem for BCH codes to a shift-register synthesis problem is demonstrated, and other applications for the algorithm are suggested.

Optimal recursive estimation with uncertain observation

Abstract: In classical estimation theory, the observation is always assumed to contain the signal to be estimated. In practice, certain observations, or sequences of observations, may contain noise alone, only the probability of occurrence of such cases being available to the estimator. An example is trajectory tracking where the signal is first detected and then the estimator is allowed to process it for tracking purposes. However, any detection decision is associated with a false-alarm probability, which is the probability that the detected signal contains only noise. Minimum mean-square estimators are derived for two different forms of this problem; 1) when it is possible that the observation at any sample time contains signal or is noise alone, independent of the situation at any other sample, and 2) when the entire sequence of observations contains signal or is only noise. The estimators derived are of recursive form. A simple example is given for illustration.

Some lower bounds on signal parameter estimation

Abstract: New bounds are presented for the maximum accuracy with which parameters of signals imbedded in white noise can be estimated. The bounds are derived by comparing the estimation problem with related optimal detection problems. They are, with few exceptions, independent of the bias and include explicitly the dependence on the a priori interval. The new results are compared with previously known results.

A general likelihood-ratio formula for random signals in Gaussian noise

Abstract: It is shown that the likelihood ratio for the detection of a random, not necessarily Gaussian, signal in additive white Gaussian noise has the same form as that for a known signal in white Gaussian noise. The role of the known signal is played by the casual least-squares estimate of the signal from the observations. However, the "correlation" integral has to be interpreted in a special sense as an Ito stochastic integral. It will be shown that the formula includes all known explicit formulas for signals in white Gaussian noise. However, and more important, the formula suggests an "estimator-correlator" philosophy for engineering approximation of the optimum receiver. Some extensions of the above result are also discussed, e.g., additive finite-variance, not necessarily Gaussian, noise plus a white Gaussian noise component. Purely colored Gaussian noise can be treated if whitening filters can be specified. The analog implementation of Ito integrals is briefly discussed. The proofs of the formulas are based on the concept of an innovation process, which has been useful in certain related problems of linear and nonlinear least-squares estimation, and on the concept of covariance factorization.

On the Viterbi decoding algorithm

Abstract: A new interpretation of the Viterbi decoding algorithm based on the state-space approach to dyamical systems is presented. In this interpretation the optimum decoder solves a generalized regulator control problem by dynamic programming techniques.

Space-filling curves: Their generation and their application to bandwidth reduction

Abstract: This paper introduces a class of finite-state algorithms which characterize self-similar space-filling curves. The curves enable one to continuously map a line onto an N -dimensional cube, and find application in compressing the bandwidth of arbitrary waveforms. The bandwidth compression is effected in return for an increased susceptibility of the signal to perturbations. The algorithms are represented in a diagrammatic form which enables one to convert the N coordinates of a point in a cube into a single number representing the distance along a space-filling curve, or vice-versa, merely by visual inspection. The diagrams are always finite in size and may be constructed by following a rather simple numerical procedure.

Asymptotically optimal discriminant functions for pattern classification

Abstract: The two category classification problem is treated. No a priori knowledge of the statistics of the classes is assumed. A sequence of labeled samples from the two classes is used to construct a sequence of approximations of a discriminant function that is optimum in the sense of minimizing the probability of misclassification but which requires knowledge of all the statistics of the classes. Depending on the assumptions made about the probability densities corresponding to the two classes, the integrated square error of the approximations converges to 0 in probability or with probability 1 . The approximations are nonparametric and recursive for each fixed point of the domain. Rates of convergence are given. The approximations are used to define a decision procedure for classifying unlabeled samples. It is shown that as the number of labeled samples used to construct the approximations increases, the resulting sequence of discriminant functions is asymptotically optimal in the sense that the probability of misclassification when using the approximations in the decision procedure converges in probability or with probability 1 , depending on the assumptions made, to the probability of misclassification of the optimum discriminant function. The results can be easily extended to the multicategory problem and to the case of arbitrary loss functions, that is, where the costs of misclassification are not necessarily equal to 1 .

An adaptive receiver for digital signaling through channels with intersymbol interference

Abstract: A tutorial paper on an adaptive receiver that is suitable for high-speed digital signaling over slowly time-varying, band-limited channels which have impulse responses that are unknown at the receiver is presented. The receiver utilizes a steepest-descent technique for adjusting its parameters to existing channel conditions. A treatment of the speed of adaptation of the receiver is included.

Communication under the Poisson regime

Abstract: By "Poisson regime" we mean a model in which intelligence is communicated by random discrete occurrences in time that obey Poisson statistics of arbitrarily time-varying mean, as for example when modulated electromagnetic radiation and background radiation at optical frequencies is incoherently detected by photon-sensitive surfaces. The problems of optimal detection of signals of arbitrary shape and of the estimation of signal amplitude and delay are treated under a maximum-likelihood criterion. Detection probabilities, delay estimation errors, and the probability of "noise threshold" in delay estimation, are derived. Some results are basically different from those of parallel problems treating known signals in Gaussian noise. The treatment is based on a representation of nonstationary Poisson processes in which the observables are the instants of the occurrences rather than their numbers in given intervals of time.

On optimum quantization

Abstract: The problem of minimizing mean-square quantization error is considered and simple closed form approximations based on the work of Max and Roe are derived for the quantization error and entropy of signals quantized by the optimum fixed- N quantizer. These approximations are then used to show that, when N is moderately large, it is better to use equl-interval quantizing than the optimum fixed- N quantizer if the signal is to be subsetiuently buffered and transmitted at a fixed bit rate. Finally, the problem of optimum quantizing in the presence of buffering is examined, and the numerical results presented for Gaussian signals indicate that equllevel quantizing yields nearly optimum results.

New binary coding results by circulants

Abstract: The Introduction contains a new circulant echelon canonical form for the perfect (23,12) Golay code and some tentative conclusions are suggested. Section I gives an account of the properties of circulant matrices A , and a number of lemmas that make it possible to determine the minimum weight of codes generated by the rows of a matrix of the form |E|A| . In Section II, it is shown that many quadratic residue codes are almost of this form. The following new minimum weight results are obtained: For the (79, 40) code, w = 15; (103, 52), w = 19; (151, 76), w = 19; (89, 45), w = 17 and for (113, 57), w = 15 . In Section III, high-quality (noncyclic) group codes are constructed by means of circulants. In some cases a definite improvement is obtained on the best previously known Bose-Chaudhuri-Hocquenghem cyclic codes (including the (31, 16) code). Methods of coding and decoding circulant codes are not discussed.

A general formulation of linear feedback communication systems with solutions

Abstract: The feedback coding problem for additive noise systems, in which the noise may be colored, nonstationary, and correlated between channels, is formulated in terms of arbitrary linear operations at the transmitting and receiving points. This rather general linear formulation provides a unified approach for deriving new results, as well as previous results obtained under more restrictive assumptions, in a straightforward manner. Thus the sequential form of the optimum linear feedback code with an average power constraint on the transmitter is derived for noiseless feedback but forward noise of arbitrary covariance. It is shown explicitly that noiseless feedback increases the capacity of a channel with colored noise. The noisy feedback problem is considered and upper and lower bounds on the performance presented.

Approximating discrete probability distributions

Abstract: The method of minimum discrimination information estimation is applied to the problem of estimating an n -dimensional discrete probability distribution in terms of lower order marginal distributions. The procedure provides a convergent iterative algorithm. The method yields regular best asymptotically normal (RBAN) estimates. The general procedure includes as a particular case that proposed by a method using dependence trees. An example is given.

Even-shift orthogonal sequences

Abstract: A class of binary sequences whose elements are either 1 or -1 and whose autocorrelation function is 0 for all even shifts except the zero shift is discussed. These sequences will be called E sequences. It is proved that every E sequence is paired with another E sequence, its mate, so that the cross-correlation function between them is 0 for all even shifts including the zero shift. Using these sequences, new complete orthogonal function sets are derived. Several other important properties of these sequences are also obtained and the relations among these sequences, Golay's complementary sequences and Welti's quaternary codes are discussed.

The intrinsic dimensionality of signal collections

Abstract: In view of the trend toward the representation of signals as physical observables characterized by vectors in an abstract signal space, rather than as time or frequency functions, it is desirable that dimensionality be defined in a manner independent of the choice of basis on which the vectors are projected. The intrinsic dimensionality of a collection of signals is defined to be equal to the number of free parameters required in a hypothetical signal generator capable of producing a close approximation to each signal in the collection. Thus defined, the dimensionality becomes a relationship between the vectors representing the signals. This relationship need not be a linear one and does not depend on the basis onto which the vectors are projected in the signal-measuring process. A digital computer program for estimating this dimensionality from the signal coefficients on an arbitrary basis has been developed. The program makes use of some results obtained from a multi-dimensional scaling problem in experimental psychology and utilizes an inverse relationship between the variance in interpoint distances within a hypersphere and the dimensionality of the hypersphere. Using this method, the results are believed to be independent of the choice of orthogonal basis, and no prior knowledge of the analytical form of the signals is assumed. The validity of the program is tested and verified by using it to estimate the dimensionality of signals of known structure.

A useful form of the Barankin lower bound and its application to PPM threshold analysis

Abstract: A form of Barankin's greatest lower bound on estimation error [7] is obtained, which is easy to compute and easy to interpret. This gives a lower bound on estimation error for non-linear modulation systems in an additive Gaussian noise back-ground. Threshold effects are included. This bound is applied to a set of pulse-position modulation waveforms designed to reduce threshold effects. It is shown that these signals do, in fact, offer reduced threshold levels (e.g., \approx 3.5 dB) with very small ( \approx \frac{1}{2} dB) degradation in large signal performance.

Intersymbol interference and probability of error in digital systems

Abstract: Intersymbol interference and additive Gaussian noise are two important sources of distortion in digital systems, and a principal goal in the analysis of such systems is the determination of the resulting probability of error Earlier related work has sought to estimate the error probability either by calculating an approximation based upon a truncated version of the random pulse train or by obtaining an upper bound which results from consideration of the worst case intersymbol interference In this paper a new upper bound is derived for the probability of error which is computationally simpler than the truncated pulse-train approximation and which never exceeds the worst case bound Moreover, the new bound is applicable in a number of cases where the worst case bound cannot be used The bound is readily evaluated and depends upon three parameters: the usual signal-to-noise ratio; the ratio of intersymbol interference power to total distortion power; and the ratio of the maximum intersymbol interference amplitude to its rms value To illustrate the utility of the bound, it is compared with the earlier methods in three cases which are representative of the most important situations occurring in practice

Measurement of random time-variant linear channels

Abstract: This paper concerns the problems of the measurability and measurement of random time-variant linear channels. With regard to measurability, a new, less stringent channel measurability criterion is proposed to supersede the BL product introduced by Kailath. This criterion involves the area of occupancy of the Doppler-delay spread function (or its dual). By using time and bandwidth constraints on the input and output of a channel, channel measurement is reduced to the measurement of a discrete set of finite parameters. Optimal measurement techniques are described and their performances determined for both known and unknown channel correlation functions.

Conjugate linear filtering

Abstract: Aspects of optimum filtering for complex valued random processes are presented. Ordinary linear filters are complemented with conjugate linear filters. It is found that the incorporation of conjugate linear filtering improves signal-to-noise ratio by a factor of two in matched filter receivers. For optimum least squares filtering the inclusion of conjugate processing reduces mean-square error by a factor as great as two; the improvement depends primarily on the degree of correlation between the real and imaginary parts of the signal process. The analysis utilizes special correlation properties of receiver noise. Also, in the absence of phase lock, conjugate linear processing offers no improvement. Finally, it is observed that in the Gaussian case the least squares nonlinear receiver for modulations consists of the derived linear-conjugate linear receiver followed by demodulators comparable to those used in practice.

Improving communication reliability by use of an intermittent feedback channel

Abstract: This paper describes and analyzes a feedback communication scheme where the feedback channel is used only to inform the transmitter, at specified times (prior to a final decision), which message the receiver considers most likely. The feedback information is used by the transmitter to modify its transmission according to a rule known also to the receiver. When orthogonal signals are used and only an average transmitter power constraint is imposed, the error probability can be made to decrease in an N -fold exponential manner with increasing

Fredholm resolvents, Wiener-Hopf equations, and Riccati differential equations

Abstract: We shall show that the solution of Fredholm equations with symmetric kernels of a certain type can be reduced to the solution of a related Wiener-Hopf integral equation. A least-squares filtering problem is associated with this equation. When the kernel has a separable form, this related problem suggests that the solution can be obtained via a matrix Riccati differential equation, which may be a more convenient form for digital computer evaluation. The Fredholm determinant is also expressed in terms of the solution to the Riccati equation; this formula can also be used for the numerical determination of eigenvalues. The relations to similar work by Anderson and Moore and by Schumitzky are also discussed.

Tree encoding of memoryless time-discrete sources with a fidelity criterion

Abstract: In this paper we show that for memoryless time-discrete sources with a bounded fidelity criterion, the limiting average distortion achievable by tree codes of rate R is D , the solution of the equation R = R(D) , where R( ) denotes the usual rate distortion function. Thus the performance of tree codes is as good as that of block codes. Some theoretical and experimental results are also discussed indicating that tree codes and corresponding encoding algorithms exist having for given values of R and D an implementation complexity that is far smaller than the one obtainable from block codes.

A construction technique for random-error-correcting convolutional codes

Abstract: A simple algorithm is presented for finding rate 1/n random-error-correcting convolutional codes. Good codes considerably longer than any now known are obtained. A discussion of a new distance measure for convolutional codes, called the free distance, is included. Free distance is particularly useful when considering decoding schemes, such as sequential decoding, which are not restricted to a fixed constraint length. It is shown how the above algorithm can be modified slightly to produce codes with known free distance. A comparison of probability of error with sequential decoding is made among the best known constructive codes of constraint length 36 .

Minimization of mean-square error for data transmitted via group codes

Abstract: We show how to find solutions to the problem considered by Mitryayev [l ] in the case where the loss power function is quadratic. This problem is to minimize mean-square error when digital data is represented by group code combinations and the a priori probability distribution is uniform. Furthermore, it is shown that there is always a solution which is linear in the sense of Mitryayev.

Rank permutation group codes based on Kendall's correlation statistic

Abstract: A coding scheme based on the properties of rank vectors is presented. The new codes are based on the theory of permutation groups by introducing a new notation for the group operation that simplifies the generation and decoding of desirable rank codes. The use of group theory is made possible by the introduction of the Kendall correlation coefficient as a measure of the distance between code words. This technique provides a method for the choice of rank vector code words superior to those that have been proposed in the past. Much of the terminology used in block coding can also be used to describe rank vector codes, but the actual quantities involved are quite different. The rank vector codes discussed in the paper offer the advantage of low sensitivity of the probability of error to the noise distribution because of the nonparametric character of rank vector detection schemes. Bounds that have been verified by extensive computer simulation have been derived for the probability of error.

A class of upper bounds on probability of error for multihypotheses pattern recognition (Corresp.)

Abstract: A class of upper bounds on the probability of error for the general multihypotheses pattern recognition problem is obtained. In particular, an upper bound in the class is shown to be a linear functional of the pairwise Bhattacharya coefficients. Evaluation of the bounds requires knowledge of a priori probabilities and of the hypothesis-conditional probability density functions. A further bound is obtained that is independent of a priori probabilities. For the case of unknown a priori probabilities and conditional probability densities, an estimate of the latter upper bound is derived using a sequence of classified samples and Kernel functions to estimate the unknown densities.

An upper bound on moments of sequential decoding effort

Abstract: In this paper we derive an infinite tree code ensemble upper bound on the u th ( u \leq 1) moments of the computational effort connected with sequential decoding governed by the Fano \footnote[1]{algorithm}. It is shown that the u th moment of the effort per decoded branch is hounded by a constant, provided the transmission rate R_{0} satisfies inequality (2), This result, although often conjectured, has previously been shown to hold only for positive integral values of u . For a wide class of discrete memoryless channels (that includes all symmetric channels), our bounds agree qualitatively with the lower bounds of Jacobs and Berlekamp [8].

Nonparametric feature selection

Abstract: Two groups of L -dimensional observations of size N_{1} and N_{2} are known to be random vector variables from two unknown probability distribution functions [1]. A method is discussed for obtaining an l -dimensional linear subspace of the observation space in which the l -variate marginal distributions are most separated, based on a nonparametric estimate of probability density functions and a distance criterion. The distance used essentially is the L_{2} norm of the difference between Parzen estimates of the two densities. An algorithm is developed that determines the subspace for which the distance between the two densities is maximized. Computer simulations are performed.

Some Remarks on BCH Bounds and Minimum Weights of Binary Primitive BCH Codes

Abstract: It is shown that if m eq 8, 12 and m > 6 , there are some binary primitive BCH codes (BCH codes in a narrow sense) of length 2^{m} - 1 whose minimum weight is greater than the BCH bound. This gives a negative answer to the question posed by Peterson [1] of whether or not the BCH bound is always the actual minimum weight of a binary primitive BCH code. It is also shown that for any even m \geq 6 , there are some binary cyclic codes of length 2^{m} - 1 that have more information digits than the primitive BCH codes of length 2^{m} - 1 with the same minimum weight.