Showing papers in "IEEE Transactions on Information Theory in 1968"

Approximating discrete probability distributions with dependence trees

Abstract: A method is presented to approximate optimally an n -dimensional discrete probability distribution by a product of second-order distributions, or the distribution of the first-order tree dependence. The problem is to find an optimum set of n - 1 first order dependence relationship among the n variables. It is shown that the procedure derived in this paper yields an approximation of a minimum difference in information. It is further shown that when this procedure is applied to empirical observations from an unknown distribution of tree dependence, the procedure is the maximum-likelihood estimate of the distribution.

On the mean accuracy of statistical pattern recognizers

Abstract: The overall mean recognition probability (mean accuracy) of a pattern classifier is calculated and numerically plotted as a function of the pattern measurement complexity n and design data set size m . Utilized is the well-known probabilistic model of a two-class, discrete-measurement pattern environment (no Gaussian or statistical independence assumptions are made). The minimum-error recognition rule (Bayes) is used, with the unknown pattern environment probabilities estimated from the data relative frequencies. In calculating the mean accuracy over all such environments, only three parameters remain in the final equation: n, m , and the prior probability p_{c} of either of the pattern classes. With a fixed design pattern sample, recognition accuracy can first increase as the number of measurements made on a pattern increases, but decay with measurement complexity higher than some optimum value. Graphs of the mean accuracy exhibit both an optimal and a maximum acceptable value of n for fixed m and p_{c} . A four-place tabulation of the optimum n and maximum mean accuracy values is given for equally likely classes and m ranging from 2 to 1000 . The penalty exacted for the generality of the analysis is the use of the mean accuracy itself as a recognizer optimality criterion. Namely, one necessarily always has some particular recognition problem at hand whose Bayes accuracy will be higher or lower than the mean over all recognition problems having fixed n, m , and p_{c} .

Logical basis for information theory and probability theory

Abstract: A new logical basis for information theory as well as probability theory is proposed, based on computing complexity.

Asymptotically efficient quantizing

Abstract: It is shown, under weak assumptions on the density function of a random variable and under weak assumptions on the error criterion, that uniform quantizing yields an output entropy which asymptotically is smaller than that for any other quantizer, independent of the density function or the error criterion. The asymptotic behavior of the rate distortion function is determined for the class of u th law loss functions, and the entropy of the uniform quantizer is compared with the rate distortion function for this class of loss functions. The extension of these results to the quantizing of sequences is also given. It is shown that the discrepancy between the entropy of the uniform quantizer and the rate distortion function apparently lies with the inability of the optimal quantizing shapes to cover large dimensional spaces without overlap. A comparison of the entropies of the uniform quantizer and of the minimum-alphabet quantizer is also given.

Signal energy distribution in time and frequency

Abstract: The Fourier representation of signals and its relation to the signal structure in time and frequency, and more generally the inherent properties of phase-modulated signals, have received considerable attention in the past. These topics have led to such seemingly unrelated studies as the representation of a signal in a combined time-frequency plane, "instantaneous power spectra," and the ambiguity function and its transform relations. It is shown in this paper that the studies can be unified by the introduction of the concept of the complex energy density function of a signal. The function is an extension and combination of the one-dimensional energy density functions in time and frequency, the energy density spectrum |\Psi(f)|^{2} , and energy density waveform |\psi (t)|^{2} . On the basis of the complex energy density function, the significance of complicated-appearing transform relations is readily understood. The new concept also conveys a good insight into the internal structure of phase-modulated signals.

Exponential error bounds for erasure, list, and decision feedback schemes

Abstract: By an extension of Gallager's bounding methods, exponential error bounds applicable to coding schemes involving erasures, variable-size lists, and decision feedback are obtained. The bounds are everywhere the tightest known.

Estimation by the nearest neighbor rule

Abstract: Let R^{\ast} denote the Bayes risk (minimum expected loss) for the problem of estimating \theta \varepsilon \Theta , given an observed random variable x , joint probability distribution F(x,\theta) , and loss function L . Consider the problem in which the only knowledge of F is that which can be inferred from samples (x_{1},\theta_{1}),(x_{2},\theta_{2}), \cdots ,(x_{n}, \theta_{n}) , where the (x_{i}, \theta_{i})'s are independently identically distributed according to F . Let the nearest neighbor estimate of the parameter \theta associated with an observation x be defined to be the parameter \theta_{n}^{'} associated with the nearest neighbor x_{n}^{'} to x . Let R be the large sample risk of the nearest neighbor rule. It will be shown, for a wide range of probability distributions, that R \leq 2R^{\ast} for metric loss functions and R = 2R^{\ast} for squared-error loss functions. A simple estimator using the nearest k neighbors yields R = R^{\ast} (1 + 1/k) in the squared-error loss case. In this sense, it can be said that at least haft the information in the infinite training set is contained in the nearest neighbor. This paper is an extension of earlier work[q from the problem of classification by the nearest neighbor rule to that of estimation. However, the unbounded loss functions in the estimation problem introduce additional problems concerning the convergence of the unconditional risk. Thus some work is devoted to the investigation of natural conditions on the underlying distribution assuring the desired convergence.

New generalizations of the Reed-Muller codes--I: Primitive codes

Abstract: First it is shown that all binary Reed-Muller codes with one digit dropped can be made cyclic by rearranging the digits. Then a natural generalization to the nonbinary case is presented, which also includes the Reed-Muller codes and Reed-Solomon codes as special cases. The generator polynomial is characterized and the minimum weight is established. Finally, some results on weight distribution are given.

A quantitative-qualitative measure of information in cybernetic systems (Corresp.)

Abstract: A Quantitative-Qualitative Measure of Information in Cybernetic Systems equal to the sum of the information supplied by each event separately. Let p and q be the probabilit.ies of the E and F events, respectively; the event E r\\ F has a probability pq if E and F are independent, and we have the equality In order to elaborate a theory of communicat ion which could be useful for designing a great variety of transmission systems, it was necessary to find a general notion capable of abstracting the various kind of signals which can be transmitted. By neglecting the particular aspect of these signals and considering them as random abstract events, it was possible to define the quantitative aspect of information based on the probability of different events. AE a matter of fact, thii simplification of the complex aspect of information led to the first great analogy between biological and technical systems considered as information transmission systems. The cybernetic analogy between man and machine consists precisely of the fact that both are control systems. This means that information is transmitted and processed in view of a goal with regard to which control signals must be efficient. The whole activity of cybernetic systems (biological or technical) is directed toward the fulfillment of a goal. The system must then possess a qualitative differentiating criterion for the signals to be transmitted. This implies the existence of a logical block able to discriminate the quality of various signals according to a given criterion. The cybernetic criterion for a qualitative differentiation of the signals is represented by the relevance, the significance, or the utility of the information they carry with respect to the goal. The occurrence of an event removes a double uncertainty: the quantitative one related to its probability of occurrence, and the qualitative one related to its utility for the fulfillment of the goal. In the following, a general formula, taking into account the two basis concepts of probability and utility, will be established. Let E,, E,, . ., En be a finite set of events representing the possible realizations of some experiment; let pl, pi, . . . , pn be the probabilities of occurrence of these events, satisfying

Simultaneous optimum detection and estimation of signals in noise

Abstract: The problem of simultaneous detection and estimation of signals in noise is formulated in the language of statistical decision theory. Optimum structures and corresponding general measures of system performance are derived under the Bayes criterion of minimum average risk for the detectors and estimators appropriate to this type of joint operation: It is shown that whereas the structures of the resulting optimum detectors require a class of modified likelihood ratios, the structures of the optimum estimators, which act on the data when there is uncertainty as to the presence of a signal [p(H_{1}) , have a common canonical form for a wide variety of operating strategies. This form is identical with that obtained for estimation alone [p(H_{1}) , even though there is generally mutual coupling between detector and extractor. A simple structure is obtained for the estimation of amplitude and waveform in the case of a quadratic cost function (least mean-square error), where it is found that the estimator which is optimum here [p(H_{1}) is the product of the corresponding Bayes estimator in the "classical" case [P(H1) = 1 ] and a simple algebraic function of the generalized likelihood ratio. In this case, one can also show that estimators that are unbiased in the classical sense remain unbiased. In parallel with the classical theory, a generalized version of unconditional maximum likelihood estimation is obtained for the "simple" cost function when p(H_{1} . It is found that estimators that are linear in the classical case [p(H_{1}) = 1 ] are nonlinear in the more general situation [p(H_{1}) , where increased structural complexity is always the rule for both detectors and estimators. A specific example involving the coherent estimation of signal amplitude illustrates the approach. Analogous extensions to prediction and filtering are formulated, making it evident that a broad area for further generalizations of classical Bayes detection and extraction theory is available for systematic investigation.

Intersymbol interference error bounds with application to ideal bandlimited signaling

Abstract: An upper bound is derived for the probability of error of a digital communication system subject to intersymbol interference and Gaussian noise. The bound is applicable to multilevel as well as binary signals and to all types of intersymbol interference. The bound agrees with the exponential portion of a normal distribution in which the larger intersymbol interference components subtract from the signal amplitude, and the smaller components add to the noise power. The results are applied to the case of random binary signaling with sin x/x pulses. It is shown that such signals are not so sensitive to timing error as is commonly believed, nor does the total signal amplitude become very large with significant probability. However, the error probability does grow very rapidly as the system bandwidth is reduced below the Nyquist band.

Sensor-array data processing for multiple-signal sources

Abstract: Techniques are considered for processing the outputs of a sensor array that is observing J distinct signal sources Three types of wideband signals are discussed: unknown, stochastic, and parameterized Narrowband signals are a special case Four types of random errors are discussed: additive sensor noise, sensor gain errors, sensor time-delay errors, and "beam-pointing' errors It is concluded that the so-called decoupled-beam data processor is a very promising technique, which can be implemented by passing the output of J individual "beams" through a J -input, J -output linear system When sensor gain-delay and beam-pointing errors are not present, the decoupled-beam data processor provides "infinite sidelobe rejection"

Multisurface method of pattern separation

Abstract: Let two sets of patterns be represented by two finite point sets in an n -dimensional Euclidean space E^{n} . If the convex hulls of the two sets do not intersect, the sets can be strictly separated by a plane. Such a plane can be constructed by the Motzkin-Schoenberg error-correction procedure or by linear programming. More often than not, however, the convex hulls of the two point sets do intersect, in which case strict separation by a plane is not possible any more. One may then resort to separation by more than one plane. In this paper, we show how two sets can be strictly separated by one or more planes or surfaces (nonlinear manifolds) via linear programming. A computer program that implements the present method has been written and successfully tested on a number of real problems.

Buffer overflow in variable length coding of fixed rate sources

Abstract: In this paper, we develop and analyze an easily instrumentable scheme for variable length encoding of discrete memoryless fixed-rate sources in which buffer overflows result in codeword erasures at locations that are perfectly specified to the user. Thus, no loss of synchronism ever occurs. We find optimal (i.e., minimizing the probability of buffer overflow) code-wold length requirements under the Kraft inequality constraint, relative to various constant transmission rates R , and show that these do not result in the minimal average code-word length. The corresponding bounds on the probability of buffer overflow provide a linkup between source coding and Renyi's generalized source entropy. We show, further, that codes having optimal word lengths can be constructed by the method of Elias, and we develop corresponding sequentially instrumented encoders and decoders. We show that the complexity of these encoders and decoders grows only linearly with the encoded message block length k , provided the size d of the coder alphabet is a power of 2 , and otherwise grows no worse than quadratically with k .

An analysis of the pseudo-randomness properties of subsequences of long m -sequences

Abstract: Expressions are derived for the moments of the distribution of weights of M -tuples or subsequences of long m -sequences. The expressions display a systematic relationship between the moments and the characteristic polynomial or generating function. An algorithm for determining the third moment, which measures skewness, is shown to provide a practical aid for selecting optimum m-sequences for various correlation-detection design problems.

The minimum variance of estimates in quantum signal detection

Abstract: A quantum-mechanical form of the Cramer-Rao inequality is derived, setting a lower bound to the variance of an unbiased estimate of a parameter of a density operator. It is applied to the estimates of parameters such as amplitude, arrival time, and carrier frequency of a coherent signal as picked up by an ideal receiver in the presence of thermal noise. The estimation of parameters of a noise-like signal is also treated.

Polynomial codes

Abstract: A class of cyclic codes is introduced by a polynomial approach that is an extension of the Mattson-Solomon method and of the Muller method. This class of codes contains several important classes of codes as subclasses, namely, BCH codes, Reed-Solomon codes, generalized primitive Reed-Muller codes, and finite geometry codes. Certain fundamental properties of this class of codes are derived. Some subclasses are shown to be majority-logic decodable.

Elliptically symmetric distributions

Abstract: Elliptically symmetric distributions are second-order distributions with probability densities whose contours of equal height are ellipses. This class includes the Gaussian and sine-wave distributions and others which can be generated from certain first-order distributions. Members of this class have several desirable features for the description of the second-order statistics of the transformation of a random signal by an instantaneous nonlinear device. In particular, they are separable in Nuttall's sense, so that the output of the device may be described in terms of equivalent gain and distortion. These distributions can also simplify the evaluation of the output autovariance because of their similarity to the Gaussian distribution. For a certain class of functions, elliptically symmetric distributions yield averages which are simply proportional to those obtained with a Gaussian distribution. Furthermore, these distributions satisfy a relation analogous to Price's theorem for Gaussian distributions. Finally, a certain subclass of these distributions can be expanded in the series representation studied by Barrett and Lampard.

On a class of majority-logic decodable cyclic codes

Abstract: A new infinite class of cyclic codes is studied. Codes of this class can be decoded in a step-by-step manner, using` majority logic. Some previously known codes fall in this class, and thus admit simpler decoding procedures. As random error-correcting codes, the codes are nearly as powerful as the Bose-Chaudhuri codes.

Group codes for the Gaussian channel (Abstr.)

Abstract: A class of equal-energy codes for use on the Gaussian channel is defined and investigated. Members of the class are eared group codes because of the manner in which they can be generated from a group of orthogonal matrices. Group codes possess an important symmetry property. Roughly speaking, all words in such a code are on an equal footing: each has the same error probability (under the assumptions of the usual model) and each has the same disposition of neighbors. A number of theorems about such codes are proved. A decomposition theorem shows every group code to be equivalent to a direct sum of certain basic group codes generated by real-irreducible representations of a finite group associated with the code. Some theorems on distances between words in group codes are demonstrated. The difficult problem of finding group codes with large nearest neighbor distance is discussed in detail and formulated in several ways. It is noted that linear (or group) codes for the binary channel can be regarded as very speciM cases of the group codes discussed. A definition of a group code for the Gaussian channel follows. An equal-energy code C with parameters M and n for this channel is a collection of M distinct unit n -vectors, X_{1}, X_{2}, \cdots , X_{M} say, that span a Euclidean n -space. An n \times n orthogonal matrix 0 is said to be a symmetry of C if the M vectors Y_{i} = 0X_{i}, i = 1, 2, \cdots , M are again the collection C . The set of all symmetries of C , say 0_{1}, 0_{2}, \cdots , 0_{g} , forms a group \cal{G}(C) under matrix multiplication. If \cal{G}(C) contains M elements 0_{\alpha_{1}}, 0_{\alpha_{2}}, \cdots , 0_{\alpha M} such that X_{i} = 0_{\alpha i}X_{1}, i = 1, 2, \cdots , M , then C is called a group code.

Nonbinary BCH decoding (Abstr.)

Abstract: The decoding of BCH codes readily reduces to the solution of a certain key equation. An iterative algorithm is presented for solving this equation over any field. Following a heuristic derivation of the algorithm, a complete statement of the algorithm and proofs of its principal properties are given. The relationship of this algorithm to the classical matrix methods and the simplification which the algorithm takes in the special case of binary codes is then discussed. The generalization of the algorithm to BCH codes with a slightly different definition, the generalization of the algorithm to decode erasures as well as errors, and the extension of the algorithm to decode more than t errors in certain eases are also presented.

A geometric treatment of the source encoding of a Gaussian random variable

Abstract: This paper gives a geometric treatment of the source encoding of a Gaussian random variable for minimum mean-square error. The first section is expository, giving a geometric derivation of Shannon's classic result [1] which explicitly shows the steps in source encoding and the properties that a near optimum code must possess. The second section makes use of the geometric insight gained in the first section to bound the performance that can be obtained with a finite block length of L random variables. It is shown that a code can be found whose performance approaches that of the rate distortion function as 1/L in mean-square error and (\ln L)/L in rate.

Optimum radar signal processing in clutter

Abstract: This paper considers the joint optimization of a class of radar signals and filters in a number of clutter-pins-noise environments. The radar signal processor in this case will be optimum in the sense that its output at the time of target detection yields the maximum ratio of peak signal power to total interference power. If the interference at the input to this signal processor is a Gaussian random process, this processor also yields the maximum probability of detection for a given value of false-alarm probability. The signals used are pulse trains and the filters are tapped delay lines. The purpose of signal design is to determine the optimum complex weighting for each pulse of the pulse train. Filter design yields the optimum complex weighting for the output taps of the delay line. Filter design for a specified signal is considered first. This is followed by combined signal and filter design and matched filter design. Constrained signal and filter design is investigated last. It should be emphasized that the optimizations require a knowledge of the clutter time-frequency distribution. For practical situations, when the clutter distribution is unknown, an adaptive filter is proposed that automatically provides the optimum filter weights for a given transmitted signal. When the clutter has a range-time extent less than the equivalent range-time extent of the signal, filter design alone yields nearly optimum performance. As the clutter becomes extended in range-time, it is necessary to consider jointly the design of signal and filter to obtain an optimum radar signal processor. In this report it is suggested that the signal be designed under the assumption of the clutter being extended over a broad range of Dopplers and that the signal processor consist of a bank of adaptive filters. Then each filter output yields the maximum ratio of peak signal to total interference power for this signal design.

A generalized concept of frequency and some applications

Abstract: The system of sine and cosine functions has been distinguished historically in communications. Whenever the term frequency is used, reference is made implicitly to these functions; hence the generally used theory of communication is based on the system of sine and cosine functions. In recent years other complete systems of orthogonal functions have been used for theoretical investigations as well as for equipment design. Analogs to Fourier series, Fourier transform, frequency, power spectra, and amplitude, phase, and frequency modulation exist for many systems of orthogonal functions. This implies that theories of communication can be worked out on the basis of these systems. Most of these theories are of academic interest only. However, for the complete system of the orthogonal Walsh functions, the implementation of circuits by modem semiconductor techniques appears to be competitive in a number of applications with the implementation of circuits for the system of sine and cosine functions.

New generalizations of the Reed-Muller codes--II: Nonprimitive codes

Abstract: In this paper a class of nonprimitive cyclic codes quite similar in structure to the original Reed-Muller codes is presented. These codes, referred to herein as nonprimitive Reed-Muller codes, are shown to possess many of the properties of the primitive codes. Specifically, two major results are presented. First the code length, number of information symbols, and minimum distance are shown to be related by means of a parameter known as the order of the code. These relationships show that for given values of code length and rate the codes have relatively large minimum distances. It is also shown that the codes are subcodes of the BCH codes of the same length and guaranteed minimum distance; thus in general the codes are not as powerful as the BCH codes. However, for most interesting values of code length and rate the difference between the two types of codes is slight. The second result is the observation that the codes can be decoded with a variation of the original algorithm proposed by Reed for the Reed-Muller codes. In other words, they are L -step orthogonalizable. Because of their large minimum distances and the simplicity of their decoders, nonprimitive Reed-Muller codes seem attractive for use in error-control systems requiring multiple random-error correction.

A 'best' mismatched filter response for radar clutter discrimination

Abstract: The response of a mismatched filter which will reduce the effects of radar clutter energy on signal detection is considered in this paper. Clutter is assumed to result from scatterers having a range-velocity distribution different from that of a desired target. A tradeoff between signal-to-clutter ratio (SCR) and signal-to-noise ratio (SNR) is made. The mathematical treatment involves controlling the distribution of the volume of the squared magnitude t/f cross-correlation function associated with a given transmitted waveform and the response of the mismatched receiving filter. A measure of this volume is minimized for variations in mismatched filter response under suitable constraints on the output signal and noise powers. The essential result is contained in a linear integral equation. A method of solution is illustrated which enables one to choose a "best" tradeoff between SCR and SNR.

On a class of processes arising in linear estimation theory

Abstract: This paper considers a class of stochastic processes, called {\em spherically invariant},which have the property that all mean-square estimation problems on them have linear solutions. It is shown that their multivariate characteristic functions are univariate functions of a quadratic form. The corresponding densities are easily found by means of the Hankel transform. Relations between spherical invariance and normality are discussed. Properties relating to the linear estimation problem are given.