Showing papers in "IEEE Transactions on Information Theory in 1976"
TL;DR: This paper suggests ways to solve currently open problems in cryptography, and discusses how the theories of communication and computation are beginning to provide the tools to solve cryptographic problems of long standing.
Abstract: Two kinds of contemporary developments in cryptography are examined. Widening applications of teleprocessing have given rise to a need for new types of cryptographic systems, which minimize the need for secure key distribution channels and supply the equivalent of a written signature. This paper suggests ways to solve these currently open problems. It also discusses how the theories of communication and computation are beginning to provide the tools to solve cryptographic problems of long standing.
14,980 citations
TL;DR: The quantity R \ast (d) is determined, defined as the infimum ofrates R such that communication is possible in the above setting at an average distortion level not exceeding d + \varepsilon .
Abstract: Let \{(X_{k}, Y_{k}) \}^{ \infty}_{k=1} be a sequence of independent drawings of a pair of dependent random variables X, Y . Let us say that X takes values in the finite set \cal X . It is desired to encode the sequence \{X_{k}\} in blocks of length n into a binary stream of rate R , which can in turn be decoded as a sequence \{ \hat{X}_{k} \} , where \hat{X}_{k} \in \hat{ \cal X} , the reproduction alphabet. The average distortion level is (1/n) \sum^{n}_{k=1} E[D(X_{k},\hat{X}_{k})] , where D(x,\hat{x}) \geq 0, x \in {\cal X}, \hat{x} \in \hat{ \cal X} , is a preassigned distortion measure. The special assumption made here is that the decoder has access to the side information \{Y_{k}\} . In this paper we determine the quantity R \ast (d) , defined as the infimum ofrates R such that (with \varepsilon > 0 arbitrarily small and with suitably large n )communication is possible in the above setting at an average distortion level (as defined above) not exceeding d + \varepsilon . The main result is that R \ast (d) = \inf [I(X;Z) - I(Y;Z)] , where the infimum is with respect to all auxiliary random variables Z (which take values in a finite set \cal Z ) that satisfy: i) Y,Z conditionally independent given X ; ii) there exists a function f: {\cal Y} \times {\cal Z} \rightarrow \hat{ \cal X} , such that E[D(X,f(Y,Z))] \leq d . Let R_{X | Y}(d) be the rate-distortion function which results when the encoder as well as the decoder has access to the side information \{ Y_{k} \} . In nearly all cases it is shown that when d > 0 then R \ast(d) > R_{X|Y} (d) , so that knowledge of the side information at the encoder permits transmission of the \{X_{k}\} at a given distortion level using a smaller transmission rate. This is in contrast to the situation treated by Slepian and Wolf [5] where, for arbitrarily accurate reproduction of \{X_{k}\} , i.e., d = \varepsilon for any \varepsilon >0 , knowledge of the side information at the encoder does not allow a reduction of the transmission rate.
3,288 citations
TL;DR: A new approach to the problem of evaluating the complexity ("randomness") of finite sequences is presented, related to the number of steps in a self-delimiting production process by which a given sequence is presumed to be generated.
Abstract: A new approach to the problem of evaluating the complexity ("randomness") of finite sequences is presented. The proposed complexity measure is related to the number of steps in a self-delimiting production process by which a given sequence is presumed to be generated. It is further related to the number of distinct substrings and the rate of their occurrence along the sequence. The derived properties of the proposed measure are discussed and motivated in conjunction with other well-established complexity criteria.
2,473 citations
TL;DR: A method of analysis is presented for the class of binary sequence generators employing linear feedback shift registers with nonlinear feed-forward operations, of special interest because the generators are capable of producing very long "unpredictable" sequences.
Abstract: A method of analysis is presented for the class of binary sequence generators employing linear feedback shift registers with nonlinear feed-forward operations. This class is of special interest because the generators are capable of producing very long "unpredictable" sequences. The period of the sequence is determined by the linear feedback connections, and the portion of the total period needed to predict the remainder is determined by the nonlinear feed-forward operations. The linear feedback shift registers are represented in terms of the roots of their characteristic equations in a finite field, and it is shown that nonlinear operations inject additional roots into the representation. The number of roots required to represent a generator is a measure of its complexity, and is equal to the length (number of stages) of the shortest linear feedback shift register that produces the same sequence. The analysis procedure can be applied to any arbitrary combination of binary shift register generators, and is also applicable to the synthesis of complex generators having desired properties. Although the discussion in this paper is limited to binary sequences, the analysis is easily extended to similar devices that generate sequences with members in any finite field.
274 citations
TL;DR: A segmentation algorithm is proposed which uses dynamic programming (Viterbi algorithm with three states) and a simpler method that makes possible the estimation of the n most probable displacements is proposed.
Abstract: Various techniques are described to measure, small displacements of television images. If two successive video frames are considered, their differences are approximately a linear combination of the components of the displacement of the object. If all the points of the frame undergo the same movement, then the velocity estimation problem is solved using linear estimation. However, if some points belong to the moving object and the others to the background, the problem can be stated in the same way only if an algorithm is available to segment the image into fixed and moving areas. Afterwards, linear estimation can be applied to the moving area only. In this paper a segmentation algorithm is proposed which uses dynamic programming (Viterbi algorithm with three states). A more complex situation arises when the points belonging to the moving area are subjected to different movements. The problem can be solved once more using dynamic programming if the displacement components are quantized into (2M + 1) (2M + 1) values, and the number of states of the Viterbi algorithm is augmented to (2M + 1)^{2} . To reduce the technical difficulties of this approach, a simpler method that makes possible the estimation of the n most probable displacements is proposed. Then the image is segmented into n moving areas with different displacements and a background area using a Viterbi algorithm with n + 1 states. Experimental results show that the precision obtainable is about 0.1 pel when the displacements are up to 2-3 pels, the object had approximate dimensions of 90 \times 90 pels, and the signal-to-noise ratio was higher than 33 dB.
273 citations
TL;DR: A discrete random variable X is to be transmitted by means of a discrete signal so that the probability of error must be exactly zero, and the problem is to minimize the signal's alphabet size.
Abstract: A discrete random variable X is to be transmitted by means of a discrete signal. The receiver has prior knowledge of a discrete random variable Y jointly distributed with X . The probability of error must be exactly zero, and the problem is to minimize the signal's alphabet size. In the case where the transmitter also has access to the value of Y , the problem is trivial and no advantage can be obtained by block coding over independent repetitions. If, however, Y is not known at the transmitter then the problem is equivalent to the chromatic number problem for graphs, and block coding may produce savings.
245 citations
TL;DR: This correspondence proposes, based on a random sample X_{1, \cdots, X_{n} generated from F, a nonparametric estimate of H(f) given by -(l/n) \sum_{i = 1}^{n} \In \hat{f}(x) , where f is the kernel estimate of f due to Rosenblatt and Parzen.
Abstract: Let F(x) be an absolutely continuous distribution having a density function f(x) with respect to the Lebesgue measure. The Shannon entropy is defined as H(f) = -\int f(x) \ln f(x) dx . In this correspondence we propose, based on a random sample X_{1}, \cdots , X_{n} generated from F , a nonparametric estimate of H(f) given by \hat{H}(f) = -(l/n) \sum_{i = 1}^{n} \In \hat{f}(x) , where \hat{f}(x) is the kernel estimate of f due to Rosenblatt and Parzen. Regularity conditions are obtained under which the first and second mean consistencies of \hat{H}(f) are established. These conditions are mild and easily satisfied. Examples, such as Gamma, Weibull, and normal distributions, are considered.
230 citations
TL;DR: A decoding rule is presented which minimizes the probability of symbol error over a time-discrete memory]ess channel for any linear error-correcting code when the codewords are equiprobable.
Abstract: A decoding rule is presented which minimizes the probability of symbol error over a time-discrete memory]ess channel for any linear error-correcting code when the codewords are equiprobable. The complexity of this rule varies inversely with code rate, making the technique particularly attractive for high rate codes. Examples are given for both block and convolutional codes.
198 citations
TL;DR: This paper considers the problem of efficient transmission of vector sources over a digital noiseless channel and gives the optimally decorrelating scheme for a source whose components are dependent and treats the problems of selecting the optimum characteristic of the encoding scheme such that the overall mean-squared error is minimized.
Abstract: This paper considers the problem of efficient transmission of vector sources over a digital noiseless channel. It treats the problem of optimal allocation of the total number of available bits to the components of a memoryless stationary vector source with independent components. This allocation is applied to various encoding schemes, such as minimum mean-square error, sample-by-sample quantization, or entropy quantization. We also give the optimally decorrelating scheme for a source whose components are dependent and treat the problems of selecting the optimum characteristic of the encoding scheme such that the overall mean-squared error is minimized. Several examples of encoding schemes, including the ideal encoder that achieves the rated istortion bound, and of sources related to a practical problem are discussed.
184 citations
TL;DR: A lower bound on the minimal mean-square error in estimating nonlinear diffusion processes is derived and the bound holds for causal and noncausal filtering.
Abstract: A lower bound on the minimal mean-square error in estimating nonlinear diffusion processes is derived. The bound holds for causal and noncausal filtering.
133 citations
TL;DR: Block codes which are uniquely decodable and capable of correcting errors are constructed in this paper for multiple-access discrete memoryless channel coding.
Abstract: In this paper, coding for a multiple-access discrete memoryless channel is investigated. Block codes which are uniquely decodable and capable of correcting errors are constructed.
TL;DR: Basic limitations on the amount of protocol information that must be transmitted in a data communication network to keep track of source and receiver addresses and of the starting and stopping of messages are considered.
Abstract: We consider basic limitations on the amount of protocol information that must be transmitted in a data communication network to keep track of source and receiver addresses and of the starting and stopping of messages. Assuming Poisson message arrivals between each communicating source-receiver pair, we find a lower bound on the required protocol information per message. This lower bound is the sum of two terms, one for the message length information, which depends only on the distribution of message lengths, and the other for the message start information, which depends only on the product of the source-receiver pair arrival rate and the expected delay for transmitting the message. Two strategies are developed which, in the limit of large numbers of sources and receivers, almost meet the lower bound on protocol information.
TL;DR: An application of the estimation schemes derived in the paper to the estimation of the state of a random time-division multiple access (ALOHA-type) computer network is presented.
Abstract: The paper presents models for discrete-time point processes (DTPP) and schemes for recursive estimation of signals randomly influencing their rates. Although the models are similar to the better known models of signals in additive Gaussian noise, DTPP differ from these in that it is possible for DTPP's to find recursive representations for the nonlinear filters. If the signal can be modeled as a finite state Markov process, then these representations reduce to explicit recursive finite-dimensional filters. The derivation of the estimation schemes, as well as the filters themselves, present a surprising similarity to the Kalman filters for signals in Gaussian noise. We present finally an application of the estimation schemes derived in the paper to the estimation of the state of a random time-division multiple access (ALOHA-type) computer network.
TL;DR: A consistent estimator is discussed which is computationally more efficient than estimators based on Parzen's estimation and its relation between the distance of a sample from the decision boundary and its contribution to the error is derived.
Abstract: The L^{ \alpha} -distance between posterior density functions (PDF's) is proposed as a separability measure to replace the probability of error as a criterion for feature extraction in pattern recognition. Upper and lower bounds on Bayes error are derived for \alpha > 0 . If \alpha = 1 , the lower and upper bounds coincide; an increase (or decrease) in \alpha loosens these bounds. For \alpha = 2 , the upper bound equals the best commonly used bound and is equal to the asymptotic probability of error of the first nearest neighbor classifier. The case when \alpha = 1 is used for estimation of the probability of error in different problem situations, and a comparison is made with other methods. It is shown how unclassified samples may also be used to improve the variance of the estimated error. For the family of exponential probability density functions (pdf's), the relation between the distance of a sample from the decision boundary and its contribution to the error is derived. In the nonparametric case, a consistent estimator is discussed which is computationally more efficient than estimators based on Parzen's estimation. A set of computer simulation experiments are reported to demonstrate the statistical advantages of the separability measure with \alpha = 1 when used in an error estimation scheme.
TL;DR: It is shown that the unit-memory codes are byte oriented in such a way as to be attractive for use in concatenated coding systems.
Abstract: It is shown that (n_{o},k_{o}) convolutional codes with unit memory always achieve the largest free distance among all codes of the same rate k_{o}/n_{o} and same number 2^{Mk_{o} of encoder states, where M is the encoder memory. A unit-memory code with maximal free distance is given at each place where this free distance exceeds that of the best code with k_{o} and n_{o} relatively prime, for all Mk_{o} \leq 6 and for R = l/2, 1/3, 1/4, 2/3 . It is shown that the unit-memory codes are byte oriented in such a way as to be attractive for use in concatenated coding systems.
TL;DR: Empirical justification is established for the common practice of applying the Kalman filter estimator to three classes of linear quadratic problems where the model statistics are not completely known, and hence specification of the filter gains is not optimum.
Abstract: In this paper, theoretical justification is established for the common practice of applying the Kalman filter estimator to three classes of linear quadratic problems where the model statistics are not completely known, and hence specification of the filter gains is not optimum. The Kalman filter is shown to be a minimax estimator for one class of problems and to satisfy a saddlepoint condition in the other two classes of problems. Equations for the worst case covariance matrices are given which allow the specifications of the minimax Kalman filter gains and the worst case distributions for the respective classes of problems. Both time-varying and time-invariant systems are treated.
TL;DR: A theorem characterizing the form of SS random vectors X is proved and the problem of detecting a known signal vector in the presence of X + N when \rho =I is looked at.
Abstract: A random n -vector X=(X_{1}, \cdots , X_{n}) is said to be spherically symmetric (SS) if its joint characteristic function (CF) can be expressed as a function of the quadratic form u \rho u \prime , where u = (u_{1}, \cdots , u_{n}) and \rho is an n \times n positive definite matrix. The investigation in this paper is concerned with the properties of such vectors and some detection problems involving them. We first prove a theorem characterizing the form of SS random vectors X and use it to find the form of the probability density functions (pdf's) of X and of X + N , where N \sim {\cal N (0, \sigma^{2}I) is an independent Gaussian vector and I is the identity matrix. Applying these results we look at the problem of detecting a known signal vector in the presence of X + N when \rho =I . For the k -ary detection problem we present two conditions under which the "minimum distance" receiver is optimum. Lastly, we discuss an application of our findings to the problem of coherent detection of binary phase-shift keyed (PSK) signals in the presence of multiple co-channel interferences and white Gaussian noise.
TL;DR: Two closely related series representations for a band-limited signal of finite energy are given, which generalize the Shannon sampling series to the case of irregularly spaced sample points.
Abstract: Two closely related series representations for a band-limited signal of finite energy are given. They both generalize the Shannon sampling series to the case of irregularly spaced sample points. One of them is itself a sampling series, and a method is indicated for calculating its terms explicitly in certain cases.
TL;DR: The sign detector is shown to be the asymptotically most robust detector when g(x) is a double-exponential density.
Abstract: The Tukey-Huber contaminated noise model is used toobtain rain-max detectors in the asymptotic case for known signals inadditive noise. According to this model, the noise density f(x) is defined by f(x) = (1 - \ )g(x) + \varepsilon h(x) for a given \varepsilon and density g(x) , with h(x) an arbitrary density from a large class. A general theorem is obtainedspecifying the most robust detector for additive contaminated noisewith g(x) satisfying certain regularity conditions. As an example,detector structures are derived by the application of the theorem for the case where g(x) belongs to the class of generalized Gaussian densities(parameterized by their rates of exponential decay). The sign detector is shown to be the asymptotically most robust detector when g(x) is a double-exponential density.
TL;DR: In this paper, the Reed-Solomon codes can be decoded with O(q \log 2 q ) additions and multiplications in GF(q) and O( q \log n 2 q) multiplications.
Abstract: Certain q -ary Reed-Solomon codes can be decoded by an algorithm requiring only O(q \log^{2} q) additions and multiplications in GF(q)
TL;DR: An iterative algorithm is developed for computing an approximation to this Markov spectrum for a regularly spaced array which approximates the desired Markov correlation function by a truncated convolution power series in an operator h .
Abstract: A constructive proof is given for the existence and uniqueness of a two-dimensional discrete Markov random field which agrees with correlation values in a nearest neighbor array. The corresponding spectrum is the two-dimensional maximum entropy (ME) spectrum whose form was discovered by Burg. An iterative algorithm is developed for computing an approximation to this Markov spectrum for a regularly spaced array. The algorithm approximates the desired Markov correlation function by a truncated convolution power series (CPS) in an operator h . The algorithm's performance is demonstrated on both simulated data and real noise data. The Markov spectral estimate can offer higher resolution than previously proposed spectral estimates.
TL;DR: In this paper the connection between the self-information of a source letter from a finite alphabet and its code-word length in a Huffman code is investigated.
Abstract: In this paper the connection between the self-information of a source letter from a finite alphabet and its code-word length in a Huffman code is investigated. Consider the set of all independent finite alphabet sources which contain a source letter a of probability p . The maximum over this set of the length of a Huffman codeword for a is determined. This maximum remains constant as p varies between the reciprocal values of two consecutive Fibonacci numbers. For the small p this maximum is approximately equal to \left[ \log_{2} \frac{1+ \sqrt{5}}{2} \right]^{-1} \approx 1.44 times the self-information.
TL;DR: A simple algorithm is given for correcting multiple errors which makes use of continued fractions and is based on residue encoding.
Abstract: Multiple-error-correcting arithmetic codes which are nonlinear are constructed by residue encoding. A simple algorithm is given for correcting multiple errors which makes use of continued fractions.
TL;DR: It is shown that Goppa codes with Goppa polynomial g(z) have the parameters: length n \leq q^{m} - s_{o} , number of check symbols n - k \lequ m (q - 1) (\deg g) , and minimum distance d \geq q (\ Deg g) + 1.
Abstract: It is shown that Goppa codes with Goppa polynomial \{g(z)\}^{q} have the parameters: length n \leq q^{m} - s_{o} , number of check symbols n - k \leq m (q - 1) (\deg g) , and minimum distance d \geq q (\deg g) + 1 , where q is a prime power, m is an integer, g(z ) is an arbitrary polynomial over GF(q^{m}) , and so is the number of roots of g(z) which belong to GF(q^{m}) . It is also shown that all binary Goppa codes of length n \leq 2^{m} - s_{o} satisfy the relation n - k \leq m (d - 1)/2 . A new class of binary codes with n \leq 2^{ m} + ms _{0}, n - k \leq m (\deg g) + s_{0} , and d \leq 2(\deg g) + 1 is constructed, as well as another class of binary codes with slightly different parameters. Some of those codes are proved superior to the best codes previously known. Finally, a decoding algorithm is given for the codes constructed which uses Euclid's algorithm.
TL;DR: The probability density and confidence intervals for the maximum entropy (or regression) method (MEM) of spectral estimation are derived using a Wishart model for the estimated covariance and asymptotic expressions are derived which are the same as those of Akaike.
Abstract: The probability density and confidence intervals for the maximum entropy (or regression) method (MEM) of spectral estimation are derived using a Wishart model for the estimated covariance. It is found that the density for the estimated transfer function of the regression filter may be interpreted as a generalization of the student's t distribution. Asymptotic expressions are derived which are the same as those of Akaike. These expressions allow a direct comparison between the performance of the maximum entropy (regression) and maximum likelihood methods under these asymptotic conditions. Confidence intervals are calculated for an example consisting of several closely space tones in a background of white noise. These intervals are compared with those for the maximum likelihood method (MLM). It is demonstrated that, although the MEM has higher peak to background ratios than the MLM, the confidence intervals are correspondingly larger. Generalizations are introduced for frequency wavenumber spectral estimation and for the joint density at different frequencies.
TL;DR: A "universal" generalization of syndrome-source-coding is formulated which provides robustly effective distortionless coding of source ensembles and can achieve arbitrarily small distortion with the number of compressed digits per source digit arbitrarily close to the entropy of a binary memoryless source.
Abstract: A method of using error-correcting codes to obtain data compression, called syndrome-source-coding, is described in which the source sequence is treated as an error pattern whose syndrome forms the compressed data. It is shown that syndrome-source-coding can achieve arbitrarily small distortion with the number of compressed digits per source digit arbitrarily close to the entropy of a binary memoryless source. A "universal" generalization of syndrome-source-coding is formulated which provides robustly effective distortionless coding of source ensembles. Two examples are given comparing the performance of noiseless universal syndrome-source-coding to 1) run-length coding and 2) Lynch-Davisson-Schalkwijk-Cover universal coding for an ensemble of binary memoryless sources.
TL;DR: This work shows how the backwards model can be used to clarify certain least squares smoothing formulas, and illustrates how this problem can be solved.
Abstract: A state-space model of a second-order random process is a representation as a linear combination of a set of state-variables which obey first-order linear differential equations driven by an input process that is both white and uncorrelated with the initial values of the state-variables. Such a representation is often called a Markovian representation. There are applications in which it is useful to consider time running backwards and to obtain corresponding backwards Markovian representations. Except in one very special circumstance, these backwards representations cannot be obtained simply by just reversing the direction of time in a forwards Markovian representation. We show how this problem can be solved, give some examples, and also illustrate how the backwards model can be used to clarify certain least squares smoothing formulas.
TL;DR: A set of functionals of general square-integrable martingales is presented which, in the case of independent-increments processes, is orthogonal and complete in the sense that every L^{2} -functional of the independent- increase process can be represented as an infinite sum of these elementary functionals.
Abstract: In analogy with the Wiener-Ito theory of multiple integrals and orthogonal polynominals, a set of functionals of general square-integrable martingales is presented which, in the case of independent-increments processes, is orthogonal and complete in the sense that every L^{2} -functional of the independent-increment process can be represented as an infinite sum of these elementary functionals. The functionals are iterated integrals of the basic martingales, similar to the multiple iterated integrals of Ito and can be also thought of as being the analogs of the powers 1,x,x^{2}, \cdots of the usual calculus. The analogy is made even clearer by observing that expanding the Doleans-Dade formula for the exponential of the process in a Taylor-like series leads again to the above elementary functionals. A recursive formula for these functionals in terms of the basic martingale and of lower order functionals is given, and several connections with the theory of reproducing kernel Hilbert spaces associated with independent-increment processes are obtained.
TL;DR: Block codes are constructed that are capable of simultaneously correcting synchronization errors in t consecutive words, for any t \geq 2e + 1 , and s or fewer substitution errors in each of t - 1 or fewer of these words under the condition that there exists at least one ungarbled word among the t consecutive Words.
Abstract: Block codes are constructed that are capable of simultaneously correcting e or fewer synchronization errors in t consecutive words, for any t \geq 2e + 1 , and s or fewer substitution errors in eachof t - 1 or fewer of these words under the condition that there exists at least one ungarbled word among the t consecutive words. Also, some new extensions of the A_{n}^{c} codes of Calabi and Hartnett are presented under the condition that synchronization and substitution errors do not coexist in the t consecutive words.
TL;DR: It is shown, in the simplest context, that in nonlinear estimation theory martingales play the same fundamental role as uncorrelation and white noise do in linear estimation.
Abstract: We describe the role of various stochastic processes, especially martingales and related concepts, in estimation theory. It is shown, in the simplest context, that in nonlinear estimation theory martingales play the same fundamental role as uncorrelation and white noise do in linear estimation.