Showing papers in "IEEE Transactions on Information Theory in 1980"
TL;DR: Jaynes's principle of maximum entropy and Kullbacks principle of minimum cross-entropy (minimum directed divergence) are shown to be uniquely correct methods for inductive inference when new information is given in the form of expected values.
Abstract: Jaynes's principle of maximum entropy and Kullbacks principle of minimum cross-entropy (minimum directed divergence) are shown to be uniquely correct methods for inductive inference when new information is given in the form of expected values. Previous justifications use intuitive arguments and rely on the properties of entropy and cross-entropy as information measures. The approach here assumes that reasonable methods of inductive inference should lead to consistent results when there are different ways of taking the same information into account (for example, in different coordinate system). This requirement is formalized as four consistency axioms. These are stated in terms of an abstract information operator and make no reference to information measures. It is proved that the principle of maximum entropy is correct in the following sense: maximizing any function but entropy will lead to inconsistency unless that function and entropy have identical maxima. In other words given information in the form of constraints on expected values, there is only one (distribution satisfying the constraints that can be chosen by a procedure that satisfies the consistency axioms; this unique distribution can be obtained by maximizing entropy. This result is established both directly and as a special case (uniform priors) of an analogous result for the principle of minimum cross-entropy. Results are obtained both for continuous probability densities and for discrete distributions.
1,774 citations
TL;DR: A probabilistic method is presented which cryptanalyzes any N key cryptosystem in N 2/3 operational with N2/3 words of memory after a precomputation which requires N operations, and works in a chosen plaintext attack and can also be used in a ciphertext-only attack.
Abstract: A probabilistic method is presented which cryptanalyzes any N key cryptosystem in N^{2/3} operational with N^{2/3} words of memory (average values) after a precomputation which requires N operations. If the precomputation can be performed in a reasonable time period (e.g, several years), the additional computation required to recover each key compares very favorably with the N operations required by an exhaustive search and the N words of memory required by table lookup. When applied to the Data Encryption Standard (DES) used in block mode, it indicates that solutions should cost between 1 and 100 each. The method works in a chosen plaintext attack and, if cipher block chaining is not used, can also be used in a ciphertext-only attack.
761 citations
TL;DR: It is shown that a source of independent identically distributed discrete random variables with joint probability mass function p(u,v) and common part w=f(u)=g(v) can be sent with arbitrarily small probability of error over a multiple access channel (MAC)
Abstract: Let \{(U_{i},V_{i})\}_{i=1}^{n} be a source of independent identically distributed (i.i.d.) discrete random variables with joint probability mass function p(u,v) and common part w=f(u)=g(v) in the sense of Witsenhausen, Gacs, and Korner. It is shown that such a source can be sent with arbitrarily small probability of error over a multiple access channel (MAC) \{\cal X_{1} \times \cal X_{2},\cal Y,p(y|x_{1},x_{2})\}, with allowed codes \{x_{l}(u), x_{2}(v)\} if there exist probability mass functions p(s), p(x_{1}|s,u),p(x_{2}|s,v) , such that H(U|V) H(V|U ) H(U,V|W) H(U,V) \mbox{where} p(s,u,v,x_{1},x_{2},y), Xl, X2, y)=p(s)p(u,v)p(x_{1}|u,s)p(x_{2}|v,s)p(y|x_{1},x_{2}). lifts region includes the multiple access channel region and the Slepian-Wolf data compression region as special cases.
475 citations
TL;DR: It was shown that homodyne detection achieves the same signal-to-noise ratio as the quantum field quadrature measurement, thus providing a receiver which realizes the linear modulation TCS performance gain found in Part I.
Abstract: In Part I of this three-part study it was shown that the use of two-photon coherent state (TCS) radiation may yield siginificant performance gains in free-space optical communicatinn if the receiver makes a quantum measurement of a single field quadrature In Part II it was shown that homodyne detection achieves the same signal-to-noise ratio as the quantum field quadrature measurement, thus providing a receiver which realizes the linear modulation TCS performance gain found in Part I Furthermore, it was shown in Part il that ff homodyne detection does exactly correspond to the field quadrature measurement, then a large binary communication performance gain is afforded by homodyne detection of antipodal TCS signals The full equivalence of honmdyne detection and single-quadrature field measurement, as well as that of heterodyne detection and two-quadrature field measurement, is established Furthermore, a heterodyne configuration which uses a TCS image-band oscillator in addition to the usual coherent state local oscillator is studied This coafiguration termed TCS heterodyne detection is shown to realize all the quantum measurements described by arbitrary TCS The foregoing results are obtained by means of a representation theorem which shows that photoemissive detection realizes the photon flux density measurement
471 citations
TL;DR: Several lower bounds for A(n,2\delta,w) are given, better than the "Gilbert bound" in most cases.
Abstract: Let A(n,2\delta,w) denote the maximum number of codewords in any binary code of length n , constant weight w , and Hamming distance 2\delta Several lower bounds for A(n,2\delta,w) are given. For w and \delta fixed, A(n,2\delta,w) \geq n^{W-\delta+l}/w! and A(n,4,w)\sim n^{w-l}/w! as n \rightarrow \infty . In most cases these are better than the "Gilbert bound." Revised tables of A(n,2 \delta,w) are given in the range n \leq 24 and \delta \leq 5 .
286 citations
TL;DR: The Rivest, Shamir, and Adleman (RSA) public-key encryption algorithm can be broken if the integer R used as the modulus can be factored as discussed by the authors.
Abstract: The Rivest, Shamir, and Adleman (RSA) public-key encryption algorithm can be broken if the integer R used as the modulus can be factored. It my however be possible to break this system without factoring R . A modification of the RSA scheme is described. For this modified version it is shown that, if the encryption procedure can be broken in a certain number of operations, then R can be factored in only a few more operations. Furthermore, this technique can also be used to produce digital signatures, in much the same manner as the RSA scheme.
285 citations
TL;DR: Three types of families of complex periodic sequences are shown to have nearly minimal correlation magnitudes and the sequences contain no zero entries.
Abstract: Three types of families of complex periodic sequences are shown to have nearly minimal correlation magnitudes. The sequences contain no zero entries. For certain pairs of sequences of period N=(p- 1)/2, p a prime, all nonpeak correlation coefficients have magnitude close to (2N)^{-1/2} .
226 citations
TL;DR: A number of new results are presented including additional kernels and lengths of polyphase complementary codes, including complementary Barker codes, and a table of known lengths of pairs, triads and quads of sequences is given.
Abstract: Previous work on polyphase complementary codes is reviewed, and a number of new results are presented including additional kernels and lengths. Generating methods from other binary and polyphase sequences are given and negative results of other existence searches reported. Advantages of specialized forms for radar and Loran-C are discussed, including complementary Barker codes. A table of known lengths of pairs, triads and quads of sequences is given.
222 citations
TL;DR: The capacity of a Poisson-type channel subject both to peak amplitude and average energy constraints is calculated, and its relation to the cutoff rate given by Snyder and Rhodes is determined in the "very noisy" case.
Abstract: The capacity of a Poisson-type channel subject both to peak amplitude and average energy constraints is calculated, and its relation to the cutoff rate given by Snyder and Rhodes is determined in the "very noisy" case.
181 citations
TL;DR: It is shown that, for determining file achievable rate region, attention may be restricted to source networks of a relatively simple structure and exponential error bounds are derived which are attainable by universal codes.
Abstract: A unified approach to multiterminal source coding problems not involving rate-distortion theory is presented. It is shown that, for determining file achievable rate region, attention may be restricted to source networks of a relatively simple structure. A product space characterizafion of the achievable rate region pinpoints the mathematical problem to be solved for getting a single letter characterization. The complexity of this problem depends on a structural condition, viz., the number of encoders of a certain kind in the source network. This approach yields all the known single-letter characterizations of achievable rate regions and a number of new ones for more complex networks. As a digression, for a class of source networks including that of Slepian and Wolf, exponential error bounds are derived which are attainable by universal codes. These bounds are tight in a neighborhood of the boundary of the achievable rate region.
175 citations
TL;DR: The Dobrushin-Tsybakov reduction of this problem to a direct one is shown to follow at once from a "disconnection principle" for conditional expectations.
Abstract: The output of a source, the first of the two arguments of a distortion function, is seen by an encoder through a noisy channel. A decoder sees the encoder's signal through the usual communication channel. The second argument of the distortion function is obtained from the decoder's output via another noisy channel. The Dobrushin-Tsybakov reduction of this problem to a direct one is shown to follow at once from a "disconnection principle" for conditional expectations. The same principle applies to more general situations such as i) dependence between the noise variables acting in the input and output channels and ii) side information available only at the decoder.
TL;DR: Gersho's bounds on the asymptotic performance of block quantizers are valid for vector distortion measures that are powers of the Euclidean or l_{2} norm, and this generalization provides a k -dimensional generalization of Gish and Pierce's results for single-symbol quantizers.
Abstract: Gersho's bounds on the asymptotic (large rate or small distortion) performance of block quantizers are valid for vector distortion measures that are powers of the Euclidean or l_{2} norm. These results are generalized to difference distortion measures that are increasing functions of the seminorm of their argument, where any seminorm is allowed. This provides a k -dimensional generalization of Gish and Pierce's results for single-symbol quantizers. When the distortion measore is a power of a seminorm the bounds are shown to be strictly better than the corresponding bounds provided by the k th-order rate-distortion functions.
TL;DR: It is shown that the reliability function of the proposed decoding algorithm is asymptotically twice that of the Viterbi decoding algorithm without repeat request, and that in certain practical situations the proposed algorithm can save about 50 percent in constraint length over the ordinary Viterba algorithm for a given performance.
Abstract: Using the Viterbi decoding algorithm with repeat request for convolutional codes is proposed, and the resulting performance is analyzed by random coding and generating function arguments and by simulation. It is shown that the reliability function of the proposed decoding algorithm is asymptotically twice that of the Viterbi decoding algorithm without repeat request, and that in certain practical situations the proposed algorithm can save about 50 percent in constraint length over the ordinary Viterbi algorithm for a given performance.
TL;DR: A unified treatment of a large class of multiterminal noiseless source coding problems including all previously studied situations is presented and a unified achievable rate region is established by a coding technique based on the typical sequence criterion.
Abstract: A unified treatment of a large class of multiterminal noiseless source coding problems including all previously studied situations is presented. A unified achievable rate region is established for this class by a coding technique based on the typical sequence criterion. This region is tight for all the previously studied situations.
TL;DR: A source matching approach to the problem of finding minimax cedes for classes of memoryless sources leads to a channel capacity problem so that Blahut's algorithm can be used to find approximations to the minimax code.
Abstract: A source matching approach is presented to the problem of finding minimax cedes for classes of memoryless sources. The approach leads to a channel capacity problem so that Blahut's algorithm can be used to find approximations to the minimax code. Closed form solutions are presented for the class of monotonic sources and for a class of Bernoulli-like sources. For extensions of finite alphabet memoryless sources, a modified Lynch-Davisson code has performance close to that of the minimax code. The exact solution to the source matching problem and the resulting codes are presented for the extensions of binary codes up to blocklength 31.
TL;DR: This chapter discusses convexity arguments in information theory from the point of view of convex sets and functions, and some examples show explicit use shown by examples.
Abstract: From its very beginning, information theory has been pervaded by convexity arguments. Much of the necessary background was developed on an ad hoc basis without reference to the knowledge available from the mathematical study of convex sets and functions. Yet explicit use shown by examples.
TL;DR: In this paper, it was shown that the Reed-Muller code holds for all m = 0, 1, 2, and for m = 3, where m is the covering radius of the code.
Abstract: Let r_{i} be the covering radius of the (2^{i},i+ 1) Reed-Muller code. It is an open question whether r_{2m+1}=2^{2_{m}}-2m holds for all m . It is known to be true for m=0,1,2 , and here it is shown to be also true for m=3 .
TL;DR: The coding theorem demonstrates the existence of {\em universal} encoders which are asymptotically optimal for every infinite sequence over a given finite alphabet.
Abstract: For every individual infinite sequence u we define a distortion-rate function d(R|u) which is shown to be an asymptotically attainable lower bound on the distortion that can be achieved for u by any finite-state encoder which operates at a fixed output information rate R . This is done by means of a coding theorem and its converse. No probabilistic characterization of u is assumed. The coding theorem demonstrates the existence of {\em universal} encoders which are asymptotically optimal for every infinite sequence over a given finite alphabet. The transmission of individual sequences via a noisy channel with a capacity C is also investigated. It is shown that, for every given sequence u and any finite-state encoder, the average distortion with respect to the channel statistics is lower bounded by d(C|u) . Furthermore d(C|u) is asymptotically attainable.
TL;DR: Universally attainable error exponents for broadcast channels with degraded message sets are obtained using a technique which generalizes that introduced by Csiszar, Korner, and Martron for the ordinary channel.
Abstract: Universally attainable error exponents for broadcast channels with degraded message sets are obtained using a technique which generalizes that introduced by Csiszar, Korner, and Martron for the ordinary channel. Lower and upper bounds to the error probabilities over a single broadcast channel are also given.
TL;DR: An algorithm for the minimum-redundancy encoding of a discrete information source is proposed and it is shown that the theoretical compression can be appmached within any desired threshold without the burden of alphabet extensions.
Abstract: An algorithm for the minimum-redundancy encoding of a discrete information source is proposed. In the case of memoryless sources it is shown that the theoretical compression can be appmached within any desired threshold without the burden of alphabet extensions (i.e., the encodhg of blocks of L primary symbols) and also irrespective of 1) the primary and secondary alphabet sizes 2) the numerical values of primary symbol probabillties, and 3) the order and structure of the encoding tree. The same algorithm is then extended to sources with memory and to cases in which there is a constraint on the statistical description of the secondary sequence (e.g., secondary symbol probabilities are given). The technique can thus be used to transform any given discrete source into any other given discrete source while minimizing the ratio of average secondary sequence length to average primary sequence length.
TL;DR: Because a function of two variables which is positive definite over a rectangle may not have a positive definite extension to the entire plane, two-dimensional Markov spectrum estimates based on estimated autocorrelation data do not always exist.
Abstract: Because a function of two variables which is positive definite over a rectangle may not have a positive definite extension to the entire plane, two-dimensional Markov spectrum estimates based on estimated autocorrelation data do not always exist. The ramifications of this fact are discussed.
TL;DR: The search algorithm represents the first use for asymmetric sources and distortion measures of a variation of a single stack algorithm proposed by Gallager, and establishes the existence of codes which attain almost any desired rate between the rate-distortion bound and the optimum entropy-coded quantizer.
Abstract: A rate-distortion theory is introduced for the optimal encoding of stationary memoryless continuous-amplitude sources with a single-letter distortion measure and reproduction alphabets of a given finite size. The theory arises from a judicious approximation of the original continuous-input discrete-output problem by one with discrete input and output. A size-constrained output alphabet rate-distortion function is defined, its coding significance is established by coding theorems, and a convergent algorithm is presented for its evaluation. The theory is applied to Gaussian sources with squared-error distortion measure. Using the algorithm for the calculation of the new rate-distortion function in this case establishes the existence of codes which attain almost any desired rate between the rate-distortion bound and the optimum entropy-coded quantizer. Furthermore, one can closely approach the rate-distortion limit with a surprisingly small number of output levels. The calculation furnishes optimal output levels, output level probabilities, and other parameters necessary for a trellis coding simulation. The search algorithm represents the first use for asymmetric sources and distortion measures of a variation of a single stack algorithm proposed by Gallager. Carrying out the simulation at a rate of 1 bit per source symbol, codes are found with 4 and 64 output levels which attain distortions smaller than that of an optimum quantizer and close to the rate-distortion bound. Furthermore, these codes attain comparable or better performance with far less search effort than previous attempts with a continuous output alphabet.
TL;DR: There exist new lower and upper bounds on the redundancy of the Huffman code which are tighter for P_{1} \geq 0.4 than those given by Shannon's first theorem or by the more recent results of Gallager.
Abstract: Some properties of Huffman codes are presented. It is shown that knowing the probability P_{1} of the most likely source letter, there exist new lower and upper bounds on the redundancy of the Huffman code which are tighter for P_{1} \geq 0.4 than those given by Shannon's first theorem or by the more recent results of Gallager. It is also shown that the new bounds are the tightest possible for P_{1} \geq 0.4 when it is supposed that PI is the only known probability.
TL;DR: A recursion relationship for the modified Bessel function is used to develop recursive formulas for functions which are expressed as a Neumann series expansion and an error analysis is presented.
Abstract: A recursion relationship for the modified Bessel function is used to develop recursive formulas for functions which are expressed as a Neumann series expansion The method is applied to devise an algorithm for calculating the generalized Q function Comparisons with some alternative schemes are discussed and an error analysis is presented
TL;DR: It is conjectured that all span- n sequences with the two-level autocorrelation property are in class PN, and some relationships between run properties and correlation properties of binary sequences are also obtained.
Abstract: Let U be the set of all binary sequences of period p=2^{n}-1 containing (p+1)/2 ones and (p-1)/2 zeros per period. There is a lattice of interesting subsets of U , the smallest of which is the set PN (the maximum-length linear shift register sequences of period p ). In between are sets with the run statistics of PN , with the correlation properties of PN , with the "span- n property" (that every nonzero subseqnonce of length n occurs in each period), and others. Results concerning the interrelationships of these subsets are obtained, examples are given to show that certain intersections of subsets are nonempty, and conjectures are formulated regarding other intersections of subsets. For example, it is conjectured that all span- n sequences with the two-level autocorrelation property are in class PN . Some relationships between run properties and correlation properties of binary sequences are also obtained.
TL;DR: The present paper describes a method for studying codes of minimum weight 6 : the supports of the words of weight 6 form what is called a center set, and all the graphs that can arise in this way are known.
Abstract: Self-orthogonal ternary codes of minimum weight 3 may be analyzed in a straightforward manner using the theory of glueing introduced in earlier papers. The present paper describes a method for studying codes of minimum weight 6 : the supports of the words of weight 6 form what is called a center set. Associated with each center set is a graph, and all the graphs that can arise in this way are known. These techniques are used to classify the ternary self-dual codes of length 20 : there are 24 inequivalent codes, 17 of which are indecomposable. Six of the codes have minimum weight 6 .
TL;DR: A method is presented which provides a criterion for detecting a change in the structure of a model generating a stochastic sequence based on the transformation of the observed sequence into a sequence of partial sums of the general innovations.
Abstract: A method is presented which provides a criterion for detecting a change in the structure of a model generating a stochastic sequence. Models that can be represented by a sequence of predictive probability distributions are considered. The method is based on the transformation of the observed sequence \{x_{n}\} into a sequence of partial sums of the general innovations, computed for the sequence \{-\log f(x_{n}|x_{n-1},x_{n-2}, \cdots ,x_{0})\} . If no change occurs the transformed sequence behaves like a Wiener process, but its mean will exhibit a monotonic growth after the process changes. Based on the properties of this transformation, fixed sample size and sequential tests for the change are constructed. The technique is applied to test for a change in the mean vector in a sequence of (generally dependent) Gaussian random variables, a change of coefficients of an autoregressive process, and a change of distribution in a sequence of discrete independent identically distributed random variables.
TL;DR: An alternate formulation of the robust hypothesis testing problem is considered in which robustness is defined in terms of a maximin game with a statistical distance criterion as a payoff function, which offers advantages over traditional error probability or risk criteria in this problem because of the greater tractability of the distance measure.
Abstract: An alternate formulation of the robust hypothesis testing problem is considered in which robustness is defined in terms of a maximin game with a statistical distance criterion as a payoff function. This distance criterion, which is a generalized version of signal-to-noise ratio, offers advantages over traditional error probability or risk criteria in this problem because of the greater tractability of the distance measure. Within this framework, a design procedure is developed which applies to a more general class of problems than do earlier robustness results based on risks. Furthermore, it is shown for the general case that when a decision rule exists that is robust in terms of risk, the same decision rule will be robust in terms of distance, a fact which supports the use of the latter criterion.
TL;DR: A simplified version of the Merkle-Hellman public key cryptographic system was shown to be breakable in this paper, which suggests some ways in which the security of their system can be enhanced.
Abstract: A simplified version of the Merkle-Hellman public key cryptographic system is breakable. While their full-fledged system seems to be resistant to the cryptanalytic attack we propose, the result suggests some ways in which the security of their system can be enhanced.