Showing papers on "Entropy (information theory) published in 1968"
•
[...]
TL;DR: It is shown that in many problems, including some of the most important in practice, this ambiguity can be removed by applying methods of group theoretical reasoning which have long been used in theoretical physics.
Abstract: In decision theory, mathematical analysis shows that once the sampling distribution, loss function, and sample are specified, the only remaining basis for a choice among different admissible decisions lies in the prior probabilities. Therefore, the logical foundations of decision theory cannot be put in fully satisfactory form until the old problem of arbitrariness (sometimes called "subjectiveness") in assigning prior probabilities is resolved. The principle of maximum entropy represents one step in this direction. Its use is illustrated, and a correspondence property between maximum-entropy probabilities and frequencies is demonstrated. The consistency of this principle with the principles of conventional "direct probability" analysis is illustrated by showing that many known results may be derived by either method. However, an ambiguity remains in setting up a prior on a continuous parameter space because the results lack invariance under a change of parameters; thus a further principle is needed. It is shown that in many problems, including some of the most important in practice, this ambiguity can be removed by applying methods of group theoretical reasoning which have long been used in theoretical physics. By finding the group of transformations on the parameter space which convert the problem into an equivalent one, a basic desideratum of consistency can be stated in the form of functional equations which impose conditions on, and in some cases fully determine, an "invariant measure" on the parameter space.
1,366 citations
••
TL;DR: It is shown that uniform quantizing yields an output entropy which asymptotically is smaller than that for any other quantizer, independent of the density function or the error criterion, and the discrepancy between the entropy of the uniform quantizer and the rate distortion function apparently lies with the inability of the optimal quantizing shapes to cover large dimensional spaces without overlap.
Abstract: It is shown, under weak assumptions on the density function of a random variable and under weak assumptions on the error criterion, that uniform quantizing yields an output entropy which asymptotically is smaller than that for any other quantizer, independent of the density function or the error criterion. The asymptotic behavior of the rate distortion function is determined for the class of
u th law loss functions, and the entropy of the uniform quantizer is compared with the rate distortion function for this class of loss functions. The extension of these results to the quantizing of sequences is also given. It is shown that the discrepancy between the entropy of the uniform quantizer and the rate distortion function apparently lies with the inability of the optimal quantizing shapes to cover large dimensional spaces without overlap. A comparison of the entropies of the uniform quantizer and of the minimum-alphabet quantizer is also given.
522 citations
••
TL;DR: The chromatic information contentIc(X) of a graphX is defined as the minimum entropy over all finite probability schemes constructed from chromatic decompositions having rank equal to the chromatic number of X as discussed by the authors.
83 citations
••
TL;DR: It is shown that certain important normalizations (position, size, pitch, etc.) are nonlinear operations.
Abstract: Pattern recognition (including sound recognition) is described mathematically as the problem to compute for any element of a given class its image in a classification set. The difficulty lies in the fact that the map may be implicitly defined by a property or must be extrapolated from prototypes. An entropy measure and an equivocation measure are defined that permit an assessment of the improvement gained (and the price in confusion paid) by a set of Linear ``features'' are identified as measures and L 2 functions, respectively. It is shown that certain important normalizations (position, size, pitch, etc.) are nonlinear operations. Finally, the method of spectral analysis which is widely used for speech analysis is examined critically. It is shown that contrary to common belief Fourier analysis is not very suitable for detecting certain speech particles (consonants, stops, etc.).
19 citations
••
17 citations
••
TL;DR: In this paper, the authors extend the ergodic theorems of information theory to spaces with an infinite invariant measure and show that the supremum f* of the conditional information given the increasing past is integrable.
Abstract: The paper extends the ergodic theorems of information theory (Shannon-MacMillan-Breiman theorems) to spaces with an infinite invariant measure. An L1 difference theorem and a pointwisa ratio theorem are proved, for the information of spreading partitions. For the validity of the theorems it is assumed that the supremum f* of the conditional information given the increasing “past” is integrable. Simple necessary and sufficient conditions for the integrability of f* are obtained in special cases: If the initial partition is composed of one state of a null-recurrent Markov chain, then f* is integrable if and only if the partition of this state according to the first return times has finite entropy.
15 citations
••
TL;DR: The Jaynes "widget problem" is reviewed as an example of an application of the principle of maximum entropy in the making of decisions, where the exact solution yields an unusual probability distribution.
Abstract: The Jaynes "widget problem" is reviewed as an example of an application of the principle of maximum entropy in the making of decisions. The exact solution yields an unusual probability distribution. The problem illustrates why some kinds of decisions can be made intuitively and accurately, but would be difficult to rationalize without the principle of maximum entropy.
13 citations
••
TL;DR: WOOLHOUSE1 remarks that the work of Shannon and Brillouin showed the fundamental relationship between information defined as I=−Σ Pi log Pi, and entropy defined in statistical terms as S=−KΣPi log Pi.
Abstract: WOOLHOUSE1 remarks that the work of Shannon and Brillouin showed the fundamental relationship between information defined as I=−ΣPi log Pi (where 0 ⩽Pi⩽1, ΣPi=1 and Pi is the relative probability of the ith symbol generated by a source), and entropy defined in statistical terms as S=−KΣPi log Pi (where ΣPi=1 and Pi is, in this case, the probability of an idealized physical system being in the state i of n possible equivalent states or complexions). It is the unwarranted extrapolation of this relationship to biological systems which, Woolhouse says, leads to erroneous conclusions. He points to the warning given by Brillouin himself, that the theory of information ignores the value or the meaning of the information which is quantified by the definition. Yet in spite of these warnings by Brillouin, the confusion is already present in his work even before its extension to biology.
13 citations
••
01 Jan 1968TL;DR: In this article, the invariant nature of a Fourier scene and a space scene as sources of a communication system is investigated with respect to their information content, and it is shown that two functions related by a finite Fourier transformation are identical.
Abstract: Two-dimensional functions are investigated with respect to their information content. Entropy relations of two functions, related by a finite Fourier transformation, are shown to be identical. The proof demonstrates the invariant nature of a Fourier scene and a space scene as sources of a communication system.
••
TL;DR: If such a statistical distribution of entropy were possible in the biological world, a similar statistical distribution should also exist in non-living systems; and for every spontaneous reaction, a small number of products with decreased entropy should result.
Abstract: IT has been suggested that a statistical distribution of entropy among a large number of entities will lead to a small percentage of them possessing a lower entropy than the mean value for the group as a whole1. To illustrate this point Campbell considered the example of the codfish laying a million eggs. This, however, is not a representative example of the entire biological world, for birds and mammals produce a very small number of eggs and the higher mammals produce only one egg at a time. Furthermore, if such a statistical distribution of entropy were possible in the biological world, a similar statistical distribution should also exist in non-living systems; and for every spontaneous reaction, a small number of products with decreased entropy should result. This is contrary to all known facts.
••
TL;DR: Numerical calculation of informational entropy shows that, at any given composition, it has its maximum value for r1 r2 = 1 and decreases very slowly with r1r2; the definition of random copolymer may be extended, therefore, to copolymers for which the products of reactivity ratios vary over a very wide range.
••
••
••
TL;DR: This paper gives a deduction from the generalized Nyquist formula based on the principle of the negative entropy of information that is free of divergence due to zero fluctuations.
Abstract: This paper gives a deduction from the generalized Nyquist formula based on the principle of the negative entropy of information. The formula obtained is free of divergence due to zero fluctuations.
••
TL;DR: An exact proof of the estimate for the e-entropy of the ordinary Brownian motion B ( t), 0 ≦ t ≦ 1, which was presented without proof by A.N. Kolmogorov is obtained.
Abstract: M.S. Pinsker [3] has given a general method of calculating the e-entropy of a Gaussian process and obtained, for example, an exact proof of the estimate for the e-entropy of the ordinary Brownian motion B ( t), 0 ≦ t ≦ 1, which was presented without proof by A.N. Kolmogorov [1].