Showing papers on "Entropy (information theory) published in 1977"

Information-Theoretical Aspects of Quantum Measurement

Abstract: We present criteria for comparing measurements on a given system from the point of view of the information they provide. These criteria lead to a concept ofinformational completeness of a set of observables, which generalizes the conventional concept of completeness. The entropy of a state with respect to an arbitrary sample space of potential measurement outcomes is defined, and then studied in the context of configuration space and fuzzy stochastic phase space.

Development of a Mathematical Formula for the Calculation of Fingerprint Probabilities Based on Individual Characteristics

Abstract: A method is developed for assigning a probability to a fingerprint, including a partial print, based on the number of individual (Galton) characteristics present. A multinomial model is used, the categories of which are the individual characteristics and combinations of them. The negative of the logarithm of the probability of any particular configuration is related to the entropy function of information theory. The parameters of the model are estimated from data (fingerprints). Confidence bounds are obtained for the negative log probability of any configuration.

Maximum entropy image restoration. I. The entropy expression

Abstract: The two entropy expressions, log B and −B logB (where B is the local brightness of the object or its spatial spectral power) used in maximum entropy (ME) image restoration, are derived as limiting cases of a general entropy formula. The brightness B is represented by the n photons emitted from a small unit area of the object and imaged in the receiver. These n photons can be distributed over z degrees of freedom in q(n,z) different ways calculated by the Bose-Einstein statistics. The entropy to be maximized is interpreted, as in the original definition of entropy by Boltzmann and Planck, as logq(n,z). This entropy expression reduces to log B and −B logB in the limits of n⪢z>1 and n⪡z, respectively. When n is interpreted as an average n¯ over an ensemble, the above two criteria remain the same (with n replaced by n¯), and in addition for the z = 1 case the logB expression, used in ME spectral power estimation, is derived for n¯⪢z=1.

Two‐dimensional maximum entropy reconstruction of radio brightness

Abstract: This paper describes a procedure for maximum entropy reconstruction of two-dimensional radio brightness maps from noisy interferometer measurements. The method defines a map that obeys the nonnegativity constraint and is, in a sense, the smoothest of all brightness distributions that agree with the visibility measurements within the errors of observation. This approach acknowledges the fact that signal-to-noise considerations have a strong influence on useful resolution; fine structure appears only to the extent justified by measurement accuracy. Iterative computing is needed to find the maximum entropy image. It is shown that the primary computational burden of maximum entropy reconstruction involves calculations that are efficiently performed by fast Fourier transform techniques. Different techniques are used depending on whether visibility data are irregularly distributed in the u,v plane or interpolated onto a rectangular lattice prior to reconstruction. The efficiency of the fast Fourier transform provides a tremendous computational advantage with the result that maximum entropy reconstruction on a moderately large grid (64×64) is practicable at reasonable cost. Several comparative examples are shown, and some of the limitations of the present theory of maximum entropy imaging are identified.

Pedagogy again: what is entropy?*

Abstract: Entropy maximizing methods are popular in geography, civil engineering, and planning, but are not easy for nonspecialists to understand. Entropy is a measure of uncertainty. A reasonable means of constructing models is to maximize uncertainty subject to the information which is given because this minimizes the observer's bias. The nature of the constraints, or given information, is important. Results provided by the entropy maximizing method are illustrated for interaction models and location models. Although entropy maximizing models do not assert that people behave in a particular manner, it is possible to provide process interpretations of such models.

Information Theory: Some Concepts and Measures

Abstract: The concepts of entropy and of information are increasingly used in spatial analysis. This paper analyses these ideas in order to show how measures of spatial distributions may be constructed from them. First, the information content of messages is examined and related to the notion of uncertainty. Then three information measures, due to Shannon, Brillouin, and Good, are derived and shown to be appropriate in analysing different spatial problems; in particular, the Shannon and Brillouin measures are extensively compared and the effects of sample size on them are investigated. The paper also develops appropriate multivariate analogues of the information measures. Finally, some comments are made on the relations between the concepts of entropy, information, and order.

On the convergence of entropy measures of a fuzzy set

Abstract: Entropy measures of a fuzzy set defined on a denumerable support are studied. A necessary and sufficient condition for the convergence of the logarithmic entropy is given. Some properties of more general measures are studied and a necessary condition for their convergence is given. Further, some propositions relating the convergence of these measures with the logarithmic one are proved.

The recovery of detailed migration patterns from aggregate data: an entropy maximizing approach

Abstract: The entropy maximization method is used to estimate inter-regional migration flow matrices for subgroups of the population. The method is presented in lucid terms and a number of practical applications are given, such as the estimation of age-specific migration flows.

Entropy and Structure Two Measures of Complexity in Television Programs

Abstract: Two measures of the complexity of television program form were compared: one based on several indicators of the Information Theory concept of entropy, the other measure based on coders' perception of several kinds of program structure. Factor analysis of complexity scores for programs yielded two dimensions for both kinds of complexity measures. Canonical correlation among the entropy and structure factors was also significant along two dimensions. The entropy and structure factors produced similar differences in the program selection patterns of viewers. It was found that one dimension of program complexity was differentially preferred by young adults and the more highly educated. The other dimension of program complexity did not produce different program preferences among viewers of different ages and education.

Angular entropy: The information content of molecular scattering angular distributions

Abstract: The concept of the angular entropy arises from consideration of the information content of a scattering pattern, i.e., an angular distribution of collision products. It is shown that information theory (I.T.) provides the framework for evaluation and interpretation of the entropy (and entropy deficiency) of an angular distribution of reactive, inelastic, or elastic scattering. The differential cross section σ (ϑ) is converted to a normalized probability density function (pdf), P (u) [u= (1/2)(1−cosϑ)], from which the angular surprisal is obtained as −lnP (u). The average over u of the surprisal yields the angular entropy deficiency. (A histogrammic approximation to the continuous pdf can provide a simple estimate of ΔS). Examples are presented of reactive and inelastic molecular scattering patterns and of various prototype angular distributions giving insight into the angular entropy. The I.T. method is also applied to elastic scattering of atoms and molecules. It inherently demands the elimination of the...

Tests for system complexity

Abstract: The number of defects or errors present while a system continues to operate is related to the complexity of a physical system or the organization of a human system. A maximum entropy model of the number of defects in a complex system has a truncated geometric distribution with a parameter that indicates system complexity. Likelihood ratio testing allows comparison of systems on the basis of their complexity, and maximum likelihood estimation can be used to estimate complexity and infer the level of the system at which the complexity is inherently determined. Further testing may determine, the validity of the maximum entropy hypothesis, and, particularly in human systems, it may be argued that ordinary activities are entropy increasing transformations so that the maximum entropy model is a reasonable approximation.

On binary sliding block codes

Abstract: Sliding block codes are an intriguing alternative to the block codes used in the development of classical information theory. The fundamental analytical problem associated with the use of a sliding block code (SBC) for source encoding with respect to a fidelity criterion is that of determining the entropy of the coder output. Several methods of calculating and of bounding the output entropy of an SBC are presented. The local and global behaviors of a well-designed SBC also are discussed. The so-called "101-coder," which eliminates all the isolated zeros from a binary input, plays a central role. It not only provides a specific example for application of the techniques developed for calculating and bounding the output entropy, but also serves as a medium for obtaining indirect insight into the problem of characterizing a good SBC. An easily implementable SBC subclass is introduced in which the outputs can be calculated by simple logic circuitry. The study of this subclass is shown to be closely linked with the theory of algebraic group codes.

Retrieval performance and information theory

Abstract: This paper challenges the meaningfulness of precision and recall values as a measure of performance of a retrieval system. Instead, it advocates the use of a normalised form of Shannon's functions (entropy and mutual information). Shannon's four axioms are replaced by an equivalent set of five axioms which are more readily shown to be pertinent to document retrieval. The applicability of these axioms and the conceptual and operational advantages of Shannon's functions are the central points of the work. The applicability of the results to any automatic classification is also outlined.

Empirical estimates of program entropy

Abstract: We wish to investigate compact representation of object programs, therefore we wish to measure entropy, the average in/onnation content of programs. This number tells how many bits, on the average, would be needed to represent a program in the best possible encoding. A collection of 114 MESA programs, comprising approximately a million characters of source text, is analyzed. For analysis purposes, the programs are represented by trees, obtained by taking the parse trees from the compiler before the code generation pass and merging some of the symbol table information into them. A new definition is given for a Markov source where the concept of "previous" is defined in terms of the tree structure, and this definition is used to model the MESA program source. The lowest entropy value for these Markov models is 1.7 bits per tree node, assuming dependencies of each node on its grandfather, father, and elder brother (order 3). These numbers compare with an approximate 10 bits per node required for a naive encoding, and an equivalent of 3.2 bits per "node of code generated by the existing compiler. Motivated by sample set limitations for higher order models, we derive an entropy formula in which the order is nonuniform. The non-uniform entropy formulas are particularly suited to trees, where we can now speak of conditional probabilities in terms of patterns, or arbitrarily shaped contexts around a node. A method called pattern refinement is presented whereby patterns are "grown", i.e., the set of nodes matching an existing pattern is divided into those matching a larger pattern and those remaining. A proof is given that the process always leads to a lower estimate unless the old and new patterns induce exactly the same conditional probabilities. The result of applying this technique to the sample was an estimate of 1.6 bits per node. Further application would reduce this number even more. Analytic solutions for the error bounds in approximating the entropy of a Markov source are very difficult to obtain, so an experimental approach is used to gauge a confidence figure for the estimate. These calculations suggest that a more accurate estimate would be 1.8 bits per node, with a standard deviation of 13%. This corresponds to an entropy of .54 bits per character of source program. The methods of this thesis can be used both to define a bound for code compression and to evaluate existing object code.

Interpolative Picture Coding Using a Subjective Criterion

Abstract: Interpolative picture coding refers to sending coded information about a few picture elements separated in space and interpolating all the rest of the picture elements. In this paper we consider sending coded information about picture elements separated by as large a distance as possible along a scan line. We study the effects of a few twodimensional interpolation strategies and evaluate the usefulness of several different error criteria required to judge the faithfulness of the interpolated signal. The error criteria are motivated by our knowledge of pictorial information processing in the human visual system. Based on the picture quality and entropy of the coded output as the criterion for judging the coding schemes, we find that error measures in which the interpolation error is filtered adaptively and compared to a varying threshold perform the best. The filter is adapted based on the spatial activity of the signal: high-bandwidth filter for low activity areas and low-bandwidth filter for high activity areas. The variation in threshold is based on the spatial masking of the interpolation error and has a high value in high activity areas and a low value in low activity areas. Our computer simulations indicate that, for head-and-head-and-shoulders-type pictures, it is possble, without affecting the picture quality, to reduce the entropy of the coded output by as much as 40 percent over that obtainable from previous element differential pulse code modulation (DPCM) system.

A New Data Compression Technique

Abstract: By coding fixed-length blocks of symbols from an information source with a minimum redundancy (the variable-length Huffman code) the source entropy is approached as a function of the block size. In the present paper it is shown that it is also possible to split the source output into variable-length blocks which can be coded with a fixed-length code such that the efficiency also converges to the entropy of the source. An algorithm for optimal splitting is given, as well as a proof of the convergence.

Information-theoretic stability and evolution criteria in irreversible thermodynamics

Abstract: Generalized information gains are used to derive stability conditions, for steady states, periodic orbits and invariant sets, and general evolution criteria, both global and local with respect to the time variable, in irreversible thermodynamics. Meixner's passivity condition and the Glansdorff-Prigogine stability and evolution criteria are found to be special cases thereof. The information gain quantities include Kullback's three kinds of divergences, the first two of which are dual to each other and yield criteria which are symmetric in the average densities of the system's extensive variables and the conjugate parameters, but which are nonsymmetric in the irreversible fluxes and forces, while the third one does not involve the entropy function of the system. Furthermore, Renyi's information gain of orderα and Csiszar'sf-divergence are treated. The latter is used to construct a most general information gain quantity as a Liapunov function and evolution criterion, which, however, for local stability and evolution conditions is still equivalent to the use of the second-order variation of the entropy.

Linear Prediction and Maximum Entropy Spectral Analysis for Radar Applications

Abstract: : For most applications in radar data processing, the Fourier transform performs satisfactorily. However, other methods of spectral analysis can offer some advantages when a data set is too short for a Fourier transform to resolve or detect important spectral features. This report describes one alternative technique, maximum entropy spectral analysis (MESA), and suggests possible radar applications including range-Doppler sizing and the coherent measurement of range rate. Practical examples demonstrate an improvement in velocity resolution and cross-range resolution. Computer codes are listed that calculate MESA power spectra for a real or complex discrete time series. (Author)

On axiomatic characterization of some non-additive measures of information

Abstract: An axiomatic characterization of non-additive measures of information associated with a pair of probability distributions having the same number of elements has been given. This quantity under additional suitable postulates leads to the non-additive Entropy, Directed-Divergence and Inaccuracy of one or more parameters.

Multichannel complex maximum entropy spectral analysis

Abstract: The Burg reflection-coefficient method for maximum entropy spectral estimation is generalized to apply to multichannel complex signals. The virtues of the single-channel Burg process, particularly superior resolution and a guarantee that all power matrices are positive definite, carry over to the generalization. Results of some numerical computations are presented in graphical form.

Information theoretic approach to migration analysis

Abstract: Information theory is employed to look at certain aspects of migration. It is suggested that the dividedness of a population into “movers” and “stayers” is better assessed by migration entropy. The notion of “migration inequality” is introduced and the principle of minimum entropy suggested as a criterion for fitting migrations models. Canadian census data are utilized for illustration purposes.

Maximum Entropy Spectrum Analysis Technique - A Review of Its Theoretical Properties.

Abstract: : The maximum entropy method estimates the power spectrum of a random process by extrapolating the autocorrelation function outside the observation interval. The extrapolation is such that a minimal number of constraints is imposed on it. The equivalence of various derivation of the maximum entropy spectrum is proved in this paper and various examples are given to illustrate the properties of the maximum entropy spectrum. Finally the relation of the maximum entropy method to other techniques for estimating the spectrum is given. (Author)