Showing papers on "Entropy (information theory) published in 1989"

Automatic thresholding of gray-level pictures using two-dimensional entropy

Abstract: Automatic thresholding of the gray-level values of an image is very useful in automated analysis of morphological images, and it represents the first step in many applications in image understanding. Recently it was shown that by choosing the threshold as the value that maximizes the entropy of the 1-dimensional histogram of an image, one might be able to separate, effectively, the desired objects from the background. This approach, however, does not take into consideration the spatial correlation between the pixels in an image. Thus, the performance might degrade rapidly as the spatial interaction between pixels becomes more dominant than the gray-level values. In this case, it becomes difficult to isolate the object from the background and human interference might be required. This was observed during studies that involved images of the stomach. The objective of this report is to extend the entropy-based thresholding algorithm to the 2-dimensional histogram. In this approach, the gray-level value of each pixel as well as the average value of its immediate neighborhood is studied. Thus, the threshold is a vector and has two entries: the gray level of the pixel and the average gray level of its neighborhood. The vector that maximizes the 2-dimensional entropy is used as the 2-dimensional threshold. This method was then compared to the conventional 1-dimensional entropy-based method. Several images were synthesized and others were obtained from the hospital files that represent images of the stomach of patients. It was found that the proposed approach performs better specially when the signal to noise ratio (SNR) is decreased. Both, as expected, yielded good results when the SNR was high (more than 12 dB).

Entropy-constrained vector quantization

Abstract: An iterative descent algorithm based on a Lagrangian formulation for designing vector quantizers having minimum distortion subject to an entropy constraint is discussed. These entropy-constrained vector quantizers (ECVQs) can be used in tandem with variable-rate noiseless coding systems to provide locally optimal variable-rate block source coding with respect to a fidelity criterion. Experiments on sampled speech and on synthetic sources with memory indicate that for waveform coding at low rates (about 1 bit/sample) under the squared error distortion measure, about 1.6 dB improvement in the signal-to-noise ratio can be expected over the best scalar and lattice quantizers when block entropy-coded with block length 4. Even greater gains are made over other forms of entropy-coded vector quantizers. For pattern recognition, it is shown that the ECVQ algorithm is a generalization of the k-means and related algorithms for estimating cluster means, in that the ECVQ algorithm estimates the prior cluster probabilities as well. Experiments on multivariate Gaussian distributions show that for clustering problems involving classes with widely different priors, the ECVQ outperforms the k-means algorithm in both likelihood and probability of error. >

Information and entropy in strange attractors

Abstract: A technique for analyzing time-series data from experiments is presented that provides estimates of four basic characteristics of a system: (1) the measure-theoretic entropy; (2) the accuracy of the measurements; (3) the number of measurements necessary to specify a system state; and (4) the best delay time T to use in order to construct phase space portraits by the method of delays. These characteristics are obtained by separating the entropy of measurements into a part due to noise and parts due to deterministic effects. For the technique to work, the noise associated with each measurement must be independent of the noise associated with all other measurements. An algorithm for implementing the analysis is presented with three examples. >

Algorithmic randomness and physical entropy.

Abstract: Algorithmic randomness provides a rigorous, entropylike measure of disorder of an individual, microscopic, definite state of a physical system. It is defined by the size (in binary digits) of the shortest message specifying the microstate uniquely up to the assumed resolution. Equivalently, algorithmic randomness can be expressed as the number of bits in the smallest program for a universal computer that can reproduce the state in question (for instance, by plotting it with the assumed accuracy). In contrast to the traditional definitions of entropy, algorithmic randomness can be used to measure disorder without any recourse to probabilities. Algorithmic randomness is typically very difficult to calculate exactly but relatively easy to estimate. In large systems, probabilistic ensemble definitions of entropy (e.g., coarse-grained entropy of Gibbs and Boltzmann's entropy H=lnW, as well as Shannon's information-theoretic entropy) provide accurate estimates of the algorithmic entropy of an individual system or its average value for an ensemble. One is thus able to rederive much of thermodynamics and statistical mechanics in a setting very different from the usual. Physical entropy, I suggest, is a sum of (i) the missing information measured by Shannon's formula and (ii) of the algorithmic information content---algorithmic randomness---present in the available data about the system. This definition of entropy is essential in describing the operation of thermodynamic engines from the viewpoint of information gathering and using systems. These Maxwell demon-type entities are capable of acquiring and processing information and therefore can ``decide'' on the basis of the results of their measurements and computations the best strategy for extracting energy from their surroundings. From their internal point of view the outcome of each measurement is definite. The limits on the thermodynamic efficiency arise not from the ensemble considerations, but rather reflect basic laws of computation. Thus inclusion of algorithmic randomness in the definition of physical entropy allows one to formulate thermodynamics from the Maxwell demon's point of view.

Object-background segmentation using new definitions of entropy

Abstract: The definition of Shannon's entropy in the context of information theory is critically examined and some of its applications to image processing problems are reviewed. A new definition of classical entropy based on the exponential behaviour of information-gain is proposed along with its justification. Its properties also include those of Shannon's entropy. The concept is then extended to fuzzy sets for defining a non-probabilistic entropy and to grey tone image for defining its global, local and conditional entropy. Based on those definitions, three algorithms are developed for image segmentation. The superiority of these algorithms is experimentally demonstrated for a set of images having various types of histogram.

Developments in Maximum Entropy Data Analysis

Abstract: The Bayesian derivation of “Classic” MaxEnt image processing (Skilling 1989a) shows that exp(αS(f,m)), where S(f,m) is the entropy of image f relative to model m, is the only consistent prior probability distribution for positive, additive images. In this paper the derivation of “Classic” MaxEnt is completed, showing that it leads to a natural choice for the regularising parameter α, that supersedes the traditional practice of setting x2=N. The new condition is that the dimensionless measure of structure -2αS should be equal to the number of good singular values contained in the data. The performance of this new condition is discussed with reference to image deconvolution, but leads to a reconstruction that is visually disappointing. A deeper hypothesis space is proposed that overcomes these difficulties, by allowing for spatial correlations across the image.

A perceptually tuned sub-band image coder with image dependent quantization and post-quantization data compression

Abstract: The authors present a 16-band subband coder arranged as four equal-width subbands in each dimension, It uses an empirically derived perceptual masking model, to set noise-level targets not only for each subband but also for each pixel in a given subband. The noise-level target is used to set the quantization levels in a DPCM (differential pulse code modulation) quantizer. The output from the DPCM quantizer is then encoded, using an entropy-based coding scheme, in either 1*1, 1*2, or 2*2 pixel blocks. The type of encoding depends on the statistics in each 4*4 subblock of a particular subband. One set of codebooks, consisting of less than 100000 entries, is used for all images, and the codebook subset used for any given image is dependent on the distribution of the quantizer outputs for that image. A block elimination algorithm takes advantage of the peaky spatial energy distribution of subbands to avoid using bits for quiescent parts of a given subband. Using this system, high-quality output is obtainable at bit rates from 0.1 to 0.9 bits/pixel, and nearly transparent quality requires 0.3 to 1.5 bits/pixel. >

Continuity properties of entropy

Abstract: On donne une borne superieure sur le defaut en semicontinuite superieure des entropies topologiques et metriques d'une auto-application C r d'une variete C r , r>1

Some asymptotic properties of the entropy of a stationary ergodic data source with applications to data compression

Abstract: Theorems concerning the entropy of a stationary ergodic information source are derived and used to obtain insight into the workings of certain data-compression coding schemes, in particular the Lempel-Siv data compression algorithm. >

Thermodynamic cost of computation, algorithmic complexity and the information metric

Abstract: Algorithmic complexity is discussed as a computational counterpart to the second law of thermodynamics. It is shown that algorithmic complexity, which is a measure of randomness, sets limits on the thermodynamic cost of computations and casts a new light on the limitations of Maxwell's demon. Algorithmic complexity can also be used to define distance between binary strings.

A gray-level threshold selection method based on maximum entropy principle

Abstract: A description is given of a gray-level threshold selection method for image segmentation that is based on the maximum entropy principle. The optimal threshold value is determined by maximizing the a posteriori entropy subject to certain inequality constraints which are derived by means of spectral measures characterizing uniformity and the shape of the regions in the image. For this purpose, the authors use both the gray-level distribution and the spatial information of an image. The effectiveness of the method is demonstrated by its performance on some real-world images. An extension of this method to chromatic images is provided. >

Finding minimum entropy codes

Abstract: To determine whether a particular sensory event is a reliable predictor of reward or punishment it is necessary to know the prior probability of that event. If the variables of a sensory representation normally occur independently of each other, then it is possible to derive the prior probability of any logical function of the variables from the prior probabilities of the individual variables, without any additional knowledge; hence such a representation enormously enlarges the scope of definable events that can be searched for reliable predictors. Finding a Minimum Entropy Code is a possible method of forming such a representation, and methods for doing this are explored in this paper. The main results are (1) to show how to find such a code when the probabilities of the input states form a geometric progression, as is shown to be nearly true for keyboard characters in normal text; (2) to show how a Minimum Entropy Code can be approximated by repeatedly recoding pairs, triples, etc. of an original 7-bit code for keyboard characters; (3) to prove that in some cases enlarging the capacity of the output channel can lower the entropy.

Hydrologic Uncertainty Measure and Network Design

Abstract: This paper presents a simple methodology, using the entropy concept, to estimate regional hydro logic uncertainty and information at both gaged and ungaged grids in a basin. The methodology described in this paper is applicable for (a) the selection of the optimum station from a dense network, using maximization of information transmission criteria, and (b) expansion of a network using data from an existing sparse network by means of the information interpolation concept and identification of the zones from minimum hydrologic information. The computation of single and joint entropy terms used in the above two cases depends upon single and multivariable probability density functions. In this paper, these terms are derived for the gamma distribution. The derived formulation for optimum hydrologic network design was tested using the data from a network of 29 rain gages on Sleeper River Experimental Watershed. For the purpose of network reduction, the watershed was divided into three subregions, and the optimum stations and their locations in each subregion were identified. To apply the network expansion methodology, only the network consisting of 13 stations was used, and feasible triangular elements were formed by joining the stations. Hydrologic information was calculated at various points on the line segments, and critical information zones were identified by plotting information contours. The entropy concept used in this paper, although derived for single and bivaviate gamma distribution, is general in type and can easily be modified for other distributions by a simple variable transformation criterion.

A Test for Normality Based on Kullback—Leibler Information

Abstract: A goodness-of-fit test (based on sample entropy) for normality was given by Vasicek. The test, however, can be applied only to the composite hypotheses. In this article an extended test of fit for normality is introduced based on Kullback—Leibler information. The Kullback—Leibler information is an extended concept of entropy, so the test can be applied not only to the composite hypotheses, but also to the simple hypotheses. The power comparisons of the proposed test with some other tests are illustrated and discussed.

An improved algorithm for computing topological entropy

Abstract: A new algorithm is presented for computing the topological entropy of a unimodal map of the interval. The accuracy of the algorithm is discussed and some graphs of the topological entropy which are obtained using the algorithm are displayed.

Transfer-matrix analysis of a two-dimensional quasicrystal.

Abstract: We investigate the quasicrystalline state of a two-dimensional binary alloy in a discrete tiling approximation. Through transfer-matrix calculations we determine the configurational entropy over a range of concentrations. We find that the entropy density is maximized by a state with tenfold symmetry at the quasicrystal concentration. Derivatives of the entropy density at its maximum yield values for the phason elastic constants. Our results confirm the existence of quasi-long-range translational order in equilibrium quasicrystalline alloys and lend support to the random-tiling model of quasicrystals.

Estimation of origin-destination matrices from traffic counts using multiobjective programming formulations

Abstract: Origin-destination (O-D) trip matrices can be estimated by methods that use traffic volume counts. Assuming that we know the proportionate usage of each link by the interzonal traffic, a system of linear equations combining the O-D flow and the observed volumes can be formulated. This system is, in general, underspecified. To obtain a unique solution, additional information, often a target trip matrix, has to be used. The estimation problem can be interpreted as a problem that has two types of objectives, one of which is to satisfy the traffic counts constraints and the other to search for a solution as “close” as possible to the target matrix. Errors are normally present in the input data, and it is therefore reasonable to allow for solutions where the observed traffic volumes are not reproduced exactly. Depending on his/her degree of uncertainty or belief in the available information, the planner can choose to give more or less weight to the different objectives. To satisfy all the constraints to equality is only one extreme case in a continuum of possibilities. In this paper, we present multiobjective programming formulations for estimating O-D matrices. The main emphasis is to point out that multiobjective theory can be used in the interpretation of the problem. In a two-objective model, an aggregated entropy measure is defined for each type of information (targets and observations), and is used as the objective. In addition, a totally disaggregated multiobjective model is presented in which one objective for each target matrix element and each traffic count observation is defined. These models are then combined to make a general model. Different approaches for estimation of the magnitude of uncertainty and for specification of the weights of the objectives are discussed.

An information theoretic approach to the distributed detection problem

Abstract: The distributed detection problem is considered from an information-theoretic point of view. An entropy-based cost function is used for system optimization. This cost function maximizes the amount of information transfer between the input and the output. Distributed detection system topologies with and without a fusion center are considered, and an optimal fusion rule and optimal decision rules are derived. >

An information theoretic approach to the distributed detection problem

Abstract: The distributed detection problem is considered from an information theoretic point of view. An entropy-based cost function is used for system optimization. This cost function maximizes the amount of information transfer between the input and the output. Distributed detection system topologies with and without a fusion center are considered and an optimal fusion rule and optimal decision rules are derived

Characterizing grain size distributions: evaluation of a new approach using a multivariate extension of entropy analysis

Abstract: In the search for environment-diagnostic descriptors of grain size distributions, the use of both least squares statistical procedures and single index or summary measures to characterize sediment distributions have come under increasing criticism. Tests of these approaches highlight the validity of the criticisms. Analysis of sieved samples using the entropy concept, noted in the sedimentological literature as a potentially powerful tool for granulometric analysis, gives a much sharper result. Used previously only in its univariate form, the multivariate extension to the entropy procedure proposed here overcomes the problem of numbers of intervals and interval width inherent in the univariate application. It groups samples in terms of the whole shape of their grain size distributions, allows for variation in the optimal number of intervals for particular samples, and permits the use of more than one set of descriptor variables to be incorporated.

Generation interval distribution characteristics of packetized variable rate video coding data streams in an ATM network

Abstract: A description is given of the packetized data stream characteristics of variable-rate video coding in an asynchronous transfer mode (ATM) network. Variable-rate video coding characteristics are presented using motion-compensated adaptive intra-interframe prediction and entropy coding. The peak/mean entropy ratio, i.e., the burstiness of the TV conference video signals, ranges from 3 to 4. A method is proposed to transmit the packets of the three outputs of the variable-rate video coder: prediction errors, motion vector information, and quantizer selection information. The packet generation interval distribution characteristics of the prediction errors and motion vector information are clarified. It is shown that the generation interval distribution of prediction errors and motion vector information are not characterized by a simple Erland distribution, but by a combination of different distributions. >

Extension of an entropy property for binary input memoryless symmetric channels

Abstract: The channel output entropy property introduced by A.D. Wyner and J. Ziv (ibid., vol.IT-19, p.769-762, Nov.1973) for a binary symmetric channel is extended to arbitrary memoryless symmetric channels with binary inputs and discrete or continuous outputs. This yields lower bounds on the achievable information rates of these channels under constrained binary inputs. Using the interpretation of entropy as a measure of order and randomness, the authors deduce that output sequences of memoryless symmetric channels induced by binary inputs are of a higher degree of randomness if the redundancy of the input binary sequence is spread in memory rather than in one-dimensional asymmetry. It is of interest to characterize the general class of schemes for which this interpretation holds. >

Nonlinear reciprocity and the maximum entropy formalism

Abstract: The maximum entropy formalism calculates a phase-space distribution, p(x) , which maximizes the information-theoretic entropy subject to specified values of internal energy and state variables αi and ηi (i = 1, …, n). The latter are averages, calculated from p , of dynamical functions Ai and Ai≡iLAi, respectively, where L is the Liouville operator. From p , one extracts a fluctuation distribution g (a, v) for the values of the A and A . A kinetic equation for the fluctuation distribution has been derived by Grabert using projection operators appropriate to a system in thermal contact with its surroundings, and g is proposed as an ansatz for its solution. From the first moments of the kinetic equation, after introduction therein of g , one extracts phenomenological equations for ⋗ai and ⋗hi whose terms can be grouped to exhibit Onsager symmetry, even when nonlinear terms are included. The groupings can be affected exactly, but not uniquely.

An analogy of the charge distribution on Julia sets with the Brownian motion

Abstract: A way to compute the entropy of an invariant measure of a hyperbolic rational map from the information given by a Ruelle–Perron–Frobenius operator of a generic Holder‐continuous function will be shown. This result was motivated by an analogy of the Brownian motion with the dynamical system given by a rational map and the maximal measure. In the case the rational map is a polynomial, then the maximal measure is the charge distribution in the Julia set. The main theorem of this paper can be seen as a large deviation result. It is a kind of Donsker–Varadhan formula for dynamical systems.