Showing papers on "Entropy (information theory) published in 1985"

Second-generation image-coding techniques

Abstract: The digital representation of an image requires a very large number of bits. The goal of image coding is to reduce this number, as much as possible, and reconstruct a faithful duplicate of the original picture. Early efforts in image coding, solely guided by information theory, led to a plethora of methods. The compression ratio, starting at 1 with the first digital picture in the early 1960s, reached a saturation level around 10:1 a couple of years ago. This certainly does not mean that the upper bound given by the entropy of the source has also been reached. First, this entropy is not known and depends heavily on the model used for the source, i.e., the digital image. Second, the information theory does not take into account what the human eye sees and how it sees. Recent progress in the study of the brain mechanism of vision has opened new vistas in picture coding. Directional sensitivity of the neurones in the visual pathway combined with the separate processing of contours and textures has led to a new class of coding methods capable of achieving compression ratios as high as 70:1. Image quality, of course, remains as an important problem to be investigated. This class of methods, that we call second generation, is the subject of this paper. Two groups can be formed in this class: methods using local operators and combining their output in a suitable way and methods using contour-texture descriptions. Four methods, two in each class, are described in detail. They are applied to the same set of original pictures to allow a fair comparison of the quality in the decoded pictures. If more effort is devoted to this subject, a compression ratio of 100:1 is within reach.

The metric entropy of diffeomorphisms Part II: Relations between entropy, exponents and dimension

Abstract: On considere f: (M,m)→(M,m) un C 2 -diffeomorphisme f d'une variete de Riemann compacte M preservant une mesure de probabilite de Borel m. Soit hm(f) l'entropie metrique de f et λ 1 (x)>...>λ r(x) (x) les exposants de Lyapunov distincts en x. On etablit une formule qui relie h m (f) et les λ r(x) (x)

Computing the Kolmogorov entropy from time signals of dissipative and conservative dynamical systems

Abstract: The extraction of the Kolmogorov (metric) entropy from an experimental time signal is discussed Theoretically we stress the concept of generators and that the existence of an expansive constant guarantees that a finite-time series would be sufficient for the calculation of the metric entropy On the basis of the theory we attempt to propose optimal algorithms which are tested on a number of examples The approach is applicable to both dissipative and conservative dynamical systems

Unbiased bits from sources of weak randomness and probabilistic communication complexity

Abstract: We introduce a general model for physical sources or weak randomness. Loosely speaking, we view physical sources as devices which output strings according to probability distributions in which no single string is too probable. The main question addressed is whether it is possible to extract alrnost unbiased random bits from such "probability bounded" sources. We show that most or the functions can be used to extract almost unbiased and independent bits from the output of any two independent "probability-bounded" sources. The number of extractable bits is within a constant factor of the information theoretic bound. We conclude this paper by establishing further connections between communication complexity and the problem discussed above. This allows us to show that most Boolean functions have linear communication complexity in a very strong sense.

Rate-distortion performance of DPCM schemes for autoregressive sources

Abstract: An analysis of the rate-distortion performance of differential pulse code modulation (DPCM) schemes operating on discrete-time auto-regressive processes is presented. The approach uses an iterative algorithm for the design of the predictive quantizer subject to an entropy constraint on the output sequence. At each stage the iterative algorithm optimizes the quantizer structure, given the probability distribution of the prediction error, while simultaneously updating the distribution of the resulting prediction error. Different orthogonal expansions specifically matched to the source are used to express the prediction error density. A complete description of the algorithm, including convergence and uniqueness properties, is given. Results are presented for rate-distortion performance of the optimum DPCM scheme for first-order Gauss-Markov and Laplace-Markov sources, including comparisons with the corresponding rate-distortion bounds. Furthermore, asymptotic formulas indicating the high-rate performance of these schemes are developed for both first-order Gaussian and Laplacian autoregressive sources.

The optimal projection/maximum entropy approach to designing low-order, robust controllers for flexible structures

Abstract: The Optimal Projection/Maximum Entropy approach to designing low-order controllers for high-order systems with parameter uncertainties is reviewed. The philosophy of representing uncertain parameters by means of Stratonovich multiplicative white noise is motivated by means of the Maximum Entropy Principle of Jaynes and statistical analysis of modal systems. The main result, the optimal projection equations for fixed-order dynamic compensation in the presence of state-, control- and measurement-dependent noise, represents a fundamental generalization of classical LQG theory.

Entropy Condition Satisfying Approximations for the Full Potential Equation of Transonic Flow

Abstract: A class of conservative difference approximations for the steady full potential equation was presented. They are, in general, easier to program than the usual density biasing algorithms, and in fact, differ only slightly from them. Rigorous proof indicated that these new schemes satisfied a new discrete entropy inequality, which ruled out expansion shocks, and that they have sharp, steady, discrete shocks. A key tool in the analysis is the construction of a new entropy inequality for the full potential equation itself. Results of some numerical experiments using the new schemes are presented.

Entropy and correlation

Abstract: A simple an intuitively interpretable correlation coefficient between two discrete random variables is defined. Its metric property is then proved.

Entropy minimax multivariate statistical modeling–i: theory

Abstract: The theory of entropy mimmax, an information theoretic approach to predictive modeling, is reviewed. Comparisons are given to other methodologies, including: Neyman-Pearson hypothesis testing, James-Stein “empirical Bayes” estimation, maximum likelihood estimation, least-squares fitting, linear regression and logistic regression. Examples are provided showing how maximum entropy expectation probabilities are computed and how minimum entropy partitions are determined. The importance of the a priori weight normalization, in establishing the coarse-grain of the minimum entropy partition, is discussed. The trial crossvalidation procedure for determining the normalization is described. Generalizations utilizing Zadeh's fuzzy entropies are provided for variables involving indistinguishability, partial or total. Specific cases are discussed of maximum likelihood estimation, illustrating ils “data range irregularity” which is avoided by methods such as entropy minimization that account for residuals over...

On diversity measures based on entropy functions

Abstract: Some properties and interrelationships among the diversity measures arising from the concept of entropy are discussed, Asymptotic distributions of the estimates of these measures are presented.

The Entropy of an Image

Abstract: We investigate the entropy expressions that have been used for image reconstruction, including the spectral analysis of time-series data. We find that one should always use the Shannon formula S = −Σ pi log pi when attempting to reconstruct the shape of an image. This produces an image that is maximally noncommittal about the set of relative proportions pi which one wishes to construct. Alternative forms of entropy, notably the Σ log fi form recommended by Burg, refer to the physical process producing the image, and produce results that are maximally noncommittal, instead, about the probability distribution function governing individual realizations of this process. It is appropriate to use a Burg-type form when attempting to predict future samples from the physical process, but not when reconstructing the proportions pi themselves.

The use of transinformation in the design of data sampling schemes for inverse problems

Abstract: The author analyses the average useful information content of data samples by using the transinformation entropy (rate of transmission) of Shannon's information theory. The author derives a simple expression for the transinformation in linear experiments with Gaussian a priori distributions. The author uses this expression to examine various schemes for sampling the image spaces of a translation invariant (sinc) and a conformally invariant (Laplace) mapping. The optimum sampling scheme is found to be considerably better than the naive sampling scheme (e.g. Nyquist) when the number of samples is small and the a priori knowledge is non-trivial.

Compression of Two-Dimensional Images

Abstract: Distortion-free compressibility of individual pictures, i.e., two-dimensional arrays of data, by finite-state encoders is investigated. For every individual infinite picture I, a quantity ρ(I) is defined, called the compressibility of I, which is shown to be the asymptotically attainable lower bound on the compression-ratio that can be achieved for I by any finite-state, information-lossless encoder. This is demonstrated by means of a constructive coding theorem and its converse that, apart from their asymptotic significance, might also provide useful criteria for finite and practical data-compression tasks. The proposed picture-compressibility is also shown to possess the properties that one would expect and require of a suitably defined concept of two-dimensional entropy for arbitrary probabilistic ensembles of infinite pictures. While the definition of ρ(I) allows the use of different machines for different pictures, the constructive coding theorem leads to a universal compression-scheme that is asymptotically optimal for every picture. The results of this paper are readily extendable to data arrays of any finite dimension. The proofs of the theorems will appear in a forthcoming paper.

Entropy Principles in Decision Making Under Risk

Abstract: This paper considers decision problems where: (1) The exact probability distribution over the states of nature is not precisely known, but certain prior information is available about the possibilities of these outcomes; (2) A prior distribution over the states of nature is known, but new constraint information about the probabilities becomes available The maximum entropy principle asserts that the probability distribution with maximum entropy, satisfying the prior knowledge, should be used in the decision problem The minimum cross-entropy principle says that the posterior distribution is the one which minimizes cross-entropy, subject to the new constraint information The entropy principles have not gone uncriticized, and this literature, together with that justifying the principles, is surveyed Both principles are illustrated in a number of situations where the distribution is either discrete or continuous The discrete distribution case with prior interval estimates based on expert opinions is considered in detail

Entropy and Search Theory

Abstract: A recent article by Pierce (1978) has brought search theory to the attention of workers in related fields that also use statistical theory. In recounting history, he noted that early workers tried to relate detection probability pp and search effort to the posterior entropy HND conditional on nondetection [Eq. (4) below] or to the “expected posterior entropy” HE = pD HD + (1 − pD) HND, discovered quickly that no general relation exists, and concluded that information theory has no useful connection with search theory.

Dice, entropy, and likelihood

Abstract: We show that a famous die experiment used by E. T. Jaynes as intuitive justification of the need for maximum entropy (ME) estimation admits, in fact, of solutions by classical, Bayesian estimation. The Bayesian answers are the maximum probable (m.a.p.) and posterior mean solutions to the problem. These depart radically from the ME solution, and are also much more probable answers.

Entropy guided deconvolution

Abstract: The day-to-day application of minimum entropy deconvolution has rarely lived up to expectations based on a few of the first selected examples. This appears to be the result of (a) the multimaxima nature of the varimax function or other similar nonlinear norms, and (b) the excessive sensitivity of the estimated filter to the probability distribution of the underlying random series. As the distribution approaches Gaussian, the variance of the filter coefficients becomes unbounded. The alternative strategy proposed consists of solving for the inverse operator as a sequence of two-term operators each having an independently determined lag. The amplitude and sign of the operator coefficient are estimated by least squares; the discrimination between minimum or maximum phase is determined by the varimax criterion.

Probability kinematics, conditionals, and entropy principles

Abstract: treatment of marginalization requires axioms that govern the interaction between composition, pairing, and margining. The axioms are as follows: (i) T.o = (Tv).; ( i i ) (ST), = $~T; (iii) ( T; S), x ~ = T,; T~, where u x v is the usual Cartesian product of propositional (measurable) maps. At this point the framework of relative probabilities is rich enough to express a number of notions providing a foundation to statistical reasoning. One such notion is independence. Given a diagram P u ; v 1--~X-~ Y x Z , we say that the propositional (measurable) maps u and v are Pindependent iff P,,v = P,; P,. It is not hard to check that this definition coincides with the usual definition of statistical independence of two random variables. (Here the pairing u; v is defined by u; v(x)= (u(x), v(x))). Unquestionably, the most important concept by far in statistics and probability kinematics is conditionalization. In fact much of our groundworkup to this point was intended for the introduction of conditionals. K I N E M A T I C S , C O N D I T I O N A L S , E N T R O P Y P R I N C I P L E S 99 We know that every propositional map u:X--> Y induces a family of equational propositions [u = y] = {xl u(x) = y} which partitions the sample space X. These propositions in turn convert every probability P on X into a standard conditional Pt,=y] (including 0), defined by the usual operations of restriction and normalization Pt,=y](A) = P(A N [u = y]): P([u = y]). Note the important fact that the conditional is a function of the fixed map u and state y. It may be possible to glue these 'local' conditionals together so as to get one 'global' conditional, a relative probability P~: Y --~ X, defined by P\"(y, A) = P[,=y](A). The problem is that we would like this definition to work for all propositions and not just the equational ones. A leading idea is to define pu by the limit of standard Bayesian conditionals P~(y, A) = Lim [P(A n u-l( U)): P,( U)], U-->y where U is a proposition satisfying P~(U) ~ 0 in ~3y, and approaching the state y, relative to a suitable convergence structure. As U becomes: increasingly smaller, still close to y, in the limit the standard conditionals (P~)u merge into a 'point' conditional P~(y,-). Of course, the limit may not exist, and even if it does exist, it is not clear what the underlying convergence structure is. It turns out that a construction on filters does the job. To avoid additional technicalities, we wind up this brief discussion of generalized conditionals by pointing out that if Y has given a topology, then the set Azy of possible proposition-open set pairs (U, V), satisfying P . ( U ) ~ O, y~ V and U c V, forms a directed system of elements, ordered by ( U, V) <( U', V') iff V' c V. Now if the net of standard Bayesian conditionals {(Pu) u l( U, V) e JC'y} converges at all, its limit is unique and is equal to P\"(y, .). In standard texts of statistics the conditional P\" : Y ~ X is a relative probability, defined implicitly by the integral equation 100 Z O L T A N D O M O T O R P( A A u-~( B)) = JB P\"(Y' A) P,( dy). (10) Although this definition does provide a natural extension of Bayesian conditionals, in general, contrary to the actual practice of statistics, it • does not single out a unique relative probability P% No matter which definition is adapted, its axiomatization leads to the following: AXIOM 4. Generalized Bayesian Conditionalization of Belief States P u I ~ X , X ~ Y

Kolmogorov Complexity, Data Compression, and Inference

Abstract: If a sequence of random variables has Shannon entropy H, it is well known that there exists an efficient description of this sequence which requires only H bits. But the entropy H of a sequence also has to do with inference. Low entropy sequences allow good guesses of their next terms. This is best illustrated by allowing a gambler to gamble at fair odds on such a sequence. The amount of money that one can make is essentially the complement of the entropy with respect to the length of the sequence.

Maximum bounded entropy: application to tomographic reconstruction

Abstract: A new restoring algorithm, maximum bounded entropy (MBE), has been investigated. It incorporates prior knowledge of both a lower and upper bound in the unknown object. Its outputs are maximum probable estimates of the object under the following conditions: (a) the photons forming the image behave as classical particles; (b) the object is assumed to be biased toward a flat gray scene in the absence of image data; (c) the object is modeled as consisting of high-gradient foreground details riding on top of a smoothly varying background that is not to be restored but rather must be estimated in a separate step; and (d) the image noise is Poisson. The resulting MBE estimator obeys the sum of maximum entropy for the occupied photon sites in the object and maximum entropy for the unoccupied sites. The result is an estimate of the object that obeys an analytic form that functionally cannot take on values outside the known bounds. The algorithm was applied to the problem of reconstructing rod cross sections due to tomographic viewing. This problem is ideal because the object consists only of upper- and lower-bound values. We found that only four projections are needed to provide a good reconstruction and that twenty projections allow for the resolution of a single pixel wide crack in one of the rods.

An answer to a question by J. Milnor

Abstract: We consider two commuting automorphismsT1,T2 of the Lebesque space (M, M, μ) such thathm,n=h(T1mT2n)<∞ whereh is the measure-theoretic entropy. Under additional assumptions we show the existence of the limits lim (1/m)hm,n wherem→∞,n→∞,m/n→ω and ω is an irrational number.

Intelligent control-operating systems in uncertain environments

Abstract: Systems operating in uncertain environments with minimum interaction with a human operator, may be managed by controls with special considerations. They result in a Hierarchically Intelligent Control System, the higher levels of which may be modeled as knowledge based system processing various types of information with Entropy as an analytic measure. The concept of Entropy of statistical Thermodynamics, is used to express the average value of the performance criterion of a feedback control, encountered at the lower levels of the system. Thus, the resulting optimal control problem may be recast as an information theoretic one, which minimizes the entropy of selecting the feedback controls. This unifies the treatment of all the levels of a Hierarchically Intelligent Control System by a mathematical programming algorithm which minimizes the sum of their entropies. The resulting "Intelligent Machine" is composed of three levels hierarchically ordered in decreasing intelligence with increasing precision: the organization level, performing information processing tasks like planning, decision making, learning and storage and retrieval of information from a long-term memory; the coordination level, dealing again with information processing tasks like learning, lower level decision making and dealing with short-term memory only and the control level, which performs the execution of various tasks through hardware using feedback control methods.

Entropy guided deconvolution of seismic signal

Abstract: a Disclosed are a process and a system for improving seismic returns and other signals representative of nontime varying series which have been degraded by interaction with unknown wavelets. Use is made of a sequence of two-term operators, each derived from the most recent version of the signal and each applied thereto to obtain the next version. The operators are selected such that their application tends to increase the entropy of the signal, and to remove the effects of the unknown wavelets and change the observed signal in a manner which tends to reveal the signal of interest.

"Entropy and information theory": critique, comments, and reply Entropy and information theory: are we missing something?

Abstract: In this paper, some issues are raised concerning entropy-maximizing and information-theoretical approaches to spatial distribution modelling It is argued that approaches based on information theory and the minimum-information principle may not always be satisfactory because they may not take into account all the known information characterizing the spatial system It is concluded that this approach is not appropriate as a basis upon which to derive spatial distribution models