Showing papers on "Entropy (information theory) published in 1981"
Book•
16 Dec 1981
TL;DR: The first part of the text as discussed by the authors provides an introduction to ergodic theory suitable for readers knowing basic measure theory, including recurrence properties, mixing properties, the Birkhoff Ergodic theorem, isomorphism, and entropy theory.
Abstract: This text provides an introduction to ergodic theory suitable for readers knowing basic measure theory. The mathematical prerequisites are summarized in Chapter 0. It is hoped the reader will be ready to tackle research papers after reading the book. The first part of the text is concerned with measure-preserving transformations of probability spaces; recurrence properties, mixing properties, the Birkhoff ergodic theorem, isomorphism and spectral isomorphism, and entropy theory are discussed. Some examples are described and are studied in detail when new properties are presented. The second part of the text focuses on the ergodic theory of continuous transformations of compact metrizable spaces. The family of invariant probability measures for such a transformation is studied and related to properties of the transformation such as topological traitivity, minimality, the size of the non-wandering set, and existence of periodic points. Topological entropy is introduced and related to measure-theoretic entropy. Topological pressure and equilibrium states are discussed, and a proof is given of the variational principle that relates pressure to measure-theoretic entropies. Several examples are studied in detail. The final chapter outlines significant results and some applications of ergodic theory to other branches of mathematics.
3,550 citations
TL;DR: The principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information) is a general method of inference about an unknown probability density when there exists a prior estimate of the density and new information in the form of constraints on expected values.
Abstract: The principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information) is a general method of inference about an unknown probability density when there exists a prior estimate of the density and new information in the form of constraints on expected values. Various fundamental properties of cross-entropy minimization are proven and collected in one place. Cross-entropy's well-known properties as an information measure are extended and strengthened when one of the densities involved is the result of cross-entropy minimization. The interplay between properties of cross-entropy minimization as an inference procedure and properties of cross-entropy as an information measure is pointed out. Examples are included and general analytic and computational methods of finding minimum cross-entropy probability densities are discussed.
384 citations
TL;DR: The power properties of an entropy-based test for uniformity on [0, 1] were investigated in this article, where the power of the test was compared with other tests of uniformity.
Abstract: The power properties of an entropy-based test are investigated when used for testing uniformity on [0, 1]. Percentage points and power against seven alternatives are reported. Compared with other tests of uniformity, the entropy-based test possesses good power properties for many alternatives. Some asymptotic null and alternative distributions are derived. For sample sizes up to 100 the table of percentage points provides a practical guide for using this test. A theory of entropy-based tests of distributional hypotheses other than uniformity is outlined.
216 citations
TL;DR: In this paper, the authors established inequalities between entropy numbers and approximation numbers for operators acting between Banach spaces, and derived inequalities between eigenvalues and entropy numbers for operator acting on a Banach space.
196 citations
TL;DR: This new method differs from previous methods in its explicit inclusion of a prior estimate of the power spectrum, and it reduces to maximum entropy spectral analysis as a special case.
Abstract: The principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information, minimum relative entropy) is summarized, discussed, and applied to the classical problem of estimating power spectra given values of the autocorrelation function. This new method differs from previous methods in its explicit inclusion of a prior estimate of the power spectrum, and it reduces to maximum entropy spectral analysis as a special case. The prior estimate can be viewed as a means of shaping the spectral estimator. Cross-entropy minimization yields a family of shaped spectral estimators consistent with known autocorrelations. Results are derived in two equivalent ways: once by minimizing the cross-entropy of underlying probability densities, and once by arguments concerning the cross-entropy between the input and output of linear filters. Several example minimum cross-entropy spectra are included.
167 citations
TL;DR: In this article, the concept of sufficiency is more or less considered through the informativity of a certain subalgebra with respect to a given alge- bra for a dynamical system of interest.
Abstract: Introduction* Since the investigation of sufficient statistics in abstract measure theoretic terms was initiated by Halmos and Savage (10), the concept of sufficiency has been developed by many mathe- matical statisticians in terms of various relations given by compar- ison of experiments, risk functions within the framework of stati- stical decision problems and so on. A characterization of sufficiency was given in (12) through the measure of Kullback-Leib ler informa- tion. The concept of sufficiency was first generalized by Umegaki (22, 23) to the noncommutative case of semi-finite von Neumann algebras with some extension of the Kullback-Leibler information (usually called the relative entropy). Later the related discussions especially concerning the relative entropy for quantum systems have been made by several authors, e.g., Araki (2, 3), Gudder and Marchand (7), and Lindblad (13). As defined precisely and explained in §§ 1 and 4 of this paper, the concept of sufficiency is more or less considered through the informativity of a certain subalgebra with respect to a given alge- bra for a dynamical system of interest. Namely, in the case that such a subalgebra is sufficient, the relative entropy on the subalgebra is equal to that on the given algebra. This fact may or may not be a reason why the concept of sufficiency has not been entered into analysis of physical systems, in which the change of entropy is thought of more relevant. The Kubo-Martin-Schwinger (KMS) condition was introduced by these three authors (11, 14) as a boundary condition of the thermal Green function. Haag, Hugenholtz and Winnink (8) showed that in the operator algebraic framework this condition is a fundamental one describing thermal equilibrium of quantum systems. The KMS condition through the Tomita-Takesaki theory now becomes a core of studying von Neumann algebras. Under the above historical basis, our main motivation of this
115 citations
TL;DR: In this paper, the authors reviewed the method of maximum entropy with emphasis on its relationship to entropy rate, Wiener filters, autoregressive processes, extrapolation, the Levinson algorithm, lattice, all-pole and all-pass filters, and stability.
Abstract: The method of maximum entropy is reviewed with emphasis on its relationship to entropy rate, Wiener filters, autoregressive processes, extrapolation, the Levinson algorithm, lattice, all-pole and all-pass filters, and stability.
110 citations
TL;DR: It is shown that many known algorithms of clustering and pattern recognition can be characterized as efforts to minimize entropy, when suitably defined.
107 citations
TL;DR: A new iterative algorithm for the maximum entropy power spectrum estimation is presented, which utilizes the computational efficiency of the fast Fourier transform (FFT) algorithm and has been empirically observed to solve the maximum Entropy Power spectrum estimation problem.
Abstract: A new iterative algorithm for the maximum entropy power spectrum estimation is presented in this paper. The algorithm, which is applicable to two-dimensional signals as well as one-dimensional signals, utilizes the computational efficiency of the fast Fourier transform (FFT) algorithm and has been empirically observed to solve the maximum entropy power spectrum estimation problem. Examples are shown to illustrate the performance of the new algorithm.
101 citations
TL;DR: In this article, an information-theoretic approach to entropy modeling is used to derive expressions for the possible number of microstates derived from the sets of choices available to households and firms in given location/transport planning situations.
Abstract: Using an information-theoretic approach to entropy modelling, expressions for the possible number of microstates are derived from the sets of choices available to households and firms in given location/transport planning situations. In planning applications, the urban space is usually subdivided into discrete zones, and various classes of householders and firms are grouped into separate activities. Nevertheless, the location sites within zones may vary considerably in accessibility, price and quality, and amenity v. cost trade-offs between location and travel many differ markedly between individuals within the same activity group. The inclusion of such variations within a random utility framework is demonstrated to be equivalent to maximizing entropy with the microstates disaggregated to the level where these variations occur. Using constraints based on spatial conditions, observed behaviour and planning policy, entropy maximization is used to determine the most probable macrostate according to each choice set specification. The resulting distributions are compared and their revelance discussed.
TL;DR: In this article, a conceptual framework for the analysis of community types pattern (mosaic) is proposed and discussed in this paper, the pattern is viewed as a system whose freedom to take any particular state from among a set of possible microstates is restrained by ecological factors.
TL;DR: For a given parametric family of densities the member of maximal (or sometimes minimal) entropy is ascertained and a natural (partial) ordering over for which the entropy functional is monotone is determined.
Abstract: Entropy functionals of probability densities feature importantly in classifying certain finite-state stationary stochastic processes, in discriminating among competing hypotheses, in characterizing Gaussian, Poisson, and other densities, in describing information processes, and in other contexts. Two general types of problems are considered. For a given parametric family of densities the member of maximal (or sometimes minimal) entropy is ascertained. Secondly, we determine a natural (partial) ordering over for which the entropy functional is monotone. The examples include the multiparameter binomial, multiparameter negative binomial, some classes of log concave densities, and others.
01 Jan 1981
TL;DR: In this article, the authors demonstrate the application of the entropy-maximizing paradigm in its traditional form, together with recent adaptations emerging from information theory, to the area of spatial and non-spatial input-output analysis.
Abstract: Interindustry transactions recorded at a macro level are simply summations of commodity shipment decisions taken at a micro level. The resulting statistical problem is to obtain minimally biased estimates of commodity flow distributions at the disaggregated level, given various forms of aggregated information. This study demonstrates the application of the entropy-maximizing paradigm in its traditional form, together with recent adaptations emerging from information theory, to the area of spatial and non-spatial input-output analysis. A clear distinction between the behavioural and statistical aspects of entropy modelling is suggested. The discussion of non-spatial input-output analysis emphasizes the rectangular and dynamic extensions of Leontief's original model, and also outlines a scheme for simple aggregation, based on a criterion of minimum loss of information. In the chapters on spatial analysis, three complementary approaches to the estimation of interregional flows are proposed. Since the static formulations cannot provide an accurate picture of the gross interregional flows between any two sectors, Leontief's dynamic framework is adapted to the problem. The study concludes by describing a hierarchical system of models to analyse feasible paths of economic development over space and time.
TL;DR: Several metrics for the quality assessment of a software system design are discussed, based on the entropy function of communication information theory, which can compute the excess entropy and thereby rank different design alternatives.
Journal Article•
TL;DR: Application of the estimation methodology is not limited to migration data by can be applied to all cross-classified data or multidimensional contingency tables and the potential of contingency table analysis are being explored for their contribution toward improved estimation.
Abstract: Data on migration flows are not often disaggregated by regions of origin and destination for subgroups of the population. These detailed flows must be inferred from aggregate information. The 2 methods presented for deducing these estimates are conceptualized as mathematical optimization problems with nonlinear objective functions and linear constraints. The individual techniques differ in the objective functions used. Bi- and multiproportional adjustment methods integrate information divergence minimizing problems and entropy maximization (a special formulation of the information divergence method). In the entropy problem no initial values of the elements to be estimated are available and they are uniformly set to equal unity. A single solution algorithm is developed for both problems. The entropy method is suited for estimating detailed migration flows on the basis of aggregate data only; the information divergent method is useful for upgrading migration tables. The 2nd method is quadratic adjustment where weighted squared deviations between estimates and initial guesses are minimized. The guesses may be derived from outdated migration tables or may represent "a priori" information on the migration pattern. A modified Friedlander method is presented which yields estimates of the appropriate sign. Proofs of the 5 theorems and solutions for 3 special entropy cases are provided. Validity of the estimation methods is measured using chi-square and absolute percentage error techniques examining the accuracy of the estimates when compared to observed data. Illustrations of the procedures using migration data from Austria and Sweden uncovered an interesting observation. A certain amount of data on the age composition of migrants is necessary to yield good estimates of detailed flow--additional initial information on age structure does not add significantly to the quality of the estimates. Additional research will increase the utility of estimation methodology. For example determination of the amount of initial information needed strategies for improvement of initial guesses and the potential of contingency table analysis are being explored for their contribution toward improved estimation. Further research of the methodologys application to updating migration tables would be valuable. Improvement of validity measures that are less susceptible to smaller flows is also needed. Application of the estimation methodology is not limited to migration data by can be applied to all cross-classified data or multidimensional contingency tables.
TL;DR: Monte Carlo methods are used to extend previous tables of critical points and power for such entropy tests to the large sample sizes likely to be desirable when evaluating the output of one or more random number generators.
Abstract: SYNOPTIC ABSTRACTPrevious work has shown how to test a simple hypothesis of uniformity on the interval (0, 1) by using spacings-based estimates of entropy. In this paper we use Monte Carlo methods to extend previous tables of critical points and power for such entropy tests to the large sample sizes likely to be desirable when evaluating the output of one or more random number generators. A comparison with asymptotic critical points and power is made. The results are used to evaluate a number of commonly used random number generators, which are of importance in such areas as bootstrapping. At least one random number generator is found unsuitable for use. Since a generator cycling on .00, .01, .02, …, .99 (to more digits) could have a sample entropy of nearly zero, this test is appropriate only for generators that pass other extensive testing, such as the TESTRAND tests (e.g., see Karian and Dudewicz (1991)).
TL;DR: A discrete-time nonlinear filtering lower bound algorithm is given for evaluating the error in a desired function of the state vector based on a rate distortion bound derived previously by the author.
Abstract: A discrete-time nonlinear filtering lower bound algorithm is given for evaluating the error in a desired function of the state vector. The algorithm is based on a rate distortion bound derived previously by the author. The problem is formulated in terms of Monte Carlo analysis. The theory of backward Markovian models is used to evaluate the conditional expectation appearing in the Bucy representation for the ease of Gauss-Markov signal models. An approximation procedure is given for the case of nonlinear signal models. In comparison with the author's previous bound the bound algorithm obtained here is tighter and does not require the difficult computation of the entropy of the state vector.
01 Jun 1981
TL;DR: In this paper, a finitary homomorphism between the corresponding one-sided Bernoulli shifts was shown for probability vectors of equal entropy with at least three non-zero components.
Abstract: If p and q are probability vectors of equal entropy each having at least three non-zero components then there exists a finitary homomorphism between the corresponding one-sided Bernoulli shifts.
TL;DR: The entropy of clusters of 3-5 adjacent spike intervals, when computed in this particular way, seems to be a useful measure or index of the informational state of neurons.
Abstract: Techniques were developed for using the classical information theory descriptor, entropy, to quantify the “uncertainty” present in neuronal spike trains. Entropy was calculated on the basis of a method that describes the relative relationships of serially ordered interspike intervals by encoding the intervals as a series of symbols, each of which depicts the relative duration of two adjacent spike intervals. Each symbol, or set of symbols has a specific fractional entropy value, derived from its probability of occurrence; moreover, fractional entropy can describe the relative amount of “information” that is associated with the relative location of a given symbol in a string of symbols.Using spike trains from 12 single neurons in the cerebellar cortex of rats, we determined: (1) the mean and S.D. of information content of each symbol in each specific position in a group of symbols (2-4 symbols/group, based on 3-5 adjacent intervals), (2) the 4-symbol groups which had the least and the most average fraction...
TL;DR: Using action potential interval data from 12 neurons in rat cerebellum, it is demonstrated that the sequential ordering of spike intervals contains both noise and signal.
Abstract: A method for detecting signal in the presence of noise in a highly specific was is described. Using action potential interval data from 12 neurons in rat cerebellum, we have demonstrated that the sequential ordering of spike intervals contains both noise and signal. We have identified and quantified the magnitude of relative entropy (uncertainty) for specified sets of interval patterns, ranging in length from 3–5 successive intervals. Some of these sets have exceptionally low entropy and thus seem to be especially meaningful as a set (‘word’) to the brain.
TL;DR: In the first part of this paper a new method of applying the Maximum Entropy Principle (MEP) is presented, which makes use of a “frequency related” entropy, and which is valid for all stationary processes.
Abstract: In the first part of this paper a new method of applying the Maximum Entropy Principle (MEP) is presented, which makes use of a “frequency related” entropy, and which is valid for all stationary processes. The method is believed valid only in the case of discrete spectra. In the second part of the paper, a method of estimating continuous spectra in the presence of noise is presented, which makes use of the Mutual Information Principle (MIP). Although the method proceeds smoothly in mathematical terms, there appear to be some difficulties in interpreting the physical meaning of some of the expressions. Examples in the use of both methods are presented, for the usual practical problem of estimating a power spectrum for a process whose autocorrelation function is partially known a priori.
TL;DR: For each n ≥ 2, the minimum value of the topological entropy of all continuous self-maps of the circle having a fixed point and a point of least period n was found in this paper.
Abstract: For each n ≥2, we find the minimum value of the topological entropies of all continuous self-maps of the circle having a fixed point and a point of least period n , and we exhibit a map with this minimal entropy.
TL;DR: In this article, the spreading of free wave packets is expressed by means of the entropy of position for a certain class of states, and connection between such formulation and the usual treatment is discussed.
Abstract: The spreading of free wave packets is expressed by means of the entropy of position for a certain class of states. Connection between such formulation and the usual treatment is discussed.
TL;DR: In this article, it was shown that the distributions over single-particle states for Boltzmann, Bose-Einstein, and Fermi-Dirac statistics describing N noninteracting identical particles follow directly from the principle of maximum entropy.
Abstract: It is demonstrated that the distributions over single-particle states for Boltzmann, Bose-Einstein, and Fermi-Dirac statistics describing N non-interacting identical particles follow directly from the principle of maximum entropy. It is seen that the notions of indistinguishability and coarse graining are secondary, if not irrelevant. A detailed examination of the structure of the Boltzmann limit is provided.
TL;DR: It is illustrated that this algorithm also solves the ME PSE problem for both 1-D and 2-D signals when the region in which the correlation function is known has any arbitrary shape that includes the origin.
Abstract: A computationally simple algorithm has been recently proposed by Lim and Malik [1] to solve the two-dimensional (2-D) maximum entropy (ME) power spectrum estimation (PSE) problem. In this note, we illustrate that this algorithm also solves the ME PSE problem for both 1-D and 2-D signals when the region in which the correlation function is known has any arbitrary shape that includes the origin.
TL;DR: Binary random variables are regarded as random vectors in a binary-field (modulo-2) linear vector space and a characteristic function is defined and related results derived using this formulation.
Abstract: Binary random variables are regarded as random vectors in a binary-field (modulo-2) linear vector space. A characteristic function is defined and related results derived using this formulation. Minimax estimation of probability distributions using an entropy criterion is investigated, which leads to an A-distribution and bilinear discriminant functions. Nonparametric classification approaches using Hamming distances and their asymptotic properties are discussed. Experimental results are presented.
TL;DR: The experimental results are shown to support the claim by Cover that committee betting can do better than that of individual subjects in the estimation of the entropy of language.
Abstract: Cover and King's gambling approach for the estimation of the entropy of language is used to estimate the entropy of the Malay language. Our experimental results are shown to support the claim by Cover that committee betting can do better than that of individual subjects.
TL;DR: It is proved that of all Markov chains with common steady-state probabilities the one whose entropy is largest is theOne whose rows are all equal to the steady- state probabilities.
Abstract: It is proved that of all Markov chains with common steady-state probabilities the one whose entropy is largest is the one whose rows are all equal to the steady-state probabilities.