On the rationale of maximum-entropy methods
01 Sep 1982-Vol. 70, Iss: 9, pp 939-952
TL;DR: The relations between maximum-entropy (MAXENT) and other methods of spectral analysis such as the Schuster, Blackman-Tukey, maximum-likelihood, Bayesian, and Autoregressive models are discussed, emphasizing that they are not in conflict, but rather are appropriate in different problems.
Abstract: We discuss the relations between maximum-entropy (MAXENT) and other methods of spectral analysis such as the Schuster, Blackman-Tukey, maximum-likelihood, Bayesian, and Autoregressive (AR, ARMA, or ARIMA) models, emphasizing that they are not in conflict, but rather are appropriate in different problems. We conclude that: 1) "Orthodox" sampling theory methods are useful in problems where we have a known model (sampling distribution) for the properties of the noise, but no appreciable prior information about the quantities being estimated. 2) MAXENT is optimal in problems where we have prior information about multiplicities, but no noise. 3) The full Bayesian solution includes both of these as special cases and is needed in problems where we have both prior information and noise. 4) AR models are in one sense a special case of MAXENT, but in another sense they are ubiquitous in all spectral analysis problems with discrete time series. 5) Empirical methods such as Blackman-Tukey, which do not invoke even a likelihood function, are useful in the preliminary, exploratory phase of a problem where our knowledge is sufficient to permit intuitive judgments about how to organize a calculation (smoothing, decimation, windows, prewhitening, padding with zeroes, etc.) but insufficient to set up a quantitative model which would do the proper things for us automatically and optimally.
Citations
More filters
University of Melbourne1, Stony Brook University2, City University of New York3, Princeton University4, University of Lausanne5, University of California, Berkeley6, University of Alaska Fairbanks7, National Institute of Water and Atmospheric Research8, Commonwealth Scientific and Industrial Research Organisation9, University of São Paulo10, University of Missouri11, Consejo Nacional de Ciencia y Tecnología12, University of Kansas13, Landcare Research14, AT&T15, McGill University16, James Cook University17, Swiss Federal Institute for Forest, Snow and Landscape Research18
TL;DR: This work compared 16 modelling methods over 226 species from 6 regions of the world, creating the most comprehensive set of model comparisons to date and found that presence-only data were effective for modelling species' distributions for many species and regions.
Abstract: Prediction of species' distributions is central to diverse applications in ecology, evolution and conservation science. There is increasing electronic access to vast sets of occurrence records in museums and herbaria, yet little effective guidance on how best to use this information in the context of numerous approaches for modelling distributions. To meet this need, we compared 16 modelling methods over 226 species from 6 regions of the world, creating the most comprehensive set of model comparisons to date. We used presence-only data to fit models, and independent presence-absence data to evaluate the predictions. Along with well-established modelling methods such as generalised additive models and GARP and BIOCLIM, we explored methods that either have been developed recently or have rarely been applied to modelling species' distributions. These include machine-learning methods and community models, both of which have features that may make them particularly well suited to noisy or sparse information, as is typical of species' occurrence data. Presence-only data were effective for modelling species' distributions for many species and regions. The novel methods consistently outperformed more established methods. The results of our analysis are promising for the use of data from museums and herbaria, especially as methods suited to the noise inherent in such data improve.
7,589 citations
Cites methods from "On the rationale of maximum-entropy..."
...Similarly, maximum entropy methods are well known in other fields (Jaynes 1982) but only recently developed for questions of species’ distributions (Phillips et al. 2006)....
[...]
01 Jan 2019
TL;DR: The book presents a thorough treatment of the central ideas and their applications of Kolmogorov complexity with a wide range of illustrative applications, and will be ideal for advanced undergraduate students, graduate students, and researchers in computer science, mathematics, cognitive sciences, philosophy, artificial intelligence, statistics, and physics.
Abstract: The book is outstanding and admirable in many respects. ... is necessary reading for all kinds of readers from undergraduate students to top authorities in the field. Journal of Symbolic Logic Written by two experts in the field, this is the only comprehensive and unified treatment of the central ideas and their applications of Kolmogorov complexity. The book presents a thorough treatment of the subject with a wide range of illustrative applications. Such applications include the randomness of finite objects or infinite sequences, Martin-Loef tests for randomness, information theory, computational learning theory, the complexity of algorithms, and the thermodynamics of computing. It will be ideal for advanced undergraduate students, graduate students, and researchers in computer science, mathematics, cognitive sciences, philosophy, artificial intelligence, statistics, and physics. The book is self-contained in that it contains the basic requirements from mathematics and computer science. Included are also numerous problem sets, comments, source references, and hints to solutions of problems. New topics in this edition include Omega numbers, KolmogorovLoveland randomness, universal learning, communication complexity, Kolmogorov's random graphs, time-limited universal distribution, Shannon information and others.
3,361 citations
TL;DR: In this article, the minimum description length (MDL) criterion is used to estimate the total number of binary digits required to rewrite the observed data, when each observation is given with some precision.
Abstract: of the number of bits required to write down the observed data, has been reformulated to extend the classical maximum likelihood principle. The principle permits estimation of the number of the parameters in statistical models in addition to their values and even of the way the parameters appear in the models; i.e., of the model structures. The principle rests on a new way to interpret and construct a universal prior distribution for the integers, which makes sense even when the parameter is an individual object. Truncated realvalued parameters are converted to integers by dividing them by their precision, and their prior is determined from the universal prior for the integers by optimizing the precision. 1. Introduction. In this paper we study estimation based upon the principle of minimizing the total number of binary digits required to rewrite the observed data, when each observation is given with some precision. Instead of attempting at an absolutely shortest description, which would be futile, we look for the optimum relative to a class of parametrically given distributions. This Minimum Description Length (MDL) principle, which we introduced in a less comprehensive form in [25], turns out to degenerate to the more familiar Maximum Likelihood (ML) principle in case the number of parameters in the models is fixed, so that the description length of the parameters themselves can be ignored. In another extreme case, where the parameters determine the data, it similarly degenerates to Jaynes's principle of maximum entropy, [14]. But the main power of the new criterion is that it permits estimates of the entire model, its parameters, their number, and even the way the parameters appear in the model; i.e., the model structure. Hence, there will be no need to supplement the estimated parameters with a separate hypothesis test to decide whether a model is adequately parameterized or, perhaps, over parameterized.
1,762 citations
Cites background from "On the rationale of maximum-entropy..."
...In the final Section 5, we show that Jaynes's principle of maximum entropy [14, 15, 16, 17] may be viewed as a particular instance of ours....
[...]
...This is of some significance because there are a number of important applications where the ML principle fails but where the maximum entropy formalism has been highly successful [16, 17]....
[...]
Book•
01 Jan 2001TL;DR: This detailed and thoroughly enhanced third edition presents a comprehensive study / reference to theories, methodologies and recent developments in solving computer vision problems based on MRFs, statistics and optimisation.
Abstract: Markov random field (MRF) theory provides a basis for modeling contextual constraints in visual processing and interpretation. It enables systematic development of optimal vision algorithms when used with optimization principles. This detailed and thoroughly enhanced third edition presents a comprehensive study / reference to theories, methodologies and recent developments in solving computer vision problems based on MRFs, statistics and optimisation. It treats various problems in low- and high-level computational vision in a systematic and unified way within the MAP-MRF framework. Among the main issues covered are: how to use MRFs to encode contextual constraints that are indispensable to image understanding; how to derive the objective function for the optimal solution to a problem; and how to design computational algorithms for finding an optimal solution. Easy-to-follow and coherent, the revised edition is accessible, includes the most recent advances, and has new and expanded sections on such topics as: Discriminative Random Fields (DRF) Strong Random Fields (SRF) Spatial-Temporal Models Total Variation Models Learning MRF for Classification (motivation + DRF) Relation to Graphic Models Graph Cuts Belief Propagation Features: Focuses on the application of Markov random fields to computer vision problems, such as image restoration and edge detection in the low-level domain, and object matching and recognition in the high-level domain Presents various vision models in a unified framework, including image restoration and reconstruction, edge and region segmentation, texture, stereo and motion, object matching and recognition, and pose estimation Uses a variety of examples to illustrate how to convert a specific vision problem involving uncertainties and constraints into essentially an optimization problem under the MRF setting Introduces readers to the basic concepts, important models and various special classes of MRFs on the regular image lattice and MRFs on relational graphs derived from images Examines the problems of parameter estimation and function optimization Includes an extensive list of references This broad-ranging and comprehensive volume is an excellent reference for researchers working in computer vision, image processing, statistical pattern recognition and applications of MRFs. It has been class-tested and is suitable as a textbook for advanced courses relating to these areas.
1,694 citations
Cites background from "On the rationale of maximum-entropy..."
...The maximum entropy criterion is simply taking this fact into account: Configurations with higher entropy are more likely because nature can generate them in more ways (Jaynes 1982)....
[...]
Book•
01 Aug 1995TL;DR: This book presents a comprehensive study on the use of MRFs for solving computer vision problems, and covers the following parts essential to the subject: introduction to fundamental theories, formulations of MRF vision models, MRF parameter estimation, and optimization algorithms.
Abstract: From the Publisher:
Markov random field (MRF) theory provides a basis for modeling contextual constraints in visual processing and interpretation. It enables us to develop optimal vision algorithms systematically when used with optimization principles. This book presents a comprehensive study on the use of MRFs for solving computer vision problems. The book covers the following parts essential to the subject: introduction to fundamental theories, formulations of MRF vision models, MRF parameter estimation, and optimization algorithms. Various vision models are presented in a unified framework, including image restoration and reconstruction, edge and region segmentation, texture, stereo and motion, object matching and recognition, and pose estimation. This book is an excellent reference for researchers working in computer vision, image processing, statistical pattern recognition, and applications of MRFs. It is also suitable as a text for advanced courses in these areas.
1,333 citations
Cites background from "On the rationale of maximum-entropy..."
...can generate them in more ways and the maximum entropy criterion is simply taking this fact into account (Jaynes 1982)....
[...]
References
More filters
TL;DR: In this article, the authors consider statistical mechanics as a form of statistical inference rather than as a physical theory, and show that the usual computational rules, starting with the determination of the partition function, are an immediate consequence of the maximum-entropy principle.
Abstract: Information theory provides a constructive criterion for setting up probability distributions on the basis of partial knowledge, and leads to a type of statistical inference which is called the maximum-entropy estimate. It is the least biased estimate possible on the given information; i.e., it is maximally noncommittal with regard to missing information. If one considers statistical mechanics as a form of statistical inference rather than as a physical theory, it is found that the usual computational rules, starting with the determination of the partition function, are an immediate consequence of the maximum-entropy principle. In the resulting "subjective statistical mechanics," the usual rules are thus justified independently of any physical argument, and in particular independently of experimental verification; whether or not the results agree with experiment, they still represent the best estimates that could have been made on the basis of the information available.It is concluded that statistical mechanics need not be regarded as a physical theory dependent for its validity on the truth of additional assumptions not contained in the laws of mechanics (such as ergodicity, metric transitivity, equal a priori probabilities, etc.). Furthermore, it is possible to maintain a sharp distinction between its physical and statistical aspects. The former consists only of the correct enumeration of the states of a system and their properties; the latter is a straightforward example of statistical inference.
12,099 citations
TL;DR: In this article, a technique for image reconstruction by a maximum entropy method is presented, which is sufficiently fast to be useful for large and complicated images and is applicable in spectroscopy, electron microscopy, X-ray crystallography, geophysics and virtually any type of optical image processing.
Abstract: Results are presented of a powerful technique for image reconstruction by a maximum entropy method, which is sufficiently fast to be useful for large and complicated images. Although our examples are taken from the fields of radio and X-ray astronomy, the technique is immediately applicable in spectroscopy, electron microscopy, X-ray crystallography, geophysics and virtually any type of optical image processing. Applied to radioastronomical data, the algorithm reveals details not seen by conventional analysis, but which are known to exist.
969 citations
TL;DR: In this paper, the authors summarize progress to date toward generalizing Gibbs' variational principle to nonequilibrium conditions, and conclude that the outlook is good in that the basic principles are believed known; but they do not yet �now whether they can be reduced to simple rules immediately useful in practice, in the way that the Gibbs phase rule is useful.
Abstract: It seems intuitively reasonable that Gibbs' variational principle de termining the conditions of heterogeneous equilibrium can be gener alized to nonequilibrium conditions. That is, a nonequilibriurn steady state should be the one that makes some kind of generalized-entropy production stationary; and even in the presence of irreversible fluxes, the condition for migrational equilibrium should still be the equality of some generalized chemical potentials. We summarize progress to date toward this goal, reviewing (a) the early history, (b) work of Onsager and first attempts at generalization, (c) the new direction the field took after 1967 with the work of Tykodi and Mitchell, and (d) the present situation and prospects. Our conclu sion will be, briefly, that the outlook is good in that the basic principles are believed known; but we do not yet �now whether they can be reduced to simple rules immediately useful in practice, in the way that the Gibbs phase rule is useful. For this, we need more experience in the technique of applying them to particular cases, and more data to test some conjectures.
400 citations
TL;DR: Maximum entropy spectral analysis is a method for the estimation of power spectra with a higher resolution than can be obtained with conventional techniques by extrapolation of the autocorrelation function in such a way that the entropy of the corresponding probability density function is maximized in each step of the extrapolation.
Abstract: Maximum entropy spectral analysis is a method for the estimation of power spectra with a higher resolution than can be obtained with conventional techniques. This is achieved by extrapolation of the autocorrelation function in such a way that the entropy of the corresponding probability density function is maximized in each step of the extrapolation. This correspondence also gives a simple interpretation of the method without entropy considerations.
268 citations