Showing papers on "Entropy (information theory) published in 1982"

Realization of Covariance Sequences

Abstract: This paper examines the problem of “positivity” in relation to the partial realization of scalar power series. An exact criterion of positivity is proved for second-order realizations. The general case is currently unsolved. Even the special results contained here show that the so-called “maximum entropy principle” cannot be applied to the realization problem in the naive sense in which it is employed by physicists. It would be better to call this principle a “prejudice” because it does not fully utilize the information inherent in the data and does not provide a realization with natural (“minimal”) mathematical properties.

Minimum Cross-Entropy Pattern Classification and Cluster Analysis

Abstract: This paper considers the problem of classifying an input vector of measurements by a nearest neighbor rule applied to a fixed set of vectors. The fixed vectors are sometimes called characteristic feature vectors, codewords, cluster centers, models, reproductions, etc. The nearest neighbor rule considered uses a non-Euclidean information-theoretic distortion measure that is not a metric, but that nevertheless leads to a classification method that is optimal in a well-defined sense and is also computationally attractive. Furthermore, the distortion measure results in a simple method of computing cluster centroids. Our approach is based on the minimization of cross-entropy (also called discrimination information, directed divergence, K-L number), and can be viewed as a refinement of a general classification method due to Kullback. The refinement exploits special properties of cross-entropy that hold when the probability densities involved happen to be minimum cross-entropy densities. The approach is a generalization of a recently developed speech coding technique called speech coding by vector quantization.

Minimum entropy quantizers and permutation codes

Abstract: Amplitude quantization and permutation encoding are two approaches to efficient digitization of analog data. It has been proven that they are equivalent in the sense that their optimum rate versus distortion performances are identical. Reviews of the aforementioned results and of work performed in the interim by several investigators are presented. Equations which must be satisfied by the thresholds of the minimum entropy quantizer that achieves a prescribed mean r th power distortion are derived, and an iterative procedure for solving them is developed. It is shown that these equations often have many families of solutions. In the case of the Laplacian distribution, for which we had previously shown that quantizers with uniformly spaced thresholds satisfy the equations when r=2 , other families of solutions with nonuniform spacing are exhibited. What had appeared to be a discrepancy between the performances of optimum permutation codes and minimum entropy quantizers is resolved by the resulting optimum quantizers, which span all entropy rates from zero to infinity.

Information rates and power spectra of digital codes

Abstract: The encoding of independent data symbols as a sequence of discrete amplitude, real variables with given power spectrum is considered. The maximum rate of such an encoding is determined by the achievable entropy of the discrete sequence with the given constraints. An upper bound to this entropy is expressed in terms of the rate distortion function for a memoryless finite alphabet source and mean-square error distortion measure. A class of simple dc-free power spectra is considered in detail, and a method for constructing Markov sources with such spectra is derived. It is found that these sequences have greater entropies than most codes with similar spectra that have been suggested earlier, and that they often come close to the upper bound. When the constraint on the power spectrum is replaced by a constraint On the variance of the sum of the encoded symbols, a stronger upper bound to the rate of dc-free codes is obtained. Finally, the optimality of the binary biphase code and of the ternary bipolar code is decided.

Why use population entropy? It determines the rate of convergence

Abstract: The population entropy introduced by Demetrius is shown to have a precise dynamical meaning as a measure of convergence rate to the stable age distribution. First the Leslie population model is transformed exactly into a Markov chain on a state space of age-classes. Next the dynamics of convergence from a nonequilibrium state to the stable state are analyzed. The results provide the first clear biological reason why entropy is a broadly useful population statistic.

Maximum entropy image reconstruction - A practical non-information-theoretic approach

Abstract: The maximum entropy method (MEM) of image reconstructtion is discussed in the context of incomplete Fourier information (as in aperture synthesis) Several current viewpoints on the conceptual foundation of the method are analysed and found to be unsatisfactory It is concluded that the MEM is a form of model-fitting, the model being a non-linear transform of a band-limited function A whole family of ’entropies’ can be constructed to give reconstructions which (a) are individually unique, (b) have sharpened peaks and (c) have flattened baselines The widely discussed 1nB and - B1nB forms of the entropy are particular cases and lead to Lorentzian and Gaussian shaped peaks respectively However, they hardly exhaust the possibilities-for example, B1/2 is equally good The two essential features of peak sharpening and baseline flattening are shown to depend on a parameter which can be controlled by adding a suitable constant to the zero spacing correlation ρ00 This process, called FLOATing, effectively tames much of the unphysical behaviour noted in earlier studies of the MEM A numerical scheme for obtaining the MEM reconstruction is described This incorporates the FLOAT feature and uses the fast Fourier transform (FFT), requiring about a hundred FFTs for convergence Using a model brightness distribution, the MEM reconstructions obtained for different entropies and different values of the resolution parameter are compared The results substantiate the theoretically deduced properties of the MEM To allow for noise in the data, the least-squares approach has been widely used It is shown that this method is biased since it leads to deterministic residuals which do not have a Gaussian distribution It is suggested that fitting the noisy data exactly has the advantage of being unbiased even though the noise appears in the final map A comparison of the strengths and weaknesses of the MEM and CLEAN suggests that the MEM already has a useful role to play in image reconstruction

Symmetropy, an entropy-like measure of visual symmetry

Abstract: A new objective measure of symmetry for single patterns, called symmetropy, is developed on two bases, the two-dimensional discrete Walsh transform of a pattern and the entropy concept in information theory. It is extended to a more general measure, called the symmetropy vector. In order to test the predictive power of the symmetropy vector, multiple regression analyses of judged pattern goodness and of judged pattern complexity were carried out. The analyses show that the symmetropy vector predicts pattern goodness and pattern complexity, as well as the amount of symmetry in a pattern. They also suggest that pattern goodness is a concept based on the holistic properties of a pattern, while pattern complexity (or simplicity) is a concept based on both holistic and partial properties of a pattern.

On the convexity of higher order Jensen differences based on entropy functions (Corresp.)

Abstract: In an earlier work, the authors introduced a divergence measure, called the first-order Jensen difference, or in short \cal j -divergence, which is based on entropy functions of degree \alpha . This provided a generalization of the measure of mutual information based on Shannon's entropy (corresponding to \alpha = 1) . It was shown that the first-order \cal j -divergence is a convex function only when a is restricted to some range. We define higher order Jensen differences and show that they are convex functions only when the underlying entropy function is of degree two. A statistical application requiring the convexity of higher order Jensen differences is indicated.

Accessibility, entropy and the distribution and assignment of traffic revisited

Abstract: An optimizing model which minimizes average generalised trip cost subject to constraints on the entropy was given in a previous paper. In this note the model is placed in a planning context. (Author/TRRL)

Statistical features of the evolution of two-dimensional turbulence

Abstract: Statistical fluid dynamics identifies a functional of the fluid energy spectrum that plays the role of Boltzmann’s entropy for fluids. Through a series of two-dimensional flow simulations we confirm the theoretical predictions for the behaviour of this entropy functional. This includes a demonstration of Loschmidt’s paradox and an examination of the effects of Rossby waves and viscosity on the behaviour of the entropy.

Application of group theory to the calculation of the configurational entropy in the cluster variation method

Abstract: The counting and classification of clusters, required for the entropy calculation in the cluster variation method, is systematically carried out for general crystal structures. The approach explicitly uses the symmetry properties of both the set of clusters and the crystal structure, in conjunction with simple group-theoretical considerations. As an illustration of the method, several new configurational entropy expressions are calculated for HCP, diamond cubic and spinel structures. For each approximation, the critical temperature of the corresponding Ising ferromagnet has been calculated.

Properties of two-dimensional maximum entropy power spectrum estimates

Abstract: In this paper we present the results of a number of experiments that have been performed to study the properties of two-dimensional maximum entropy spectral estimates. The results presented include studies on the resolution differences for real and complex data, resolution properties of the spectral estimates, and the determination of the relative power of the sinusoids from the spectral estimates. The results also include the effects of signal-to-noise ratio, size and shape of the known autocorrelation region, and data length and initial phase of a sinusoid, on the spectral estimates. In most cases studied, the properties of two-dimensional maximum entropy spectral estimates can be viewed as simple extensions of their one-dimensional counterparts.

Information theory and the spectrum of isotropic turbulence

Abstract: The problem of determining the spectrum of isotropic turbulence can be thought of as one of finding the most appropriate joint probability distribution for the flow taken as a whole. From the point of view of information theory, what one means by the most appropriate distribution is clearly defined and easily justified; it is the probability distribution that maximises the information theory entropy, subject to whatever constraints one can impose on the flow. In this work, the relevant constraints are taken to be the Reynolds number and energy dissipation rate of the flow, energy balance (on average) at every point in wavenumber space, and adherence to the Navier-Stokes equations. Using these constraints, it is shown that the maximum entropy formalism leads to a pair of coupled equations describing the distribution of energy in the turbulent spectrum, and the correlations between the amplitudes of velocity components with nearly identical wavenumbers. Although solutions to these equations are not presented, it develops that if a power-law solution exists, it can only be the Kolmogorov law E(k) varies as k-53/. In arriving at this result, a useful concept is that of the 'turbulent temperature', defined as the reciprocal of the derivative of the entropy with respect to the local energy dissipation rate. This quantity plays a role directly analogous to the thermodynamic temperature, governing the rate of energy exchange between different wavenumbers. It is found that, within the spectrum's inertial subrange, the turbulent temperature is virtually constant, with only a minute temperature gradient required to drive the energy cascade.

A Phenomenological Theory of Socioeconomic Systems with Spatial Interactions

Abstract: This paper deals with a class of models which describe spatial interactions and are based on Jaynes's principle. The variables entering these models can be partitioned in four groups: (a) probability density distributions (for example, relative traffic flows), (b) expected values (average cost of travel), (c) their duals (Lagrange multipliers, traffic impedance coefficient), and (d) operators transforming probabilities into expected values.The paper presents several dual formulations replacing the problem of maximizing entropy in terms of the group of variables (a) by equivalent extreme problems involving groups (b)-(d). These problems form the basis of a phenomenological theory. The theory makes it possible to derive useful relationships among groups (b) and (c). There are two topics discussed: (1) practical application of the theory (with examples), (2) the relationship between socioeconomic modelling and statistical mechanics.

Information of partitions with applications to random access communications

Abstract: The minimum amount of information and the asymptotic minimum amount of entropy of a random partition which separates the points of a Poisson point process are found. Related information theoretic bounds are applied to yield an upper bound to the throughput of a random access broadcast channel. It is shown that more information is needed to separate points by partitions consisting of intervals than by general partitions. This suggests that single-interval conflict resolution algorithms may not achieve maximum efficiency.

Fuzzy design of military information systems

Abstract: On the supposition that military operations are by nature fuzzy endeavours, some fuzzy set theory concepts are applied to the analysis of combat. More specifically, the military command and control function, which directs combat operations, is examined from the perspective of a supporting management information system (MIS), Since a military commander's perspective of the situation can be as important as the facts of the situation, a partitioned data base to support the MIS is introduced. One part contains data which may be either sharp or fuzzy; the other part, information relating to context. It is suggested that probability concepts apply to the former, and that possibility concepts apply to the latter. Data fusion is defined as the joining of the two partitions to provide information in context to the relevant field commander and his staff. Based on a simple military scenario, the design implications of this fusion are explored. A number of examples using recent fuzzy set results are applied to a conceptual fuzzy data base. In particular, both probabilistic and possibilistic entropy appear to operate within the data base, and structure it serves. Both entropy concepts lead to the introduction of information context regeneration along the command network serviced by the fuzzy MIS. The idea of fuzzy containment in a real system is discussed. It is asserted that both the classical hierarchial structure and the centralized control structures of the military serve to keep fuzziness within bounds. The concept of contextual noise is introduced. Throughout, fuzzy set theory is advanced as a conceptual framework for design rather than a detailed design tool. This suggestion is based partially on the evidence that the universe of fuzzy data and information is much larger than that of sharp data. Moreover, in military situations, there may only be fuzzy information.

Minimum Information Stochastic Modelling of Linear Systems with a Class of Parameter Uncertainies

Abstract: This paper considers the problem of mean-square optimal control for a linear system with stochastic parameters and limited prior information. For specific application to flexible mechanical systems, consideration is limited to the class of multiplicative parameter perturbations of skew-hermitian type. To avoid ad hoc assumptions regarding a priori statistics, a prior probability assignment is induced from available data through use of a maximum entropy principle. Moreover, we discern a minimum set of a priori data which is just sufficient to induce a well-defined maximum entropy probability assignment. The statistical-description induced by this minimum data set is tantamount to a form of Stratonovich state-dependent noise.

Remarks on Shannon's secrecy systems

Abstract: The paper contains three improvements of Shannon's theory of secrecy systems: 1. By a very simple construction we obtain ciphers which are with respect to natural security measures as good as Shannon's 'random ciphers'. 2. For this construction it is unnecessary to assume that the messages are essentially equally likely. Shannon made this assumption in order to the make his 'random cipher' approach work. 3. Furthermore we construct optimal ciphers under the rather robust assumption that only a bound on the entropy of the source is known to the communicators and that the cryptanalyst is still granted to know the message statistic exactly. Finally we construct worst codes for the binary symmetric channel and emphasize the importance of this 'dual coding problem' for cryptography.

Noise Scaling of Symbolic Dynamics Entropies

Abstract: An increasing body of experimental evidence supports the belief that random behavior observed in a wide variety of physical systems is due to underlying deterministic dynamics on a low-dimensional chaotic attractor. The behavior exhibited by a chaotic attractor is predictable on short time scales and unpredictable (random) on long time scales. The unpredictability, and so the attractor’s degree of chaos, is effectively measured by the entropy. Symbolic dynamics is the application of information theory to dynamical systems. It provides experimentally applicable techniques to compute the entropy, and also makes precise the difficulty of constructing predictive models of chaotic behavior. Furthermore, symbolic dynamics offers methods to distinguish the features of different kinds of randomness that may be simultaneously present in any data set: chaotic dynamics, noise in the measurement process, and fluctuations in the environment.

Cross-entropy minimization given fully decomposable subset and aggregate constraints (Corresp.)

Abstract: The principle of maximum entropy and the principle of minimum cross-entropy (minimum directed divergence, minimum discrimination information) have been applied recently to problems in queuing theory and computer-system performance modeling. These information-theoretic principles estimate probability distributions based on information in the form of known expected values. In the case of queuing theory and computer-system modeling, the known expected values arise from rate balance equations. This correspondence concerns situations in which the system state probabilities decompose into disjoint subsets and in which the known expected values are either expectations conditional on a specific subset or expectations involving aggregate subset probabilities. New properties of minimum cross-entropy distributions are derived and an efficient method of computing these distributions is derived. Computational examples are included. In the case of queuing theory and computer-system modeling, the disjoint subsets correspond to internal device states, and the aggregate probabilities correspond to overall device states. The results here apply when one has both rate balance equations for device equilibrium involving internal device state probabilities, as well as rate balance equations for system equilibrium involving aggregate device state probabilities.

Spatial Analysis of Interacting Economies: The Role of Entropy and Information Theory in Spatial Input-Output Modeling

Abstract: 6. 2 Basic Model Characteristics 185 6. 3 A Closed Model Approach to Interregional Estimation 189 7 Towards an Integrated System of Models for National and Regional Development 205 7. 1 Introduction 205 7. 2 In Search of a Framework for Integration 207 7. 3 National Development Scenarios 222 7. 4 The National-Regional Interlace 231 7. 5 Regional Development Scenarios 236 7. 6 Concluding Remarks 244 Appendixes 253 A Basic Microstate Descriptions 253 B Incomplete Prior Information: A Simple Example 257 C Computing Capital Coefficients and Turnpike Solutions: The DYNIO Package 259 D Minimizing Information Losses in Simple Aggregation: Two Test Problems 274 E Computing Interregional and Intersectoral Flows: 276 References 287 Index 305 vi LIST OF FIGURES 1. 1 A Three-Dimensional Guide to Later Chapters 12 2. 1 Historical Development of the Entropy Concept 32 2. 2 Selected Applications of Information Theory to Input-Output Analysis and Interaction Modelling 48 3. 1 The Bose-Einstein Analogy 58 5. 1 The Dog-Leg Input-Output Table 159 7. 1 A General Multilevel Social System 219 7. 2 The Hierarchical System of Models 219 7. 3 Choice of Production Techniques 230 7. 4 The National-Regional Interface 235 7. 5 A Sequential Compromise Procedure 243 7. 6 The Integrated Modelling System 246 vii LIST OF TABLES 3. 1 Production-Constrained Microstate Descriptions 59 3. 2 Production-Constrained Entropy Formulae 62 3. 3 Production-Constrained Solutions 65 3. 4 Doubly-Constrained Solutions 73 4. 1 The Static Input-Output Table 85 4.

Channel Entropy and Primitive Approximation

Abstract: For stationary channels, channel entropy is defined as the supremum of the conditional output entropy over all stationary input sources. If the channel is $\bar{d}$-continuous and conditionally almost block independent, channel entropy gives an upper bound to the output rate of the channel. Furthermore, the channel can be approximated arbitrarily well in the $\bar{d}$-metric by any primitive channel whose noise source entropy exceeds channel entropy and cannot be so approximated if the noise source entropy is less than channel entropy.

Instabilities of R nyi Entropies

Abstract: We show that for systems with a large number of microstates R~nyi entropies do not represent experimentally observable quantities except the R~nyi entropy that coincides with the Shannon entropy

Fuzzy Set Theoretic Approach: A Tool for Speech and Image Recognition

Abstract: The paper consists of three parts. In the first part of the paper, a self-supervised learning algorithm with the concept of guard zones around the class representative vectors has been presented for vowel sound recognition. In the second part, an algorithm consisting of histogram equalisation technique followed by a further enhancement using fuzzy S and π membership functions is described for detecting the small variation in grey levels and identifying the different regional contours of x-ray images. Finally, a quantitative measure of image-quality is provided by the terms “index of fuzziness”, “entropy” and “π-ness” of a fuzzy set.

Entropy of mixing of liquid metal alloys

Abstract: An analysis of the dependence of entropy of mixing on the concentration of the constituents in alloy has been presented, based on the hard-sphere reference system. The parameters of the hard-sphere system, as calculated by fitting the excess entropy at equiatomic concentration, provide results comparable with those obtained by the variational method. The alloying properties are found to be sensitive to the atomic composition of the alloy.

Entropy and Martingales in Markov Chain Models

Abstract: The concept of entropy in models is discussed with particular reference to the work of P.A.P. Moran. For a vector-valued Markov chain {X}) whose states are relative-frequency (proportion) tables corresponding to a physical mixing model of a number N of particles over n urns, the definition of entropy may be based on the usual information-theoretic concept applied to the probability distribution given by the expectation W(Xk). The model is used for a brief probabilistic assessment of the relationship between Boltzmann's H-Theorem, the Ehrenfest urn model, and Poincare's considerations on the mixing of liquids and card shuffling, centred on the property of an ultimately uniform distribution of a single particle. It is then generalized to the situation where the total number of particles fluctuates over time, and martingale results are used to establish