Showing papers on "Entropy (information theory) published in 1970"

Generalized information functions

Abstract: The concept of information functions of type β ( β > 0) is introduced and discussed. By means of these information functions the entropies of type β are defined. These entropies have a number of interesting algebraic and analytic properties similar to Shannon's entropy. The capacity of type β ( β > 1) of a discrete constant channel is defined by means of the entropy of type β . Examples are given for the computation of the capacity of type β , from which the Shannon's capacity can be derived as the limiting case β = 1.

On the entropy of context-free languages

Abstract: The information theoretical concept of the entropy (channel capacity) of context-free languages and its relation to the structure generating function is investigated in the first part of this paper. The achieved results are applied to the family of pseudolinear grammars. In the second part, relations between context-free grammars, infinite labelled digraphs and infinite nonnegative matrices are exhibited. Theorems on the convergence parameter of infinite matrices are proved and applied to the evaluation of the entropy of certain context-free languages. Finally, a stochastic process is associated with any context-free language generated by a deterministic labelled digraph, such that the stochastic process is equivalent to the language in the sense that both have the same entropy.

The Use of the Concept of Entropy in System Modelling

Abstract: The concept of "entropy" is widely used in the physical sciences and, recently, has aroused interest in the social sciences, especially in building models of urban and regional systems to be used for planning purposes. Most papers referring to the concept focus on the particular model being developed; this paper focuses on the concept itself. The first section is an introductory one which explains the use of system models in planning processes. In the second section, four views are presented of entropy: as related to probability and uncertainty; as a statistic of a probability distribution; in relation to Bayesian inference; and as a measurable system property. In the third part of the paper, the types of application of the concept are discussed in turn, and a number of other applications of the concept of "entropy" outside the usual physical science range are noted. The final section is a summary and concluding discussion, including a comparison of the approach of the statistician and entropy maximizer to system modelling.

Increasing the efficiency of quicksort

Abstract: A method is presented for the analysis of various generalzotions of quicksort. The average asymptotic number of comparisons needed is shown to be ozn log2 (n). A formula is derived expressing o~ in terms of the probability distribution of the \"bound\" of a partition. This formula assumes a partcdarly simple form for a generalization already considered by Hoare, namely, choice of the bound as median of a random sample. The main contribution of this paper is another generaization of quicksort, which uses a bounding interval instead of a single element as bound. This generalization turns c,~t to be easy to implement in a computer program. A numerical approximation shows that a = 1.140 for this version 0f qvicksort compared with 1.386 for the original. This implies a decrease in number of comparisons of 18 percent; actual tests showed about 15 percent saving in computing time.

Employment Concentration in the Common Market: An Entropy Approach

Abstract: Entropy has been proposed as a meaningful index of industrial concentration. It is shown that necessarily accurate estimates of entropy can be obtained from grouped data. Using the suggested procedure, employment concentration as measured by entropy is estimated for 26 two-digit industries in each of the six Common Market nations. The relationship between firm and industry size and concentration is then analysed, as are the international similarities in industrial structure.

Derivation of the Negative Exponential Model by an Entropy Maximising Method

Abstract: All the inhabitants of a city who participate in the choice of a place of residence are assumed to have a propensity to visit the urban centre. The distribution of residential locations is represented by a probability density surface whose horizontal plane projection is coextensive with that of the city. A spatially continuous system is thus defined in which it is shown that, under conditions of maximum entropy and subject to specific normalisation and cost constraints, the population is distributed in accordance with the negative exponential model of urban population densities.

Information and Inference

Abstract: I. Information and Induction.- On Semantic Information.- Bayesian Information Usage.- Experimentation as Communication with Nature.- II. Information and Some Problems of the Scientific Method.- On the Information Provided by Observations.- Quantitative Tools for Evaluating Scientific Systematizations.- Qualitative Information and Entropy Structures.- III. Information and Learning.- Learning and the Structure of Information.- IV. New Applications of Information Concepts.- Surface Information and Depth Information.- Towards a General Theory of Auxiliary Concepts and Definability in First-Order Theories.- Index of Names.- Index of Subjects.

Entropy and data compression

Abstract: Optimum compression ratio defined for Markov sources for compression algorithm efficiency and data compression schemes

Probability, Information and Entropy

Abstract: The claim that information theory provides a foundation for statistical thermodynamics which is independent of the use of ensembles can be sustained only if probabilities can be determined numerically without treating them as frequencies. The use of Shannon's measure of information to assign probabilities is correct only in the same conditions as justify Gibbs's use of ensembles.

Some statistical methods in machine intelligence research

Abstract: About a dozen examples are given of the use of statistical methods in research on machine intelligence, most, though not all, previously known, but not previously brought together. The topics include the application of rationality to the research as a whole; the trading of immediate gain for information; adaptive control without the identification of a model, by using smoothing techniques; phoneme recognition using distinctive features and their derivatives; the compiling of dictionaries; “botryology” or concept formation by clump finding; information retrieval; medical diagnosis; game playing and its relationship to theorem proving; design of an alphabet or of a vocabulary; and artificial neural networks. Among the statistical themes that are emphasized are the estimation of probabilities; the use of amounts of information and of evidence as substitutes for utility when utility is difficult to estimate; decision trees; “evolving” probabilities; and maximum, minimum, and minimax entropy in diagnosis. In this survey of methods it has been necessary at several points to make do with references to the literature.

Measures of uncertainty mathematical programming and physics

Abstract: The first section gives the measure of uncertainty given by Shannon (1948) and the generalizations thereof by Schvitzenberger (1954), Kullback (1959), Renyi (1961,1965), Kapur (1967, 1968), and Rathie (1970). It gives some postulates characterizing Shannon's entropy, Renyi's entropy of order a and our entropy of order a and type P. It also gives some properties of this most general type of entropy. In the second section an optimization problem is formulated and solved in the case of Shannon's and Renyi's entropies by the use of the principle of optimality. It is shown that this principle fails to solve the problem in the case of entropy of order a and type (3 and this leads to an interesting problem in non-linear integer fractional functional programming. In the third section, we discuss the connection between the concepts of entropy in information theory and physics and show how Shannon's entropy leads to Boltzman distribution of statistical mechanics but fails to give the Fermi-Dirac and BoseEinstein distributions of quantum mechanics. We find the entropies which lead to these distributions, but these do not satisfy an important property satisfied by Shannons entropy. This may give us some insight into quantum mechanical systems . In the fourth and last section, we obtain some properties of Bose-Einstein and Fermi-Dirac entropies obtained in the third section.

The noncomputability of the channel capacity of context-sensitive languages

Abstract: It is proved that there exists no algorithm (in the sense of recursive function theory) which applies to any context sensitive grammar and which allows one to This suggests that there exists no general calculation method to compute the entropy of the language generated by a context sensitive grammar.

Information theory and approximation of bandlimited functions

Abstract: For bandlimited functions, simultaneous approximation of a function and several of its derivatives is considered. Concomitant entropy estimates are obtained. A feasible algorithm for the transmission of information is discussed. This algorithm has been applied to the design of a class of PCM systems.1

On estimating the entropy of random fields

Abstract: Random fields entropy estimation technique taking into account higher than immediately adjacent spatial dependencies

Qualitative Information and Entropy Structures

Abstract: Information theory deals with the mathematical properties of communication models, which are usually defined in terms of concepts like channel, source, information, entropy, capacity, code, and which satisfy certain conditions and axioms.

Probabilities in context-free programmed grammars,

Abstract: : Context-free programmed grammars with probabilities attached to the 'go-to' fields are studied as realistic models for syntactical information sources. The model is formally defined and examples of its output are given. Simplifications that follow from the imposition of leftmost derivations are displayed. Source models for languages that are known to be context-free are studied, and for these a first-order Markov approximating source is obtained; standard methods are then used to calculate its entropy. (Author)

Information transmission theory

Abstract: : The theory of information transmission deals with the construction of methods for the coding and decoding of messages and signals with the object of transmitting them along communication channels, and the establishment of principles for the comparison of the effectiveness, noiseproof feature and feasibility of these methods. The principles of this theory are discussed: the conditions under which the transmission of information is possible; the information-theory aspects of the theory of information transmission and certain pertinent mathematical works are specified. In statistical coding the principal problems are: discretization messages; measurement of statistical properties of messages and statistical coding, i.e., the reduction of the sequence of digits by utilizing the statistical properties of messages.

Characterization and Modeling of Real Communication Channels

Abstract: : The paper presents new descriptive and generative models for the error-cluster and error gap patterns which occur in the binary, discrete-time stochastic processes observed as outputs of digital communication channels having memory. The slope of the error-gap distribution is used to uncover relationships between various channel models. One characterizes the memory mu of a process of error density Pe by its relative deviation in average conditional entropy from the discrete memoryless channel (D.M.C.), which one proves has maximum entropy for the class of (finite and infinite memory length) processes of density Pe. One obtains an upper bound for mu for real channels, derive mu for the general discrete renewal process from the error gap probability mass function (EGPMF) and prove that it is a lower bound for any processes having the same EGPMF. One demonstrates some limitations of finite error-free state models by showing that their EGPMF is bounded from above by a geometric series. To estimate the counting distribution with flexibility we introduce conditional gap distributions and multigap statistics; one uses these in implementing a denumerable Markov Chain model which, free from finite state model limitations and more general than renewal processes, allows the derivation of all classical statistics including entropy. (Author)

The Principle of Conservation of Entropy

Abstract: Let us go back to (1.8). If it is (approximately) true for the best possible encoding, then it must be true for arbitrary uniquely decodable coding. After the encoding we get a new random sequence of signals; we may consider it as a new information source. Let us denote by y. Further, H(y) denotes the information contant of one signal of y (for a moment heuristically). If the information is material-like, then the information contant of one information signal must be distributed on the L code signals which transmit it: (2.17) It is called the principle of conservation of entropy [2] and we shall now formulate precisely and prove it.

On the classical approximation in the quantum statistics of equivalent particles

Abstract: It is shown here that the microcanonical ensemble for a system of noninteracting bosons and fermions contains a subensemble of state vectors for which all particles of the system are distinguishable. This “IQC” (inner quantum-classical) subensemble is therefore fully classical, except for a rather extreme quantization of particle momentum and position, which appears as the natural price that must be paid for distinguishability. The contribution of the IQC subensemble to the entropy is readily calculated, and the criterion for this to be a good approximation to the exact entropy is a logarithmically strengthened form of the usual criterion for the validity of classical statistics in terms of the thermal de Broglie wavelength and the average volume per particle. Thus, it becomes possible to derive the Maxwell-Boltzmann distribution directly from the ensemble in the classical limit, using fully classical reasoning about the distinguishability of particles. The entropy is additive—theN! factor of the Boltzmann count cancels out in the course of the calculation, and the “N! paradox” is thereby resolved. The method of “correct Boltzmann counting” and the lowest term of the Wigner-Kirkwood series for the partition function are seen to be partly based on the IQC subensemble, and their partly nonclassical nature is clarified. The clear separation in the full ensemble of classical and nonclassical components makes it possible to derive the classical statistics of indistinguishable particles from their quantum statistics in a controlled, explicit way. This is particularly important for nonequilibrium theory. The treatment of molecular collisions along too-literally classical lines turns out to require exorbitantly high temperatures, although there are suggestions of indirect ways in which classical nonequilibrium theory might be justified at ordinary temperatures. The applicability of exact classical ergodic and mixing theory to systems at ordinary temperatures is called into question, although the general idea of coarse-graining is confirmed. The concepts on which the IQC idea is based are shown to give rise to a series development of thermostatistical quantities, starting with the distinguishable-particle approximation.