Showing papers on "Entropy (information theory) published in 2009"

Some Research Problems in Uncertainty Theory

Abstract: In addition to the four axioms of uncertainty theory, this paper presents the flfth axiom called product measure axiom. This paper also gives an operational law of independent uncertain variables and a concept of entropy of continuous uncertain variables. Based on the uncertainty theory, a new uncertain calculus is proposed and applied to uncertain difierential equation, flnance, control, flltering and dynamical systems. Finally, an uncertain inference will be presented. c

Entropy: a consolidation manager for clusters

Abstract: Clusters provide powerful computing environments, but in practice much of this power goes to waste, due to the static allocation of tasks to nodes, regardless of their changing computational requirements. Dynamic consolidation is an approach that migrates tasks within a cluster as their computational requirements change, both to reduce the number of nodes that need to be active and to eliminate temporary overload situations. Previous dynamic consolidation strategies have relied on task placement heuristics that use only local optimization and typically do not take migration overhead into account. However, heuristics based on only local optimization may miss the globally optimal solution, resulting in unnecessary resource usage, and the overhead for migration may nullify the benefits of consolidation.In this paper, we propose the Entropy resource manager for homogeneous clusters, which performs dynamic consolidation based on constraint programming and takes migration overhead into account. The use of constraint programming allows Entropy to find mappings of tasks to nodes that are better than those found by heuristics based on local optimizations, and that are frequently globally optimal in the number of nodes. Because migration overhead is taken into account, Entropy chooses migrations that can be implemented efficiently, incurring a low performance overhead.

On the effectiveness of secret key extraction from wireless signal strength in real environments

Abstract: We evaluate the effectiveness of secret key extraction, for private communication between two wireless devices, from the received signal strength (RSS) variations on the wireless channel between the two devices. We use real world measurements of RSS in a variety of environments and settings. Our experimental results show that (i) in certain environments, due to lack of variations in the wireless channel, the extracted bits have very low entropy making these bits unsuitable for a secret key, (ii) an adversary can cause predictable key generation in these static environments, and (iii) in dynamic scenarios where the two devices are mobile, and/or where there is a significant movement in the environment, high entropy bits are obtained fairly quickly. Building on the strengths of existing secret key extraction approaches, we develop an environment adaptive secret key generation scheme that uses an adaptive lossy quantizer in conjunction with Cascade-based information reconciliation [7] and privacy amplification [14]. Our measurements show that our scheme, in comparison to the existing ones that we evaluate, performs the best in terms of generating high entropy bits at a high bit rate. The secret key bit streams generated by our scheme also pass the randomness tests of the NIST test suite [21] that we conduct.

An informational approach to the global optimization of expensive-to-evaluate functions

Abstract: In many global optimization problems motivated by engineering applications, the number of function evaluations is severely limited by time or cost. To ensure that each evaluation contributes to the localization of good candidates for the role of global minimizer, a sequential choice of evaluation points is usually carried out. In particular, when Kriging is used to interpolate past evaluations, the uncertainty associated with the lack of information on the function can be expressed and used to compute a number of criteria accounting for the interest of an additional evaluation at any given point. This paper introduces minimizers entropy as a new Kriging-based criterion for the sequential choice of points at which the function should be evaluated. Based on stepwise uncertainty reduction, it accounts for the informational gain on the minimizer expected from a new evaluation. The criterion is approximated using conditional simulations of the Gaussian process model behind Kriging, and then inserted into an algorithm similar in spirit to the Efficient Global Optimization (EGO) algorithm. An empirical comparison is carried out between our criterion and expected improvement, one of the reference criteria in the literature. Experimental results indicate major evaluation savings over EGO. Finally, the method, which we call IAGO (for Informational Approach to Global Optimization), is extended to robust optimization problems, where both the factors to be tuned and the function evaluations are corrupted by noise.

Entropy Minimization for Shadow Removal

Abstract: Recently, a method for removing shadows from colour images was developed (Finlayson et al. in IEEE Trans. Pattern Anal. Mach. Intell. 28:59---68, 2006) that relies upon finding a special direction in a 2D chromaticity feature space. This "invariant direction" is that for which particular colour features, when projected into 1D, produce a greyscale image which is approximately invariant to intensity and colour of scene illumination. Thus shadows, which are in essence a particular type of lighting, are greatly attenuated. The main approach to finding this special angle is a camera calibration: a colour target is imaged under many different lights, and the direction that best makes colour patch images equal across illuminants is the invariant direction. Here, we take a different approach. In this work, instead of a camera calibration we aim at finding the invariant direction from evidence in the colour image itself. Specifically, we recognize that producing a 1D projection in the correct invariant direction will result in a 1D distribution of pixel values that have smaller entropy than projecting in the wrong direction. The reason is that the correct projection results in a probability distribution spike, for pixels all the same except differing by the lighting that produced their observed RGB values and therefore lying along a line with orientation equal to the invariant direction. Hence we seek that projection which produces a type of intrinsic, independent of lighting reflectance-information only image by minimizing entropy, and from there go on to remove shadows as previously. To be able to develop an effective description of the entropy-minimization task, we go over to the quadratic entropy, rather than Shannon's definition. Replacing the observed pixels with a kernel density probability distribution, the quadratic entropy can be written as a very simple formulation, and can be evaluated using the efficient Fast Gauss Transform. The entropy, written in this embodiment, has the advantage that it is more insensitive to quantization than is the usual definition. The resulting algorithm is quite reliable, and the shadow removal step produces good shadow-free colour image results whenever strong shadow edges are present in the image. In most cases studied, entropy has a strong minimum for the invariant direction, revealing a new property of image formation.

Gas entropy in a representative sample of nearby X-ray galaxy clusters (REXCESS): relationship to gas mass fraction

Abstract: (Abridged) We examine the radial entropy distribution and its scaling using 31 nearby galaxy clusters from the Representative XMM-Newton Cluster Structure Survey (REXCESS). The entropy profiles are robustly measured at least out to R_1000 in all systems and out to R_500 in 13 systems. Compared to theoretical expectations, the observed distributions show a radial and mass-dependent excess entropy that is greater and extends to larger radii in lower mass systems. At R_500, the mass dependence and entropy excess are both negligible within the uncertainties. Mirroring this behaviour, the scaling of gas entropy is shallower than self-similar in the inner regions, but steepens with radius, becoming consistent with self-similar at R_500. The dispersion in scaled entropy in the inner regions is linked to the presence of cool cores and dynamical activity; at larger radii the dispersion decreases by a factor of two and the dichotomy between subsamples disappears. Parameterising the profiles with a power law plus constant model, there are two peaks in central entropy K_0; however, we cannot distinguish between a bimodal or a left-skewed distribution. The outer slopes are correlated with system temperature; their distribution is unimodal with a median value of 0.98. Renormalising the dimensionless entropy profiles by the gas mass fraction profile f_gas(< R), leads to a remarkable reduction in the scatter, implying that gas mass fraction variations with radius and mass are the cause of the observed entropy properties. We discuss a tentative scenario to explain the behaviour of the entropy and gas mass fraction in the REXCESS sample, in which extra heating and merger mixing maintains an elevated central entropy level in the majority of the population, and a smaller fraction of systems develops a cool core.

Entropy Inference and the James-Stein Estimator, with Application to Nonlinear Gene Association Networks

Abstract: We present a procedure for effective estimation of entropy and mutual information from small-sample data, and apply it to the problem of inferring high-dimensional gene association networks. Specifically, we develop a James-Stein-type shrinkage estimator, resulting in a procedure that is highly efficient statistically as well as computationally. Despite its simplicity, we show that it outperforms eight other entropy estimation procedures across a diverse range of sampling scenarios and data-generating models, even in cases of severe undersampling. We illustrate the approach by analyzing E. coli gene expression data and computing an entropy-based gene-association network from gene expression data. A computer program is available that implements the proposed shrinkage estimator.

Entropy measures for networks: toward an information theory of complex topologies.

Abstract: The quantification of the complexity of networks is, today, a fundamental problem in the physics of complex systems. A possible roadmap to solve the problem is via extending key concepts of information theory to networks. In this Rapid Communication we propose how to define the Shannon entropy of a network ensemble and how it relates to the Gibbs and von Neumann entropies of network ensembles. The quantities we introduce here will play a crucial role for the formulation of null models of networks through maximum-entropy arguments and will contribute to inference problems emerging in the field of complex networks.

Robust ISAR Range Alignment via Minimizing the Entropy of the Average Range Profile

Abstract: In this letter, a novel global approach to range alignment for inverse synthetic aperture radar (ISAR) image formation is presented. The algorithm is based on the minimization of the entropy of the average range profile (ARP), and the processing chain is capable of exploiting the efficiency of the fast Fourier transform. With respect to the existing global methods, the new one requires no exhaustive search operation and eliminates the necessity of the parametric model for the relative offset among the range profiles. The derivation of the algorithm indicates that the presented methodology is essentially an iterative solution to a set of simultaneous equations, and its robustness is also ensured by the iterative structure. Some alternative criteria, such as the maximum contrast of the ARP, can be introduced into the algorithm with a minor change in the entropy-based method. The convergence and robustness of the presented algorithm have been validated by experimental ISAR data.

Entropy of network ensembles.

Abstract: In this paper we generalize the concept of random networks to describe network ensembles with nontrivial features by a statistical mechanics approach. This framework is able to describe undirected and directed network ensembles as well as weighted network ensembles. These networks might have nontrivial community structure or, in the case of networks embedded in a given space, they might have a link probability with a nontrivial dependence on the distance between the nodes. These ensembles are characterized by their entropy, which evaluates the cardinality of networks in the ensemble. In particular, in this paper we define and evaluate the structural entropy, i.e., the entropy of the ensembles of undirected uncorrelated simple networks with given degree sequence. We stress the apparent paradox that scale-free degree distributions are characterized by having small structural entropy while they are so widely encountered in natural, social, and technological complex systems. We propose a solution to the paradox by proving that scale-free degree distributions are the most likely degree distribution with the corresponding value of the structural entropy. Finally, the general framework we present in this paper is able to describe microcanonical ensembles of networks as well as canonical or hidden-variable network ensembles with significant implications for the formulation of network-constructing algorithms.

Identification of a threshold value for the DEMATEL method using the maximum mean de-entropy algorithm to find critical services provided by a semiconductor intellectual property mall

Abstract: To deal with complex problems, structuring them through graphical representations and analyzing causal influences can aid in illuminating complex issues, systems, or concepts. The DEMATEL method is a methodology which can confirm interdependence among variables and aid in the development of a chart to reflect interrelationships between variables, and can be used for researching and solving complicated and intertwined problem groups. The end product of the DEMATEL process is a visual representation-the impact-relations map-by which respondents organize their own actions in the world. In order to obtain a suitable impact-relations map, an appropriate threshold value is needed to obtain adequate information for further analysis and decision-making. In the existing literature, the threshold value has been determined through interviews with respondents or judged by the researcher. In most cases, it is hard and time-consuming to aggregate the respondents and make a consistent decision. In addition, in order to avoid subjective judgments, a theoretical method to select the threshold value is necessary. In this paper, we propose a method based on the entropy approach, the maximum mean de-entropy algorithm, to achieve this purpose. Using a real case to find the interrelationships between the services of a Semiconductor Intellectual Property Mall as an example, we will compare the results obtained from the respondents and from our method, and show that the impact-relations maps from these two methods could be the same.

Entropy of interval-valued fuzzy sets based on distance and its relationship with similarity measure

Abstract: This article proposes a new axiomatic definition of entropy of interval-valued fuzzy sets (IVFSs) and discusses its relation with similarity measure. First, we propose an axiomatic definition of entropy for IVFS based on distance which is consistent with the axiomatic definition of entropy of a fuzzy set introduced by De Luca, Termini and Liu. Next, some formulae are derived to calculate this kind of entropy. Furthermore we investigate the relationship between entropy and similarity measure of IVFSs and prove that similarity measure can be transformed by entropy. Finally, a numerical example is given to show that the proposed entropy measures are more reasonable and reliable for representing the degree of fuzziness of an IVFS.

Generalized information and entropy measures in physics

Abstract: The formalism of statistical mechanics can be generalized by starting from more general measures of information than the Shannon entropy and maximizing those subject to suitable constraints. We discuss some of the most important examples of information measures that are useful for the description of complex systems. Examples treated are the Renyi entropy, Tsallis entropy, Abe entropy, Kaniadakis entropy, Sharma-Mittal entropies, and a few more. Important concepts such as the axiomatic foundations, composability and Lesche stability of information measures are briefly discussed. Potential applications in physics include complex systems with long-range interactions and metastable states, scattering processes in particle physics, hydrodynamic turbulence, defect turbulence, optical lattices, and quite generally driven nonequilibrium systems with fluctuations of temperature.

Shannon and entanglement entropies of one- and two-dimensional critical wave functions

Abstract: We study the Shannon entropy of the probability distribution resulting from the ground-state wave function of a one-dimensional quantum model. This entropy is related to the entanglement entropy of a Rokhsar-Kivelson-type wave function built from the corresponding two-dimensional classical model. In both critical and massive cases, we observe that it is composed of an extensive part proportional to the length of the system and a subleading universal constant ${S}_{0}$. In $c=1$ critical systems (Tomonaga-Luttinger liquids), we find that ${S}_{0}$ is a simple function of the boson compactification radius. This finding is based on a field-theoretical analysis of the Dyson-Gaudin gas related to dimer and Calogero-Sutherland models. We also performed numerical demonstrations in the dimer models and the spin-1/2 $XXZ$ chain. In a massive (crystal) phase, ${S}_{0}$ is related to the ground-state degeneracy. We also examine this entropy in the Ising chain in a transverse field as an example showing a $c=1/2$ critical point.

First- and Second-Order Expectation Semirings with Applications to Minimum-Risk Training on Translation Forests

Abstract: Many statistical translation models can be regarded as weighted logical deduction. Under this paradigm, we use weights from the expectation semiring (Eisner, 2002), to compute first-order statistics (e.g., the expected hypothesis length or feature counts) over packed forests of translations (lattices or hypergraphs). We then introduce a novel second-order expectation semiring, which computes second-order statistics (e.g., the variance of the hypothesis length or the gradient of entropy). This second-order semiring is essential for many interesting training paradigms such as minimum risk, deterministic annealing, active learning, and semi-supervised learning, where gradient descent optimization requires computing the gradient of entropy or risk. We use these semirings in an open-source machine translation toolkit, Joshua, enabling minimum-risk training for a benefit of up to 1.0 bleu point.

Asymptotics of the Fast Diffusion Equation via Entropy Estimates

Abstract: We consider non-negative solutions of the fast diffusion equation ut = Δum with m ∈ (0, 1) in the Euclidean space \({{\mathbb R}^d}\), d ≧ 3, and study the asymptotic behavior of a natural class of solutions in the limit corresponding to t → ∞ for m ≧ mc = (d − 2)/d, or as t approaches the extinction time when m < mc. For a class of initial data, we prove that the solution converges with a polynomial rate to a self-similar solution, for t large enough if m ≧ mc, or close enough to the extinction time if m < mc. Such results are new in the range m ≦ mc where previous approaches fail. In the range mc < m < 1, we improve on known results.

Approximate entropy for all signals

Abstract: In this study, computer simulation examples consisting of various signals with different complexity were compared. It was found that neither approximate entropy (ApEn) nor sample entropy (SampEn) methods was accurate in measuring signals' complexity when the recommended values (e.g., m = 2 and r = 0.1-0.2 times the standard deviation of the signal) were strictly adhered to. However, when we selected the maximum ApEn value as determined by considering many different r values, we were able to correctly discern a signal's complexity for both synthetic and experimental data. However, this requires that many different choices of r values need to be considered. This is a very cumbersome and time-consuming process. Thus, the primary goal of the present work is to illustrate our recently developed method that can automatically select the appropriate tolerance threshold value r, which corresponds to the maximum ApEn value, without resorting to the calculation of ApEn for each of the threshold values selected in the range of zero and one times the standard deviation.

Estimation of Mutual Information: A Survey

Abstract: A common problem found in statistics, signal processing, data analysis and image processing research is the estimation of mutual information, which tends to be difficult. The aim of this survey is threefold: an introduction for those new to the field, an overview for those working in the field and a reference for those searching for literature on different estimation methods. In this paper comparison studies on mutual information estimation is considered. The paper starts with a description of entropy and mutual information and it closes with a discussion on the performance of different estimation methods and some future challenges.

On the Capacity of the Discrete-Time Poisson Channel

Abstract: The large-inputs asymptotic capacity of a peak-power and average-power limited discrete-time Poisson channel is derived using a new firm (nonasymptotic) lower bound and an asymptotic upper bound. The upper bound is based on the dual expression for channel capacity and the notion of capacity-achieving input distributions that escape to infinity. The lower bound is based on a lower bound on the entropy of a conditionally Poisson random variable in terms of the differential entropy of its conditional mean.

Dense saliency-based spatiotemporal feature points for action recognition

Abstract: Several spatiotemporal feature point detectors have been used in video analysis for action recognition. Feature points are detected using a number of measures, namely saliency, cornerness, periodicity, motion activity etc. Each of these measures is usually intensity-based and provides a different trade-off between density and informativeness. In this paper, we use saliency for feature point detection in videos and incorporate color and motion apart from intensity. Our method uses a multi-scale volumetric representation of the video and involves spatiotemporal operations at the voxel level. Saliency is computed by a global minimization process constrained by pure volumetric constraints, each of them being related to an informative visual aspect, namely spatial proximity, scale and feature similarity (intensity, color, motion). Points are selected as the extrema of the saliency response and prove to balance well between density and informativeness. We provide an intuitive view of the detected points and visual comparisons against state-of-the-art space-time detectors. Our detector outperforms them on the KTH dataset using nearest-neighbor classifiers and ranks among the top using different classification frameworks. Statistics and comparisons are also performed on the more difficult Hollywood human actions (HOHA) dataset increasing the performance compared to current published results.

Invariance Entropy for Control Systems

Abstract: For continuous time control systems, this paper introduces invariance entropy as a measure for the amount of information necessary to achieve invariance of weakly invariant compact subsets of the state space. Upper and lower bounds are derived; in particular, finiteness is proven. For linear control systems with compact control range, the invariance entropy is given by the sum of the real parts of the unstable eigenvalues of the uncontrolled system. A characterization via covers and corresponding feedbacks is provided.

Scale-Rotation Invariant Pattern Entropy for Keypoint-Based Near-Duplicate Detection

Abstract: Near-duplicate (ND) detection appears as a timely issue recently, being regarded as a powerful tool for various emerging applications. In the Web 2.0 environment particularly, the identification of near-duplicates enables the tasks such as copyright enforcement, news topic tracking, image and video search. In this paper, we describe an algorithm, namely Scale-Rotation invariant Pattern Entropy (SR-PE), for the detection of near-duplicates in large-scale video corpus. SR-PE is a novel pattern evaluation technique capable of measuring the spatial regularity of matching patterns formed by local keypoints. More importantly, the coherency of patterns and the perception of visual similarity, under the scenario that there could be multiple ND regions undergone arbitrary transformations, respectively, are carefully addressed through entropy measure. To demonstrate our work in large-scale dataset, a practical framework composed of three components: bag-of-words representation, local keypoint matching and SR-PE evaluation, is also proposed for the rapid detection of near-duplicates.

The work value of information

Abstract: We present quantitative relations between work and information that are valid both for finite sized and internally correlated systems as well in the thermodynamical limit. We suggest work extraction should be viewed as a game where the amount of work an agent can extract depends on how well it can guess the micro-state of the system. In general it depends both on the agent's knowledge and risk-tolerance, because the agent can bet on facts that are not certain and thereby risk failure of the work extraction. We derive strikingly simple expressions for the extractable work in the extreme cases of effectively zero- and arbitrary risk tolerance respectively, thereby enveloping all cases. Our derivation makes a connection between heat engines and the smooth entropy approach. The latter has recently extended Shannon theory to encompass finite sized and internally correlated bit strings, and our analysis points the way to an analogous extension of statistical mechanics.

Nonextensive Information Theoretic Kernels on Measures

Abstract: Positive definite kernels on probability measures have been recently applied to classification problems involving text, images, and other types of structured data. Some of these kernels are related to classic information theoretic quantities, such as (Shannon's) mutual information and the Jensen-Shannon (JS) divergence. Meanwhile, there have been recent advances in nonextensive generalizations of Shannon's information theory. This paper bridges these two trends by introducing nonextensive information theoretic kernels on probability measures, based on new JS-type divergences. These new divergences result from extending the the two building blocks of the classical JS divergence: convexity and Shannon's entropy. The notion of convexity is extended to the wider concept of q-convexity, for which we prove a Jensen q-inequality. Based on this inequality, we introduce Jensen-Tsallis (JT) q-differences, a nonextensive generalization of the JS divergence, and define a k-th order JT q-difference between stochastic processes. We then define a new family of nonextensive mutual information kernels, which allow weights to be assigned to their arguments, and which includes the Boolean, JS, and linear kernels as particular cases. Nonextensive string kernels are also defined that generalize the p-spectrum kernel. We illustrate the performance of these kernels on text categorization tasks, in which documents are modeled both as bags of words and as sequences of characters.

Natural selection maximizes Fisher information.

Abstract: In biology, information flows from the environment to the genome by the process of natural selection. However, it has not been clear precisely what sort of information metric properly describes natural selection. Here, I show that Fisher information arises as the intrinsic metric of natural selection and evolutionary dynamics. Maximizing the amount of Fisher information about the environment captured by the population leads to Fisher's fundamental theorem of natural selection, the most profound statement about how natural selection influences evolutionary dynamics. I also show a relation between Fisher information and Shannon information (entropy) that may help to unify the correspondence between information and dynamics. Finally, I discuss possible connections between the fundamental role of Fisher information in statistics, biology and other fields of science.

Randomized shortest-path problems: Two related models

Abstract: This letter addresses the problem of designing the transition probabilities of a finite Markov chain (the policy) in order to minimize the expected cost for reaching a destination node from a source node while maintaining a fixed level of entropy spread throughout the network (the exploration). It is motivated by the following scenario. Suppose you have to route agents through a network in some optimal way, for instance, by minimizing the total travel cost---nothing particular up to now---you could use a standard shortest-path algorithm. Suppose, however, that you want to avoid pure deterministic routing policies in order, for instance, to allow some continual exploration of the network, avoid congestion, or avoid complete predictability of your routing strategy. In other words, you want to introduce some randomness or unpredictability in the routing policy (i.e., the routing policy is randomized). This problem, which will be called the randomized shortest-path problem (RSP), is investigated in this work. The global level of randomness of the routing policy is quantified by the expected Shannon entropy spread throughout the network and is provided a priori by the designer. Then, necessary conditions to compute the optimal randomized policy---minimizing the expected routing cost---are derived. Iterating these necessary conditions, reminiscent of Bellman's value iteration equations, allows computing an optimal policy, that is, a set of transition probabilities in each node. Interestingly and surprisingly enough, this first model, while formulated in a totally different framework, is equivalent to Akamatsu's model (1996), appearing in transportation science, for a special choice of the entropy constraint. We therefore revisit Akamatsu's model by recasting it into a sum-over-paths statistical physics formalism allowing easy derivation of all the quantities of interest in an elegant, unified way. For instance, it is shown that the unique optimal policy can be obtained by solving a simple linear system of equations. This second model is therefore more convincing because of its computational efficiency and soundness. Finally, simulation results obtained on simple, illustrative examples show that the models behave as expected.

Shannon's entropy for combining the efficiency results of different DEA models: Method and application

Abstract: Data envelopment analysis (DEA) was initially proposed by Charnes, Cooper, and Rhodes for evaluating the decision making units (DMUs), via calculating their relative efficiencies. To do this, there exist many different DEA models and different DEA viewpoints. Hence, selecting the best model (viewpoint) or the way of combining different models (viewpoints) for ranking DMUs is a main question in applied DEA. This paper provides a methodology, based upon Shannon's entropy formula, for combining the efficiency results of different DEA models (viewpoints). An application of the provided method in educational departments of a university in Iran is also illustrated.