scispace - formally typeset
Search or ask a question

Showing papers on "Entropy (information theory) published in 2010"


Book
22 Oct 2010
TL;DR: Students, practitioners and researchers interested in statistical signal processing, computational intelligence, and machine learning will find the theory to understand the basics, the algorithms to implement applications, and exciting but still unexplored leads that will provide fertile ground for future research in this book.
Abstract: This book presents the first cohesive treatment of Information Theoretic Learning (ITL) algorithms to adapt linear or nonlinear learning machines both in supervised or unsupervised paradigms. ITL is a framework where the conventional concepts of second order statistics (covariance, L2 distances, correlation functions) are substituted by scalars and functions with information theoretic underpinnings, respectively entropy, mutual information and correntropy. ITL quantifies the stochastic structure of the data beyond second order statistics for improved performance without using full-blown Bayesian approaches that require a much larger computational cost. This is possible because of a non-parametric estimator of Renyis quadratic entropy that is only a function of pairwise differences between samples. The book compares the performance of ITL algorithms with the second order counterparts in many engineering and machine learning applications. Students, practitioners and researchers interested in statistical signal processing, computational intelligence, and machine learning will find in this book the theory to understand the basics, the algorithms to implement applications, and exciting but still unexplored leads that will provide fertile ground for future research.

802 citations


DissertationDOI
01 Jan 2010
TL;DR: The principle of maximum causal entropy is introduced, a general technique for applying information theory to decision-theoretic, game-the theoretical, and control settings where relevant information is sequentially revealed over time.
Abstract: Predicting human behavior from a small amount of training examples is a challenging machine learning problem. In this thesis, we introduce the principle of maximum causal entropy, a general technique for applying information theory to decision-theoretic, game-theoretic, and control settings where relevant information is sequentially revealed over time. This approach guarantees decision-theoretic performance by matching purposeful measures of behavior (Abbeel & Ng, 2004), and/or enforces game-theoretic rationality constraints (Aumann, 1974), while otherwise being as uncertain as possible, which minimizes worst-case predictive log-loss (Grunwald & Dawid, 2003). We derive probabilistic models for decision, control, and multi-player game settings using this approach. We then develop corresponding algorithms for efficient inference that include relaxations of the Bellman equation (Bellman, 1957), and simple learning algorithms based on convex optimization. We apply the models and algorithms to a number of behavior prediction tasks. Specifically, we present empirical evaluations of the approach in the domains of vehicle route preference modeling using over 100,000 miles of collected taxi driving data, pedestrian motion modeling from weeks of indoor movement data, and robust prediction of game play in stochastic multi-player games.

519 citations


Journal ArticleDOI
TL;DR: A multicriteria fuzzy decision-making method based on weighted correlation coefficients using entropy weights is proposed under intuitionistic fuzzy environment for some situations where the information about criteria weights for alternatives is completely unknown.

354 citations


Journal ArticleDOI
TL;DR: A multicriteria fuzzy decision-making method based on weighted correlation coefficients using entropy weights is proposed under interval-valued intuitionistic fuzzy environment for the some situations where the information about criteria weights for alternatives is completely unknown.

296 citations


Journal ArticleDOI
TL;DR: In this paper, a recently discovered duality relation between (nonsmooth) min- and max-entropies is extended to the smooth case and it is shown that the smooth min-entropy of a system A conditioned on a system B equals the negative of the smooth max-Entropy of Acondition on a purifying system C.
Abstract: In classical and quantum information theory, operational quantities such as the amount of randomness that can be extracted from a given source or the amount of space needed to store given data are normally characterized by one of two entropy measures, called smooth min-entropy and smooth max-entropy, respectively. While both entropies are equal to the von Neumann entropy in certain special cases (e.g., asymptotically, for many independent repetitions of the given data), their values can differ arbitrarily in the general case. In this paper, a recently discovered duality relation between (nonsmooth) min- and max-entropies is extended to the smooth case. More precisely, it is shown that the smooth min-entropy of a system A conditioned on a system B equals the negative of the smooth max-entropy of A conditioned on a purifying system C. This result immediately implies that certain operational quantities (such as the amount of compression and the amount of randomness that can be extracted from given data) are related. We explain how such relations have applications in cryptographic security proofs.

292 citations


Journal ArticleDOI
TL;DR: Experimental results indicate that the proposed approach cannot only reliably discriminate among different fault categories, but identify the level of fault severity, so the approach has possibility for bearing incipient fault diagnosis.
Abstract: A bearing fault diagnosis method has been proposed based on multi-scale entropy (MSE) and adaptive neuro-fuzzy inference system (ANFIS), in order to tackle the nonlinearity existing in bearing vibration as well as the uncertainty inherent in the diagnostic information. MSE refers to the calculation of entropies (e.g. appropriate entropy, sample entropy) across a sequence of scales, which takes into account not only the dynamic nonlinearity but also the interaction and coupling effects between mechanical components, thus providing much more information regarding machinery operating condition in comparison with traditional single scale-based entropy. ANFIS can benefit from the decision-making under uncertainty enabled by fuzzy logic as well as from learning and adaptation that neural networks provide. In this study, MSE and ANFIS are employed for feature extraction and fault recognition, respectively. Experiments were conducted on electrical motor bearings with three different fault categories and several levels of fault severity. The experimental results indicate that the proposed approach cannot only reliably discriminate among different fault categories, but identify the level of fault severity. Thus, the proposed approach has possibility for bearing incipient fault diagnosis.

291 citations


Journal ArticleDOI
TL;DR: The generalized beta-generated (GBG) distributions as discussed by the authors are the most tractable class of distributions and have tractable properties: moments, generating function, quantiles, deviations and reliability.
Abstract: This article introduces generalized beta-generated (GBG) distributions. Sub-models include all classical beta-generated, Kumaraswamy-generated and exponentiated distributions. They are maximum entropy distributions under three intuitive conditions, which show that the classical beta generator skewness parameters only control tail entropy and an additional shape parameter is needed to add entropy to the centre of the parent distribution. This parameter controls skewness without necessarily differentiating tail weights. The GBG class also have tractable properties: we present various expansions for moments, generating function, quantiles, deviations and reliability. The model parameters are estimated by maximum likelihood and the usefulness of the new class is illustrated by means of some real data sets.

290 citations


Journal ArticleDOI
05 Jan 2010-Entropy
TL;DR: The goal of this paper is the extension of the Shannon entropy method for the imprecise data, especially interval and fuzzy data cases.
Abstract: Finding the appropriate weight for each criterion is one of the main points in Multi Attribute Decision Making (MADM) problems. Shannon’s entropy method is one of the various methods for finding weights discussed in the literature. However, in many real life problems, the data for the decision making processes cannot be measured precisely and there may be some other types of data, for instance, interval data and fuzzy data. The goal of this paper is the extension of the Shannon entropy method for the imprecise data, especially interval and fuzzy data cases.

273 citations


Journal ArticleDOI
TL;DR: A new objective weighting method that employs intuitionistic fuzzy (IF) entropy measures to solve multiple-attribute decision-making problems in the context of intuitionistically fuzzy sets based on the credibility of the input data is proposed.

263 citations


Proceedings ArticleDOI
21 Jun 2010
TL;DR: This work presents the principle of maximum causal entropy—an approach based on causally conditioned probabilities that can appropriately model the availability and influence of sequentially revealed side information.
Abstract: The principle of maximum entropy provides a powerful framework for statistical models of joint, conditional, and marginal distributions. However, there are many important distributions with elements of interaction and feedback where its applicability has not been established. This work presents the principle of maximum causal entropy—an approach based on causally conditioned probabilities that can appropriately model the availability and influence of sequentially revealed side information. Using this principle, we derive models for sequential data with revealed information, interaction, and feedback, and demonstrate their applicability for statistically framing inverse optimal control and decision prediction tasks.

262 citations


Journal ArticleDOI
TL;DR: A critical synthesis of diverging implementations of the Rao quadratic entropy index in the recent literature is presented and a new extension of the index for measuring b-diversity is proposed, allowing meaningful comparisons of partitioning across species, phylogenetic and functional diversities within the same mathematical framework.
Abstract: A methodology for partitioning of biodiversity into a, b and g components has long been debated, resulting in different mathematical frameworks. Recently, use of the Rao quadratic entropy index has been advocated since it allows comparison of various facets of diversity (e.g. taxonomic, phylogenetic and functional) within the same mathematical framework. However, if not well implemented, the Rao index can easily yield biologically meaningless results and lead into a mathematical labyrinth. As a practical guideline for ecologists, we present a critical synthesis of diverging implementations of the index in the recent literature and a new extension of the index for measuring b-diversity. First, we detail correct computation of the index that needs to be applied in order not to obtain negative b-diversity values, which are ecologically unacceptable, and elucidate the main approaches to calculate the Rao quadratic entropy at different spatial scales. Then, we emphasize that, similar to other entropy measures, the Rao index often produces lower-than-expected b-diversity values. To solve this, we extend a correction based on equivalent numbers, as proposed by Jost (2007), to the Rao index. We further show that this correction can be applied to additive partitioning of diversity and not only its multiplicative form. These developments around the Rao index open up an exciting avenue to develop an estimator of turnover diversity across different environmental and temporal scales, allowing meaningful comparisons of partitioning across species, phylogenetic and functional diversities within the same mathematical framework. We also propose a set of R functions, based on existing developments, which perform different key computations to apply this framework in biodiversity science.

Journal ArticleDOI
TL;DR: Almost lossless analog compression for analog memoryless sources in an information-theoretic framework, in which the compressor or decompressor is constrained by various regularity conditions, in particular linearity of the compressor and Lipschitz continuity of the decompressor.
Abstract: In Shannon theory, lossless source coding deals with the optimal compression of discrete sources Compressed sensing is a lossless coding strategy for analog sources by means of multiplication by real-valued matrices In this paper we study almost lossless analog compression for analog memoryless sources in an information-theoretic framework, in which the compressor or decompressor is constrained by various regularity conditions, in particular linearity of the compressor and Lipschitz continuity of the decompressor The fundamental limit is shown to the information dimension proposed by Renyi in 1959

Journal ArticleDOI
01 Feb 2010
TL;DR: The experiment results show that, compared with three other multi-objective optimization evolutionary algorithms, the proposed MOSADE is able to find better spread of solutions with better convergence to the Pareto front and preserve the diversity of Pare to optimal solutions more efficiently.
Abstract: A self-adaptive differential evolution algorithm incorporate Pareto dominance to solve multi-objective optimization problems is presented. The proposed approach adopts an external elitist archive to retain non-dominated solutions found during the evolutionary process. In order to preserve the diversity of Pareto optimality, a crowding entropy diversity measure tactic is proposed. The crowding entropy strategy is able to measure the crowding degree of the solutions more accurately. The experiments were performed using eighteen benchmark test functions. The experiment results show that, compared with three other multi-objective optimization evolutionary algorithms, the proposed MOSADE is able to find better spread of solutions with better convergence to the Pareto front and preserve the diversity of Pareto optimal solutions more efficiently.

Journal ArticleDOI
TL;DR: The results demonstrate the capability of the proposed MODE approach to generate well-distributed Pareto optimal non-dominated solutions of multi-objective EED problem and confirms its potential for solving other power systems multi- objective optimization problems.

Journal ArticleDOI
01 Feb 2010-Oikos
TL;DR: It is shown that in many cases the fitness value of a developmental cue, when measured this way, is exactly equal to the reduction in uncertainty about the environment, as described by the mutual information.
Abstract: Biologists measure information in different ways. Neurobiologists and researchers in bioinformatics often measure information using informationtheoretic measures such as Shannon’s entropy or mutual information. Behavioral biologists and evolutionary ecologists more commonly use decisiontheoretic measures, such the value of information, which assess the worth of information to a decision maker. Here we show that these two kinds of measures are intimately related in the context of biological evolution. We present a simple model of evolution in an uncertain environment, and calculate the increase in Darwinian fitness that is made possible by information about the environmental state. This fitness increase — the fitness value of information — is a composite of both Shannon’s mutual information and the decision-theoretic value of information. Furthermore, we show that in certain cases the fitness value of responding to a cue is exactly equal to the mutual information between the cue and the environment. In general the Shannon entropy of the environment, which seemingly fails to take anything about organismal fitness into account, nonetheless imposes an upper bound on the fitness value of information.

Proceedings ArticleDOI
13 Jun 2010
TL;DR: A new visual saliency measure called Site Entropy Rate (SER) is proposed to compute the average information transmitted from a node to all the others during the random walk on the graphs/network to explain the center-surround mechanism from computation aspect.
Abstract: In this paper, we propose a new computational model for visual saliency derived from the information maximization principle. The model is inspired by a few well acknowledged biological facts. To compute the saliency spots of an image, the model first extracts a number of sub-band feature maps using learned sparse codes. It adopts a fully-connected graph representation for each feature map, and runs random walks on the graphs to simulate the signal/information transmission among the interconnected neurons. We propose a new visual saliency measure called Site Entropy Rate (SER) to compute the average information transmitted from a node (neuron) to all the others during the random walk on the graphs/network. This saliency definition also explains the center-surround mechanism from computation aspect. We further extend our model to spatial-temporal domain so as to detect salient spots in videos. To evaluate the proposed model, we do extensive experiments on psychological stimuli, two well known image data sets, as well as a public video dataset. The experiments demonstrate encouraging results that the proposed model achieves the state-of-the-art performance of saliency detection in both still images and videos.

Journal ArticleDOI
TL;DR: A video coding architecture is described that is based on nested and pre-configurable quadtree structures for flexible and signal-adaptive picture partitioning that was ranked among the five best performing proposals, both in terms of subjective and objective quality.
Abstract: -A video coding architecture is described that is based on nested and pre-configurable quadtree structures for flexible and signal-adaptive picture partitioning. The primary goal of this partitioning concept is to provide a high degree of adaptability for both temporal and spatial prediction as well as for the purpose of space-frequency representation of prediction residuals. At the same time, a leaf merging mechanism is included in order to prevent excessive partitioning of a picture into prediction blocks and to reduce the amount of bits for signaling the prediction signal. For fractional-sample motion-compensated prediction, a fixed-point implementation of the maximal-order minimum-support algorithm is presented that uses a combination of infinite impulse response and FIR filtering. Entropy coding utilizes the concept of probability interval partitioning entropy codes that offers new ways for parallelization and enhanced throughput. The presented video coding scheme was submitted to a joint call for proposals of ITU-T Visual Coding Experts Group and ISO/IEC Moving Picture Experts Group and was ranked among the five best performing proposals, both in terms of subjective and objective quality.

Journal ArticleDOI
TL;DR: It was found that the optimal set of summary statistics was highly dataset specific, suggesting that more generally there may be no globally-optimal choice, which argues for a new selection for each dataset even if the model and target of inference are unchanged.
Abstract: How best to summarize large and complex datasets is a problem that arises in many areas of science. We approach it from the point of view of seeking data summaries that minimize the average squared error of the posterior distribution for a parameter of interest under approximate Bayesian computation (ABC). In ABC, simulation under the model replaces computation of the likelihood, which is convenient for many complex models. Simulated and observed datasets are usually compared using summary statistics, typically in practice chosen on the basis of the investigator's intuition and established practice in the field. We propose two algorithms for automated choice of efficient data summaries. Firstly, we motivate minimisation of the estimated entropy of the posterior approximation as a heuristic for the selection of summary statistics. Secondly, we propose a two-stage procedure: the minimum-entropy algorithm is used to identify simulated datasets close to that observed, and these are each successively regarded as observed datasets for which the mean root integrated squared error of the ABC posterior approximation is minimized over sets of summary statistics. In a simulation study, we both singly and jointly inferred the scaled mutation and recombination parameters from a population sample of DNA sequences. The computationally-fast minimum entropy algorithm showed a modest improvement over existing methods while our two-stage procedure showed substantial and highly-significant further improvement for both univariate and bivariate inferences. We found that the optimal set of summary statistics was highly dataset specific, suggesting that more generally there may be no globally-optimal choice, which argues for a new selection for each dataset even if the model and target of inference are unchanged.

Proceedings ArticleDOI
14 Mar 2010
TL;DR: It is shown that when using the log-sum-exp function to approximate the optimal value of any combinatorial problem, the solution can be interpreted as the stationary probability distribution of a class of time-reversible Markov chains.
Abstract: Many important network design problems can be formulated as a combinatorial optimization problem. A large number of such problems, however, cannot readily be tackled by distributed algorithms. The Markov approximation framework studied in this paper is a general technique for synthesizing distributed algorithms. We show that when using the log-sum-exp function to approximate the optimal value of any combinatorial problem, we end up with a solution that can be interpreted as the stationary probability distribution of a class of time- reversible Markov chains. Certain carefully designed Markov chains among this class yield distributed algorithms that solve the log-sum-exp approximated combinatorial network optimization problem. By three case studies, we illustrate that Markov approximation technique not only can provide fresh perspective to existing distributed solutions, but also can help us generate new distributed algorithms in various domains with provable performance. We believe the Markov approximation framework will find applications in many network optimization problems, and this paper serves as a call for participation.

Journal ArticleDOI
TL;DR: In this paper, the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies is studied and a duality between the upper and lower bounds for joint entropy is developed.
Abstract: Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as inequalities of Han, Fujishige, and Shearer. A duality between the upper and lower bounds for joint entropy is developed. All of these results are shown to be special cases of general, new results for submodular functions-thus, the inequalities presented constitute a richly structured class of Shannon-type inequalities. The new inequalities are applied to obtain new results in combinatorics, such as bounds on the number of independent sets in an arbitrary graph and the number of zero-error source-channel codes, as well as determinantal inequalities in matrix theory. A general inequality for relative entropies is also developed. Finally, revealing connections of the results to literature in economics, computer science, and physics are explored.

Journal ArticleDOI
TL;DR: An information-theoretic framework for flow visualization with a special focus on streamline generation is presented, and it is shown that the framework can effectively visualize 2D and 3D flow data.
Abstract: The process of visualization can be seen as a visual communication channel where the input to the channel is the raw data, and the output is the result of a visualization algorithm. From this point of view, we can evaluate the effectiveness of visualization by measuring how much information in the original data is being communicated through the visual communication channel. In this paper, we present an information-theoretic framework for flow visualization with a special focus on streamline generation. In our framework, a vector field is modeled as a distribution of directions from which Shannon's entropy is used to measure the information content in the field. The effectiveness of the streamlines displayed in visualization can be measured by first constructing a new distribution of vectors derived from the existing streamlines, and then comparing this distribution with that of the original data set using the conditional entropy. The conditional entropy between these two distributions indicates how much information in the original data remains hidden after the selected streamlines are displayed. The quality of the visualization can be improved by progressively introducing new streamlines until the conditional entropy converges to a small value. We describe the key components of our framework with detailed analysis, and show that the framework can effectively visualize 2D and 3D flow data.

Journal ArticleDOI
TL;DR: A new information entropy measure of interval-valued intuitionistic fuzzy set (IvIFS) is proposed by using membership interval and non-membership interval of IvIFS, which complies with the extended form of Deluca-Termini axioms for fuzzy entropy.

Posted Content
TL;DR: For the first time, it is proved the almost sure consistency of these estimators and upper bounds on their rates of convergence, the latter of which under the assumption that the density underlying the sample is Lipschitz continuous.
Abstract: We present simple and computationally efficient nonparametric estimators of Renyi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over $\R^d$. The estimators are calculated as the sum of $p$-th powers of the Euclidean lengths of the edges of the `generalized nearest-neighbor' graph of the sample and the empirical copula of the sample respectively. For the first time, we prove the almost sure consistency of these estimators and upper bounds on their rates of convergence, the latter of which under the assumption that the density underlying the sample is Lipschitz continuous. Experiments demonstrate their usefulness in independent subspace analysis.

Book
15 Aug 2010
TL;DR: The author revealed that Gaussian Measures on Function Spaces and the Scale -Rotation- and Translation-Invariant Gaussian Distribution Mode lII: Images Made Up of Independent Objects is a stable property of the Discrete Gaussian Pyramid.
Abstract: Preface Notation What Is Pattern Theory? The Manifesto of Pattern Theory The Basic Types of Patterns Bayesian Probability Theory: Pattern Analysis and Pattern Synthesis English Text and Markov Chains Basics I: Entropy and Information Measuring the n-gram Approximation with Entropy Markov Chains and the n-gram Models Words Word Boundaries via Dynamic Programming and Maximum Likelihood Machine Translation via Bayes' Theorem Exercises Music and Piece wise Gaussian Models Basics III: Gaussian Distributions Basics IV: Fourier Analysis Gaussian Models for Single Musical Notes Discontinuities in One-Dimensional Signals The Geometric Model for Notes via Poisson Processes Related Models Exercises Character Recognition and Syntactic Grouping Finding Salient Contours in Images Stochastic Models of Contours The Medial Axis for Planar Shapes Gestalt Laws and Grouping Principles Grammatical Formalisms Exercises Contents Image Texture, Segmentation and Gibbs Models Basics IX: Gibbs Fields (u + v)-Models for Image Segmentation Sampling Gibbs Fields Deterministic Algorithms to Approximate the Mode of a Gibbs Field Texture Models Synthesizing Texture via Exponential Models Texture Segmentation Exercises Faces and Flexible Templates Modeling Lighting Variations Modeling Geometric Variations by Elasticity Basics XI: Manifolds, Lie Groups, and Lie Algebras Modeling Geometric Variations by Metrics on Diff Comparing Elastic and Riemannian Energies Empirical Data on Deformations of Faces The Full Face Model Appendix: Geodesics in Diff and Landmark Space Exercises Natural Scenes and their Multiscale Analysis High Kurtosis in the Image Domain Scale Invariance in the Discrete and Continuous Setting The Continuous and Discrete Gaussian Pyramids Wavelets and the "Local" Structure of Images Distributions Are Needed Basics XIII: Gaussian Measures on Function Spaces The Scale -Rotation- and Translation-Invariant Gaussian Distribution Mode lII: Images Made Up of Independent Objects Further Models Appendix: A Stability Property of the Discrete Gaussian Pyramid Exercises Bibliography Index

Journal ArticleDOI
TL;DR: In this article, Rao et al. defined the cumulative residual entropy and the dynamic cumulative past entropy as new measures of uncertainty in reliability and survival studies, and studied the relationship between these measures and the mean residual life function.

Journal ArticleDOI
TL;DR: Although the two-dimensional piecewise nonlinear chaotic maps presented in this paper aims at image encryption, it is not just limited to this area and can be widely applied in other information security fields.

Journal ArticleDOI
TL;DR: In this article, the authors introduce an approach to the measurement of locational phenomena in a spatial hierarchy using entropy statistics, which is suitable for the study of spatial aggregation and information in problems involving equal-area zoning.
Abstract: This paper introduces an approach to the measurement of locational phenomena in a spatial hierarchy using entropy statistics. A number of such statistics suitable for the study of spatial aggregation are derived, and each of these statistics is decomposed at different levels of the spatial hierarchy using principles of decomposition first applied by Theil. These decomposition statistics are compared with the variance analysis applied by Moellering and Tobler and with the spatial entropy measure suggested by Curry. The use of these statistics is then illustrated by data from the Reading subregion and New York City, and the paper is concluded with an analysis of a possible role for entropy and information in problems involving equal-area zoning.

Proceedings ArticleDOI
19 Apr 2010
TL;DR: EbAT - Entropy-based Anomaly Testing - offering novel methods that detect anomalies by analyzing for arbitrary metrics their distributions rather than individual metric thresholds is proposed.
Abstract: The online detection of anomalies is a vital element of operations in data centers and in utility clouds like Amazon EC2. Given ever-increasing data center sizes coupled with the complexities of systems software, applications, and workload patterns, such anomaly detection must operate automatically, at runtime, and without the need for prior knowledge about normal or anomalous behaviors. Further, detection should function for different levels of abstraction like hardware and software, and for the multiple metrics used in cloud computing systems. This paper proposes EbAT - Entropy-based Anomaly Testing - offering novel methods that detect anomalies by analyzing for arbitrary metrics their distributions rather than individual metric thresholds. Entropy is used as a measurement that captures the degree of dispersal or concentration of such distributions, aggregating raw metric data across the cloud stack to form entropy time series. For scalability, such time series can then be combined hierarchically and across multiple cloud subsystems. Experimental results on utility cloud scenarios demonstrate the viability of the approach. EbAT outperforms threshold-based methods with on average 57.4% improvement in accuracy of anomaly detection and also does better by 59.3% on average in false alarm rate with a ‘near-optimum’ threshold-based method.

Journal ArticleDOI
TL;DR: Transfer Entropy as mentioned in this paper is a model-free measure designed as the Kullback-Leibler distance of transition probabilities, which allows to determine information transfer without being restricted to linear dynamics.
Abstract: We apply the concept of transfer entropy to quantify information flows between financial time series. Transfer entropy is a model-free measure designed as the Kullback-Leibler distance of transition probabilities. This approach allows to determine information transfer without being restricted to linear dynamics. We further develop a bootstrap procedure in order to allow for statistical inference of the estimates. In our empirical application, we examine the importance of the CDS and bond market for the process of pricing credit risk as well as the dynamic relation between market risk and credit risk proxied by the iTraxx Europe and the VIX.

Journal ArticleDOI
TL;DR: It is shown that both the traditional confidence-based active learning and semi-supervised learning approaches can be improved by maximizing the lattice entropy reduction over the whole dataset.