Showing papers on "Multivariate mutual information published in 2002"

The mutual information: detecting and evaluating dependencies between variables.

Abstract: Motivation: Clustering co-expressed genes usually requires the definition of ‘distance’ or ‘similarity’ between measured datasets, the most common choices being Pearson correlation or Euclidean distance. With the size of available datasets steadily increasing, it has become feasible to consider other, more general, definitions as well. One alternative, based on information theory, is the mutual information, providing a general measure of dependencies between variables. While the use of mutual information in cluster analysis and visualization of large-scale gene expression data has been suggested previously, the earlier studies did not focus on comparing different algorithms to estimate the mutual information from finite data. Results: Here we describe and review several approaches to estimate the mutual information from finite datasets. Our findings show that the algorithms used so far may be quite substantially improved upon. In particular when dealing with small datasets, finite sample effects and other sources of potentially misleading results have to be taken into account.

Four Top Reasons Mutual Information Does Not Quantify Neural Information Processing

Abstract: Mutual information between stimulus and response has been advocated as an information theoretic measure of a neural system’s capability to process information. Once calculated, the result is a single number that supposedly captures the system’s information characteristics over the range of stimulus conditions used to measure it. I show that mutual information is a flawed measure, the standard approach to measuring it has theoretical difficulties, and that relating capacity to information processing capability is quite complicated.

Mutual Information and Relevant Variables for Predictions

Abstract: In this chapter we propose a method to select from a possibly large set of observable quantities a minimal subset yielding (nearly) all relevant information on the quantity we are going to predict. We derive the theoretical background and give numerical hints and examples, including results for some daily dollar exchange rates. Our approach essentially profits from the availability of a fast algorithm for mutual information analysis.

Distribution of mutual information for robust feature selection

Abstract: Mutual information is widely used in artificial intelligence, in a descriptive way, to measure the stochastic dependence of discrete random variables. In order to address questions such as the reliability of the empirical value, one must consider sample-to-population inferential approaches. This paper deals with the distribution of mutual information, as obtained in a Bayesian framework by using second-order Dirichlet prior distributions. We derive reliable and quickly computable analytical approximations for the distribution of mutual information. We concentrate on the mean, variance, skewness, and kurtosis. For the mean we also provide an exact expression. The results are applied to the problem of selecting features for incremental learning and classification of the naive Bayes classifier. A fast, newly defined filter is shown to outperform the traditional approach based on empirical mutual information on a number of real data sets. A theoretical development allows the above methods to be extended to incomplete samples in an easy and effective way. Further experiments on incomplete data sets support the extension of the proposed filter to the case of missing data.

Inferring Genetic Networks from Expression Data with Mutual Information

Abstract: Recent methods to infer genetic networks are based on identifying gene interactions by similarities in expression profiles. These methods are founded on the assumption that interacting genes share higher similarities in their expression profiles than non-interacting genes. In this dissertation this assumption is validated when using mutual information as a similarity measure. Three algorithms that calculate mutual information between expression data are developed: 1) a basic approach implemented with the histogram technique; 2) an extension of the basic approach that takes into consideration time delay between expression profiles; 3) an extension of the basic approach that takes into consideration that genes are regulated in a complex manner by multiple genes. In our experiments we compare the mutual information distributions for profiles of interacting and non-interacting genes. The results show that interacting genes do not share higher mutual information in their expression profiles than non-interacting genes, thus contradicting the basic assumption that similarity measures need to fulfil. This indicates that mutual information is not appropriate as similarity measure, which contradicts earlier proposals.