The formalism of probabilistic graphical models provides a unifying framework for capturing complex dependencies among random variables, and building large-scale multivariate statistical models. Graphical models have become a focus of research in many statistical, computational and mathematical fields, including bioinformatics, communication theory, statistical physics, combinatorial optimization, signal and image processing, information retrieval and statistical machine learning. Many problems that arise in specific instances — including the key problems of computing marginals and modes of probability distributions — are best studied in the general setting. Working with exponential family representations, and exploiting the conjugate duality between the cumulant function and the entropy for exponential families, we develop general variational representations of the problems of computing likelihoods, marginal probabilities and most probable configurations. We describe how a wide variety of algorithms — among them sum-product, cluster variational methods, expectation-propagation, mean field methods, max-product and linear programming relaxation, as well as conic programming relaxations — can all be understood in terms of exact or approximate forms of these variational representations. The variational approach provides a complementary alternative to Markov chain Monte Carlo as a general source of approximation methods for inference in large-scale statistical models.

/pdf/graphical-models-exponential-families-and-variational-3i6dwlfta1.pdf

Graphical Models, Exponential Families, and Variational Inference

Methods of information geometry

Information Theory and Reliable Communication

This book provides a broad survey of models and efficient algorithms for Nonnegative Matrix Factorization (NMF) This includes NMFs various extensions and modifications, especially Nonnegative Tensor Factorizations (NTF) and Nonnegative Tucker Decompositions (NTD) NMF/NTF and their extensions are increasingly used as tools in signal and image processing, and data analysis, having garnered interest due to their capability to provide new insights and relevant information about the complex latent relationships in experimental data sets It is suggested that NMF can provide meaningful components with physical interpretations; for example, in bioinformatics, NMF and its extensions have been successfully applied to gene expression, sequence analysis, the functional characterization of genes, clustering and text mining As such, the authors focus on the algorithms that are most useful in practice, looking at the fastest, most robust, and suitable for large-scale models Key features: Acts as a single source reference guide to NMF, collating information that is widely dispersed in current literature, including the authors own recently developed techniques in the subject area Uses generalized cost functions such as Bregman, Alpha and Beta divergences, to present practical implementations of several types of robust algorithms, in particular Multiplicative, Alternating Least Squares, Projected Gradient and Quasi Newton algorithms Provides a comparative analysis of the different methods in order to identify approximation error and complexity Includes pseudo codes and optimized MATLAB source codes for almost all algorithms presented in the book The increasing interest in nonnegative matrix and tensor factorizations, as well as decompositions and sparse representation of data, will ensure that this book is essential reading for engineers, scientists, researchers, industry practitioners and graduate students across signal and image processing; neuroscience; data mining and data analysis; computer science; bioinformatics; speech processing; biomedical engineering; and multimedia

Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation

A wide variety of distortion functions, such as squared Euclidean distance, Mahalanobis distance, Itakura-Saito distance and relative entropy, have been used for clustering. In this paper, we propose and analyze parametric hard and soft clustering algorithms based on a large class of distortion functions known as Bregman divergences. The proposed algorithms unify centroid-based parametric clustering approaches, such as classical kmeans , the Linde-Buzo-Gray (LBG) algorithm and information-theoretic clustering, which arise by special choices of the Bregman divergence. The algorithms maintain the simplicity and scalability of the classical kmeans algorithm, while generalizing the method to a large class of clustering loss functions. This is achieved by first posing the hard clustering problem in terms of minimizing the loss in Bregman information, a quantity motivated by rate distortion theory, and then deriving an iterative algorithm that monotonically decreases this loss. In addition, we show that there is a bijection between regular exponential families and a large class of Bregman divergences, that we call regular Bregman divergences. This result enables the development of an alternative interpretation of an efficient EM scheme for learning mixtures of exponential family distributions, and leads to a simple soft clustering algorithm for regular Bregman divergences. Finally, we discuss the connection between rate distortion theory and Bregman clustering and present an information theoretic analysis of Bregman clustering algorithms in terms of a trade-off between compression and loss in Bregman information.

/pdf/clustering-with-bregman-divergences-29l7xomi4s.pdf

Clustering with Bregman Divergences

A measurement process is logically reversible if the premeasurement density operator of the measured system is uniquely determined from the postmeasurement density operator and the outcome of the measurement. This paper analyzes the necessary and sufficient condition for a measurement process to be logically reversible and discusses specific examples on quantum-nondemolition measurements, quantum counting, and measurement of spin systems. It is shown that for any sharp measurement we can construct a logically reversible measurement that continuously approaches the sharp measurement with a decrease in the measurement error. A general condition for a measurement process to be reversed by another with a nonzero probability of success is given, and the implications of such physical reversibility are discussed. \textcopyright{} 1996 The American Physical Society.

Logical reversibility in quantum measurement: General theory and specific examples

In the Arimoto-Blahut (1972) algorithm for computing the capacity of a classical discrete memoryless channel, maximization of the mutual information is converted to more tractable alternating maximization of a two-variable function. We apply the same idea to computation of the capacity of a quantum memoryless channel.

Algorithms of Arimoto-Blahut type for computing quantum channel capacity

The capacity of quantum channel with product input states was formulated by the quantum coding theorem. However, whether entangled input states can enhance the quantum channel is still open. It turns out that this problem is reduced to aspecial case of the more general problem whether the capacity of product quantum channel exhibits additivity. In the present study, we apply one of the quantum Arimoto-Blahut type algorithms to the latter problem. The results suggest that the additivity of product quantum channel capacity always holds and that entangled input states cannot enhance the quantum channel capacity.

Numerical Experiments on The Capacity of Quantum Channel with Entangled Input States

A new type of sufficient condition is provided for a probability distribution on the nonnegative integers to be given an optimal D-ary prefix code by a Huffman-type algorithm. In the justification of our algorithm, we introduce two new (essentially one) concepts as the definition of the "optimality" of a prefix D-ary code, which are shown to be equivalent to that defined in the traditional way. These new concepts of the optimality are meaningful even for the case where the Shannon entropy H(P) diverges.

Huffman coding with an infinite alphabet

The hypothesis testing problem for two quantum states is treated. We show a new inequality between the errors of the first kind and the second kind, which complements the result of Hiai and Petz (1991) to establish the quantum version of Stein's lemma. The inequality is also used to show a bound on the probability of errors of the first kind when the power exponent for the probability of errors of the second kind exceeds the quantum relative entropy, which yields the strong converse in quantum hypothesis testing. Finally, we discuss the relation between the bound and the power exponent derived by Han and Kobayashi (1989) in classical hypothesis testing.

Hiroshi Nagaoka

Papers

Logical reversibility in quantum measurement: General theory and specific examples

Algorithms of Arimoto-Blahut type for computing quantum channel capacity

Numerical Experiments on The Capacity of Quantum Channel with Entangled Input States

Huffman coding with an infinite alphabet

Strong Converse and Stein's Lemma in Quantum Hypothesis Testing