We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model.

/pdf/latent-dirichlet-allocation-4gmscb06en.pdf

Latent dirichlet allocation

We propose a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams [6], and Hof-mann's aspect model, also known as probabilistic latent semantic indexing (pLSI) [3]. In the context of text modeling, our model posits that each document is generated as a mixture of topics, where the continuous-valued mixture proportions are distributed as a latent Dirichlet random variable. Inference and learning are carried out efficiently via variational algorithms. We present empirical results on applications of this model to problems in text modeling, collaborative filtering, and text classification.

/pdf/latent-dirichlet-allocation-rkwbqihm3p.pdf

Latent Dirichlet Allocation

The emergence of order in natural systems is a constant source of inspiration for both physical and biological sciences. While the spatial order characterizing for example the crystals has been the basis of many advances in contemporary physics, most complex systems in nature do not offer such high degree of order. Many of these systems form complex networks whose nodes are the elements of the system and edges represent the interactions between them. 
Traditionally complex networks have been described by the random graph theory founded in 1959 by Paul Erdohs and Alfred Renyi. One of the defining features of random graphs is that they are statistically homogeneous, and their degree distribution (characterizing the spread in the number of edges starting from a node) is a Poisson distribution. In contrast, recent empirical studies, including the work of our group, indicate that the topology of real networks is much richer than that of random graphs. In particular, the degree distribution of real networks is a power-law, indicating a heterogeneous topology in which the majority of the nodes have a small degree, but there is a significant fraction of highly connected nodes that play an important role in the connectivity of the network. 
The scale-free topology of real networks has very important consequences on their functioning. For example, we have discovered that scale-free networks are extremely resilient to the random disruption of their nodes. On the other hand, the selective removal of the nodes with highest degree induces a rapid breakdown of the network to isolated subparts that cannot communicate with each other. 
The non-trivial scaling of the degree distribution of real networks is also an indication of their assembly and evolution. Indeed, our modeling studies have shown us that there are general principles governing the evolution of networks. Most networks start from a small seed and grow by the addition of new nodes which attach to the nodes already in the system. This process obeys preferential attachment: the new nodes are more likely to connect to nodes with already high degree. We have proposed a simple model based on these two principles wich was able to reproduce the power-law degree distribution of real networks. Perhaps even more importantly, this model paved the way to a new paradigm of network modeling, trying to capture the evolution of networks, not just their static topology.

/pdf/statistical-mechanics-of-complex-networks-1zfjpvxipu.pdf

Statistical mechanics of complex networks

Inspired by empirical studies of networked systems such as the Internet, social networks, and biological networks, researchers have in recent years developed a variety of techniques and models to help us understand or predict the behavior of these systems. Here we review developments in this field, including such concepts as the small-world effect, degree distributions, clustering, network correlations, random graph models, models of network growth and preferential attachment, and dynamical processes taking place on networks.

/pdf/the-structure-and-function-of-complex-networks-34deompc9l.pdf

The Structure and Function of Complex Networks

In recent years, spectral clustering has become one of the most popular modern clustering algorithms. It is simple to implement, can be solved efficiently by standard linear algebra software, and very often outperforms traditional clustering algorithms such as the k-means algorithm. On the first glance spectral clustering appears slightly mysterious, and it is not obvious to see why it works at all and what it really does. The goal of this tutorial is to give some intuition on those questions. We describe different graph Laplacians and their basic properties, present the most common spectral clustering algorithms, and derive those algorithms from scratch by several different approaches. Advantages and disadvantages of the different spectral clustering algorithms are discussed.

/pdf/a-tutorial-on-spectral-clustering-536pmg9onu.pdf

A tutorial on spectral clustering

Exchangeability and related topics

Let $(\mathscr{R}(k), k \geq 1)$ be random trees with $k$ leaves, satisfying a consistency condition: Removing a random leaf from $\mathscr{R}(k)$ gives $\mathscr{R}(k - 1)$. Then under an extra condition, this family determines a random continuum tree $\mathscr{L}$, which it is convenient to represent as a random subset of $l_1$. This leads to an abstract notion of convergence in distribution, as $n \rightarrow \infty$, of (rescaled) random trees $\mathscr{J}_n$ on $n$ vertices to a limit continuum random tree $\mathscr{L}$. The notion is based upon the assumption that, for fixed $k$, the subtrees of $\mathscr{J}_n$ determined by $k$ randomly chosen vertices converge to $\mathscr{R}(k)$. As our main example, under mild conditions on the offspring distribution, the family tree of a Galton-Watson branching process, conditioned on total population size equal to $n$, can be rescaled to converge to a limit continuum random tree which can be constructed from Brownian excursion.

/pdf/the-continuum-random-tree-iii-2638ivy8ct.pdf

The Continuum Random Tree III

For simple random walk on aN-vertex graph, the mean time to cover all vertices is at leastcN log(N), wherec>0 is an absolute constant. This is deduced from a more general result about stationary finite-state reversible Markov chains. Under weak conditions, the covering time for such processes is at leastc times the covering time for the corresponding i.i.d. process.

Lower bounds for covering times for reversible Markov chains and random walks on graphs

Author(s): Aldous, DJ | Abstract: Consider N particles, which merge into clusters according to the following rule: a cluster of size x and a cluster of size y merge at (stochastic) rate K(x, y)/N, were AT is a specified rate kernel. This Marcus-Lushnikov model of stochastic coalescence and the underlying deterministic approximation given by the Smoluchowski coagulation equations have an extensive scientific literature. Some mathematical literature (Kingman's coalescent in population genetics; component sizes in random graphs) implicitly studies the special cases K(x, y) = 1 and K(x, y) = xy. We attempt a wide-ranging survey. General kernels are only now starting to be studied rigorously; so many interesting open problems appear. © 1999 ISI/BS.

/pdf/deterministic-and-stochastic-models-for-coalescence-1vtarx7rxw.pdf

Deterministic and stochastic models for coalescence (aggregation and coagulation): a review of the mean-field theory for probabilists

(1981). The Central Limit Theorem for Real and Banach Valued Random Variables. Technometrics: Vol. 23, No. 3, pp. 307-307.

David Aldous

Papers

Exchangeability and related topics

The Continuum Random Tree III

Lower bounds for covering times for reversible Markov chains and random walks on graphs

Deterministic and stochastic models for coalescence (aggregation and coagulation): a review of the mean-field theory for probabilists

The Central Limit Theorem for Real and Banach Valued Random Variables