We present conditional random fields , a framework for building probabilistic models to segment and label sequence data. Conditional random fields offer several advantages over hidden Markov models and stochastic grammars for such tasks, including the ability to relax strong independence assumptions made in those models. Conditional random fields also avoid a fundamental limitation of maximum entropy Markov models (MEMMs) and other discriminative Markov models based on directed graphical models, which can be biased towards states with few successor states. We present iterative parameter estimation algorithms for conditional random fields and compare the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.

/pdf/conditional-random-fields-probabilistic-models-for-qeo4l25dzg.pdf

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

https://ssli.ee.washington.edu/people/duh/projects/CRFintro.pdf

Probabilistic Models for Segmenting and Labeling Sequence Data

The formalism of probabilistic graphical models provides a unifying framework for capturing complex dependencies among random variables, and building large-scale multivariate statistical models. Graphical models have become a focus of research in many statistical, computational and mathematical fields, including bioinformatics, communication theory, statistical physics, combinatorial optimization, signal and image processing, information retrieval and statistical machine learning. Many problems that arise in specific instances — including the key problems of computing marginals and modes of probability distributions — are best studied in the general setting. Working with exponential family representations, and exploiting the conjugate duality between the cumulant function and the entropy for exponential families, we develop general variational representations of the problems of computing likelihoods, marginal probabilities and most probable configurations. We describe how a wide variety of algorithms — among them sum-product, cluster variational methods, expectation-propagation, mean field methods, max-product and linear programming relaxation, as well as conic programming relaxations — can all be understood in terms of exact or approximate forms of these variational representations. The variational approach provides a complementary alternative to Markov chain Monte Carlo as a general source of approximation methods for inference in large-scale statistical models.

/pdf/graphical-models-exponential-families-and-variational-3i6dwlfta1.pdf

Graphical Models, Exponential Families, and Variational Inference

The authors define ordered subset processing for standard algorithms (such as expectation maximization, EM) for image restoration from projections. Ordered subsets methods group projection data into an ordered sequence of subsets (or blocks). An iteration of ordered subsets EM is defined as a single pass through all the subsets, in each subset using the current estimate to initialize application of EM with that data subset. This approach is similar in concept to block-Kaczmarz methods introduced by Eggermont et al. (1981) for iterative reconstruction. Simultaneous iterative reconstruction (SIRT) and multiplicative algebraic reconstruction (MART) techniques are well known special cases. Ordered subsets EM (OS-EM) provides a restoration imposing a natural positivity condition and with close links to the EM algorithm. OS-EM is applicable in both single photon (SPECT) and positron emission tomography (PET). In simulation studies in SPECT, the OS-EM algorithm provides an order-of-magnitude acceleration over EM, with restoration quality maintained. >

Accelerated image reconstruction using ordered subsets of projection data

The concept of maximum entropy can be traced back along multiple threads to Biblical times. Only recently, however, have computers become powerful enough to permit the widescale application of this concept to real world problems in statistical estimation and pattern recognition. In this paper, we describe a method for statistical modeling based on maximum entropy. We present a maximum-likelihood approach for automatically constructing maximum entropy models and describe how to implement this approach efficiently, using as examples several problems in natural language processing.

/pdf/a-maximum-entropy-approach-to-natural-language-processing-57psgipa7w.pdf

A maximum entropy approach to natural language processing

Say that a probability distribution $\{p_i; i \in I\}$ over a finite set $I$ is in "product form" if (1) $p_i = \pi_i\mu \prod^d_{s=1} \mu_s^{b_si}$ where $\pi_i$ and $\{b_{si}\}$ are given constants and where $\mu$ and $\{\mu_s\}$ are determined from the equations (2) $\sum_{i \in I} b_{si} p_i = k_s, s = 1, 2, \cdots, d$; (3) $\sum_{i \in I} p_i = 1$. Probability distributions in product form arise from minimizing the discriminatory information $\sum_{i \in I} p_i \log p_i/\pi_i$ subject to (2) and (3) or from maximizing entropy or maximizing likelihood. The theory of the iterative scaling method of determining (1) subject to (2) and (3) has, until now, been limited to the case when $b_{si} = 0, 1$. In this paper the method is generalized to allow the $b_{si}$ to be any real numbers. This expands considerably the list of probability distributions in product form which it is possible to estimate by maximum likelihood.

/pdf/generalized-iterative-scaling-for-log-linear-models-4ru8c6hl95.pdf

D. Ratcliff

Papers

Generalized Iterative Scaling for Log-Linear Models