We present conditional random fields , a framework for building probabilistic models to segment and label sequence data. Conditional random fields offer several advantages over hidden Markov models and stochastic grammars for such tasks, including the ability to relax strong independence assumptions made in those models. Conditional random fields also avoid a fundamental limitation of maximum entropy Markov models (MEMMs) and other discriminative Markov models based on directed graphical models, which can be biased towards states with few successor states. We present iterative parameter estimation algorithms for conditional random fields and compare the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.

/pdf/conditional-random-fields-probabilistic-models-for-qeo4l25dzg.pdf

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

https://ssli.ee.washington.edu/people/duh/projects/CRFintro.pdf

Probabilistic Models for Segmenting and Labeling Sequence Data

This paper presents a Bayesian method for constructing probabilistic networks from databases. In particular, we focus on constructing Bayesian belief networks. Potential applications include computer-assisted hypothesis testing, automated scientific discovery, and automated construction of probabilistic expert systems. We extend the basic method to handle missing data and hidden (latent) variables. We show how to perform probabilistic inference by averaging over the inferences of multiple belief networks. Results are presented of a preliminary evaluation of an algorithm for constructing a belief network from a database of cases. Finally, we relate the methods in this paper to previous work, and we discuss open problems.

/pdf/a-bayesian-method-for-the-induction-of-probabilistic-4fd3nluuju.pdf

A Bayesian Method for the Induction of Probabilistic Networks from Data

Hidden Markov models (HMMs) are a powerful probabilistic tool for modeling sequential data, and have been applied with success to many text-related tasks, such as part-of-speech tagging, text segmentation and information extraction. In these cases, the observations are usually modeled as multinomial distributions over a discrete vocabulary, and the HMM parameters are set to maximize the likelihood of the observations. This paper presents a new Markovian sequence model, closely related to HMMs, that allows observations to be represented as arbitrary overlapping features (such as word, capitalization, formatting, part-of-speech), and defines the conditional probability of state sequences given observation sequences. It does this by using the maximum entropy framework to fit a set of exponential models that represent the probability of a state given an observation and the previous state. We present positive experimental results on the segmentation of FAQ’s.

/pdf/maximum-entropy-markov-models-for-information-extraction-and-4zvhogmtcx.pdf

Maximum Entropy Markov Models for Information Extraction and Segmentation

We report on improved practical algorithms for lattice basis reduction. We present a variant of the L3-algorithm with “deep insertions” and a practical algorithm for blockwise Korkine-Zolotarev reduction, a concept extending L3-reduction, that has been introduced by Schnorr (1987). Empirical tests show that the strongest of these algorithms solves almost all subset sum problems with up to 58 random weights of arbitrary bit length within at most a few hours on a UNISYS 6000/70 or within a couple of minutes on a SPARC 2 computer.

/pdf/lattice-basis-reduction-improved-practical-algorithms-and-3e22lna5en.pdf

Lattice basis reduction: improved practical algorithms and solving subset sum problems

Introduction to Probabilistic Automata

http://www.eecs.harvard.edu/cs286r/courses/fall11/papers/EP84.pdf

A note on cake cutting

This paper offers an axiomatic characterization of the probabilistic relation “X is independent of Y (written (X, Y))”, where X and Y are two disjoint sets of variables. Four axioms for (X, Y) are presented and shown to be complete. Based on these axioms, a polynomial membership algorithm is developed to decide whether any given independence statement (X, Y) logically follows from a set Σ of such statements, i.e., whether (X, Y) holds in every probability distribution that satisfies Σ. The complexity of the algorithm is O(|Σ| · k2 + |Σ| · n), where |Σ| is the number of given statements, n is the number of variables in Σ ∪ {(X, Y)}, and k is the number of variables in (X, Y).

https://escholarship.org/content/qt9q16j91d/qt9q16j91d.pdf?t=lrh1xe

Axioms and algorithms for inferences involving probabilistic independence

Graphoids: graph-based logic for reasoning about relevance relations or when would x tell you more about y if you already know z?

Probabilistie finite automata (p.a.) as introduced in this paper are intended to generalize deterministic and nondeterministie finite automata (f.a.) in the sense of Rabin and Scott (1959). Although the notion of p.a. was mentioned in a popular book by Ashby (1958) no precise definition has been given in literature except for a recent paper by Rabin (1963). Another a t tempt to generalize the f.a. in the sense of Moore (1956) was made by Carlyle (1961) and a particular form of p.a. is investigated by Tsetslin (1961) and Davis (1961), but all the above papers t reat the p.a. from a different point of view than that assumed by Rabin (1963) and here. An at tempt to define the p.a. in the sense of this paper, and to investigate some of their properties, was already made by the author in his (unpublished) master's thesis undertaken at the Hebrew University of Jerusalem; still, as our point of view is very close to that of Rabin (1963) we shall adopt here (with certain modifications) the basic definitions of tha t paper. In order to keep this paper within reasonable limits we shall omit the proofs of some of the theorems.

Azaria Paz

Papers

Introduction to Probabilistic Automata

A note on cake cutting

Axioms and algorithms for inferences involving probabilistic independence

Graphoids: graph-based logic for reasoning about relevance relations or when would x tell you more about y if you already know z?

Some aspects of probabilistic automata