Data Mining Practical Machine Learning Tools and Techniques

語用論(Pragmatics)を考える

From Discourse to Logic

Inference and Disputed Authorship: The Federalist

"You shall know a word by the company it keeps!" With this slogan, J. R. Firth drew attention to a fact that language scholars had intuitively known for a long time: In natural language, words are not combined randomly into phrases and sentences, constrained only by the rules of syntax. They have a tendency to appear in certain recurrent combinations. As there are many possible reasons for words to go together, a broad range of linguistic and extra-linguistic phenomena can be found among the recurrent combinations, making them a goldmine of information for linguistics, natural language processing and related fields. There are compound nouns ("black box"), fixed and opaque idioms ("kick the bucket"), lexical selection ("a pride of lions", "heavy smoker") and formulaic expressions ("have a nice day"). They can often tell us something about the meaning of a word or even the concept behind the word (think of combinations like "dark night" and "bright day"), an idea that has inspired latent semantic analysis and similar vector space models of word meaning.
With modern computers it is easy to extract evidence for recurrent word pairs from huge text corpora, often aided by linguistic pre-processing and annotation (so that specific combinations, e.g. noun+verb can be targeted). However, the raw data - in the form of frequency counts for word pairs – are not always meaningful as a measure for the amount of "glue" between two words. Provided that both words are sufficiently frequent, their cooccurrences might be pure coincidence. Therefore, a statistical interpretation of the frequency data is necessary, which determines the degree of statistical association between the words and whether there is enough evidence to rule out chance as a factor. For this purpose, association measures are applied, which assign a score to each word pair based on the observed frequency data. The higher this score is, the stronger and more certain the association between the two words.
Even forty years ago, at the Symposium on Statistical Association Methods for Mechanized Documentation, there was a bewildering multitude of measures to choose from, but hardly any guidelines to help with the decision. This situation hasn't changed very much over the last forty years. We are still far away from a thorough understanding of association measures and there is not even a standard reference where one could look up precise definitions and related information. My thesis aims to fill this gap.
The first, encyclopedic part of the thesis begins with a description of the formal and statistical prerequisites. Intended primarily as a reference for students and researchers, it also addresses the limits of the statistical models. The following chapter presents a comprehensive repository of association measures, which are organised into thematic groups. An explicit equation is given for each measure, using a consistent notation in terms of observed and expected frequencies.
The second, methodological part suggests new approaches to the study of association measures, with an emphasis on empirical results and intuitive understanding. A cornerstone of this approach is a geometric interpretation of cooccurrence data and association measure. Measures are visualised as surfaces in a three-dimensional "coordinate space". The properties of each measure are determined by the geometric shapes of the respective surfaces.
Empirical results are obtained from evaluation studies, which test the performance of association measures in a collocation extraction task. In addition to its relevance for real-life applications, a carefully designed evaluation can reveal important properties of the association measures. Unfortunately, it is becoming clear the evaluation results cannot easily be generalised. For this reason it is desirable to carry out more evaluation experiments under different conditions. In order to reduce the necessary amount of manual work, evaluation can be performed on random samples from a set of candidates. Appropriate significance tests correct for the higher degree of uncertainty.
Finally, there is a third, computational aspect to the thesis. It is accompanied by an open-source software toolkit, which was used to perform experiments and produce graphs for the thesis. The unique feature of this software toolkit is that the current release includes all the data, scripts and explanations needed to replicate (almost) all the results found in the book. 
Das gemeinsame Vorkommen von Wortern in naturlicher Sprache - sei es in unmittelbarer Nachbarschaft, innerhalb desselben Satzes oder in einer bestimmten syntaktischen Relation - stellt eine zentrale Wissensquelle fur die maschinelle Sprachverarbeitung dar. Frequenzdaten fur solche Kookkurrenzen konnen leicht aus Textkorpora gewonnen werden, wobei in den meisten Fallen zunachst eine linguistische Vorverarbeitung erfolgt. Eine mathematische Auswertung erlaubt dann, die Ergebnisse uber das spezifische Extraktionskorpus hinaus zu verallgemeinern und auf statistische Assoziationen zwischen den Vorkommen der beteiligten Worter in der Sprache insgesamt (oder zumindest in einer Teilsprache) zu schliesen.
Das gebrauchlichste Verfahren hierfur sind sogenannte Assoziationsmase, die ausgehend von der im Korpus ermittelten Frequenzinformation eine Bewertungszahl berechnen: je hoher dieser Wert, desto starker ist die mutmasliche Assoziation. Dabei stutzt sich das Mas lediglich auf die Kookkurrenzhaufigkeit und auf die Haufigkeiten der einzelnen Worter.
Die so gewonnene Information last sich in vielfaltiger Weise anwenden, unter anderem zur Desambiguierung von syntaktischen Analysen, zur Identifikation von Satz- und Phrasengrenzen, zur Verbesserung von stochastischen Sprachmodellen, zur Lesartendesambiguierung und anderen Klassifikationsaufgaben, sowie zur Bestimmung von semantischen Ahnlichkeiten zwischen Wortern wie Synonymie und Hyponymie. Andererseits bieten statistische Assoziationen einen wichtigen Anhaltspunkt fur die Identifikation lexikalisierter Wortverbindungen, sogenannter Kollokationen.
Bereits zur Zeit der ersten computerlinguistischen Arbeiten mit Kookkurrenzdaten und Kollokationen stand eine nahezu unuberschaubare Vielfalt von Assoziationsmasen zur Verfugung: man bediente sich bei diversen Fachgebieten, allen voran naturlich der mathematischen Statistik. Gleichzeitig war man sich der Schwierigkeit bewust, ein fur die jeweilige Aufgabe geeignetes Mas zu finden, besonders da sich die in der mathematischen Statistik angefuhrten Argumente nicht ohne weiteres auf Wortkookkurrenzen ubertragen lassen.
Vierzig Jahre spater versucht die vorliegende Arbeit nun, diese Lucke zu schliesen. In einem enzyklopadischen Teil werden zunachst die formalen und statistischen Grundlagen zusammengetragen, auf denen die Assoziationsmase basieren. Diese Aufstellung soll als Referenz fur weitere Untersuchungen und Anwendungen dienen, es werden jedoch auch die Grenzen der statistischen Modelle angesprochen. Im nachsten Kapitel folgt eine Auflistung aller dem Autor bekannten Assoziationsmase. Fur jedes Mas wird eine explizite Formel in einheitlicher Notation angegeben, um die Umsetzung in einem Computerprogramm zu erleichtern und Misverstandnisse zu vermeiden. Die Assoziationsmase sind nach ihrem mathematischen Hintergrund in Gruppen eingeteilt, um so dem Leser einen besseren Uberblick zu verschaffen. Zugleich werden feine Unterschiede und uberraschende Ahnlichkeiten zwischen den Masen hervorgehoben.
Der zweite, methodische Teil will neue Anstose fur die Forschung zur Theorie und Anwendung von Assoziationsmasen geben, mit einem Schwerpunkt auf empirische Forschung und intuitives Verstehen. Eine zentrale Rolle spielt dabei eine geometrische Interpretation der Mase, die als Flachen in einem dreidimensionalen "Haufigkeitsraum" veranschaulicht werden. Die Eigenschaften eines Assoziationsmases werden dabei durch die geometrische Form der ihm entsprechenden Flachen bestimmt. 
Neben diesem intuitiven Ansatz werden auch mathematische und empirische Methoden beschrieben. Die mathematische Diskussion liefert dabei eine theoretisch fundierte Begrundung fur den Einsatz von Haufigkeitsschwellwerten: aufgrund der besonderen Verteilung von Worthaufigkeiten (bekannt unter dem Namen Zipfsches Gesetz), sind statistische Inferenzen fur Worter oder Wortpaare, die weniger als 5 Vorkommen haben, prinzipiell unzuverlassig.
Schlieslich stellen die empirischen Untersuchungen im letzten Kapitel eine Verbindung zu Anwendungen und zum linguistischen Kollokationsbegriff her. Hier werden die Assoziationsmase zur Kollokationsextraktion eingesetzt und auf diese Weise evaluiert. Neben der praktischen Relevanz kann eine sorgfaltige Evaluation einiges uber die besonderen Eigenschaften eines Mases verraten. Leider zeigt sich immer deutlicher, das sich Evaluationsergebnisse nicht auf andere Situationen ubertragen lassen. Um eine grosere Anzahl von Evaluationsexperimenten unter verschiedenen Bedingungen durchfuhren zu konnen, wird eine Evaluationsmethode vorgeschlagen, die auf Stichproben basiert. Geeignete Signifikanztests stellen sicher, das es sich bei den Ergebnissen solcher Evaluationen nicht um zufallige "Ausrutscher" handelt.

/pdf/the-statistics-of-word-cooccurrences-word-pairs-and-2nwwdy04gy.pdf

The statistics of word cooccurrences : word pairs and collocations

This paper reports on the TIGER Treebank, a corpus of currently 40,000 syntactically annotated German newspaper sentences. We describe what kind of information is encoded in the treebank and introduce the different representation formats that are used for the annotation and exploitation of the treebank. We explain the different methods used for the annotation: interactive annotation, using the tool ANNOTATE, and LFG parsing. Furthermore, we give an account of the annotation scheme used for the TIGER treebank. This scheme is an extended and improved version of the NEGRA annotation scheme and we illustrate in detail the linguistic extensions that were made concerning the annotation in the TIGER project. The main differences are concerned with coordination, verb-subcategorization, expletives as well as proper nouns. In addition, the paper also presents the query tool TIGERSearch that was developed in the project to exploit the treebank in an adequate way. We describe the query language which was designed to facilitate a simple formulation of complex queries; furthermore, we shortly introduce TIGER in, a graphical user interface for query input. The paper concludes with a summary and some directions for future work.

TIGER: Linguistic Interpretation of a German Corpus

This paper deals with the representation of multi-level linguistic annotations. It proposes an XML-based, generic stand-off architecture and presents an example instantiation. Application scenarios that profit from this architecture are sketched out. In recent years, corpus linguistics has become more and more important to a broad community, including people working in theoretical, applied and computational linguistics. To many of them, speech and text corpora represent a rich source of data and phenomena, forming the basis of their research. Benefit from such data is even more important if the data is annotated by suitable information, allowing for fast and effective retrieval of relevant data. Whereas corpora of the first generation featured part-of-speech and syntactic annotations (e.g. PennTreebank [MSM93], TIGER corpus [BDE04]), the focus has now switched to properties beyond the (morpho-)syntactic level. Recent corpora are annotated by semantic information (PropBank [KP02], FrameNet [JPB03], SALSA [EKPP03]), pragmatic information (Penn Discourse TreeBank [MPJW04], RST Discourse Treebank [CMO03], Potsdam Commentary Corpus [Ste04]), and dialogue structure (Switchboard SWBD-DAMSL [JSB97]). Annotations often have to be carried out manually — reliable (semi-)automatic tools exist only for the annotation of part of speech and syntax, and are restricted to well-researched languages like English or German. Moreover, hand-annotated training material is a prerequisite for the development of automatic tools. As a consequence, corpora and annotations ought to be reusable so that a large community can profit from the data. To this end, various standardization efforts have been launched. Standardization of linguistic data concerns (see, e.g., [Sch05]): (i) The physical data structure: here, XML has become the widely-recognized standard format. (ii) The logical data structure: i.e., the data models that are used to model the phenomena and their properties (e.g. hierarchical structures like trees or graphs for syntax annotations 1The research reported in this paper was jointly financed by the German Research Foundation (DFG, SFB632) and the Federal Ministry of Education and Research (BMBF grant no. 03WKH22). Many thanks go to my colleagues, especially Michael Götze, for helpful discussions of the topics addressed in this paper. vs. time-aligned tiers for speech and dialogue annotations). Examples of data models are annotation graphs [BL01] and the NITE Object Model [CKO03b]. (iii) Content: in several initiatives, XML applications for specific linguistic annotations have been developed. For instance, TEI2 (“Text Encoding Initiative”, [SB94]) defines highly-detailed DTDs for encoding all kinds of bibliographic and other information; XCES3 (“XML-based Corpus Encoding Standard”) provides DTDs for the annotation of chunks, alignment, etc. More recently, however, it has been recognized that these standardized DTDs often do not meet application-specific needs. Hence, abstract, generic XML formats have been proposed that allow for the formal integration of application-specific annotations [IR01]. For the conceptual integration of specific annotations, so-called data category repositories as well as linguistic ontologies have been developed. They define reference categories, with precise semantics and examples, that specific annotation tags ought to be mapped to (see, e.g., DOLCE4, “Descriptive Ontology for Linguistic and Cognitive Engineering”). This papers deals with the formal integration of specific annotations. It first addresses the subject of stand-off architecture (sec. 1). We then propose an XML-based representation of linguistic annotation and present an example application (instantiation) in some detail (sec. 2). We also sketch out some application scenarios that profit from such a flexible architecture (sec. 3) and address related approaches (sec. 4). 1 Stand-off Architecture As early as in the mid-nineties, the topic of “stand-off annotation” has been discussed (see, e.g., [TM97]). This term describes the situation where primary data (e.g., the source text) and annotations of this data are stored in separate files. Stand-off annotation might seem problematic, because there is no immediate connection between the text and its annotation; hence, whenever the source text is modified, extra care has to be taken to synchronize its annotation. Similarly, human inspection of the data becomes cumbersome. On the other hand, however, stand-off annotation has the great advantage of leaving the source text untouched. It thus allows for annotating text that cannot be modified for whatever reasons, e.g., because it is a text available on the Internet. Moreover, whereas XML as such does not easily account for overlapping segments and conflicting hierarchies,5 they can be marked in a natural way in stand-off annotation: by distributing annotations over different files. That is, not only is the source text separated from its annotations, but individual annotations are separated from each other as well. This way, annotations at different levels can be created and modified independently of each other. Finally, competing, alternative annotations can even be represented, e.g. variants of part-of-speech annotations that are output of different tools. 2http://www.tei-c.org/ 3http://www.cs.vassar.edu/XCES/ 4http://www.loa-cnr.it/DOLCE.html 5Different methods have been proposed to accommodate conflicting markup into XML. We will come back to them below. One of the first proposals for stand-off annotation of linguistic corpora is [DBD98]. An ISO working group is currently developing the stand-off based LAF6 (“Linguistic Annotation Framework” [IRdlC03]). Some recent corpora like the ANC (“American National Corpus” [RI04]) are encoded in stand-off architecture. In our approach presented in this paper, we also subscribe to the principles of stand-off annotation. 2 A Generic XML Format Our format defines generic XML elements like <mark> (markable), <feat> (feature), and <struct> (structure), which indicate which data type the annotation conforms to. We assume that primary data is stored in a file that optionally specifies a header, followed by a tag <body>, which contains the source text. Annotations are stored in separate files; they may refer to the source text or to other annotations. These relations are encoded by means of XLinks and XPointers. We distinguish three different types of annotations: markables, structures, and features. (i) Markables: <mark> tags specify text positions or spans of text (or spans of other markables) that can be annotated by linguistic information. For instance, <mark> tags might indicate tokens by specifying ranges of the source text, cf. fig. 1. (ii) Structures: <struct> tags are special types of markables. Similar to <mark> tags, they specify objects that then can serve as anchors for annotations. Whereas <mark> tags define simple types of anchors (flat spans of text or markables), a <struct> tag represents a complex anchor involving relations between arbitrarily many markables (including <struct> elements). Relations (<rel>) can be further specified by an attribute type, e.g. as undirected or directed (= pointers). Put differently, a <structList> specifies a complete tree or graph, which consists of single tree fragments specified by the <struct> tags, cf. fig. 1. (iii) Features: <feat> tags specify information annotated to markables or structures, which are referred to by xlink attributes. The type of information (e.g., “part of speech”) is encoded by an attribute type, cf. fig. 2. For instance, the information encoded by the first <feat> in fig. 2 can be paraphrased as follows: Take the token that is defined by the tag <mark> with the ID attribute id="tok 1" and assign the part of speech “ART” (article) to that token. We intend to adopt the idea of [CKO03a] by assuming that admissible feature values (such as “NN”, normal/common noun, or “NE”, named entity) may be complex types and are organized in a type hierarchy. For instance, “NN” and “NE” might be subtypes of the more general type “N”, noun. <feat> tags then point to some type in the hierarchy (which is stored separately), thus specifying the value of the annotated property, cf. fig. 3.7 6ISO Technical TC37/SC4, http://www.tc37sc4.org 7Type hierarchies have to be defined by the user or they may be derived from annotation schemes that incorporate hierarchies, cf. the schemes used by the annotation tool MMAX. In case no hierarchy is defined, the features will be organized in a flat list. The stand-off architecture allows the user to experiment with different hierarchies. Further examples of annotations are sketched out below. They illustrate that annotations may stem from different sources (see the attribute source) and encode various types of information. Categorial annotation (anchored to constituents) <header sfb id="rabin1.const cat" type="categories" source="TIGERcorpus"/> <featList xml:base="rabin1.const.xml"> <feat xlink:href="#syn 1" value="PN"/>  <feat xlink:href="#syn 2" value="PP"/>  ... Coreference annotation, marking coreferential expressions such as pronouns (referred to xlink:href attributes) and their antecedents (identified by target attributes) <header sfb id="rabin1.coref" type="coreference" source="MMAXcoref"/> <featList> <feat xlink:href="rabin1.tok.xml#tok 19" (sein) target="rabin1.const.xml#syn 9" (Der Rabin-Attentäter Jigal Amir) value="identity"/> ... Document structure: headers, paragraphs, lists, etc. (anchored to markables that refer to tokens) <header sfb id="rabin1.div" type="divisions"/> <markList xmlns:xlink="http://www.w3.org/1999/xlink" xml:base="rabin1.tok.xml"> <mark id="div 1" xlink:href="#xpointer(id(’tok 1’)/range-to(id(’tok 390’)"/> <mark id="div 2" xlink:href="#xpointer(id(’tok 1’)/range-to(id(’tok 89’)"/> ... <header sfb id="rabin1.div docstr" type="documentStructure"/> <featL

XML-based Stand-off Representation and Exploitation of Multi-Level Linguistic Annotation.

We present a general framework for integrating annotations from different tools and tag sets. When annotating corpora at multiple linguistic levels, annotators may use different expert tools for different phenomena or types of annotation. These tools employ different data models and accompanying approaches to visualization, and they produce different output formats. For the purposes of uniformly processing these outputs, we developed a pivot format called PAULA, along with converters to and from tool formats. Different annotations are not only integrated at the level of data format, but are also joined on the level of conceptual representation. For this purpose, we introduce OLiA, an ontology of linguistic annotations that mediates between alternative tag sets that cover the same class of linguistic phenomena. All components are integrated in the linguistic information system ANNIS : Annotation tool output is converted to the pivot format PAULA and read into a database where the data can be visualized, queried, and evaluated across multiple layers. For cross-tag set querying and statistical evaluation, ANNIS uses the ontology of linguistic annotations. Finally, ANNIS is also tied to a machine learning component for semiautomatic annotation.

A Flexible Framework for Integrating Annotations from Different Tools and Tag Sets

This paper deals with normalization of language data from Early New High German. We describe an unsupervised, rulebased approach which maps historical wordforms to modern wordforms. Rules are specified in the form of context-aware rewrite rules that apply to sequences of characters. They are derived from two aligned versions of the Luther bible and weighted according to their frequency. The evaluation shows that our approach (83%‐91% exact matches) clearly outperforms the baseline (65%).

http://www.linguistics.rub.de/~petran/papers/ranlp2011.pdf

Rule-Based Normalization of Historical Texts

We report on the evaluation of information structural annotation according to the Linguistic Information Structure Annotation Guidelines (LISA, (Dipper et al., 2007)). The annotation scheme differentiates between the categories of information status, topic, and focus. It aims at being language-independent and has been applied to highly heterogeneous data: written and spoken evidence from typologically diverse languages. For the evaluation presented here, we focused on German texts of different types, both written texts and transcriptions of spoken language, and analyzed the annotation quantitatively and qualitatively.

/pdf/annotation-of-information-structure-an-evaluation-across-4x1wox1e41.pdf

Stefanie Dipper

Papers

TIGER: Linguistic Interpretation of a German Corpus

XML-based Stand-off Representation and Exploitation of Multi-Level Linguistic Annotation.

A Flexible Framework for Integrating Annotations from Different Tools and Tag Sets

Rule-Based Normalization of Historical Texts

Annotation of Information Structure: an Evaluation across different Types of Texts