A long-standing challenge for computers communicating with humans is to pass the Turing test, i.e., to communicate in such a way that it is impossible for humans to determine whether they are talking to a computer or another human being. The field of natural language understanding — which studies automatic means of capturing the semantics of textual content — plays a central part in this long-term goal of artificial intelligence research. Natural language understanding can itself be understood at different levels. In this thesis, we make contributions to automatic understanding of text at the level of words, short texts, and full documents. Understanding texts at the word level, means understanding how words relate to each other semantically. For example, do two words or phrases mean approximately the same thing? Does a particular word still mean the same thing it used to, say, 50 years ago? Or, as is the question central to the first part of this thesis, can we automatically detect which words people used in different periods in time to refer to a particular concept? The question we ask ourselves in the second part of this thesis is: can we automatically determine, from the word-level up, if two sentences have a similar meaning? Finally, in the third part of this thesis, document-level text understanding is the focus of our interest. In particular, we study multiple approaches to the reading comprehension task, where a computer reads a document and answers questions about it.

background

xTAS and ThemeStreams

Text understanding for computers

Text categorization is a fundamental task in document processing, allowing the automated handling of enormous streams of documents in electronic form. One difficulty in handling some classes of documents is the presence of different kinds of textual errors, such as spelling and grammatical errors in email, and character recognition errors in documents that come through OCR. Text categorization must work reliably on all input, and thus must tolerate some level of these kinds of problems. We describe here an N-gram-based approach to text categorization that is tolerant of textual errors. The system is small, fast and robust. This system worked very well for language classification, achieving in one test a 99.8% correct classification rate on Usenet newsgroup articles written in different languages. The system also worked reasonably well for classifying articles from a number of different computer-oriented newsgroups according to subject, achieving as high as an 80% correct classification rate. There are also several obvious directions for improving the system`s classification performance in those cases where it did not do as well. The system is based on calculating and comparing profiles of N-gram frequencies. First, we use the system to compute profiles on training set data that represent the variousmore » categories, e.g., language samples or newsgroup content samples. Then the system computes a profile for a particular document that is to be classified. Finally, the system computes a distance measure between the document`s profile and each of the category profiles. The system selects the category whose profile has the smallest distance to the document`s profile. The profiles involved are quite small, typically 10K bytes for a category training set, and less than 4K bytes for an individual document. Using N-gram frequency profiles provides a simple and reliable way to categorize documents in a wide range of classification tasks.« less

/pdf/n-gram-based-text-categorization-2g56luruq3.pdf

N-gram-based text categorization

In February 2008, the New York Times published an unusual chart of box office revenues for 7500 movies over 21 years. The chart was based on a similar visualization, developed by the first author, that displayed trends in music listening. This paper describes the design decisions and algorithms behind these graphics, and discusses the reaction on the Web. We suggest that this type of complex layered graph is effective for displaying large data sets to a mass audience. We provide a mathematical analysis of how this layered graph relates to traditional stacked graphs and to techniques such as ThemeRiver, showing how each method is optimizing a different ldquoenergy functionrdquo. Finally, we discuss techniques for coloring and ordering the layers of such graphs. Throughout the paper, we emphasize the interplay between considerations of aesthetics and legibility.

/pdf/stacked-graphs-geometry-aesthetics-3xfq8leqvp.pdf

xTAS and ThemeStreams

Citations

Text understanding for computers

References

N-gram-based text categorization

Stacked Graphs – Geometry & Aesthetics

Related Papers (2)

Zwevende of geemancipeerde kiezers

Moord of Stervenshulp