Showing papers on "Question answering published in 2001"

Discovery of inference rules for question-answering

Abstract: One of the main challenges in question-answering is the potential mismatch between the expressions in questions and the expressions in texts. While humans appear to use inference rules such as ‘X writes Y’ implies ‘X is the author of Y’ in answering questions, such rules are generally unavailable to question-answering systems due to the inherent difficulty in constructing them. In this paper, we present an unsupervised algorithm for discovering inference rules from text. Our algorithm is based on an extended version of Harris’ Distributional Hypothesis, which states that words that occurred in the same contexts tend to be similar. Instead of using this hypothesis on words, we apply it to paths in the dependency trees of a parsed corpus. Essentially, if two paths tend to link the same set of words, we hypothesize that their meanings are similar. We use examples to show that our system discovers many inference rules easily missed by humans.

Natural language question answering: the view from here

Abstract: As users struggle to navigate the wealth of on-line information now available, the need for automated question answering systems becomes more urgent. We need systems that allow a user to ask a question in everyday language and receive an answer quickly and succinctly, with sufficient context to validate the answer. Current search engines can return ranked lists of documents, but they do not deliver answers to the user.Question answering systems address this problem. Recent successes have been reported in a series of question-answering evaluations that started in 1999 as part of the Text Retrieval Conference (TREC). The best systems are now able to answer more than two thirds of factual questions in this evaluation.

Scaling question answering to the Web

Abstract: The wealth of information on the web makes it an attractive resource for seeking quick answers to simple, factual questions such as \who was the rst American in space?" or \what is the second tallest mountain in the world?" Yet today's most advanced web search services (e.g., Google and AskJeeves) make it surprisingly tedious to locate answers to such questions. In this paper, we extend question-answering techniques, rst studied in the information retrieval literature, to the web and experimentally evaluate their performance. First we introduce Mulder, which we believe to be the rst general-purpose, fully-automated questionanswering system available on the web. Second, we describe Mulder's architecture, which relies on multiple search-engine queries, natural-language parsing, and a novel voting procedure to yield reliable answers coupled with high recall. Finally, we compare Mulder's performance to that of Google and AskJeeves on questions drawn from the TREC-8 question track. We nd that Mulder's recall is more than a factor of three higher than that of AskJeeves. In addition, we nd that Google requires 6.6 times as much user e ort to achieve the same level of recall as Mulder.

Scaling question answering to the web

Abstract: The wealth of information on the web makes it an attractive resource for seeking quick answers to simple, factual questions such as “who was the first American in space?” or “what is the second tallest mountain in the world?” Yet today's most advanced web search services (e.g., Google and AskJeeves) make it surprisingly tedious to locate answers to such questions. In this paper, we extend question-answering techniques, first studied in the information retrieval literature, to the web and experimentally evaluate their performance.First we introduce Mulder, which we believe to be the first general-purpose, fully-automated question-answering system available on the web. Second, we describe Mulder's architecture, which relies on multiple search-engine queries, natural-language parsing, and a novel voting procedure to yield reliable answers coupled with high recall. Finally, we compare Mulder's performance to that of Google and AskJeeves on questions drawn from the TREC-8 question answering track. We find that Mulder's recall is more than a factor of three higher than that of AskJeeves. In addition, we find that Google requires 6.6 times as much user effort to achieve the same level of recall as Mulder.

Data-intensive Question Answering

Abstract: Utilisation de la redondance des reponses elles-memes pour ameliorer le resultat final de la recherche d'information- redondance due a la tres grande quantite d'informations disponibles actuellement

Overview of the TREC 2001 Question Answering Track

Abstract: The TREC question answering track is an effort to bring the benefits of large-scale evaluation to bear on the question answering problem. In its third year, the track continued to focus on retrieving small snippets of text that contain an answer to a question. However, several new conditions were added to increase the realism, and the difficulty, of the task. In the main task, questions were no longer guaranteed to have an answer in the collection; systems returned a response of 'NIL' to indicate their belief that no answer was present. In the new list task, systems assembled a set of instances as the response for a question, requiring the ability to distinguish among instances found in multiple documents. Another new task, the context task, required systems to track discourse objects through a series of questions.

Exploiting redundancy in question answering

Abstract: Our goal is to automatically answer brief factual questions of the form ``When was the Battle of Hastings?'' or ``Who wrote The Wind in the Willows?''. Since the answer to nearly any such question can now be found somewhere on the Web, the problem reduces to finding potential answers in large volumes of data and validating their accuracy. We apply a method for arbitrary passage retrieval to the first half of the problem and demonstrate that answer redundancy can be used to address the second half. The success of our approach depends on the idea that the volume of available Web data is large enough to supply the answer to most factual questions multiple times and in multiple contexts. A query is generated from a question and this query is used to select short passages that may contain the answer from a large collection of Web data. These passages are analyzed to identify candidate answers. The frequency of these candidates within the passages is used to ``vote'' for the most likely answer. The approach is experimentally tested on questions taken from the TREC-9 question-answering test collection. As an additional demonstration, the approach is extended to answer multiple choice trivia questions of the form typically asked in trivia quizzes and television game shows.

The TREC question answering track

Abstract: The Text REtrieval Conference (TREC) question answering track is an effort to bring the benefits of large-scale evaluation to bear on a question answering (QA) task. The track has run twice so far, first in TREC-8 and again in TREC-9. In each case, the goal was to retrieve small snippets of text that contain the actual answer to a question rather than the document lists traditionally returned by text retrieval systems. The best performing systems were able to answer about 70% of the questions in TREC-8 and about 65% of the questions in TREC-9. While the 65% score is a slightly worse result than the TREC-8 scores in absolute terms, it represents a very significant improvement in question answering systems. The TREC-9 task was considerably harder than the TREC-8 task because TREC-9 used actual users’ questions while TREC-8 used questions constructed for the track. Future tracks will continue to challenge the QA community with more difficult, and more realistic, question answering tasks.

Patterns of Potential Answer Expressions as Clues to the Right Answers.

Abstract: The core of our question-answering mechanism is searching for predefined patterns of textual expressions that may be interpreted as answers to certain types of questions The presence of such patterns in analyzed answer-string candidates may provide evidence of the right answer The answer-string candidates are created by cutting up relatively-large source documents passages containing the query terms or their synonyms/substitutes indicative patterns The specificity of our approach is: placing the use of indicative patterns in the core of the QA approach; aiming at the comprehensive and systematic use of such indicators; defining various structural types of the indicative patterns, including nontrivial and sophisticated ones; developing accessory techniques that ensure effective performance of the approach We believe that the use of indicative patterns for question answering can be considered as a special case of the more general approach to text information retrieval that contrasts with linguisticsoriented methodology Introduction We decided to participate in the TREC-10 Question Answering track with a purpose to test certain specific features of the text processing technology we are developing in the framework of our CrossReader project This technology is aimed at presenting to a user the needed information directly, ie instead of documents, or sources containing potentially relevant information The query-relevant sentences or short passages are extracted from the processed documents and judiciously arranged; so, new full texts emerge that are focused precisely on the user's subject (Subbotin, 1993; Gilyarevskii, 1993; Perez, 2001) The latest version of this technology the TextRoller system uses not only key words, but also positive and negative patterns for choosing and arranging text items For the TREC-10 Question Answering task we have developed a variant of our basic technology that searches for candidate answers using key words (from the question text) and chooses the most probable answer using patterns The participation at TREC-10 was a test for some basic mechanisms of our technology Now, after this test was successfully passed, these mechanisms will be implemented in the new TextRoller versions Basic Features of the Applied Approach It seems that many systems participating at TREC QA track represent the question (or its reformulations) as a set of entities and relations between them in order to compare these entities and relations with those of candidate answers texts; the answer candidate that correlates at the highest degree with the question text gets the highest score By contrast, our QA system checks the answer candidates for the presence of certain predefined indicators (patterns) to which scores were assigned beforehand, ie independently of the question text analysis Candidate snippets containing the highest-scored indicators are chosen as final answers It is obvious from the above that the applied approach does not require NLP or knowledge-based analysis of the question text This text is considered as just a string consisting of various substrings These are used, first of all, for composing queries helping to retrieve passages containing answer candidates If present in candidate answers texts, they are considered as a condition of applicability of a given indicative pattern for a given question, but they do not influence the score of the pattern (as said above, it is predefined beforehand) The efficiency of this approach depends on the quantity and diversification of predefined indicative patterns as well as on the recall of passages containing candidate answers We could not rely on the presence of predefined patterns in the texts of candidate answers for every question If case of neither pattern was found, the system used the more common way to choose among candidate answers basing on lexical similarity between the question and an answer snippet From 289 answer strings that were correct responses 193 did contain the patterns Non-matching any patterns, but containing question (query) terms were 64 Other (containing minor indicators, such as capitalized words, or randomly selected) 32 To some extent, many QA-Track participants (at TRECs 8 and 9) had used what we call the indicative patterns The specificity of our approach is: placing the use of indicative patterns in the core of the QA approach; aiming at the comprehensive and systematic use of such indicators; defining various structural types of the indicative patterns, including nontrivial and sophisticated ones; developing accessory techniques that ensure effective performance of the approach In (Sanda Harabagiu et al, 2000) the term "definition patterns" was introduced as "associated with questions that inquire about definitions" This kind of patterns was widely used by our QA system, although in many cases they were effective in combination with some additional indicators (see section "How patterns work") It is also noteworthy that we did not confine the use of these patterns to questions inquiring about definitions We assume, in general, that there should not be one-to-one correspondence between a given pattern and a question type The same pattern can be applicable in answering many types of questions (getting a different score for each question type) The Library of Indicative Patterns The indicative patterns used by our QA system are sequences or combinations of certain string elements, such as letters, punctuation marks, spaces, tokens (such as "&", "%", or "$"), digits, and words/phrases that are accumulated in special lists (see Fig 1) Fig 1 The general approach The way we defined indicative patterns is totally heuristic and inductive At the initial stage the indicative patterns lists are accumulated basing on expressions that can be interpreted as answers to the questions of a definite type For example: "Milan, Italy" present in any text passage can be considered (completely independently from the whole sense of the passage) as an answer to the question "Where is Milan" So, a pattern for the "Where" question type may be created: "city name; comma; country name" The string "Mozart (1756-1791)" contains answers to the questions about Mozart's birth and death dates, allowing construction of the pattern: "a person's name; parenthesis; four digits; dash; four digits; parenthesis " We studied texts systematically with the purpose of identifying expressions that may serve as models for answer patterns Some patterns components can be used for searching more complex structure patterns The validity of a pattern for a given question type (and its score) can be tested in large text corpora The library of patterns can never be complete Identifying patterns while studying text corpora is a research field by itself, accumulating special knowledge on cooccurencies of text elements (characters, strings, as well as definite classes of characters and strings) So, it can be found that the string "Mr " at a certain frequency level precedes one or two capitalized words, and the string "Jr" follows such words, etc Thus, we can accumulate the knowledge on "typical" combinations and correlations of strings that correspond to personal names, to a persons age, to locations, dates, activities, etc This requires the use of sophisticated tools and special methods This knowledge area can become important not only for QA, but also for other text retrieval tasks For example, we use such methodology for extracting and ordering of sentences resulting in a coherent description of the requested subject The Structure of Indicative Patterns A pattern may include a constant part and a variable part The latter can be represented by a query term or even an unknown term (the answer word/phrase proper that occupies a definite position in the sequence of pattern elements) We distinguish between two pattern categories: the first represents a complete structure while the second is a composite structure of specific pattern elements (see above) For TREC-10 we had prepared 51 lists of various patterns elements; for each question category 5 15 of such lists were applied for recognition of potential answers (see Fig 2)

Toward semantics-based answer pinpointing

Abstract: We describe the treatment of questions (Question-Answer Typology, question parsing, and results) in the Weblcopedia question answering system.

High performance question/answering

Abstract: In this paper we present the features of a Question/Answering (Q/A) system that had unparalleled performance in the TREC-9 evaluations. We explain the accuracy of our system through the unique characteristics of its architecture: (1) usage of a wide-coverage answer type taxonomy; (2) repeated passage retrieval; (3) lexico-semantic feedback loops; (4) extraction of the answers based on machine learning techniques; and (5) answer caching. Experimental results show the effects of each feature on the overall performance of the Q/A system and lead to general conclusions about Q/A from large text collections.

Parsing and question classification for question answering

Abstract: This paper describes machine learning based parsing and question classification for question answering. We demonstrate that for this type of application, parse trees have to be semantically richer and structurally more oriented towards semantics than what most treebanks offer. We empirically show how question parsing dramatically improves when augmenting a semantically enriched Penn treebank training corpus with an additional question treebank.

Identifying Collocations for Recognizing Opinions

Abstract: Subjectivity in natural language refers to aspects of language used to express opinions and evaluations (Banfield, 1982; Wiebe, 1994). There are numerous applications for which knowledge of subjectivity is relevant, including genre detection, information extraction, and information retrieval. This paper shows promising results for a straightforward method of identifying collocational clues of subjectivity, as well as evidence of the usefulness of these clues for recognizing opinionated documents.

Logic Form Transformation of WordNet and its Applicability to Question Answering

Abstract: WordNet is a rich source of world knowledge from which formal axioms can be derived. In this paper we present a method for transforming the WordNet glosses into logic forms and further into axioms. The transformation of WordNet glosses into logic forms is useful for theorem proving and other applications. The paper demonstrates the utility of the WordNet axioms in a question answering system to rank and extract answers.

Learning search engine specific query transformations for question answering

Abstract: We introduce a method for learning query transformations that improves the ability to retrieve answers to questions from an information retrieval system. During the training stage the method involves automatically learning phrase features for classifying questions into different types, automatically generating candidate query transformations from a training set of question/answer pairs, and automatically evaluating the candidate transforms on target information retrieval systems such as real-world general purpose search engines. At run time, questions are transformed into a set of queries, and re-ranking is performed on the documents retrieved. We present a prototype search engine, Tritus, that applies the method to web search engines. Blind evaluation on a set of real queries from a web search engine log shows that the method significantly outperforms the underlying web search engines as well as a commercial search engine specializing in question answering.

The use of external knowledge in factoid QA

Abstract: This paper describes recent development in the Webclopedia QA (Question Answering) system, focusing on the use of knowledge resources such as WordNet and a QA typology to improve the basic operations of candidate answer retrieval, ranking, and answer matching.

IBM's Statistical Question Answering system: TREC-10

Abstract: We describe herein the IBM Statistical Question Answering system for TREC-10 in detail. Based on experiences in TREC-9, we have adapted our system to deal with definition type questions and furthermore completed the trainability aspect of our question-answering system. The experiments performed in this evaluation confirmed our hypothesis that post-processing the IR engine results can achieve the same performance as incorporating query expansion terms into the retrieval engine.

Analyses for elucidating current question answering technology

Abstract: In this paper, we take a detailed look at the performance of components of an idealized question answering system on two different tasks: the TREC Question Answering task and a set of reading comprehension exams. We carry out three types of analysis: inherent properties of the data, feature analysis, and performance bounds. Based on these analyses we explain some of the performance results of the current generation of Q/A systems and make predictions on future work. In particular, we present four findings: (1) Q/A system performance is correlated with answer repetition; (2) relative overlap scores are more effective than absolute overlap scores; (3) equivalence classes on scoring functions can be used to quantify performance bounds; and (4) perfect answer typing still leaves a great deal of ambiguity for a Q/A system because sentences often contain several items of the same type.

A Survey of Concept-based Information Retrieval Tools on the Web

Abstract: . In order to solve the problem of information overkill on the web current information retrieval tools need to be improved. Much more "intelligence" should be embedded to search tools to manage effectively search, retrieval, filtering and presenting relevant information. This can be done by concept-based (or ontology driven) information retrieval, which is considered as one of the high-impact technologies for the next ten years. Nevertheless, most of commercial products of search and retrieval category do not report about concept-based search features. The paper provides an overview of conceptbased information retrieval techniques and software tools currently available as prototypes or commercial products. Tools are evaluated using feature classification, which incorporates general characteristics of tools and their information retrieval features.

SiteQ: Engineering high performance QA system using lexico-semantic pattern matching and shallow NLP

Abstract: In TREC-10, we participated in the web track (only ad-hoc task) and the QA track (only main task). In the QA track, our QA system (SiteQ) has general architecture with three processing steps: question processing, passage selection and answer processing. The key technique is LSP's (Lexico-Semantic Patterns) that are composed of linguistic entries and semantic types. LSP grammars constructed from various resources are used for answer type determination and answer matching. We also adapt AAD (Abbreviafion-Appositive-Definition) processing for the queries that answer type cannot be determined or expected, encyclopedia search for increasing the matching coverage between query terms and passages, and pivot detection for the distance calculation with answer candidates. We used two-level answer types consisted of 18 upper-level types and 47 lower-level types. Semantic category dictionary, WordNet, POS combined with lexicography and a stemmer were all applied to construct the LSP knowledge base. CSMT (Category Sense-code Mapping Table) fried to find answer types using the matching between semantic categories and sense-codes from WordNet. Evaluation shows that MRR for 492 questions is 0.320 (strict), which is considerably higher than the average MRR of other 67 runs.- In the Web track, we focused on the effectiveness of both noun phrase extraction and our new PRF (Pseudo Relevance Feedback). We confirmed that our query expansion using PRF with TSV function adapting TF factor contributed to better performance, but noun phrases did not contribute much. It needs more observations for us to make elaborate rules of tag patterns for the construction of better noun phrases.

Natural Language Processing and User Modeling: Synergies and Limitations

Abstract: The fields of user modeling and natural language processing have been closely linked since the early days of user modeling. Natural language systems consult user models in order to improve their understanding of users' requirements and to generate appropriate and relevant responses. At the same time, the information natural language systems obtain from their users is expected to increase the accuracy of their user models. In this paper, we review natural language systems for generation, understanding and dialogue, focusing on the requirements and limitations these systems and user models place on each other. We then propose avenues for future research.

Question answering in TREC

Abstract: Traditional text retrieval systems return a ranked list of documents in response to a user's request. While a ranked list of documents can be an appropriate response for the user, frequently it is not. Usually it would be better for the system to provide the answer itself instead of requiring the user to search for the answer in a set of documents. The Text REtrieval Conference (TREC) is sponsoring a question answering "track" to foster research on the problem of retrieving answers rather than document lists.TREC is a workshop series sponsored by the National Institute of Standards and Technology and the U.S. Department of Defense [7]. The purpose of the conference series is to encourage research on text retrieval for realistic applications by providing large test collections, uniform scoring procedures, and a forum for organizations interested in comparing results. The conference has focused primarily on the traditional IR problem of retrieving a ranked list of documents in response to a statement of information need, but has also included other tasks, called tracks, that focus on new areas or particularly difficult aspects of information retrieval. A question answering track was introduced in TREC-8 1999. The track has generated wide-spread interest in the QA problem [2, 3, 4], and has documented significant improvements in question answering system effectiveness in its two-year history.This paper provides a brief summary of the findings of the TREC question answering track to date and discusses the future directions of the track. The paper is extracted from a fuller description of the track given in "The TREC Question Answering Track" [8]. Complete details about the TREC question answering track can be found in the TREC proceedings.

The Role of Lexico-Semantic Feedback in Open-Domain Textual Question-Answering

Abstract: This paper presents an open-domain textual Question-Answering system that uses several feedback loops to enhance its performance. These feedback loops combine in a new way statistical results with syntactic, semantic or pragmatic information derived from texts and lexical databases. The paper presents the contribution of each feedback loop to the overall performance of 76% human-assessed precise answers.

Answering complex, list and context questions with LCC's Question-Answering Server

Abstract: This paper presents the architecture of the Question-Answering server (QAS) developed at the Language Computer Corporation (LCC) and used in the TREC-10 evaluations.

Mining the web for answers to natural language questions

Abstract: The web is now becoming one of the largest information and knowledge repositories. Many large scale search engines (Google, Fast, Northern Light, etc.) have emerged to help users find information. In this paper, we study how we can effectively use these existing search engines to mine the Web and discover the "correct" answers to factual natural language questions.We propose a probabilistic algorithm called QASM (Question Answering using Statistical Models) that learns the best query paraphrase of a natural language question. We validate our approach for both local and web search engines using questions from the TREC evaluation. We also show how this algorithm can be combined with another algorithm (AnSel) to produce precise answers to natural language questions.

Building a statistical machine translation system from scratch: how much bang for the buck can we expect?

Abstract: We report on our experience with building a statistical MT system from scratch, including the creation of a small parallel Tamil-English corpus, and the results of a task-based pilot evaluation of statistical MT systems trained on sets of ca. 1300 and ca. 5000 parallel sentences of Tamil and English data. Our results show that even with apparently incomprehensible system output, humans without any knowledge of Tamil can achieve performance rates as high as 86% accuracy for topic identification, 93% recall for document retrieval, and 64% recall on question answering (plus an additional 14% partially correct answers).

High-performance, open-domain question answering from large text collections

Abstract: Vast amounts of information, covering virtually every topic of interest, are now electronically accessible in the form of internal text databases of commercial institutions, encyclopedia, newswire services, or in the form of the unstructured, continuously growing World Wide Web. Situated at the frontier of information retrieval and natural language processing, open-domain question answering is a challenging task, involving the extraction of brief, relevant answer strings from large text collections, in response to users' questions. The design of novel, robust models for capturing the semantics of natural language questions, finding relevant text snippets, and selecting the most relevant answer when several candidates have been identified, is essential for high-precision question answering. A relational representation encodes lexical, relational and semantic information in an integrated model applying to both questions and candidate answers. The representation impacts all stages of question answering, including question processing—when the category of the expected answers is detected, passage retrieval—when relevant passages are identified in the text collection, and answer extraction—when the actual answers are found. The theoretical contributions of the thesis are reflected in a fully-implemented architecture, whose performance was evaluated within the Question Answering track of the DARPA-sponsored Text REtrieval Conference (TREC). The theoretical concepts developed in the thesis are instrumental to the extraction of correct answers as response to a test set of 893 fact-seeking questions from a 3 Gigabyte text collection. Experimental results also show important qualitative improvements with respect to output from Web search engines, and unveil some of the challenges and desired features of next-generation text search technologies.

Japanese Named Entity Recognition based on a Simple Rule Generator and Decision Tree Learning

Abstract: Named entity (NE) recognition is a task in which proper nouns and numerical information in a document are detected and classified into categories such as person, organization, location, and date. NE recognition plays an essential role in information extraction systems and question answering systems. It is well known that hand-crafted systems with a large set of heuristic rules are difficult to maintain, and corpus-based statistical approaches are expected to be more robust and require less human intervention. Several statistical approaches have been reported in the literature. In a recent Japanese NE workshop, a maximum entropy (ME) system outperformed decision tree systems and most hand-crafted systems. Here, we propose an alternative method based on a simple rule generator and decision tree learning. Our experiments show that its performance is comparable to the ME approach. We also found that it can be trained more efficiently with a large set of training data and that it improves readability.

Question answering system and storage medium

Abstract: A question answering system includes a storage for storing question information together with first time information, a storage for storing answer information with respect to the question information together with solver information and second time information, and an evaluating part for evaluating a solver indicated by the solver information based on the first and second time information.