Showing papers presented at "Cross-Language Evaluation Forum in 2002"

Evaluation of cross-language information retrieval systems

Abstract: Die Online-Fachbuchhandlung beck-shop.de ist spezialisiert auf Fachbücher, insbesondere Recht, Steuern und Wirtschaft. Im Sortiment finden Sie alle Medien (Bücher, Zeitschriften, CDs, eBooks, etc.) aller Verlage. Ergänzt wird das Programm durch Services wie Neuerscheinungsdienst oder Zusammenstellungen von Büchern zu Sonderpreisen. Der Shop führt mehr als 8 Millionen Produkte.

Cross-Language Retrieval Experiments at CLEF 2002

Abstract: This paper describes monolingual, bilingual, and multilingual retrieval experiments using CLEF 2002 test collection. The paper presents a technique for incorporating blind relevance feedback into a document ranking formula based on logistic regression analysis. Both blind relevance feedback and decompounding in German or Dutch are shown to be effective in monolingual and bilingual retrieval. The amount of improvement of performance by decompounding varies from one set of topics to another. The simple raw-score merging strategy in multilingual retrieval can be effective if the individual ranked lists of documents, one for each document language, are produced using the same retrieval system under similar conditions. The performance of English to French bilingual retrieval using a large parallel corpus as the translation resource is comparable to that using machine translation systems.

Report on CLEF 2002 Experiments: Combining Multiple Sources of Evidence

Abstract: In our second participation in the CLEF retrieval tasks, our first objective was to propose better and more general stopword lists for various European languages (namely, French, Italian, German, Spanish and Finnish) along with improved, simpler and efficient stemming procedures. Our second goal was to propose a combined query-translation approach that could cross language barriers and also an effective merging strategy based on logistic regression for accessing the multilingual collection. Finally, within the Amaryllis experiment, we wanted to analyze how a specialized thesaurus might improve retrieval effectiveness.

Linguistic and Statistical Analysis of the CLEF Topics

Abstract: This paper reports on an analysis of the CLEF 2001 topics. In particular, we investigated potential correlations between features of the topics and the performance of retrieval systems. Although there are some weak relations, we claim that the properties of the CLEF topics do not influence the results of the retrieval systems. We found just one correlation for the English topics. The more linguistic challenges contained in the topic texts, the better the systems performed. However, no correlation for the length of a topic could be found.

Merging Mechanisms in Multilingual Information Retrieval

Abstract: This paper considers centralized and distributed architectures for multilingual information retrieval. Several merging strategies, including raw-score merging, round-robin merging, normalized-score merging, and normalized-by-top-k merging, were investigated. The effects of translation penalty on merging was also examined. The experimental results show that the centralized approach is better than the distributed approach. In the distributed approach, the normalized-by-top-k merging with translation penalty outperforms other merging strategies, except for raw-score merging. Because the performances of English to other languages are similar, raw-score merging gives better performance in our experiments. However, raw-score merging is not workable in practice if different IR systems are adopted.

CLEF 2002 — Overview of Results

Abstract: In its third year, the CLEF campaign has again seen considerable growth on multiple fronts. While the explosive growth in the number of participants has slowed somewhat, the number of actual experiments has grown considerably, as has their complexity (more data to process and more languages to handle). The main tracks of the CLEF 2002 campaign attracted 37 participating groups who submitted nearly 300 different experiments. In this overview, a description of the tracks and tasks, and a summary of the principal research results are given. As for the last two years, we have also examined the multilingual test collection produced as a result of the campaign with respect to the completeness of its relevance assessments, with very favorable findings.

The CLEF 2002 Interactive Track

Abstract: In the CLEF 2002 Interactive Track, research groups interested in the design of systems to support interactive Cross-Language Retrieval used a shared experiment design to explore aspects of that question. Participating teams each compared two systems, both supporting a full retrieval task where users had to select relevant documents given a (native language) topic and a (foreign language) document collection. The two systems being compared at each site should differ in (at least) one of these aspects: a) support for document selection (how the system describes the content of a document written in a foreign language), b) support for query translation (how the system interacts with the user in order to obtain an optimal translation of the query), and c) support for query refinement (how the system helps the user refine their query based on previous search results). This paper describes the shared experiment design and summarizes preliminary results from the five teams that submitted runs.

SICS at CLEF 2002: Automatic query expansion using Random indexing

Abstract: Vector space techniques can be used for extracting semantically similar words from the co-occurrence statistics of words in large text data collections. We have used a technique called Random Indexing to accumulate context vectors for Swedish, French and Italian. We have then used the context vectors to perform automatic query expansion. In this paper, we report on our CLEF 2002 experiments on Swedish, French and Italian monolingual query expansion.

Scalable Multilingual Information Access

Abstract: The third Cross-Language Evaluation Forum workshop (CLEF-2002) provides the unprecedented opportunity to evaluate retrieval in eight different languages using a common set of topics and a uniform assessment methodology. This year the Johns Hopkins University Applied Physics Laboratory participated in the monolingual, bilingual, and multilingual retrieval tasks. We contend that information access in a plethora of languages requires approaches that are inexpensive in developer and run-time costs. In this paper we describe a simplified approach that seems suitable for retrieval in many languages; we also show how good retrieval is possible over many languages, even when translation resources are scarce, or when query-time translation is infeasible. In particular, we investigate the use of character n-grams for monolingual retrieval, CLIR between related languages using partial morphological matches, and translation of document representations to an interlingua for computationally efficient retrieval against multiple languages.

Experiments in 8 European Languages with Hummingbird SearchServer™ at CLEF 2002

Abstract: Hummingbird submitted ranked result sets for all Monolingual Information Retrieval tasks of the Cross-Language Evaluation Forum (CLEF) 2002. Enabling stemming in SearchServer increased average precision by 16 points in Finnish, 9 points in German, 4 points in Spanish, 3 points in Dutch, 2 points in French and Italian, and 1 point in Swedish and English. Accent-indexing increased average precision by 3 points in Finnish and 2 points in German, but decreased it by 2 points in French and 1 point in Italian and Swedish. Treating apostrophes as word separators increased average precision by 3 points in French and 1 point in Italian. Confidence intervals produced using the bootstrap percentile method were found to be very similar to those produced using the standard method; both were of similar width to rank-based intervals for differences in average precision, but substantially narrower for differences in Precision@10.

Resolving Translation Ambiguity Using Monolingual Corpora

Abstract: In the bilingual track of CLEF 2002, focusing on word translation ambiguity, we experimented with several techniques for choosing the best target translation for each source query word by using co-occurrence statistics in a reference corpus consisting of documents in the target language. Our techniques give one best translation per source query word. We also experimented with combining these word choice results in the final translated query. The source query languages were Spanish, Chinese, and Japanese; the target language documents were in English. For Spanish-to-English retrieval, the best recall and average precision of Spanish-to-English retrieval reached 95% and 97%, respectively, of the recall and average precision of an English monolingual retrieval run. For Chinese-to-English text retrieval, the recall and average precision reached 89% and 60%, respectively, of the English run. For Japanese-to-English text retrieval, the best recall and average precision reached 82% and 69%, respectively, of the English run.

SINAI at CLEF 2002: Experiments with Merging Strategies

Abstract: For our first participation in the CLEF multilingual task, we present a new approach to obtain a single list of relevant documents for CLIR systems based on query translation. This new approach, which we call two-step RSV, is based on the re-indexing of the retrieval documents according to the query vocabulary, and it performs noticeably better than traditional methods1.

Eurospider at CLEF 2002

Abstract: For the CLEF 2002 campaign, Eurospider participated in the multilingual and German monolingual tasks. Our main focus was on trying new merging strategies for our multilingual experiments. In this paper, we describe the setup of our system, the characteristics of our experiments, and give an analysis of the results.

Improving the Automatic Retrieval of Text Documents

Abstract: This paper reports on a statistical stemming algorithm based on link analysis. Considering that a word is formed by a prefix (stem) and a suffix, the key idea is that the interlinked prefixes and suffixes form a community of sub-strings. Thus, discovering these communities means searching for the best word splits that give the best word stems. The algorithm has been used in our participation in the CLEF 2002 Italian monolingual task. The experimental results show that stemming improves text retrieval effectiveness. They also show that the effectiveness level of our algorithm is comparable to that of an algorithm based on a-priori linguistic knowledge.

Italian Monolingual Information Retrieval with PROSIT

Abstract: PROSIT (PRObabilistic Sifting of Information Terms) is a novel probabilistic information retrieval system that combines a term-weighting model based on deviation from randomness with information-theoretic query expansion. We report on the application of PROSIT to the Italian monolingual task at CLEF. We experimented with both standard PROSIT and with enhanced versions. In particular, we studied the use of bigrams and coordination level-based retrieval within the PROSIT framework. The main findings of our research are that (i) standard PROSIT was quite effective, with an average precision of 0.5116 on CLEF 2001 queries and 0.5019 on CLEF 2002 queries, (ii) bigrams were useful provided that they were incorporated into the main algorithm, and (iii) the benefits of coordination level-based retrieval were unclear.

A Multilingual Approach to Multilingual Information Retrieval

Abstract: Multilingual IR is usually carried out by first performing cross-language IR on separate collections, each for a language. Once a set of answers has been found in each language, all the sets are merged to produce a unique answer list. In our experiments of CLEF2002, we propose a truly multilingual approach in which the documents in different languages are mixed in the same collection. Indexes are associated with a language tag so as to distinguish homographs in different languages. The indexing and retrieval processes can then be done once for all the languages. No result merging is required. This paper describes our first tests in CLEF2002.

Interactive Cross-Language Searching: Phrases Are Better than Terms for Query Formulation and Refinement

Abstract: This paper summarizes the participation of the UNED group in the CLEF 2002 Interactive Track. We focused on interactive query formulation and refinement, comparing two approaches: a) a reference system that assists the user to provide adequate translations for terms in the query; and b) a proposed system that assists the user to formulate the query as a set of relevant phrases, and to select promising phrases in the documents to enhance the query. All collected evidence indicates that the phrase-based approach is preferable: the official Fα=0.8 measure is 65% better for the proposed system, and all users in our experiment preferred the phrase-based system as a simpler and faster way of searching.

Cross-Language Spoken Document Retrieval on the TREC SDR Collection

Abstract: This paper presents preliminary experiments on cross-language spoken document retrieval (SDR) carried out on a benchmark assembled at ITC-irst. The benchmark is based on resources used in the last two spoken document retrieval tracks at the TREC conference, which are available on the Internet. They include automatic transcripts of American English broadcast news, short topics written in English, and relevance assessments. The extension from monolingual to cross-language SDR was obtained by translating all topics into five European languages: Dutch, French, German, Italian, and Spanish. In this paper preliminary experiments on the last four languages are presented. Translations of the topics will be used to run a pilot track in CLEF 2003.

CLEF 2002 Cross-Language Spoken Document Retrieval Pilot Track Report

Abstract: The current expansion in collections of natural language based digital documents in various media and languages is creating challenging opportunities for automatically accessing the information contained in these documents. This paper describes the CLEF 2002 pilot track investigation of Cross-Language Spoken Document Retrieval (CLSDR) combining information retrieval, cross-language translation and speech recognition. The experimental investigation is based on the TREC-8 and TREC-9 SDR evaluation tasks, augmented to form a CLSDR task. The original task of retrieving English language spoken documents using English request topics is compared with cross-language retrieval using French, German, Italian and Spanish topic translations. The results of the pilot track establish baseline performance levels and indicate that pseudo relevance feedback and contemporaneous text document collections can be used to improve CLSDR performance.

IR-n System at CLEF-2002

Abstract: Passage Retrieval is an alternative to traditional document-oriented Information Retrieval. These systems use contiguous text fragments (or passages) instead of full documents as the basic unit of information. The IR-n system is a passage retrieval system that uses groups of contiguous sentences as units of information. This paper reports on experiments with the IR-n system at CLEF-2002 where it has obtained considerably better results than in the previous participation in CLEF-2001.

Combining Evidence for Cross-Language Information Retrieval

Abstract: This paper describes the official runs of our team for CLEF 2002. We took part in the monolingual tasks for each of the seven non-English languages for which CLEF provides document collections (Dutch, Finnish, French, German, Italian, Spanish, and Swedish). We also conducted our first experiments for the bilingual task (English to Dutch, and English to German), and took part in the GIRT and Amaryllis tasks. Finally, we experimented with the combination of runs.

CLEF 2002 Methodology and Metrics

Abstract: We give a detailed presentation of the organization of the CLEF 2002 evaluation campaign, focusing mainly on the core tracks. This includes a discussion of the evaluation approach adopted, explanations of the tracks and tasks and the underlying motivations, a description of the test collections, and an outline of the guidelines for the participants. The paper concludes with indications of the techniques used for results calculation and analysis.

Portuguese-English Experiments Using Latent Semantic Indexing

Abstract: This paper reports the work of Middlesex University in the CLEF bilingual task. We have carried out experiments using Portuguese queries to retrieve documents in English. The approach used was Latent Semantic Indexing, which is an automatic method not requiring dictionaries or thesauri. We have also run a monolingual version of the system to work as a baseline. Here we describe in detail the methods used and give an analysis of the results obtained.

Using Thesauri in Cross-Language Retrieval of German and French Indexed Collections

Abstract: For CLEF 2002, Berkeley’s Group One experimented with Russian, French and English as query languages, and investigated thesaurus-aided retrieval for the special CLEF collections GIRT and Amaryllis. Two techniques were used to locate source language topic terms within the controlled vocabulary and replace them with the document language thesaurus terms to form the query sent against the collection index. This form of controlled vocabulary-aided translation is called thesaurus matching. Results show that thesaurus-aided cross-language retrieval performs slightly worse than machine translation retrieval on average, but can yield decidedly better results for particular queries. In addition, Berkeley submitted runs to the monolingual and bilingual (French and German) CLEF main tasks. We found that bilingual retrieval sometimes outperforms monolingual retrieval and postulate reasons to explain this phenomenon.

Assessing Automatically Extracted Bilingual Lexicons for CLIR in Vertical Domains: XRCE Participation in the GIRT Track of CLEF 2002

Abstract: In this paper, we describe the approach we used in the Cross-Language Evaluation Forum CLEF 2002, and more specifically in the GIRT Task. The approach is based on (1) the extraction of two bilingual lexicons, one from parallel corpora and the other one from comparable corpora, (2) the optimal combination of these bilingual lexicons for Cross-Language Information Retrieval and (3) the combination with monolingual IR on parallel corpora. While our original submission to CLEF2002 was restricted to short queries (using only the title field), we present here the results extended to complete queries.

Experiments in Term Expansion Using Thesauri in Spanish

Abstract: This paper presents some experiments carried out this year in the Spanish monolingual task at CLEF2002 The objective is to continue our research on term expansion Last year we presented results regarding stemming Now, our effort is centred on term expansion using thesauri Many words that derive from the same stem have a close semantic content However other words with very different stems also have semantically close senses In this case, the analysis of the relationships between words in a document collection can be used to construct a thesaurus of related terms The thesaurus can then be used to expand a term with the best related terms This paper describes some experiments carried out to study term expansion using association and similarity thesauri

CLIR at NTCIR Workshop 3: Cross-Language and Cross-Genre Retrieval

Abstract: This paper introduces the NTCIR Workshops, a series of evaluation workshops that are designed to enhance research in information access technologies, such as information retrieval, text summarization, question answering, information extraction, and text mining, by providing large-scale test collections and a forum for researchers A brief history and descriptions of tasks, participants, test collections and CLIR evaluation at the workshops, and a brief overview of the third NTCIR Workshop are given To conclude, some thoughts on future directions are suggested

Océ at CLEF 2002

Abstract: This report describes the work done by the Information Retrieval Group at Oce Technologies B.V., for the 2002 edition of the Cross-Language Evaluation Forum (CLEF). We participated in the mono, cross and multi-lingual tasks, using BM25 for ranking, Ergane, Logos and BabelFish for translation and the Knowledge Concepts semantic network for stemming and morphological expansion. We performed a brute-force parameter search using the 2001 topics and relevance assessments to determine for each language the pair of BM25 parameters that yielded the highest average precision. These parameters were used to query the 2002 topics.

Domain Specific Retrieval Experiments with MIMOR at the University of Hildesheim

Abstract: For our first participation in CLEF we chose the domain specific GIRT corpus We implemented the adaptive fusion model MIMOR (Multiple Indexing and Method-Object Relations) which is based on relevance feedback The linear combination of several retrieval engines was optimized As a basic retrieval engine, IRF from NIST was employed The results are promising For several topics, our runs achieved a performance above the average The optimization based on topics and relevance judgements from CLEF 2001 proved to be a fruitful strategy

UTACLIR @ CLEF 2002 — Bilingual and Multilingual Runs with a Unified Process

Abstract: The UTACLIR system of University of Tampere uses a dictionary-based CLIR approach. The idea of UTACLIR is to recognize distinct source key types and process them accordingly. The linguistic resources utilized by the framework include morphological analysis or stemming in indexing, normalization of topic words, stop word removal, splitting of compounds, translation utilizing bilingual dictionaries, handling of non-translated words, phrase composition of compounds in the target language, and constructing structured queries. UTACLIR was shown to perform consistently with different language pairs. The greatest differences in performance are due to the translation dictionary used.