Showing papers presented at "Cross-Language Evaluation Forum in 2007"

CLEF2006 question answering experiments at Tokyo Institute of Technology

Abstract: In this paper we present the experiments performed at Tokyo Institute of Technology for the CLEF2006 Multiple Language Question Answering ([email protected]) track. Our approach to QA centres on a non-linguistic, data-driven, statistical classification model that uses the redundancy of the web to find correct answers. For the cross-language aspect we employed publicly available web-based text translation tools to translate the question from the source into the corresponding target language, then used the corresponding mono-lingual QA system to find the answers. The hypothesised correct answers were then projected back on to the appropriate closed-domain corpus. Correct and supported answer performance on the mono-lingual tasks was around 14% for both Spanish and French. Performance on the cross-language tasks ranged from 5% for Spanish-English, to 12% for French-Spanish. Our method of projecting answers onto documents was shown not to work well: in the worst case on the French-English task we lost 84% of our otherwise correct answers. Ignoring the need for correct support information the exact answer accuracy increased to 29% and 21% correct on the Spanish and French mono-lingual tasks, respectively.

The University of Amsterdam at CLEF@QA 2007

Abstract: We describe the official runs of our team for the CLEF 2004 ad hoc tasks. We took part in the monolingual task (for Finnish, French, Portuguese, and Russian), in the bilingual task (for Amharic to English, and English to Portuguese), and, finally, in the multilingual task. In the CLEF 2004 evaluation exercise we participated in all three ad hoc retrieval tasks. We took part in the monolingual tasks for four non-English languages, Finnish, French, Portuguese, and Russian. The Portuguese language was new for CLEF 2004. Our participation in the monolingual task was a further continuation of our earlier efforts to monolingual retrieval [11, 5, 6]. Our first aim was to continue our experiments with a number of language-dependent techniques, in particular stemming algorithms for all European languages [14], and compound splitting for the compound rich Finnish language. A second aim was to continue our experiments with languageindependent techniques, in particular the use of character n-grams, where we may also index leading and ending character sequences, and retain the original words. Our third aim was to experiment with combinations of runs. We took part in the bilingual task, this year focusing on Amharic into English, and on English to Portuguese. Our bilingual runs were motivated by the following aims. Our first aim was to experiment with a language for which resources are few and far between, Amharic, and to see how far we could get by combining the scarcely available resources. Our second aim was to experiment with the relative effectiveness of a number of translation resources: machine translation [16] versus a parallel corpus [7], and query translation versus collection translation. Our third aim was to evaluate the effectiveness of our monolingual retrieval approaches for imperfectly translated queries, shedding light on the robustness of these approaches. Finally, we continued our participation for the multilingual task, where we experimented with straightforward ways of query translation, using machine translation whenever available, and a translation dictionary otherwise. We also experimented with combination methods using runs made on varying types of indexes. In Section 2 we describe the FlexIR system as well as the approaches used for each of the tasks in which we participated. Section 3 describes our official retrieval runs for CLEF 2004. In Section 4 we discuss the results we have obtained. Finally, in Section 5, we offer some conclusions regarding our document retrieval efforts.

CLEF2007 Image Annotation Task: an SVM-based Cue Integration Approach

Abstract: This paper presents the algorithms and results of our participation to the medical image annotation task of ImageCLEFmed 2007. We proposed, as a general strategy, a multi-cue approach where images are represented both by global and local descriptors, so to capture difierent types of information. These cues are combined during the classiflcation step following two alternative SVM-based strategies. The flrst algorithm, called Discriminative Accumulation Scheme (DAS), trains an SVM for each feature type, and considers as output of each classifler the distance from the separating hyperplane. The flnal decision is taken on a linear combination of these distances: in this way cues are accumulated, thus even when they both are misleaded the flnal result can be correct. The second algorithm uses a new Mercer kernel that can accept as input difierent feature types while keeping them separated. In this way, cues are selected and weighted, for each class, in a statistically optimal fashion. We call this approach Multi Cue Kernel (MCK). We submitted several runs, testing the performance of the single-cue SVM and of the two cue integration methods. Our team was called BLOOM (BLance∞Or-tOMed.im2) from the name of our sponsors. The DAS algorithm obtained a score of 29.9, which ranked flfth among all submissions. We submitted two versions of the MCK algorithm, one using the one-vs-all multiclass extension of SVMs and the other using the one-vs-one extension. They scored respectively 26.85 and 27.54, ranking flrst and second among all submissions.

SemEval-2007 Task 01: Evaluating WSD on Cross-Language Information Retrieval

Abstract: This paper presents a first attempt of an application-driven evaluation exercise of WSD. We used a CLIR testbed from the Cross Lingual Evaluation Forum. The expansion, indexing and retrieval strategies where fixed by the organizers. The participants had to return both the topics and documents tagged with WordNet 1.6 word senses. The organization provided training data in the form of a pre-processed Semcor which could be readily used by participants. The task had two participants, and the organizer also provide an in-house WSD system for comparison.

Overview of the WiQA Task at CLEF 2006

Abstract: We describe WiQA 2006, a pilot task aimed at studying question answering using Wikipedia. Going beyond traditional factoid questions, the task considered at WiQA 2006 was to return--given an source page from Wikipedia--to identify snippets from other Wikipedia pages, possibly in languages different from the language of the source page, that add new and important information to the source page, and that do so without repetition. A total of 7 teams took part, submitting 20 runs. Our main findings are two-fold: (i) while challenging, the tasks considered at WiQA are do-able as participants achieved impressive scores as measured in terms of yield, mean reciprocal rank, and precision, (ii) on the bilingual task, substantially higher scores were achieved than on the monolingual tasks.

A study on the use of stemming for monolingual ad-hoc Portuguese information retrieval

Abstract: For UFRGS's first participation in CLEF our goal was to compare the performance of heavier and lighter stemming strategies using the Portuguese data collections for monolingual Ad-hoc retrieval. The results show that the safest strategy was to use the lighter alternative (reducing plural forms only). On a query-by-query analysis, full stemming achieved the highest improvement but also the biggest decrease in performance when compared to no stemming. In addition, statistical tests showed that the only significant improvement in terms of mean average precision, precision at ten and number of relevant retrieved was achieved by our lighter stemmer.

IPAL at ImageClef 2007 Mixing Features, Models and Knowledge

Abstract: This paper presents IPAL ad-hoc photographic retrieval and medical image retrieval results in the ImageClef 2007 campaign. For the photo task, IPAL group is ranked at the 3rd place among 20 participants. The MAP of our best run is 0.2833, which is ranked at the 6th place among the 476 runs. The IPAL system is based on the mixed modality search, i.e. textual and visual modalities. Compare with our results in 2006, our results are significantly enhanced by extracting multiple low-level visual content descriptors and fusing multiple CBIR. Several text based image search (TBIR) engines are also developed such as the language model (LM) approach, the latent semantic indexing (LSI) approach. We also have used external knowledge like Wordnet, and Wikipedia for document expansion. Then the cross-modality pseudo-relevance feedback is applied to boost each individual modality. Linear fusion is used to combine different ranking lists. Combining the CBIR and TBIR outperforms the individual modality search. On medical side, our run ranks 19 among 111. We continue to use a conceptual indexing with vector space weighting, but we add this year a Bayesian network on concepts extracted from UMLS meta-thesaurus.

MSRA Columbus at GeoCLEF 2006

Abstract: This paper describes the participation of Columbus Project of Microsoft Research Asia (MSRA) in GeoCLEF 2006. We participated in the Monolingual GeoCLEF evaluation (EN-EN) and submitted five runs based on different methods. In this paper, we describe our geographic information retrieval system, discuss the results and draw following conclusions: 1) if we just extract locations from the topics automatically without any expansions as geo-terms, the retrieval performance is barely satisfactory; 2) automatic query expansion weakens the performance; 3) if the queries are expanded manually, the performance is significantly improved.

Forostar: a system for GIR

Abstract: We detail our methods for generating and applying co-occurrence models for the purpose of placename disambiguation. We explain in detail our use of co-occurrence models for placename disambiguation using a model generated from Wikipedia. The presented system is split into two stages: a batch text & geographic indexer and a real time query engine. Four alternative query constructions and six methods of generating a geographic index are compared. The paper concludes with a full description of future work and ways in which the system could be optimised.

Amharic-English Information Retrieval

Abstract: We describe Amharic-English cross lingual information retrieval experiments in the ad hoc bilingual tracks of the CLEF 2006. The query analysis is supported by morphological analysis and part of speech tagging while we used two machine readable dictionaries supplemented by online dictionaries for term lookup in the translation process. Out of dictionary terms were handled using fuzzy matching and Lucene[4] was used for indexing and searching. Four experiments that differed in terms of utilized fields in the topic set, fuzzy matching, and term weighting, were conducted. The results obtained are reported and discussed.

University of Alicante at QA@CLEF2006: answer validation exercise

Abstract: In this paper we present an Answer Validation System which is based on the combination of word overlap and Latent Semantic Indexing modules. The main contribution of our work consist in the adaptation of our already developed machine-learning textual entailment system MLEnt to the multilingual Answer Validation exercise.

What happened in CLEF 2006

Abstract: The organization of the CLEF 2006 evaluation campaign is described and details are provided concerning the tracks, test collections, evaluation infrastructure, and participation.

Bilingual Question Answering Using CINDI QA at QA@CLEF 2007.

Abstract: This article presents the first participation of the CINDI group in the Multiple Language Question Answering Cross Language Evaluation Forum (QA@CLEF). We participated in a track using French as source language and English as target language. CINDI_QA first uses an online translation tool to convert the French input question into an English sentence. Second, a Natural Language Parser extracts keywords such as verbs, nouns, adjectives and capitalized entities from the query. Third, synonyms of those keywords are generated thanks to a Lexical Reference module. Fourth, our integrated Searching and Indexing component localises the candidate answers from the QA@CLEF data collection. Finally, the candidates are matched against our existing set of templates to decide on the best answer to return to the user. Out of eight runs submitted this year, CINDI_QA ranked second and third with an overall accuracy of 13%.

Brown at CL-SR'07: Retrieving conversational speech in English and Czech

Abstract: Brown’s entry to the Cross-Language Speech Retrieval (CL-SR) track at the 2007 Cross Language Evaluation Forum (CLEF) 1 was based on the language model (LM) paradigm for retrieval [17]. For English, our system introduced two minor enhancements to the basic unigram: we extended Dirichlet smoothing (popular with unigram modeling) to bigrams, and we smoothed the collection LM to compensate for the small collection size. For Czech, time-constraints restricted us to using a basic unigram model, though we did apply Czech-specific stemming. While our English system performed well in the evaluation and showed the utility of our enhancements, several aspects of it were rushed and need to be addressed in future work. Our Czech system did not perform competitively but did provide us with a useful first experience in non-English retrieval.

Location identification for the geographic information retrieval

Abstract: In this paper we identify location names that appear in queries written in Indonesian using geographic gazeeter We built the gazeeter by collecting geographic information from a number of geographic resources We translated an Indonesian query set into English using a machine translation technique We also made an attempt to improve the retrieval effectiveness using a query expansion technique The result shows that identifying locations in the queries and applying the query expansion technique can help improve the retrieval effectiveness for certain queries

Applying logic forms and statistical methods to CL-SR performance

Abstract: This paper describes a CL-SR system that employs two different techniques: the first one is based on NLP rules that consist on applying logic forms to the topic processing while the second one basically consists on applying the IR-n statistical search engine to the spoken document collection. The application of logic forms to the topics allows to increase the weight of topic terms according to a set of syntactic rules. Thus, the weights of the topic terms are used by IR-n system in the information retrieval process.

The University of Amsterdam at the CLEF Cross Language Speech Retrieval Track 2007

Abstract: In this paper we present the contents of the University of Amsterdam submission in the CLEF Cross Language Speech Retrieval 2007 English task. We describe the eects of using character n-grams and field combinations on both monolingual English retrieval, and crosslingual Dutch to English retrieval.

Report of MIRACLE team for the Ad-Hoc track in CLEF 2007

Abstract: This paper presents the 2007 MIRACLE’s team approach to the AdHoc Information Retrieval track. The work carried out for this campaign has been reduced to monolingual experiments, in the standard and in the robust tracks. No new approaches have been attempted in this campaign, following the procedures established in our participation in previous campaigns. For this campaign, runs were submitted for the following languages and tracks: - Monolingual: Bulgarian, Hungarian, and Czech. - Robust monolingual: French, English and Portuguese. There is still some room for improvement around multilingual named entities recognition.

Text retrieval and blind feedback for the ImageCLEFphoto task

Abstract: In this paper we will describe Berkeley's approach to the ImageCLEFphoto task for CLEF 2006. This year is the first time that we have participated in ImageCLEF, and we chose to primarily establish a baseline for the Cheshire II system for this task, while we had originally hoped to use GeoCLEF methods for this task, in the end time constraints led us to restrict our submissions to the basic required runs for the task.