In digital libraries image retrieval queries can be based on the similarity of objects, using several feature attributes like shape, texture, color or text. Such multi-feature queries return a ranked result set instead of exact matches. Besides, the user wants to see only the k top-ranked objects. We present a new algorithm called Quick-Combine (European patent pending, nr. EP 00102651.7) for combining multi-feature result lists, guaranteeing the correct retrieval of the k top-ranked results. For score aggregation virtually any combining function can be used, including weighted queries. Compared to Fagin’s algorithm we have developed an improved termination condition in tuned combination with a heuristic control flow adopting itself narrowly to the particular score distribution. Top-ranked results can be computed and output incrementally. We show that we can dramatically improve performance, in particular for non-uniform score distributions. Benchmarks on practical data indicate efficiency gains by a factor of 30. For very skewed data observed speed-up factors are even larger. These performance results scale through different database sizes and numbers of result sets to combine.

/pdf/optimizing-multi-feature-queries-for-image-databases-1fze2tjn9e.pdf

Optimizing Multi-Feature Queries for Image Databases

Applications like multimedia databases or enterprise-wide information management systems have to meet the challenge of efficiently retrieving the best-matching objects from vast collections of data. We present a new algorithm, called Stream-Combine, for processing multi-feature queries on heterogeneous data sources. Stream-Combine is self-adapting to different data distributions and to the specific kind of the combining function. Furthermore, we present a new retrieval strategy that essentially speeds up the output of relevant objects.

Towards efficient multi-feature queries in heterogeneous environments

Pervasive Internet services today promise to provide users with a quick and convenient access to a variety of commercial applications However, due to unsuitable architectures and poor performance user acceptance is still low To be a major success mobile services have to provide device-adapted content and advanced value-added Web services Innovative enabling technologies like XML and wireless communication may for the first time provide a facility to interact with online applications anytime anywhere We present a prototype implementing an efficient multimedia middleware approach towards ubiquitous value-added services using an auction house as a sample application Advanced multi-feature retrieval technologies are combined with enhanced content delivery to show the impact of modern enterprise information systems on today’s e-commerce applications

/pdf/an-xml-based-multimedia-middleware-for-mobile-online-1xb73ze5eg.pdf

An XML-Based Multimedia Middleware for Mobile Online Auctions.

Contemporary digital libraries and archives of music scores focus mainly on providing efficient storage and access methods for their data. However, digital archives of historical music scores can enable musicologists not only to easily store and access research material, but also to derive new knowledge from existing data. In this paper we present the first steps in building a digital archive of historical music scores from the 17th and 18th century. Along with the architectural and accessibility aspects of the system, we describe an integrated approach for classification and identification of the scribes of music scores.

Towards a Digital Document Archive for Historical Handwritten Music Scores

In digital libraries queries are often based on the similarity of objects, using several feature attributes like colors, texture or full-text searches. Such multi-feature queries return a ranked result set instead of exact matches. Recently we presented a new algorithm called Quick-Combine [5] for combining multi-feature result lists, guaranteeing the correct retrieval of the k top-ranked results. As benchmarks on practical data promise that we can dramatically improve performance, we want to discuss interesting applications of Quick-Combine in different areas. The applications for the optimization in ranked query models are manifold. Generally speaking we believe that all kinds of federated searches can be supported like e.g. content-based retrieval, knowledge management systems or multi-classifier combination. 1. Query Optimization For Ranked Query Models Today‘s handling of multimedia data in information systems such as images, video or audio files poses an increasingly demanding problem. The query evaluation model typically does not retrieve a set of exact matches but rather a ranked result set, where an aggregated score is attached to each object returned. Only a few topranked results are normally of interest to the user. A query could for example ask for the top 10 objects from an image collection that are most similar to a fixed image in terms of visual properties; a query type which is often referred to as 'query by visual example'. Query optimization needs to be adapted to this essentially different query model for multimedia data. Some systems have already been implemented, e.g. visual retrieval systems like IBM's QBIC [1] or Virage's VIR [2]. Database applications and middlewares like GARLIC [3] or the HERON project [4] have already started to use the capabilities of visual retrieval. A major challenge in all of these systems is that similarity between different objects cannot be defined precisely. To handle queries on similarity different kinds of information on the multimedia objects have to be stored. For example in the case of images this could be color histograms, shapes of occurring objects, features on textures and layout or related fulltext information describing the object. Queries on similarity do not have to focus on one single feature. In general multimedia queries will refer to at least some different features simultaneously. According to a potentially weighted combining function for each database object an aggregated score value is computed. The results are then sorted according to their scores and are returned with a rank number the top-ranked object has the best score value of the entire database and so on. A query focusing on a single feature is called atomic. Complex multimedia queries are combinations of atomic subqueries. 2. The Quick-Combine Approach The optimization challenge is to combine several ranked results of atomic queries in order to determine the k overall best objects from the database. A naive approach would calculate the aggregated score for all database objects according to a given combining function. With growing database sizes, this obviously results in unacceptable response times, requiring a linear scan of the entire database. But as only the k top objects have to be returned, not all database objects have to be accessed. In general atomic queries can be posed in two ways. The first type is searching the database using a suitable index and retrieving the objects in the database ordered by descending score for a single feature, which we refer to as enlarging an output stream or sorted access. On the other hand a specific object's score in each atomic output stream could be of interest. This case is referred to as random access. In 1996 Fagin presented an approach [8] to process a complex query consisting of several atomic subqueries that may use any monotonous combining function, as for example the maximum or arithmetical mean. Fagin's work has also strongly influenced statistical approaches as [9]. There a different query model is presented using not only ranking expressions for query processing, but a filter condition like the WHERE-clause in SQL. During processing first the filter condition is evaluated and later on only objects satisfying the condition are ranked according to a ranking expression. Using the Quick-Combine algorithm [5], the overall number of necessary accesses can be minimized with an adequate control flow and an efficient test of termination. For these tasks Quick-Combine does not only use the information of ranks in output streams, but also the scores which are assigned to all objects in each output stream, the specific form of the combining function and heuristics to force early termination. QuickCombine is applicable for every monotonic combining function, even in the case of maximum and minimum, and is capable of processing weighted queries. Quick-Combine can even return the overall best database objects successively. Thus the top-scored object of the whole collection can be delivered to the user as soon as it is found, which of course also applies to all following ranks up to k, when the algorithm finally terminates. If the user is already satisfied by the first few ranks, the query execution for large values of k can be stopped during processing. In combining different ranked result sets it is not only a problem how far the output streams have to be expanded, but also the sequence in which they are expanded and aggregated scores of objects are calculated. Since a correct result set is always guaranteed, the final result set will of course not differ, if the sequence is changed, but different sequences may strongly influence the efficient execution of the algorithm. To find an optimal order the Quick-Combine algorithm uses a set of indicators that rely on heuristic considerations and help to choose always the stream whose further expansion will have the greatest effect on the final result set. But the indicators are not only useful for equally weighted queries. Streams with low weights should naturally be regarded less important than highly weighted streams which should get prior evaluation. We will give an example of the composition and usefulness of these indicators in section 3.3 where the problem of multiclassifier combination is described. Fig. 1: Architecture of the HERON-system The HERON system [4] has been used to test Quick-Combine on a collection of heraldic images. Besides components for query composition, user specific image delivery and efficient format conversions, the HERON-middleware features a combining engine that can use different visual retrieval systems and databases (cf. Fig. 1). For practical numbers of objects to retrieve Quick-Combine even accesses 30 times less objects than previous approaches guaranteeing a correct result set. From our practical tests we gained insight into distributions that really occur in image retrieval. We argue that typical distributions from visual retrieval systems consist of a low percentage of objects having high and medium score values and a high percentage having very low scores. If text retrieval is included the percentage having high and medium scores even decreases. QuickCombine was therefore tested on synthetic data, where different score distributions in the output streams have been analyzed. For these distributions similar benchmark results have been proven [5]. In these tests QuickCombine showed that even skewed data distributions can be handled efficiently and the algorithm scales with both, an increasing number of streams to combine and a growing database size. 3. Applications of Quick-Combine The HERON-project investigates the capabilities of content-based retrieval in digital libraries. However, QuickCombine seems not only useful in the field of content-based retrieval for multimedia data, but further typical areas of applications can be: • Feature lists within the SQL/MM standard • Content management in digital libraries • Enterprise information portals • Knowledge management systems • Multi-classifier combination 3.1. The SQL/MM Standard Together with the development of visual retrieval engines and multimedia databases an extension of the database query language SQL towards multimedia applications has been proposed. The SQL/MM standard [6] was defined to enable the handling of multimedia data in object-relational databases systems. Part 5 deals among others with content-based retrieval for still images, specifying image datatypes and methods for manipulation. This new standard will strongly influence the capabilities of SQL-engines in future multimedia databases. The standard provides an adequate datatype SI_StillImage to handle image data. This datatype provides a container for basic data and consists of several attributes that do not represent all information about the image. The retrieval methods focus on four different content-based methods. For image searches low-level features have to be used. The standard only implements average color, histogram color, positional color and texture features. But it also provides a composed feature called SI_FeatureList. It allows retrieval by any weighted combination of the basic features. The specific calculation of the combined result sets and the types of supported combining functions, however, is left to the implementation. Thus already four different output streams may be necessary for content-based retrieval, which of course have to be combined using an algorithm like QuickCombine. Besides these feature lists complex SQL/MM queries almost always will involve several text-based output streams, too. Thus in future multimedia SQL queries the combination of between 5 and 10 different features can be assumed. To perform such multimedia queries efficiently high speed iterators for sorted accesses relying on multi-dimensional k-nearest-neighbor searches, e.g. [10], and effective multi-dimensional index structures for random acces

/pdf/applications-of-quick-combine-for-ranked-query-models-3avop8cqxc.pdf

Applications of Quick-Combine for Ranked Query Models.

The interdisciplinary HERON project investigates the impact of multimedia applications from the humanities, in particular heraldry, on future database technology. We present first evaluation results of querying image databases by visual content. Also the requirements of a digital workbench for art historians are described. Here we present an approach how to tackle the complex problem of exchanging multimedia documents over the internet.

/pdf/the-heron-project-multimedia-database-support-for-history-521aqiwkp7.pdf

Katharina Erber-Urch

Papers

The HERON Project — Multimedia Database Support for History and Human Sciences