Existing approaches for optimizing queries in data integration use decoupled strategies--attempting to optimize coverage and cost in two separate phases. Since sources tend to have a variety of access limitations, such phased optimization of cost and coverage can unfortunately lead to expensive planning as well as highly inefficient plans. In this paper we present techniques for joint optimization of cost and coverage of the query plans. Our algorithms search in the space of parallel query plans that support multiple sources for each subgoal conjunct. The refinement of the partial plans takes into account the potential parallelism between source calls, and the binding compatibilities between the sources included in the plan. We start by introducing and motivating our query plan representation. We then briefly review how to compute the cost and coverage of a parallel plan. Next, we provide both a System-R style query optimization algorithm as well as a greedy local search algorithm for searching in the space of such query plans. Finally we present a simulation study that demonstrates that the plans generated by our approach will be significantly better, both in terms of planning cost, and in terms of plan execution cost, compared to the existing approaches.

/pdf/joint-optimization-of-cost-and-coverage-of-query-plans-in-3anhfih7fe.pdf

Joint optimization of cost and coverage of query plans in data integration

In the last years, alerting systems have gained strengthened attention. Several systems have been implemented. For the evaluation and cooperation of these systems, the following problems arise: The systems and their models are not compatible, and existing models are only appropriate for a subset of conceivable application domains. Due to modeling differences, a simple integration of different alerting systems is impossible. What is needed, is a unified model that covers the whole variety of alerting service applications. This paper provides a unified model for alerting services that captures the special constraints of most application areas. The model can serve as a basis for an evaluation of alerting service implementations. In addition to the unified model, we define a general profile structure by which clients can specify their interest. This structure is independent of underlying profile definition languages. To eliminate drawbacks of the existing non-cooperating solitary services we introduce a new technique, the Mediating Alerting Service (MediAS). It establishes the cooperation of alerting services in an hierarchical and parallel way.

/pdf/a-unified-model-of-internet-scale-alerting-services-26w96fvxiy.pdf

A Unified Model of Internet Scale Alerting Services

Recent work in data integration has shown the importance of statistical information about the coverage and overlap of sources for efficient query processing. Despite this recognition, there are no effective approaches for learning the needed statistics. The key challenge in learning such statistics is keeping the number of needed statistics low enough to have the storage and learning costs manageable. In this paper, we present a set of connected techniques that estimate the coverage and overlap statistics, while keeping the needed statistics tightly under control. Our approach uses a hierarchical classification of the queries and threshold-based variants of familiar data mining techniques to dynamically decide the level of resolution at which to learn the statistics. We describe the details of our method, and, present experimental results demonstrating the efficiency of the learning algorithms and the effectiveness of the learned statistics over both controlled data sources and in the context of BibFinder with autonomous online sources.

/pdf/effectively-mining-and-using-coverage-and-overlap-statistics-35qtcsjs30.pdf

Effectively mining and using coverage and overlap statistics for data integration

The ability to physically interconnect many distributed, autonomous and heterogeneous software systems on a large scale presents new opportunities for sharing and reuse of existing, and for the cre ...

/pdf/query-processing-for-peer-mediator-databases-33pg4kfjeg.pdf

Query Processing for Peer Mediator Databases

This paper presents a compile-time placement method of mobile relational operators MROs in a large scale environment. MROs are self adaptive to changing runtime conditions by deciding their execution place if they discover compile-time estimation errors. Proposed placement methods tend to have a main drawback with MROs running over a large scale environment: their focus is on finding optimal performance depending on single-point estimation at compile-time, instead of optimal performance over an estimation interval. We propose: (i) to determine the migration space of a MRO including the sites on which the MRO is allowed to migrate during its execution, and (ii) to find the robust site which will allow acceptable response time in an estimation interval. Performance study shows that, with a risk of loosing around 6% in response time, it is possible to gain up to 300% with the proposed robust placement.

Robust Placement of Mobile Relational Operators for Large Scale Distributed Query Optimization

We present the systematic design and development of a distributed query scheduling service (DQS) in the context of DIOM, a distributed and interoperable query mediation system.26 DQS consists of an extensible architecture for distributed query processing, a three-phase optimization algorithm for generating efficient query execution schedules, and a prototype implementation. Functionally, two important execution models of distributed queries, namely moving query to data or moving data to query, are supported and combined into a unified framework, allowing the data sources with limited search and filtering capabilities to be incorporated through wrappers into the distributed query scheduling process. Algorithmically, conventional optimization factors (such as join order) are considered separately from and refined by distributed system factors (such as data distribution, execution location, heterogeneous host capabilities), allowing for stepwise refinement through three optimization phases: Compilation, parallelization, site selection and execution. A subset of DQS algorithms has been implemented in Java to demonstrate the practicality of the architecture and the usefulness of the distributed query scheduling algorithm in optimizing execution schedules for inter-site queries.

/pdf/distributed-query-scheduling-service-an-architecture-and-its-4237k10mco.pdf

Calton Pu

Papers

Distributed query scheduling service: an architecture and its implementation