The development of the Internet in recent years has made it possible and useful to access many different information systems anywhere in the world to obtain information. While there is much research on the integration of heterogeneous information systems, most commercial systems stop short of the actual integration of available data. Data fusion is the process of fusing multiple records representing the same real-world object into a single, consistent, and clean representation.This article places data fusion into the greater context of data integration, precisely defines the goals of data fusion, namely, complete, concise, and consistent data, and highlights the challenges of data fusion, namely, uncertain and conflicting data values. We give an overview and classification of different ways of fusing data and present several techniques based on standard and advanced operators of the relational algebra and SQL. Finally, the article features a comprehensive survey of data integration systems from academia and industry, showing if and how data fusion is performed in each.

Data fusion

We witness a rapid increase in the number of structured information sources that are available online, especially on the WWW. These sources include commercial databases on product information, stock market information, real estate, automobiles, and entertainment. We would like to use the data stored in these databases to answer complex queries that go beyond keyword searches. We face the following challenges: (1) Several information sources store interrelated data, and any query-answering system must understand the relationships between their contents. (2) Many sources are not full-featured database systems and can answer only a small set of queries over their data (for example, forms on the WWW restrict the set of queries one can (3) Since the number of sources is very large, effective techniques are needed to prune the set of information sources accessed to answer a query. (4) The details of interacting with each source vary greatly. We describe the Information Manifold, an implemented system that provides uniform access to a heterogeneous collection of more than 100 information sources, many of them on the WWW. IM tackles the above problems by providing a mechanism to describe declaratively the contents and query capabilities of available information sources. There is a clean separation between the declarative source description and the actual details of interacting with an information source. We describe algorithms that use the source descriptions to prune effciently the set of information sources for a given query and practical algorithms to generate executable query plans. The query plans we generate can inolve querying several information sources and combining their answers. We also present experimental studies that indicate that the architecture and algorithms used in the Information Manifold scale up well to several hundred information sources

https://www.seas.upenn.edu/~zives/03s/cis650/lro96.PDF

Querying Heterogeneous Information Sources Using Source Descriptions

language, designed for querying semistructured data. Semistructured data is becoming more and more prevalent, e.g., in structured documents such as HTML and when performing simple integration of data from multiple sources. Traditional data models and query languages are inappropriate, since semistructured data often is irregular: some data is missing, similar concepts are represented using different types, heterogeneous sets are present, or object structure is not fully known. Lorel is a user-friendly language in the SQL/OQL style for querying such data effectively. For wide applicability, the simple object model underlying Lorel can be viewed as an extension of the ODMG data model and the Lorel language as an extension of OQL. The main novelties of the Lorel language are: (i) the extensive use of coercion to relieve the user from the strict typing of OQL, which is inappropriate for semistructured data; and (ii) powerful path expressions, which permit a flexible form of declarative navigational access and are particularly suitable when the details of the structure are not known to the user. Lorel also includes a declarative update language. Lorel is implemented as the query language of the Lore prototype database management system at Stanford. Information about Lore can be found at http://www-db.stanford.edu/lore. In addition to presenting the Lorel language in full, this paper briefly describes the Lore system and query processor. We also briefly discuss a second implementation of Lorel on top of a conventional object-oriented database management system, the O2 system.

The Lorel Query Language for Semistructured Data

The goal of the Tsimmis Project is to develop tools that facilitate the rapid integration of heterogeneous information sources that may include both structured and unstructured data. This paper gives an overview of the project, describing components that extract properties from unstructured objects, that translate information into a common object model, that combine information from several sources, that allow browsing of information, and that manage constraints across heterogeneous sites. Tsimmis is a joint project between Stanford and the IBM Almaden Research Center. 1 Overview A common problem facing many organizations today is that of multiple, disparate information sources and repositories, including databases, object stores, knowledge bases, file systems, digital libraries, information retrieval systems, and electronic mail systems. Decision makers often need information from multiple sources, but are unable to get and fuse the required information in a timely fashion due to the diffculties of accessing the different systems, and due to the fact that the information obtained can be inconsistent and contradictory. Research sponsored by the Wright Laboratory, Aeronautical Systems Center, Air Force Material Command, USAF, under Grant Number F33615-93-1-1339. The US Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the offcial policies or endorsements, either express or implied, of Wright Laboratory or the US Government. This work was also supported by the Reid and Polly Anderson Faculty Scholar Fund, the Center for Integrated Systems at Stanford University, and by Equipment Grants from Digital Equipment Corporation and IBM Corporation. The goal of the TSIMMIS 1 project is to provide tools for accessing, in an integrated fashion, multiple informati

The TSIMMIS project: Integration of heterogeneous information sources

We address the problem of providing integrated access to diverse and dynamic information sources. We explain how this problem differs from the traditional database integration problem and we focus on one aspect of the information integration problem, namely information exchange. We define an object-based information exchange model and a corresponding query language that we believe are well suited for integration of diverse information sources. We describe how, the model and language have been used to integrate heterogeneous bibliographic information sources. We also describe two general-purpose libraries we have implemented for object exchange between clients and servers. >

Object exchange across heterogeneous information sources

Pegasus, a heterogeneous multidatabase management system that responds to the need for effective access and management of shared data across in a wide range of applications, is described. Pegasus provides facilities for multidatabase applications to access and manipulate multipole autonomous heterogeneous distributed object-oriented relational, and other information systems through a uniform interface. It is a complete data management system that integrates various native and local databases. Pegasus takes advantage of object-oriented data modeling and programming capabilities. It uses both type and function abstractions to deal with mapping and integration problems. Function implementation can be defined in an underlying database language or a programming language. Data abstraction and encapsulation facilities in the Pegasus object model provide an extensible framework for dealing with various kinds of heterogeneities in the traditional database systems and nontraditional data sources. >

The Pegasus heterogeneous multidatabase system

Pegasus is an object-oriented multidatabase system being developed at Hewlett-Packard Laboratories. The goal of the system is to provide facilities for multidatabase applications for accessing and manipulating multiple autonomous heterogeneous object-oriented, relational and other databases. This paper gives an overview of the architecture, describes the common object model and gives the justification for adapting the model for a multidatabase system. It describes the database environment of the system, discusses how to manage schemas consisting of types and functions from multiple databases. It briefly describes the features of the data language and gives an example. It discusses a method for dealing with object identification issues in a multidatabase system, reviews schema integration issues and briefly mentions schema mapping to relational models. >

Multidatabase management in Pegasus

Pegasus, a heterogeneous multidatabase management system that is under development, is described. The goal of the system is to provide facilities for applications to access and manipulate multiple autonomous heterogeneous object-oriented, relational, and other information systems. Pegasus defines a common object model for unifying the data models of the underlying systems. The data language of Pegasus allows transparent as well as explicit access to multiple information systems in a single declarative statement. The Pegasus architecture is described, and attention is given to data integration and query processing in Pegasus. >

Pegasus: a system for seamless integration of heterogeneous information sources

The paper identifies various problems in integrating schema, data and methods in heterogeneous systems and describes the strategies used in Pegasus to deal with them. Pegasus is an object-oriented multidatabase system that is being developed at Hewlett-Packard Laboratories. The goal of the first running prototype is to provide facilities for multidatabase applications to access and manipulate multiple autonomous heterogeneous object-oriented and relational databases. Pegasus provides both type and procedural abstractions to support integration. Type abstraction is used to organize objects from multiple databases in categories that are related by having some common properties. The common properties of objects in different databases may be in incompatible forms or in inconsistent states. The system provides procedural abstraction that enables the schema integrator to develop arbitrary mappings between data from different sources and to write necessary transformations to convert object properties to a common form. Because of the importance of existing relational databases, some important integration techniques are demonstrated through the object-oriented views over SQL databases. >

http://ieeexplore.ieee.org/iel2/378/4717/00183245.pdf

A. Rafii

Papers

The Pegasus heterogeneous multidatabase system

Multidatabase management in Pegasus

Pegasus: a system for seamless integration of heterogeneous information sources

Integration strategies in Pegasus object oriented multidatabase system