Showing papers on "Distributed database published in 2009"

Ensuring data storage security in Cloud Computing

Abstract: Cloud Computing has been envisioned as the next-generation architecture of IT Enterprise. In contrast to traditional solutions, where the IT services are under proper physical, logical and personnel controls, Cloud Computing moves the application software and databases to the large data centers, where the management of the data and services may not be fully trustworthy. This unique attribute, however, poses many new security challenges which have not been well understood. In this article, we focus on cloud data storage security, which has always been an important aspect of quality of service. To ensure the correctness of users' data in the cloud, we propose an effective and flexible distributed scheme with two salient features, opposing to its predecessors. By utilizing the homomorphic token with distributed verification of erasure-coded data, our scheme achieves the integration of storage correctness insurance and data error localization, i.e., the identification of misbehaving server(s). Unlike most prior works, the new scheme further supports secure and efficient dynamic operations on data blocks, including: data update, delete and append. Extensive security and performance analysis shows that the proposed scheme is highly efficient and resilient against Byzantine failure, malicious data modification attack, and even server colluding attacks.

System Level Concurrency Control for Distributed Database Systems

Abstract: A distributed database system is one in which the database is spread among several sites and application programs “move” from site to site to access and update the data they need. The concurrency control is that portion of the system that responds to the read and write requests of the application programs. Its job is to maintain the global consistency of the distributed database while ensuring that the termination of the application programs is not prevented by phenomena such as deadlock. We assume each individual site has its own local concurrency control which responds to requests at that site and can only communicate with concurrency controls at other sites when an application program moves from site to site, terminates, or aborts.This paper presents designs for several distributed concurrency controls and demonstrates that they work correctly. It also investigates some of the implications of global consistency of a distributed database and discusses phenomena that can prevent termination of application programs.

Cassandra: structured storage system on a P2P network

Abstract: Cassandra is a distributed storage system for managing structured data that is designed to scale to a very large size across many commodity servers, with no single point of failure. Reliability at massive scale is a very big challenge. Outages in the service can have significant negative impact. Hence Cassandra aims to run on top of an infrastructure of hundreds of nodes (possibly spread across different datacenters). At this scale, small and large components fail continuously; the way Cassandra manages the persistent state in the face of these failures drives the reliability and scalability of the software systems relying on this service. Cassandra has achieved several goals--scalability, high performance, high availability and applicability. In many ways Cassandra resembles a database and shares many design and implementation strategies with databases. Cassandra does not support a full relational data model; instead, it provides clients with a simple data model that supports dynamic control over data layout and format.

Systems and methods for managing replicated database data

Abstract: Systems and methods for replicating database data and generating read-only copies of the replicated data in a clean shutdown state. For example, systems can include a tracking module (e.g., a filter driver) that monitors transactions from a database application to a source storage device to generate log entries having at least one marker indicating a known good state of the application. The systems further include a computer coupled to a target storage device comprising a database and log files. The computer processes the transactions, based on the log entries, to replicate data to the target storage device; performs a first snapshot on data stored in the database and log files; replays into the database data stored in the log files; performs another snapshot on the database; and reverts the database back to a state in which the database existed at the time of the first snapshot.

Distributed data-parallel computing using a high-level programming language

Abstract: The Dryad and DryadLINQ systems offer a new programming model for large scale data-parallel computing. They generalize previous execution environments such as SQL and MapReduce in three ways: by providing a general-purpose distributed execution engine for data-parallel applications; by adopting an expressive data model of strongly typed .NET objects; and by supporting general-purpose imperative and declarative operations on datasets within a traditional high-level programming language. A DryadLINQ program is a sequential program composed of LINQ expressions performing arbitrary side-effect-free operations on datasets, and can be written and debugged using standard .NET development tools. The DryadLINQ system automatically and transparently translates the data-parallel portions of the program into a distributed execution plan which is passed to the Dryad execution platform. Dryad, which has been in continuous operation for several years on production clusters made up of thousands of computers, ensures efficient, reliable execution of this plan on a large compute cluster. This paper describes the programming model, provides a high-level overview of the design and implementation of the Dryad and DryadLINQ systems, and discusses the tradeoffs and connections to parallel and distributed databases.

Dependable and Secure Sensor Data Storage with Dynamic Integrity Assurance

Abstract: Recently, distributed data storage has gained increasing popularity for efficient and robust data management in wireless sensor networks (WSNs). But the distributed architecture also makes it challenging to build a highly secure and dependable yet lightweight data storage system. On the one hand, sensor data are subject to not only Byzantine failures, but also dynamic pollution attacks, as along the time the adversary may modify/pollute the stored data by compromising individual sensors. On the other hand, the resource-constrained nature of WSNs precludes the applicability of heavyweight security designs. To address the challenges, we propose a novel dependable and secure data storage scheme with dynamic integrity assurance in this paper. Based on the principle of secret sharing and erasure coding, we first propose a hybrid share generation and distribution scheme to achieve reliable and fault-tolerant initial data storage by providing redundancy for original data components. To further dynamically ensure the integrity of the distributed data shares, we then propose an efficient data integrity verification scheme exploiting the technique of algebraic signatures. The proposed scheme enables individual sensors to verify in one protocol execution all the pertaining data shares simultaneously in the absence of the original data. Extensive security and performance analysis shows that the proposed schemes have strong resistance against various attacks and are practical for WSNs.

Lifecycle management of facilities components using radio frequency identification and building information model

Abstract: The AECOO industry is highly fragmented; therefore, efficient information sharing and exchange between various players are evidently needed. Furthermore, the information about facility components should be managed throughout the lifecycle and be easily accessible for all players in the AECOO industry. BIM is emerging as a method of creating, sharing, exchanging and managing the information throughout the lifecycle between all the stakeholders. RFID, on the other hand, has emerged as an automatic data collection and information storage technology, and has been used in different applications in AECOO. This research proposes permanently attaching RFID tags to facility components where the memory of the tags is populated with accumulated lifecycle information of the components taken from a standard BIM database. This information is used to enhance different processes throughout the lifecycle. In addition, this research suggests storing other types of BIM information (e.g., floor plans) on RFID tags which is not necessarily related to the components themselves. Having BIM data chunks stored on tags provides a distributed database of BIM and allows data access for different players who do not have real-time access to a central database. In this research, a conceptual RFID-based system structure and data storage/retrieval design are elaborated. The value adding benefits and scope of impact of the proposed approach are discussed. To explore the technical feasibility of the proposed approach, two case studies have been implemented and tested.

Two-Party Computation Model for Privacy-Preserving Queries over Distributed Databases.

Abstract: Many existing privacy-preserving techniques for querying distributed databases of sensitive information do not scale for large databases due to the use of heavyweight cryptographic techniques. In addition, many of these protocols require several rounds of interactions between the participants which may be impractical in wide-area settings. At the other extreme, a trusted party based approach does provide scalability but it forces the individual databases to reveal private information to the central party. This paper shows how to perform various privacypreserving operations in a scalable manner under the honest-but-curious model. Our system provides the same level of scalability as a trusted central party based solution while providing privacy guarantees without the need for heavyweight cryptography. The key idea is to develop an alternative system model using a Two-Party Query Computation Model comprising of a randomizer and a computing engine which do not reveal any information between themselves. We also show how one can replace the randomizer by a lightweight key-agreement protocol. We formally prove the privacy-preserving properties of our protocols and demonstrate the scalability and practicality of our system using a real-world implementation.

Joining Tables in Multiple Heterogeneous Distributed Databases

Abstract: A method for joining tables in multiple heterogeneous distributed databases implemented by at least two data sources accessible to a federal database server over a network includes: transmitting from the federated database server a sub-command to a first of the data sources responsive to the federated database server receiving a data query; retrieving, with the federated database server, block data from the first data source related to the data query using block fetching according to the sub-command; transmitting, with the federated database server, at least a portion of the block data to a second of the data sources together with an instruction for the second data source to perform a join operation on the portion of the block data and a data table stored by the second data source related to the query; and retrieving a result of the join operation with the federated database server.

Cassandra: a structured storage system on a P2P network

Abstract: Cassandra is a distributed storage system for managing structured data that is designed to scale to a very large size across many commodity servers, with no single point of failure. Reliability at massive scale is a very big challenge. Outages in the service can have significant negative impact. Hence Cassandra aims to run on top of an infrastructure of hundreds of nodes (possibly spread across different datacenters). At this scale, small and large components fail continuously; the way Cassandra manages the persistent state in the face of these failures drives the reliability and scalability of the software systems relying on this service. Cassandra has achieved several goals -- scalability, high performance, high availability and applicability. In many ways Cassandra resembles a database and shares many design and implementation strategies with databases. Cassandra does not support a full relational data model; instead, it provides clients with a simple data model that supports dynamic control over data layout and format.

FDAC: Toward Fine-Grained Distributed Data Access Control in Wireless Sensor Networks

Abstract: Distributed sensor data storage and retrieval has gained increasing popularity in recent years for supporting various applications. While distributed architecture enjoys a more robust and fault-tolerant wireless sensor network (WSN), such architecture also poses a number of security challenges especially when applied in mission-critical applications such as battle field and e-healthcare. First, as sensor data are stored and maintained by individual sensors and unattended sensors are easily subject to strong attacks such as physical compromise, it is significantly harder to ensure data security. Second, in many mission-critical applications, fine-grained data access control is a must as illegal access to the sensitive data may cause disastrous result and/or prohibited by the law. Last but not least, sensors usually are resource-scarce, which limits the direct adoption of expensive cryptographic primitives. To address the above challenges, we propose in this paper a distributed data access control scheme that is able to fulfill fine-grained access control over sensor data and is resilient against strong attacks such as sensor compromise and user colluding. The proposed scheme exploits a novel cryptographic primitive called attribute-based encryption (ABE), tailors, and adapts it for WSNs with respect to both performance and security requirements. The feasibility of the scheme is demonstrated by experiments on real sensor platforms. To our best knowledge, this paper is the first to realize distributed fine-grained data access control for WSNs.

Asynchronous view maintenance for VLSD databases

Abstract: The query models of the recent generation of very large scale distributed (VLSD) shared-nothing data storage systems, including our own PNUTS and others (e.g. BigTable, Dynamo, Cassandra, etc.) are intentionally simple, focusing on simple lookups and scans and trading query expressiveness for massive scale. Indexes and views can expand the query expressiveness of such systems by materializing more complex access paths and query results. In this paper, we examine mechanisms to implement indexes and views in a massive scale distributed database. For web applications, minimizing update latencies is critical, so we advocate deferring the work of maintaining views and indexes as much as possible. We examine the design space, and conclude that two types of view implementations, called remote view tables (RVTs) and local view tables (LVTs), provide good tradeoff between system throughput and minimizing view staleness. We describe how to construct and maintain such view tables, and how they can be used to implement indexes, group-by-aggregate views, equijoin views and selection views. We also introduce and analyze a consistency model that makes it easier for application developers to cope with the impact of deferred view maintenance. An empirical evaluation quantifies the maintenance costs of our views, and shows that they can significantly improve the cost of evaluating complex queries.

Distributed Skyline Retrieval with Low Bandwidth Consumption

Abstract: We consider skyline computation when the underlying data set is horizontally partitioned onto geographically distant servers that are connected to the Internet. The existing solutions are not suitable for our problem, because they have at least one of the following drawbacks: (1) applicable only to distributed systems adopting vertical partitioning or restricted horizontal partitioning, (2) effective only when each server has limited computing and communication abilities, and (3) optimized only for skyline search in subspaces but inefficient in the full space. This paper proposes an algorithm, called feedback-based distributed skyline (FDS), to support arbitrary horizontal partitioning. FDS aims at minimizing the network bandwidth, measured in the number of tuples transmitted over the network. The core of FDS is a novel feedback-driven mechanism, where the coordinator iteratively transmits certain feedback to each participant. Participants can leverage such information to prune a large amount of local data, which otherwise would need to be sent to the coordinator. Extensive experimentation confirms that FDS significantly outperforms alternative approaches in both effectiveness and progressiveness.

Master monitoring mechanism for a geographical distributed database

Abstract: The present invention faces the issue of data replication in different database nodes of a geographically distributed database wherein clients cannot always perform any database-related operation in the closest database node. Thus, the present invention provides for an enhanced distributed database system with a plurality of nodes, each node arranged for storing a replica of at least one partition of data, and a method of handling said distributed database system comprising the steps of: partitioning data to a number of partitions; replicating each partition into a number of replicas; for each partition, distributing the number of replicas amongst database nodes; activating more than one node; monitoring at each active node events of: latest updating of each replica, replica status, status of local resources in charge of each replica, and connectivity status of each replica; upon activation o deactivation of a node, determining which node is considered current master node for each partition in charge of current mast replica; for any request received in a node to read/write data, determining the current master node in charge of the current master replica, and routing said request to said current master node.

Distributed media fingerprint repositories

Abstract: Media fingerprints, which are each derived from and uniquely correspond to a portion of media content, are stored over a distributed database. An instance of one or more information containers of the distributed database are each disposed over two or more computing devices, which are communicatively linked over a data network. The media fingerprints are distributed among the information containers based on a criterion that relates individually to each of the media fingerprints. Upon a query directed to one of the media fingerprints, at least one of the two or more computing devices is selected based on the criterion. The query is executed over the distributed database instance of the selected computing device.

Apparatus and method for integrating map-reduce into a distributed relational database

Abstract: A computer readable storage medium includes executable instructions to define a map-reduce document that coordinates processing of data in a distributed database The map-reduce document complies with a map-reduce specification that integrates map-reduce functions with queries in a query language The operations specified by the map-reduce document are executed in the distributed database

A Generic Local Algorithm for Mining Data Streams in Large Distributed Systems

Abstract: In a large network of computers or wireless sensors, each of the components (henceforth, peers) has some data about the global state of the system. Much of the system's functionality such as message routing, information retrieval and load sharing relies on modeling the global state. We refer to the outcome of the function (e.g., the load experienced by each peer) as the \emph{model} of the system. Since the state of the system is constantly changing, it is necessary to keep the models up-to-date. Computing global data mining models e.g. decision trees, k-means clustering in large distributed systems may be very costly due to the scale of the system and due to communication cost, which may be high. The cost further increases in a dynamic scenario when the data changes rapidly. In this paper we describe a two step approach for dealing with these costs. First, we describe a highly efficient \emph{local} algorithm which can be used to monitor a wide class of data mining models. Then, we use this algorithm as a feedback loop for the monitoring of complex functions of the data such as its k-means clustering. The theoretical claims are corroborated with a thorough experimental analysis.

Fragmentation Design for Efficient Query Execution over Sensitive Distributed Databases

Abstract: The balance between privacy and utility is a classical problem with an increasing impact on the design of modern information systems. On the one side it is crucial to ensure that sensitive information is properly protected; on the other side, the impact of protection on the workload must be limited as query efficiency and system performance remain a primary requirement. We address this privacy/efficiency balance proposing an approach that, starting from a flexible definition of confidentiality constraints on a relational schema, applies encryption on information in a parsimonious way and mostly relies on fragmentation to protect sensitive associations among attributes. Fragmentation is guided by workload considerations so to minimize the cost of executing queries over fragments. We discuss the minimization problem when fragmenting data and provide a heuristic approach to its solution.

A Distributed Algorithm to Enumerate All Maximal Cliques in MapReduce

Abstract: Structure mining plays an important part in the researches in biology, physics, internet or telecommunications in recently emerging network science. As a main task in this area, the problem of maximal clique enumeration has attracted much interest and been studied in variant avenues in prior works. However, most of these works mainly rely on single chip computational capacity and have been constrained by local optimization. Thus it is an impossible mission for these methods to process terabytes datasets. In this paper, to extract maximal cliques from graphs, we propose a general enumeration process in a distributed manner on cluster system with the help of MapReduce. Graph is firstly split into small subgraphs automatically. Then a novel key-based clique enumeration algorithm is proposed based on subgraphs. We demonstrate that our algorithm has a high parallelism and a prominent performance on extremely huge graphs. Our method is implemented to fully utilize MapReduce execution mechanism and the experiments are soundly discussed as using such a powerful distributed platform. However we not only show the scalability and efficiency of the algorithm but also share some critical experience in using MapReduce computing model.

Distributed database recovery

Abstract: A method and apparatus for recovery a cluster database is provided. Database recovery is divided among a plurality of surviving database server instances in the cluster. A surviving instance is responsible for recovering data blocks to which the surviving instance is assigned. One form of block-to-instance assignment may be based on mastership of the blocks. If a data block is locked by a surviving instance at the time of failure, then no recovery of that data block may be necessary. Else, if a copy of a data block that is to be recovered is stored on a surviving node in the cluster, then one or more redo records are applied to that copy (if necessary). A redo record that corresponds to that data block might not need to be applied to the copy if the redo record reflects changes (to the data block) that are already reflected in the copy.

An evolutionary genetic algorithm for optimization of distributed database queries

Abstract: High performance low cost PC hardware, and high speed LAN/WAN technologies make distributed database(DDB) systems an attractive research area. Since Dynamic programming is not feasible for optimizing queries in a DDB, we propose a GA based query optimizer and compare its performance to random and optimal algorithms. We analyzed a set of possible GA parameters and determined that two-point truncate technique using GA gives the best results. New mutation and crossover operators have also been defined and experimentally analyzed. We performed experiments on a synthetic database with replicated relations, but no horizontal or vertical fragmentation. Network links are assumed to be gigabit Ethernet. Comparisons with optimal results found by exhaustive search show that our new GA formulation performs only 20% off the optimal results and we have achieved a 50% improvement over a previous GA based algorithm.

Low power mode in cloud storage systems

Abstract: We consider large scale, distributed storage systems with a redundancy mechanism; cloud storage being a prime example. We investigate how such systems can reduce their power consumption during low-utilization time intervals by operating in a low-power mode. In a low power mode, a subset of the disks or nodes are powered down, yet we ask that each data item remains accessible in the system; this is called full coverage. The objective is to incorporate this option into an existing system rather than redesign the system. When doing so, it is crucial that the low power option should not affect the performance or other important characteristics of the system during full-power (normal) operation. This work is a comprehensive study of what can or cannot be achieved with respect to full coverage low power modes. The paper addresses this question for generic distributed storage systems (where the key component under investigation is the placement function of the system) as well as for specific popular system designs in the realm of storing data in the cloud. Our observations and techniques are instrumental for a wide spectrum of systems, ranging from distributed storage systems for the enterprise to cloud data services. In the cloud environment where low cost is imperative, the effects of such savings are magnified by the large scale.

Parallel OLAP query processing in database clusters with data replication

Abstract: We consider the problem of improving the performance of OLAP applications in a database cluster (DBC), which is a low cost and effective parallel solution for query processing. Current DBC solutions for OLAP query processing provide for intra-query parallelism only, at the cost of full replication of the database. In this paper, we propose more efficient distributed database design alternatives which combine physical/virtual partitioning with partial replication. We also propose a new load balancing strategy that takes advantage of an adaptive virtual partitioning to redistribute the load to the replicas. Our experimental validation is based on the implementation of our solution on the SmaQSS DBC middleware prototype. Our experimental results using the TPC-H benchmark and a 32-node cluster show very good speedup.

Scalability for Virtual Worlds

Abstract: Networked virtual environments (net-VEs) are the next wave of digital entertainment, with Massively Multiplayer Online Games (MMOs) a very popular instance. Current MMO architectures are server-centric in that all game logic is executed at the servers of the company hosting the game. This architecture has lead to severe scalability problems, in particular since MMOs require realistic graphics and game physics – computationally expensive tasks that are currently computed centrally. We propose a distributed action based protocol for net-VEs that pushes most computation to the computers of the players and thereby achieves massive scalability. The key feature of our proposal is a novel distributed consistency model that allows us to explore the tradeoff between scalability, computational complexity at the server, and consistency. We investigate our model both theoretically and through a comprehensive experimental evaluation.

Accelerating MapReduce with Distributed Memory Cache

Abstract: MapReduce is a partition-based parallel programming model and framework enabling easy development of scalable parallel programs on clusters of commodity machines. In order to make time-intensive applications benefit from MapReduce on small scale clusters, this paper proposes a new method to improve the performance of MapReduce by using distributed memory cache as a high speed access between map tasks and reduce tasks. Map outputs sent to the distributed memory cache can be gotten by reduce tasks as soon as possible. Experiment results show that our prototype’s performance is much better than that of the original on small scale clusters. To our knowledge, this is the first effort to accelerate MapReduce with the help of distributed memory cache.

Peer-to-peer overlays for real-time communication: security issues and solutions

Abstract: Peer to peer (P2P) networks offer higher robustness against failure, easier configuration and are generally more economical as compared to their client-server counterparts. This has made it reasonable for resource consuming and typically centralized applications like voice over IP (VoIP) and, in general, real-time communication to adapt and exploit the benefits of P2P. Such a migration needs to address a new set of P2P specific security problems. We go over some of the known issues found in common P2P networks. We then analyze the relevance of such issues and the applicability of existing solutions when using P2P architectures for real-time communication.

Scalable relational database replication

Abstract: A relational database replication system includes a client, at least one primary database, a plurality of secondary databases and replication agents which coordinate database transactions. The system provides a high level of performance, reliability, and scalability with an end result of efficient and accurate duplication of transactions between the primary and secondary databases. In one implementation, the client transmits sets of database update statements to the primary database and primary agent in parallel; the primary agent replicates the statements to at least one secondary agent. A transaction prepare and commit process is coordinated between the primary database and the primary agent, which in turn coordinates with the at least one secondary agent. Databases can be partitioned into individual smaller databases, called shards, and the system can operate in a linearly scalable manner, adding clients, databases and replication agents without requiring central coordination or components that cause bottlenecks.

VoroGame: A Hybrid P2P Architecture for Massively Multiplayer Games

Abstract: Peer-to-peer (P2P) architectures have recently become very popular in massively multiplayer games (MMGs). While P2P gaming offers high scalability compared to client/server architectures, it introduces several major issues related to data distribution and game state consistency. In this paper, we report our initial version of VoroGame, a P2P architecture for MMGs that addresses these issues by combining a structured P2P overlay based on a distributed hash table (DHT) for data distribution, with a Voronoi diagram used for virtual game world decomposition and semantic overlay support. The resulting hybrid architecture enables a fully distributed management of data and state information, and ensures efficient dissemination of game state updates to relevant peers.

Managing database object placement on multiple storage devices

Abstract: A method, information processing system, and computer program storage product optimize the placement of database objects on a multiplicity of storage devices. A set of database objects are placed on a first storage device in a multiplicity of storage devices. Each storage device comprises differing characteristics. A query workload is run on the set of database objects that have been placed on the first storage device. Profiling information associated with the query workload that is running is collected. A subset of database objects is selected from the set of the database objects to be stored on a second storage device. The second storage device is a separate physical device from, and performs faster than, the first storage device. The subset of database objects is stored on the second storage device and all remaining database objects in the set of database objects on the first storage device.