https://pure.mpg.de/pubman/item/item_2503017_5/component/file_2560806/ams_2503017.pdf

Challenges and perspectives of metaproteomic data analysis

Spinnaker is an experimental datastore that is designed to run on a large cluster of commodity servers in a single datacenter. It features key-based range partitioning, 3-way replication, and a transactional get-put API with the option to choose either strong or timeline consistency on reads. This paper describes Spinnaker's Paxos-based replication protocol. The use of Paxos ensures that a data partition in Spinnaker will be available for reads and writes as long a majority of its replicas are alive. Unlike traditional master-slave replication, this is true regardless of the failure sequence that occurs. We show that Paxos replication can be competitive with alternatives that provide weaker consistency guarantees. Compared to an eventually consistent datastore, we show that Spinnaker can be as fast or even faster on reads and only 5% to 10% slower on writes.

Using Paxos to Build a Scalable, Consistent, and Highly Available Datastore

The next generation of high-performance networks with remote direct memory access (RDMA) capabilities requires a fundamental rethinking of the design of distributed in-memory DBMSs. These systems are commonly built under the assumption that the network is the primary bottleneck and should be avoided at all costs, but this assumption no longer holds. For instance, with InfiniBand FDR 4×, the bandwidth available to transfer data across the network is in the same ballpark as the bandwidth of one memory channel. Moreover, RDMA transfer latencies continue to rapidly improve as well. In this paper, we first argue that traditional distributed DBMS architectures cannot take full advantage of high-performance networks and suggest a new architecture to address this problem. Then, we discuss initial results from a prototype implementation of our proposed architecture for OLTP and OLAP, showing remarkable performance improvements over existing designs.

/pdf/the-end-of-slow-networks-it-s-time-for-a-redesign-8c75s04a7n.pdf

The end of slow networks: it's time for a redesign

Next generation high-performance RDMA-capable networks will require a fundamental rethinking of the design and architecture of modern distributed DBMSs. These systems are commonly designed and optimized under the assumption that the network is the bottleneck: the network is slow and "thin", and thus needs to be avoided as much as possible. Yet this assumption no longer holds true. With InfiniBand FDR 4x, the bandwidth available to transfer data across network is in the same ballpark as the bandwidth of one memory channel, and it increases even further with the most recent EDR standard. Moreover, with the increasing advances of RDMA, the latency improves similarly fast. In this paper, we first argue that the "old" distributed database design is not capable of taking full advantage of the network. Second, we propose architectural redesigns for OLTP, OLAP and advanced analytical frameworks to take better advantage of the improved bandwidth, latency and RDMA capabilities. Finally, for each of the workload categories, we show that remarkable performance improvements can be achieved.

The End of Slow Networks: It's Time for a Redesign

The common wisdom is that distributed transactions do not scale. But what if distributed transactions could be made scalable using the next generation of networks and a redesign of distributed databases? There would no longer be a need for developers to worry about co-partitioning schemes to achieve decent performance. Application development would become easier as data placement would no longer determine how scalable an application is. Hardware provisioning would be simplified as the system administrator can expect a linear scale-out when adding more machines rather than some complex sub-linear function, which is highly application specific.In this paper, we present the design of our novel scalable database system NAM-DB and show that distributed transactions with the very common Snapshot Isolation guarantee can indeed scale using the next generation of RDMA-enabled network technology without any inherent bottlenecks. Our experiments with the TPC-C benchmark show that our system scales linearly to over 6.5 million new-order (14.5 million total) distributed transactions per second on 56 machines.

/pdf/the-end-of-a-myth-distributed-transactions-can-scale-18k8z1dy3k.pdf

The end of a myth: distributed transactions can scale

Database scale-out is commonly implemented by partitioning data across several database instances. This approach, however, has several restrictions. In particular, partitioned databases are inflexible in large-scale deployments and assume a partition-friendly workload in order to scale. In this paper, we analyze an alternative architecture design for distributed relational databases that overcomes the limitations of partitioned databases. The architecture is based on two fundamental principles: We decouple query processing and transaction management from data storage, and we share data across query processing nodes. The combination of these design choices provides scalability, elasticity, and operational flexibility without making any assumptions on the workload. As a drawback, sharing data among multiple database nodes causes synchronization overhead. To address this limitation, we introduce techniques for scalable transaction processing in shared-data environments. Specifically, we describe mechanisms for efficient data access, concurrency control, and data buffering. In combination with new hardware trends, the techniques enable performance characteristics that top state-of-the-art partitioned databases.

On the Design and Scalability of Distributed Shared-Data Databases

Modern data-centric flows in the telecommunications industry require real time analytical processing over a rapidly changing and large dataset. The traditional approach of separating OLTP and OLAP workloads cannot satisfy this requirement. Instead, a new class of integrated solutions for handling hybrid workloads is needed. This paper presents an industrial use case and a novel architecture that integrates key-value-based event processing and SQL-based analytical processing on the same distributed store while minimizing the total cost of ownership. Our approach combines several well-known techniques such as shared scans, delta processing, a PAX-fashioned storage layout, and an interleaving of scanning and delta merging in a completely new way. Performance experiments show that our system scales out linearly with the number of servers. For instance, our system sustains event streams of 100,000 events per second while simultaneously processing 100 ad-hoc analytical queries per second, using a cluster of 12 commodity servers. In doing so, our system meets all response time goals of our telecommunication customers; that is, 10 milliseconds per event and 100 milliseconds for an ad-hoc analytical query. Moreover, our system beats commercial competitors by a factor of 2.5 in analytical and two orders of magnitude in update performance.

Analytics in Motion: High Performance Event-Processing AND Real-Time Analytics in the Same Database

Method for querying and updating entries in a data base The invention relates to a method for querying and updating entries in a data base, the data base comprising a main data structure for storing data base entries and a delta data structure for storing new entries, the method comprising the following steps: receiving (101) a plurality of data base queries, aggregating (103) a received plurality of data base queries to obtain a batch of data base queries, performing (105) a shared scan of the main data structure with the batch of data base queries, wherein the data base entries in the main data structure are queried with respect to each data base query in the batch of data base queries, after the step of performing (105) the shared scan, merging (107) the main data structure with the delta data structure to update the main data structure with the received new entries.

Method for querying and updating entries in a data base

An event processing system (100) is configured to process a stream of events (101) operating on a database system, the event processing system (100) comprising an event load balancing unit (103), a plurality of event computing nodes (105a, 105b, 105c) and a plurality of event state stores (109a, 109b, 109c) which are separated from the event computing nodes (105a, 105b, 105c), wherein the event load balancing unit (103) is configured to route the stream of events (101) to the plurality of event computing nodes (105a, 105b, 105c) according to an event load balancing criteria; wherein the plurality of event state stores (109a, 109b, 109c) are configured to store states of the plurality of event computing nodes (105a, 105b, 105c) for maintaining a state of the event processing; and wherein the plurality of event computing nodes (105a, 105b, 105c) are configured to process the events (101) received from the event load balancing unit (103), to change their states in accordance to the processing of the events and to update the plurality of event state stores (109a, 109b, 109c) based on their changed states.

Event processing system

To scale-out databases today, partitioning data across several database instances is the common technique. This approach is typically referred to as shared-nothing architecture. In this paper, we propose a new alter- native architecture design for distributed relational databases based on two fundamental principles: We logically decouple query processing and transaction management from data storage and we share data across all query processing nodes. The combination of these design choices provides operational exibility, a property that enables cost-ecient database in- frastructures and that becomes increasingly important with the growing market penetration of cloud computing. As a drawback, sharing data among several database nodes adds substantial synchronization overhead. We present techniques for ecient data access and concurrency control to counter this eect. These techniques are enabled by recent hardware trends such as in-memory storage and low-latency networking and pro- vide scalability characteristics that top state-of-the-art shared-nothing databases.

Thomas Etter

Papers

On the Design and Scalability of Distributed Shared-Data Databases

Analytics in Motion: High Performance Event-Processing AND Real-Time Analytics in the Same Database

Method for querying and updating entries in a data base

Event processing system

On the Design and Scalability of Distributed Shared-Memory Databases