While the use of MapReduce systems (such as Hadoop) for large scale data analysis has been widely recognized and studied, we have recently seen an explosion in the number of systems developed for cloud data serving. These newer systems address "cloud OLTP" applications, though they typically do not support ACID transactions. Examples of systems proposed for cloud serving use include BigTable, PNUTS, Cassandra, HBase, Azure, CouchDB, SimpleDB, Voldemort, and many others. Further, they are being applied to a diverse range of applications that differ considerably from traditional (e.g., TPC-C like) serving workloads. The number of emerging cloud serving systems and the wide range of proposed applications, coupled with a lack of apples-to-apples performance comparisons, makes it difficult to understand the tradeoffs between systems and the workloads for which they are suited. We present the "Yahoo! Cloud Serving Benchmark" (YCSB) framework, with the goal of facilitating performance comparisons of the new generation of cloud data serving systems. We define a core set of benchmarks and report results for four widely used systems: Cassandra, HBase, Yahoo!'s PNUTS, and a simple sharded MySQL implementation. We also hope to foster the development of additional cloud benchmark suites that represent other classes of applications by making our benchmark tool available via open source. In this regard, a key feature of the YCSB framework/tool is that it is extensible--it supports easy definition of new workloads, in addition to making it easy to benchmark new systems.

/pdf/benchmarking-cloud-serving-systems-with-ycsb-3u47swhmki.pdf

Benchmarking cloud serving systems with YCSB

The popularity of the Web and Internet commerce provides many extremely large datasets from which information can be gleaned by data mining. This book focuses on practical algorithms that have been used to solve key problems in data mining and which can be used on even the largest datasets. It begins with a discussion of the map-reduce framework, an important tool for parallelizing algorithms automatically. The authors explain the tricks of locality-sensitive hashing and stream processing algorithms for mining data that arrives too fast for exhaustive processing. The PageRank idea and related tricks for organizing the Web are covered next. Other chapters cover the problems of finding frequent itemsets and clustering. The final chapters cover two applications: recommendation systems and Web advertising, each vital in e-commerce. Written by two authorities in database and Web technologies, this book is essential reading for students and practitioners alike.

/pdf/mining-of-massive-datasets-2wssu726p1.pdf

Mining of Massive Datasets

Computer networks lack a general control paradigm, as traditional networks do not provide any network-wide management abstractions. As a result, each new function (such as routing) must provide its own state distribution, element discovery, and failure recovery mechanisms. We believe this lack of a common control platform has significantly hindered the development of flexible, reliable and feature-rich network control planes.To address this, we present Onix, a platform on top of which a network control plane can be implemented as a distributed system. Control planes written within Onix operate on a global view of the network, and use basic state distribution primitives provided by the platform. Thus Onix provides a general API for control plane implementations, while allowing them to make their own trade-offs among consistency, durability, and scalability.

/pdf/onix-a-distributed-control-platform-for-large-scale-2nyb613mj2.pdf

Onix: a distributed control platform for large-scale production networks

Spanner is Google's scalable, multi-version, globally-distributed, and synchronously-replicated database. It is the first system to distribute data at global scale and support externally-consistent distributed transactions. This paper describes how Spanner is structured, its feature set, the rationale underlying various design decisions, and a novel time API that exposes clock uncertainty. This API and its implementation are critical to supporting external consistency and a variety of powerful features: nonblocking reads in the past, lock-free read-only transactions, and atomic schema changes, across all of Spanner.

/pdf/spanner-google-s-globally-distributed-database-57sdsy7wpa.pdf

Spanner: Google's globally-distributed database

Recent technological advancements have led to a deluge of data from distinctive domains (e.g., health care and scientific sensors, user-generated data, Internet and financial companies, and supply chain systems) over the past two decades. The term big data was coined to capture the meaning of this emerging trend. In addition to its sheer volume, big data also exhibits other unique characteristics as compared with traditional data. For instance, big data is commonly unstructured and require more real-time analysis. This development calls for new system architectures for data acquisition, transmission, storage, and large-scale data processing mechanisms. In this paper, we present a literature survey and system tutorial for big data analytics platforms, aiming to provide an overall picture for nonexpert readers and instill a do-it-yourself spirit for advanced audiences to customize their own big-data solutions. First, we present the definition of big data and discuss big data challenges. Next, we present a systematic framework to decompose big data systems into four sequential modules, namely data generation, data acquisition, data storage, and data analytics. These four modules form a big data value chain. Following that, we present a detailed survey of numerous approaches and mechanisms from research and industry communities. In addition, we present the prevalent Hadoop framework for addressing big data challenges. Finally, we outline several evaluation benchmarks and potential research directions for big data systems.

Toward Scalable Systems for Big Data Analytics: A Technology Tutorial

We describe PNUTS, a massively parallel and geographically distributed database system for Yahoo!'s web applications. PNUTS provides data storage organized as hashed or ordered tables, low latency for large numbers of concurrent requests including updates and queries, and novel per-record consistency guarantees. It is a hosted, centrally managed, and geographically distributed service, and utilizes automated load-balancing and failover to reduce operational complexity. The first version of the system is currently serving in production. We describe the motivation for PNUTS and the design and implementation of its table storage and replication layers, and then present experimental results.

/pdf/pnuts-yahoo-s-hosted-data-serving-platform-tirb704wyn.pdf

PNUTS: Yahoo!'s hosted data serving platform

A system for storing a distributed hash table. The system includes a storage unit, a tablet controller, a router, and a transaction bank. The storage unit has a plurality of tablets forming a hash table and each of the tablets includes multiple records. The tablet controller maintains a relationship between each tablet and the storage unit. The router hashes a record's key to determine the tablet associated with each record. Further, the router distributes messages from clients to the storage units based on the tablet-storage unit relationship thereby serving as a layer of indirection. The transaction bank propagates updates made in one record to all other replicas of the record.

System for storing distributed hashtables

A system for a distributed database implementing a dynamic mastership strategy. The system includes a multiple data centers, each having a storage unit to store a set of records. Each data center stores its own replica of the set of records and each record includes a field that indicates which data center is assigned to be the master for that record. Since each of the data centers can he geographically distributed, one record may be more efficiently edited with the master being one geographic region while another record, possibly belonging to a different user, may be more efficiently edited with the master being located in another geographic region.

Asynchronously replicated database system using dynamic mastership

Methods and apparatuses are provided for dynamically reorganizing the data within a replicated database system. One method, for example, includes performing a split operation across a plurality of replicated databases with regard to an existing partition therein, wherein the existing partition comprises a plurality of data records and the two new partitions each include at least a portion of the plurality of data records, and allowing at least one type of access to the plurality of data records during the split operation.

Dynamic data reorganization to accommodate growth across replicated databases

A technique is described that reduces the complexity and resource consumption associated with performing record expiry in a distributed database system. In accordance with the technique, a record is checked to see if it has expired only when it has been accessed for a read or a write. If at the time of a read a record is determined to have expired, then it is not served. If at the time of a write a record is determined to have expired, then the write is treated as an insertion of a new record, and steps are taken to treat the insertion consistently with regard to the previous expired version. A background process is used to delete records that have not been written to or actively deleted by a client after expiration.

Daniel Weaver

Papers

PNUTS: Yahoo!'s hosted data serving platform

System for storing distributed hashtables

Asynchronously replicated database system using dynamic mastership

Dynamic data reorganization to accommodate growth across replicated databases

Decentralized record expiry