We present Resilient Distributed Datasets (RDDs), a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner. RDDs are motivated by two types of applications that current computing frameworks handle inefficiently: iterative algorithms and interactive data mining tools. In both cases, keeping data in memory can improve performance by an order of magnitude. To achieve fault tolerance efficiently, RDDs provide a restricted form of shared memory, based on coarse-grained transformations rather than fine-grained updates to shared state. However, we show that RDDs are expressive enough to capture a wide class of computations, including recent specialized programming models for iterative jobs, such as Pregel, and new applications that these models do not capture. We have implemented RDDs in a system called Spark, which we evaluate through a variety of user applications and benchmarks.

/pdf/resilient-distributed-datasets-a-fault-tolerant-abstraction-1jmy0nuhco.pdf

Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing

Stochastic Gradient Descent (SGD) is a popular algorithm that can achieve state-of-the-art performance on a variety of machine learning tasks. Several researchers have recently proposed schemes to parallelize SGD, but all require performance-destroying memory locking and synchronization. This work aims to show using novel theoretical analysis, algorithms, and implementation that SGD can be implemented without any locking. We present an update scheme called HOGWILD! which allows processors access to shared memory with the possibility of overwriting each other's work. We show that when the associated optimization problem is sparse, meaning most gradient updates only modify small parts of the decision variable, then HOGWILD! achieves a nearly optimal rate of convergence. We demonstrate experimentally that HOGWILD! outperforms alternative schemes that use locking by an order of magnitude.

/pdf/hogwild-a-lock-free-approach-to-parallelizing-stochastic-2zrjsnvj8d.pdf

Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications

https://dl.acm.org/doi/pdf/10.1145/2934664

Apache Spark: a unified engine for big data processing

HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Spanner is Google's scalable, multi-version, globally-distributed, and synchronously-replicated database. It is the first system to distribute data at global scale and support externally-consistent distributed transactions. This paper describes how Spanner is structured, its feature set, the rationale underlying various design decisions, and a novel time API that exposes clock uncertainty. This API and its implementation are critical to supporting external consistency and a variety of powerful features: nonblocking reads in the past, lock-free read-only transactions, and atomic schema changes, across all of Spanner.

/pdf/spanner-google-s-globally-distributed-database-57sdsy7wpa.pdf

Spanner: Google's globally-distributed database

Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data. By combining multi-level execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system scales to thousands of CPUs and petabytes of data, and has thousands of users at Google. In this paper, we describe the architecture and implementation of Dremel, and explain how it complements MapReduce-based computing. We present a novel columnar storage representation for nested records and discuss experiments on few-thousand node instances of the system.

/pdf/dremel-interactive-analysis-of-web-scale-datasets-4p5uh12hsi.pdf

Dremel: interactive analysis of web-scale datasets

Dremel is a scalable, interactive ad hoc query system for analysis of read-only nested data. By combining multilevel execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system scales to thousands of CPUs and petabytes of data, and has thousands of users at Google. In this paper, we describe the architecture and implementation of Dremel, and explain how it complements MapReduce-based computing. We present a novel columnar storage representation for nested records and discuss experiments on few-thousand node instances of the system.

Google's Dremel was one of the first systems that combined a set of architectural principles that have become a common practice in today's cloud-native analytics tools, including disaggregated storage and compute, in situ analysis, and columnar storage for semistructured data. In this paper, we discuss how these ideas evolved in the past decade and became the foundation for Google BigQuery.

/pdf/dremel-a-decade-of-interactive-sql-analysis-at-web-scale-jwdoeb8f7u.pdf

Dremel: a decade of interactive SQL analysis at web scale

A computer system accesses a collection of data records. Each record in the collection includes a plurality of data values and a plurality of data elements that identify semantics of corresponding data values from the plurality of data values. One or more of the data records each include multiple instances of a same data element and include data values that correspond to the multiple instances of the same data element. The computer system generates a collection of columnar stripes. The collection of columnar stripes includes the data values from each data record in the collection of data records. Each columnar stripe in the collection of columnar stripes includes all data values that correspond to a specific data element from each of the records in the collection of records.

Columnar storage representations of records

Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

Matthew B. Tolton

Papers

Dremel: interactive analysis of web-scale datasets

Dremel: interactive analysis of web-scale datasets

Dremel: a decade of interactive SQL analysis at web scale

Columnar storage representations of records

Repartitioning data in a distributed computing system