Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for l1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.

/pdf/distributed-optimization-and-statistical-learning-via-the-8s9gtddw0d.pdf

Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers

We present Resilient Distributed Datasets (RDDs), a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner. RDDs are motivated by two types of applications that current computing frameworks handle inefficiently: iterative algorithms and interactive data mining tools. In both cases, keeping data in memory can improve performance by an order of magnitude. To achieve fault tolerance efficiently, RDDs provide a restricted form of shared memory, based on coarse-grained transformations rather than fine-grained updates to shared state. However, we show that RDDs are expressive enough to capture a wide class of computations, including recent specialized programming models for iterative jobs, such as Pregel, and new applications that these models do not capture. We have implemented RDDs in a system called Spark, which we evaluate through a variety of user applications and benchmarks.

/pdf/resilient-distributed-datasets-a-fault-tolerant-abstraction-1jmy0nuhco.pdf

Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing

In this paper, we review the background and state-of-the-art of big data. We first introduce the general background of big data and review related technologies, such as could computing, Internet of Things, data centers, and Hadoop. We then focus on the four phases of the value chain of big data, i.e., data generation, data acquisition, data storage, and data analysis. For each phase, we introduce the general background, discuss the technical challenges, and review the latest advances. We finally examine the several representative applications of big data, including enterprise management, Internet of Things, online social networks, medial applications, collective intelligence, and smart grid. These discussions aim to provide a comprehensive overview and big-picture to readers of this exciting area. This survey is concluded with a discussion of open problems and future directions.

Big Data: A Survey

The initial design of Apache Hadoop [1] was tightly focused on running massive, MapReduce jobs to process a web crawl. For increasingly diverse companies, Hadoop has become the data and computational agora---the de facto place where data and computational resources are shared and accessed. This broad adoption and ubiquitous usage has stretched the initial design well beyond its intended target, exposing two key shortcomings: 1) tight coupling of a specific programming model with the resource management infrastructure, forcing developers to abuse the MapReduce programming model, and 2) centralized handling of jobs' control flow, which resulted in endless scalability concerns for the scheduler. In this paper, we summarize the design, development, and current state of deployment of the next generation of Hadoop's compute platform: YARN. The new architecture we introduced decouples the programming model from the resource management infrastructure, and delegates many scheduling functions (e.g., task fault-tolerance) to per-application components. We provide experimental evidence demonstrating the improvements we made, confirm improved efficiency by reporting the experience of running YARN on production environments (including 100% of Yahoo! grids), and confirm the flexibility claims by discussing the porting of several programming frameworks onto YARN viz. Dryad, Giraph, Hoya, Hadoop MapReduce, REEF, Spark, Storm, Tez.

/pdf/apache-hadoop-yarn-yet-another-resource-negotiator-26ypcfjhgi.pdf

Apache Hadoop YARN: yet another resource negotiator

The popularity of the Web and Internet commerce provides many extremely large datasets from which information can be gleaned by data mining. This book focuses on practical algorithms that have been used to solve key problems in data mining and which can be used on even the largest datasets. It begins with a discussion of the map-reduce framework, an important tool for parallelizing algorithms automatically. The authors explain the tricks of locality-sensitive hashing and stream processing algorithms for mining data that arrives too fast for exhaustive processing. The PageRank idea and related tricks for organizing the Web are covered next. Other chapters cover the problems of finding frequent itemsets and clustering. The final chapters cover two applications: recommendation systems and Web advertising, each vital in e-commerce. Written by two authorities in database and Web technologies, this book is essential reading for students and practitioners alike.

/pdf/mining-of-massive-datasets-2wssu726p1.pdf

Mining of Massive Datasets

The MMXTM technology and SSE/SSE2 (streaming-SIMD-extensions) introduced a variety of SIMD instructions that can exploit data parallelism in numerical and multimedia applications. In particular, new saturation and clipping instructions can boost the performance of applications that make extensive use of such operations. Unfortunately, due to the lack of support for saturation and clipping operators in e.g. C/C++ or Fortran, these operations must be explicitly coded with conditional constructs that test the value of operands before actual wrap-around arithmetic is performed. As a result, inlineassembly or language extensions are most commonly used to exploit the new instructions. In this paper, we explore an alternative approach, where the compiler automatically maps high-level saturation and clipping idioms onto efficient low-level instructions. The effectiveness of this approach is demonstrated with some experiments.

Automatic detection of saturation and clipping idioms

javar: A prototype Java restructuring compiler

Many practical computing problems concern large graphs. Standard examples include the Web graph and various social networks. The scale of these graphs—in some cases billions of vertices, trillions of edges—poses challenges to their efficient processing. Despite the ubiquity of large graphs and their commercial importance, we know of no scalable general-purpose system for implementing graph algorithms in a distributed environment. To address distributed processing of real-life graphs, we defined a model of computation and realized it through a scalable and fault-tolerant system called Pregel, with an expressive and flexible API. The high-level organization of Pregel programs is inspired by Valiant’s Bulk Synchronous Parallel model. Pregel computations consist of a sequence of iterations, called superstep s. During a superstep the framework invokes a userdefined Compute() function for each vertex, conceptually in parallel. The function specifies behavior at a single vertex v and a single superstep S. It can read messages sent to v in superstep S − 1, send messages to other vertices that will be received at superstep S + 1, and modify the state of v and its outgoing edges. Messages are typically sent along outgoing edges, but a message may be sent to any vertex whose identifier is known. A program terminates when all vertices declare that they are done. The input and output are both directed graphs. They are often but not always isomorphic, because vertices and edges can be added and removed during computation. Userdefined handlers are applied to resolve conflicts for concurrent mutations. The vertex-centric flavor of programming in Pregel is similar to the MapReduce model in that programmers focus on a local action, processing a single item at a time, which the system then lifts to computation on a large dataset. The synchronicity of the Pregel model simplifies writing correct programs and simplifies reasoning about the inter-

Pregel: a system for large-scale graph processing - "ABSTRACT"

A note on native level 1 BLAS in Java

This paper provides a manual for javab, a prototype tool that supports the automatic detection and exploitation of implicit loop parallelism in JVM bytecode. Implicit parallelism is made explicit by means of the multi-threading mechanism provided by the JVM. Automatically exploiting implicit parallelism at bytecode level can be done independently from the source program and platform from which the bytecode was obtained, and independently from the platform on which the bytecode eventually will run. The parallelized bytecode remains architecturally neutral and may exhibit speedup on any platform that supports the true parallel execution of JVM threads. This project is supported by DARPA under contract ARPA F19628–94–C–0057 through a sub-contract from Syracuse University. © 1998 John Wiley & Sons, Ltd.

Aart J. C. Bik

Papers

Automatic detection of saturation and clipping idioms

javar: A prototype Java restructuring compiler

Pregel: a system for large-scale graph processing - "ABSTRACT"

A note on native level 1 BLAS in Java

A prototype bytecode parallelization tool