Dryad is a general-purpose distributed execution engine for coarse-grain data-parallel applications. A Dryad application combines computational "vertices" with communication "channels" to form a dataflow graph. Dryad runs the application by executing the vertices of this graph on a set of available computers, communicating as appropriate through flies, TCP pipes, and shared-memory FIFOs.The vertices provided by the application developer are quite simple and are usually written as sequential programs with no thread creation or locking. Concurrency arises from Dryad scheduling vertices to run simultaneously on multiple computers, or on multiple CPU cores within a computer. The application can discover the size and placement of data at run time, and modify the graph as the computation progresses to make efficient use of the available resources.Dryad is designed to scale from powerful multi-core single computers, through small clusters of computers, to data centers with thousands of computers. The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.

/pdf/dryad-distributed-data-parallel-programs-from-sequential-1k73e2cctm.pdf

Dryad: distributed data-parallel programs from sequential building blocks

In this paper, we review the background and state-of-the-art of big data. We first introduce the general background of big data and review related technologies, such as could computing, Internet of Things, data centers, and Hadoop. We then focus on the four phases of the value chain of big data, i.e., data generation, data acquisition, data storage, and data analysis. For each phase, we introduce the general background, discuss the technical challenges, and review the latest advances. We finally examine the several representative applications of big data, including enterprise management, Internet of Things, online social networks, medial applications, collective intelligence, and smart grid. These discussions aim to provide a comprehensive overview and big-picture to readers of this exciting area. This survey is concluded with a discussion of open problems and future directions.

Big Data: A Survey

The initial design of Apache Hadoop [1] was tightly focused on running massive, MapReduce jobs to process a web crawl. For increasingly diverse companies, Hadoop has become the data and computational agora---the de facto place where data and computational resources are shared and accessed. This broad adoption and ubiquitous usage has stretched the initial design well beyond its intended target, exposing two key shortcomings: 1) tight coupling of a specific programming model with the resource management infrastructure, forcing developers to abuse the MapReduce programming model, and 2) centralized handling of jobs' control flow, which resulted in endless scalability concerns for the scheduler. In this paper, we summarize the design, development, and current state of deployment of the next generation of Hadoop's compute platform: YARN. The new architecture we introduced decouples the programming model from the resource management infrastructure, and delegates many scheduling functions (e.g., task fault-tolerance) to per-application components. We provide experimental evidence demonstrating the improvements we made, confirm improved efficiency by reporting the experience of running YARN on production environments (including 100% of Yahoo! grids), and confirm the flexibility claims by discussing the porting of several programming frameworks onto YARN viz. Dryad, Giraph, Hoya, Hadoop MapReduce, REEF, Spark, Storm, Tez.

/pdf/apache-hadoop-yarn-yet-another-resource-negotiator-26ypcfjhgi.pdf

Apache Hadoop YARN: yet another resource negotiator

The size of data sets being collected and analyzed in the industry for business intelligence is growing rapidly, making traditional warehousing solutions prohibitively expensive. Hadoop [3] is a popular open-source map-reduce implementation which is being used as an alternative to store and process extremely large data sets on commodity hardware. However, the map-reduce programming model is very low level and requires developers to write custom programs which are hard to maintain and reuse.

/pdf/hive-a-warehousing-solution-over-a-map-reduce-framework-3731d7kldx.pdf

Hive: a warehousing solution over a map-reduce framework

There is currently considerable enthusiasm around the MapReduce (MR) paradigm for large-scale data analysis [17]. Although the basic control flow of this framework has existed in parallel SQL database management systems (DBMS) for over 20 years, some have called MR a dramatically new computing model [8, 17]. In this paper, we describe and compare both paradigms. Furthermore, we evaluate both kinds of systems in terms of performance and development complexity. To this end, we define a benchmark consisting of a collection of tasks that we have run on an open source version of MR as well as on two parallel DBMSs. For each task, we measure each system's performance for various degrees of parallelism on a cluster of 100 nodes. Our results reveal some interesting trade-offs. Although the process to load data into and tune the execution of parallel DBMSs took much longer than the MR system, the observed performance of these DBMSs was strikingly better. We speculate about the causes of the dramatic performance difference and consider implementation concepts that future systems should take from both kinds of architectures.

/pdf/a-comparison-of-approaches-to-large-scale-data-analysis-2ueig83wcw.pdf

A comparison of approaches to large-scale data analysis

Companies providing cloud-scale services have an increasing need to store and analyze massive data sets such as search logs and click streams. For cost and performance reasons, processing is typically done on large clusters of shared-nothing commodity machines. It is imperative to develop a programming model that hides the complexity of the underlying system but provides flexibility by allowing users to extend functionality to meet a variety of requirements.In this paper, we present a new declarative and extensible scripting language, SCOPE (Structured Computations Optimized for Parallel Execution), targeted for this type of massive data analysis. The language is designed for ease of use with no explicit parallelism, while being amenable to efficient parallel execution on large clusters. SCOPE borrows several features from SQL. Data is modeled as sets of rows composed of typed columns. The select statement is retained with inner joins, outer joins, and aggregation allowed. Users can easily define their own functions and implement their own versions of operators: extractors (parsing and constructing rows from a file), processors (row-wise processing), reducers (group-wise processing), and combiners (combining rows from two inputs). SCOPE supports nesting of expressions but also allows a computation to be specified as a series of steps, in a manner often preferred by programmers. We also describe how scripts are compiled into efficient, parallel execution plans and executed on large clusters.

/pdf/scope-easy-and-efficient-parallel-processing-of-massive-data-1519e7zk5j.pdf

SCOPE: easy and efficient parallel processing of massive data sets

Method and system for assessing the performance of a server application that acquires performance information from the perspective of a simulated user and has significantly reduced hardware requirements. Particularly, actual user behavior is modeled so that accurate determinations can be made as to the number of users a given server application can adequately support. User behavior is modeled in a client profile that contains user parameters corresponding to the nature, timing, and frequency of user activities in operating a client that in turn corresponds to client tasks. A plurality of processes and process threads are initiated to contact a server as a plurality of simulated clients from a single client computer, each simulated client making a separate logical connection to the server. A task scheduler will schedule the simulated client tasks that are determined for each simulated user by reference to the user parameters in the client profile throughout a work day. The scheduler also introduces a random element so that the tasks simulate natural variability in user behavior. User receivable response times for the task corresponding to simulated user activity are maintained in a log file and the 95th percentile time or score for each task type is calculated. The individual task type scores may be weighted and averaged together to arrive at a weighted average response time. The weighted average response time can then be used as a threshold value to determine the total number of users a server application can adequately support.

Method, system, and product for assessing a server application performance

Companies providing cloud-scale data services have increasing needs to store and analyze massive data sets, such as search logs, click streams, and web graph data. For cost and performance reasons, processing is typically done on large clusters of tens of thousands of commodity machines. Such massive data analysis on large clusters presents new opportunities and challenges for developing a highly scalable and efficient distributed computation system that is easy to program and supports complex system optimization to maximize performance and reliability. In this paper, we describe a distributed computation system, Structured Computations Optimized for Parallel Execution (Scope), targeted for this type of massive data analysis. Scope combines benefits from both traditional parallel databases and MapReduce execution engines to allow easy programmability and deliver massive scalability and high performance through advanced optimization. Similar to parallel databases, the system has a SQL-like declarative scripting language with no explicit parallelism, while being amenable to efficient parallel execution on large clusters. An optimizer is responsible for converting scripts into efficient execution plans for the distributed computation engine. A physical execution plan consists of a directed acyclic graph of vertices. Execution of the plan is orchestrated by a job manager that schedules execution on available machines and provides fault tolerance and recovery, much like MapReduce systems. Scope is being used daily for a variety of data analysis and data mining applications over tens of thousands of machines at Microsoft, powering Bing, and other online services.

/pdf/scope-parallel-databases-meet-mapreduce-1bk82xb2lu.pdf

SCOPE: parallel databases meet MapReduce

A system and method for replica administration without data loss is disclosed. In a replication environment where data is replicated around a network and where any system can make changes to the data, data loss may occur if one copy of the data is deleted before changes made to that copy are replicated to other systems in the network. The present invention describes a robust administration environment which prevents inadvertent data loss by verifying that changes made to a local copy of the data reside on at least one other system in the network. The system and method of the present invention also provide a mechanism to allow an administrator to bypass such safeguards in appropriate circumstances in order to handle special cases such as total removal of the data from the network. The replica administration environment is implemented by defining various states that represent the level of participation in the replication of the data. For example, an active state can indicate full participation while a deleted state can indicate no participation. In addition to an active state and a deleted state, one or more intermediate states may be defined. The checks and safeguards can be performed in these intermediate states. In appropriate circumstances, an administrator may force the transition from certain of the intermediate states to either the active state in order to return a system to full participation or to the deleted state in order to bypass the safeguards of the present invention.

Replica administration without data loss in a store and forward replication enterprise

A system and method for distributed conflict resolution between different versions of the same data object which are replicated across a computer network is disclosed. The invention can be utilized with and adapted to any desired replication process. The conflict resolution process allows identification and resolution of conflicts with little or no increased message traffic on the network. The conflict resolution process is independent of the order in which conflicts are resolved and all replica nodes in the enterprise will eventually resolve the conflict in exactly the same manner. The method is adapted for resolution of conflicts between data objects or between objects which define the properties of sets of data objects. The final selection of which version of the object should be maintained and which version of the object should be deleted can be fully automated, completely manual, or a combination of the two.

Darren A. Shakib

Papers

SCOPE: easy and efficient parallel processing of massive data sets

Method, system, and product for assessing a server application performance

SCOPE: parallel databases meet MapReduce

Replica administration without data loss in a store and forward replication enterprise

System and method for distributed conflict resolution between data objects replicated across a computer network