The Hadoop Distributed File System (HDFS) is designed to store very large data sets reliably, and to stream those data sets at high bandwidth to user applications. In a large cluster, thousands of servers both host directly attached storage and execute user application tasks. By distributing storage and computation across many servers, the resource can grow with demand while remaining economical at every size. We describe the architecture of HDFS and report on experience using HDFS to manage 25 petabytes of enterprise data at Yahoo!.

/pdf/the-hadoop-distributed-file-system-14rwlvr6z7.pdf

The Hadoop Distributed File System

In this paper, we review the background and state-of-the-art of big data. We first introduce the general background of big data and review related technologies, such as could computing, Internet of Things, data centers, and Hadoop. We then focus on the four phases of the value chain of big data, i.e., data generation, data acquisition, data storage, and data analysis. For each phase, we introduce the general background, discuss the technical challenges, and review the latest advances. We finally examine the several representative applications of big data, including enterprise management, Internet of Things, online social networks, medial applications, collective intelligence, and smart grid. These discussions aim to provide a comprehensive overview and big-picture to readers of this exciting area. This survey is concluded with a discussion of open problems and future directions.

Big Data: A Survey

Recent technological advancements have led to a deluge of data from distinctive domains (e.g., health care and scientific sensors, user-generated data, Internet and financial companies, and supply chain systems) over the past two decades. The term big data was coined to capture the meaning of this emerging trend. In addition to its sheer volume, big data also exhibits other unique characteristics as compared with traditional data. For instance, big data is commonly unstructured and require more real-time analysis. This development calls for new system architectures for data acquisition, transmission, storage, and large-scale data processing mechanisms. In this paper, we present a literature survey and system tutorial for big data analytics platforms, aiming to provide an overall picture for nonexpert readers and instill a do-it-yourself spirit for advanced audiences to customize their own big-data solutions. First, we present the definition of big data and discuss big data challenges. Next, we present a systematic framework to decompose big data systems into four sequential modules, namely data generation, data acquisition, data storage, and data analytics. These four modules form a big data value chain. Following that, we present a detailed survey of numerous approaches and mechanisms from research and industry communities. In addition, we present the prevalent Hadoop framework for addressing big data challenges. Finally, we outline several evaluation benchmarks and potential research directions for big data systems.

Toward Scalable Systems for Big Data Analytics: A Technology Tutorial

A prominent parallel data processing tool MapReduce is gaining significant momentum from both industry and academia as the volume of data to analyze grows rapidly. While MapReduce is used in many areas where massive data analysis is required, there are still debates on its performance, efficiency per node, and simple abstraction. This survey intends to assist the database and open source communities in understanding various technical aspects of the MapReduce framework. In this survey, we characterize the MapReduce framework and discuss its inherent pros and cons. We then introduce its optimization strategies reported in the recent literature. We also discuss the open issues and challenges raised on parallel data analysis with MapReduce.

https://www.cs.arizona.edu/~bkmoon/papers/sigmodrec11.pdf

Parallel data processing with MapReduce: a survey

Support of high performance queries on large volumes of spatial data becomes increasingly important in many application domains, including geospatial problems in numerous fields, location based services, and emerging scientific applications that are increasingly data- and compute-intensive. The emergence of massive scale spatial data is due to the proliferation of cost effective and ubiquitous positioning technologies, development of high resolution imaging technologies, and contribution from a large number of community users. There are two major challenges for managing and querying massive spatial data to support spatial queries: the explosion of spatial data, and the high computational complexity of spatial queries. In this paper, we present Hadoop-GIS - a scalable and high performance spatial data warehousing system for running large scale spatial queries on Hadoop. Hadoop-GIS supports multiple types of spatial queries on MapReduce through spatial partitioning, customizable spatial query engine RESQUE, implicit parallel spatial query execution on MapReduce, and effective methods for amending query results through handling boundary objects. Hadoop-GIS utilizes global partition indexing and customizable on demand local spatial indexing to achieve efficient query processing. Hadoop-GIS is integrated into Hive to support declarative spatial queries with an integrated architecture. Our experiments have demonstrated the high efficiency of Hadoop-GIS on query response and high scalability to run on commodity clusters. Our comparative experiments have showed that performance of Hadoop-GIS is on par with parallel SDBMS and outperforms SDBMS for compute-intensive queries. Hadoop-GIS is available as a set of library for processing spatial queries, and as an integrated software package in Hive.

/pdf/hadoop-gis-a-high-performance-spatial-data-warehousing-4kutkj0tdp.pdf

Hadoop GIS: a high performance spatial data warehousing system over mapreduce

Increasingly, organizations capture, transform and analyze enormous data sets. Prominent examples include internet companies and e-science. The Map-Reduce scalable dataflow paradigm has become popular for these applications. Its simple, explicit dataflow programming model is favored by some over the traditional high-level declarative approach: SQL. On the other hand, the extreme simplicity of Map-Reduce leads to much low-level hacking to deal with the many-step, branching dataflows that arise in practice. Moreover, users must repeatedly code standard operations such as join by hand. These practices waste time, introduce bugs, harm readability, and impede optimizations.Pig is a high-level dataflow system that aims at a sweet spot between SQL and Map-Reduce. Pig offers SQL-style high-level data manipulation constructs, which can be assembled in an explicit dataflow and interleaved with custom Map- and Reduce-style functions or executables. Pig programs are compiled into sequences of Map-Reduce jobs, and executed in the Hadoop Map-Reduce environment. Both Pig and Hadoop are open-source projects administered by the Apache Software Foundation.This paper describes the challenges we faced in developing Pig, and reports performance comparisons between Pig execution and raw Map-Reduce execution.

/pdf/building-a-high-level-dataflow-system-on-top-of-map-reduce-3ejyj9bo8a.pdf

Building a high-level dataflow system on top of Map-Reduce: the Pig experience

A method attributes credit (802) for conversions to a plurality of event types. A plurality of probabilities that a single impression of each of a plurality of respective event types causes a conversion by one of a plurality of users are defined (804). A processor is used to select values for the plurality of probabilities so as to minimize a total squared error (806) based on differences between an observed outcome for each user and a respective expected outcome for each user calculated from the plurality of probabilities (808).

Methods to attribute conversions for online advertisement campaigns

A device, system, and method are directed towards combining a plurality of queries to a database into a combined execution plan. The plurality of queries is received. The queries may be Structured Query Language (SQL) statements. The database may be a relational database. The plurality of queries is transformed into a plurality of parse trees. A determination is made whether the plurality of queries operates on at least the same portion of the same table. If so, then the plurality of query trees is query-optimized. The plurality of query trees are combined into a master query tree based on similar nodes in the plurality of query trees. A split node in the master query tree represents non-similarities between the plurality of query trees. The master query tree is transformed into an execution plan. The execution plan is applied to a database to return at least one result.

Clustered query support for a database query engine

A directory server includes a supplier server, a consumer server in communication with the supplier server, a plurality of pluggable services that manage replication of data contained within the directory server from the supplier server to the consumer server, and a change log maintained on the consumer server of data replicated to the consumer server. The replication of data is managed by the plurality of pluggable services using the change log.

Replication architecture for a directory server

A method of addressing an entry in a directory server including generating a unique identifier for the entry, creating an encoded address by encoding the unique identifier into a distinguished name, and specifying the entry using the encoded address for a plurality of operations. A method of addressing an entry in a directory server, including generating a unique identifier for the entry, creating an encoded address by encoding the unique identifier into a control, and specifying the entry using the encoded address for a plurality of operations. A unique identifier-based addressing system for a directory server, including a unique identifier generated for an entry and an encoded address created by encoding the unique identifier into a distinguished name. The entry is specified using the encoded address for a plurality of operations.

Olga Natkovich

Papers

Building a high-level dataflow system on top of Map-Reduce: the Pig experience

Methods to attribute conversions for online advertisement campaigns

Clustered query support for a database query engine

Replication architecture for a directory server

Uniqueid-based addressing in a directory server