Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Proceedings ArticleDOI
ROBOTune: High-Dimensional Configuration Tuning for Cluster-Based Data Analytics
Muhib Khan,Weikuan Yu +1 more
TL;DR: In this article, the authors proposed a robust tuning framework called RobOTune, which performs parameter selection through a Random Forests based model to reduce the dimensionality of analytics configuration space and employs Bayesian optimization to overcome the complex nature of the configuration-performance relationship.
Posted Content
GraphBolt: Streaming Graph Approximations on Big Data.
TL;DR: This work presents GraphBolt, an innovative model for approximate graph processing, implemented in Apache Flink, and analyzes it with the case study of the PageRank algorithm, perhaps the most famous measure of vertex centrality used to rank websites in search engine results.
Posted Content
Parallel Index-based Stream Join on a Multicore CPU
TL;DR: In this article, the Partitioned In-memory Merge-Tree (PIM) data structure is introduced to address the challenges that arise when indexing highly dynamic data, which are common in streaming settings.
Journal ArticleDOI
Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme
Hellen Paz,M. A. G. Maia,Fernando Moraes,Ricardo Lustosa,Lilia Costa,Samuel Macêdo,Marcos Barreto,Anderson Ara +7 more
TL;DR: An analysis of Brazilian public data from the Bolsa Família Programme, comprising a large data set with 1.26 billion observations, is presented to understand how this social program acts in different cities, as well as to identify potentially important variables reflecting its utilization rate.
Proceedings ArticleDOI
Analyzing and Optimizing Java Code Generation for Apache Spark Query Plan
TL;DR: Two types of problems were analyzed by inspecting generated code, namely, access to column-oriented storage and to a primitive-type array, and optimizations that can eliminate inefficient code were devised to solve the issues.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.