Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Journal ArticleDOI
On-the-fly scheduling versus reservation-based scheduling for unpredictable workflows:
TL;DR: It is shown how these big data workflows have a unique set of characteristics that pose challenges for leveraging HPC methodologies, particularly in scheduling, and how on-the-fly scheduling approaches can deliver benefits in both system-level and user-level performance measures.
Journal ArticleDOI
Optimal design of urban transportation planning based on big data
Wei Sai,Hongzhi Wang +1 more
TL;DR: The optimization design of urban traffic planning based on big data is studied, and the traffic signal quality prediction model is established by using the improved regression tree algorithm and smote method to reconstruct the sample data set to achieve the balance.
Journal ArticleDOI
Boosting evolutionary algorithm configuration
TL;DR: New algorithmic ideas to improve state-of-the-art solver configurators automatically by tuning are presented, including a forward-simulation method to improve parallel performance, an improvement to the configuration process itself, and a new technique for instance-specific solver configuration.
Book ChapterDOI
Deterministic Model for Distributed Speculative Stream Processing
Igor Kuralenok,Artem Trofimov,Artem Trofimov,Nikita Marshalkin,Nikita Marshalkin,Boris Novikov,Boris Novikov +6 more
TL;DR: A speculative model based on MapReduce-complete set of operations that allows for determinism and low-latency is introduced that can outperform existing solutions due to low overhead of optimistic synchronization.
Proceedings ArticleDOI
Towards Making Distributed RDF Processing FLINKer
TL;DR: This position paper proposes to manage large RDF datasets in Flink, a well-known scalable distributed Big Data processing framework.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.