Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Proceedings ArticleDOI
Flare: optimizing apache spark with native compilation for scale-up architectures and medium-size data
Grégory M. Essertel,Ruby Y. Tahboub,James M. Decker,Kevin J. Brown,Kunle Olukotun,Tiark Rompf +5 more
TL;DR: Flare is presented, an accelerator module for Spark that delivers order of magnitude speedups on scale-up architectures for a large class of applications, Inspired by query compilation techniques from main-memory database systems, which incorporates a code generation strategy designed to match the unique aspects of Spark and the characteristics of scale- up architectures.
Journal ArticleDOI
A distributed approach to multi-objective evolutionary generation of fuzzy rule-based classifiers from big data
TL;DR: This paper proposes the first distributed multi-objective evolutionary approach to learn concurrently the rule and data bases of FRBCs by maximizing accuracy and minimizing complexity, and presents that the distributed version can efficiently extract compact rule bases with high accuracy, preserving the interpretability of the rule base, and can manage big datasets even with modest hardware support.
Proceedings Article
dShark: A General, Easy to Program and Scalable Framework for Analyzing In-network Packet Traces
TL;DR: dShark allows intuitive groupings of packets across multiple traces that are robust to header transformations and capture noise, offering simple streaming data abstractions for network operators.
Proceedings ArticleDOI
Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines
TL;DR: Rhino provides a handover protocol and a state migration protocol to consistently and efficiently migrate stream processing among servers and reconfigures a running query 15 times faster than the state-of-the-art, and reduces latency by three orders of magnitude upon a reconfiguration.
Journal ArticleDOI
Using big GPS trajectory data analytics for vehicle miles traveled estimation
TL;DR: A scalable map-matching module that considers both the spatiotemporal information of GPS waypoint sequences and topologic information of road network for the State of Maryland while striking a balance between matching accuracy and computing time is developed.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.