Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Book ChapterDOI
Recent Advances in Scalable Network Generation1
TL;DR: In this paper , a short overview over publicly available software packages as well as implementations of single random graph models is given. But the focus is on the scalability of a network generation algorithm is connected to the assumed model of computation.
Proceedings ArticleDOI
StragglerHelper: Alleviating Straggling in Computing Clusters via Sharing Memory Access Patterns
Wenjie Liu,Ping Huang,Xubin He +2 more
TL;DR: This paper proposes StragglerHelper which conveys the memory access characteristics experienced by the forerunner to the straggglers such that stragglers can be sped up due to the accurately informed memory prefetching.
Posted Content
Split-Correctness in Information Extraction
TL;DR: In this article, the authors propose a formal framework for split-correctness within the formalism of document spanners, which allows text analysis systems to devise query plans with parallel evaluation on segments for accelerating the processing of large documents.
Proceedings ArticleDOI
Automation of the Reliability Modelling Using GSDFD Framework
TL;DR: The studies confirmed the flexibility of the developed framework for automating tasks with non-linear solutions and revealed ways to further modernize the framework to reduce manual coding by including a description of the input parameters in the input pipe.
Proceedings ArticleDOI
Watching the Grid: Utility-Independent Measurements of Electricity Reliability in Accra, Ghana
Noah Klugman,Joshua Adkins,Emily Paszkiewicz,Molly G. Hickman,Matthew Podolsky,Jay Taneja,Prabal Dutta +6 more
TL;DR: In this paper, the authors introduce PowerWatch, an agile methodology to directly measure customer experience and aggregated grid performance without relying on the utility for deployment or management, and evaluate the PowerWatch methodology by deploying 462 sensors in homes and businesses in Accra, Ghana for over a year.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.