Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Posted Content
SCALPEL3: a scalable open-source library for healthcare claims databases
Emmanuel Bacry,Stéphane Gaïffas,Fanny Leroy,Maryan Morel,Dinh Phong Nguyen,Youcef Sebiat,Dian Sun +6 more
TL;DR: SCALPEL3 makes studies based on SNDS much easier and more scalable than the existing framework, and is now used at the agency collecting SNDS data, at the French Ministry of Health and soon at the National Health Data Hub in France.
Journal ArticleDOI
Equivalence classes and conditional hardness in massively parallel computations
TL;DR: In this article , it was shown that the problem of classifying a graph as one cycle versus two cycles can be solved in O(log n) rounds in the MPC model under the L⊈MPC(o(logN) conjecture.
Proceedings ArticleDOI
Distributed Parallel Analysis Engine for High Energy Physics Using AWS Lambda
Jacek Kuśnierz,Maciej Malawski,Vincenzo Eduardo Padulano,Enric Tejedor Saavedra,Pedro Alonso-Jordá +4 more
TL;DR: In this paper, the authors explore the possibility of running such analyses on serverless services in public cloud using a purely stateless environment and demonstrate the excellent speedup in parallel stage of processing in their benchmarks.
Proceedings ArticleDOI
Performance Evaluation of Big Data Processing Strategies for Neuroimaging
TL;DR: It is concluded that Big Data processing strategies are worth developing for neuroimaging applications because in-memory computing alone will not speed-up current functional MRI analyses unless coupled with data locality and processing around 280 subjects concurrently.
ReportDOI
The ISTI Rapid Response on Exploring Cloud Computing 2018
Carleton Coffrin,James Arnold,Stephan Eidenbenz,Derek Aberle,John Ambrosiano,Zachary K. Baker,Sara Brambilla,Michael J. I. Brown,K. Nolan Carter,Pinghan Chu,Patrick Conry,Keeley R. Costigan,Ariane Eberhardt,David Fobes,Adam Gausmann,Sean Harris,Donovan Heimer,Marlin Holmes,Bill Junor,Csaba Kiss,Steve P. Linger,Rodman R. Linn,Li-Ta Lo,J. K. MacCarthy,Omar Marcillo,Clay McGinnis,Alexander McQuarters,Eric Michalak,Arvind Mohan,Matthew A. Nelson,Diane Oyen,Nidhi Parikh,Donatella Pasqualini,Aaron Scott Pope,Reid B. Porter,Chris Rawlings,Hannah Reinbolt,Reid D. Rivenburgh,Philip Romero,Kevin Schoonover,Alexei N. Skurikhin,Daniel R. Tauritz,Dima Tretiak,Zhehui Wang,James Wernicke,Brad Wolfe,Phillip J. Wolfram,Jonathan Woodring +47 more
TL;DR: This report describes eighteen projects that explored how commercial cloud computing services can be utilized for scientific computation at national laboratories, ranging from deploying proprietary software in a cloud environment to leveraging established cloud-based analytics workflows for processing scientific datasets.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.