Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Proceedings ArticleDOI
Toward RDB to NoSQL: transforming data with metamorfose framework
TL;DR: This paper presents a novel approach to convert relational databases (RDB) to document and column family NoSQLs using a set of directed acyclic graphs (DAG) representing the target NoSQL model.
Proceedings ArticleDOI
Dynamic Container-based Resource Management Framework of Spark Ecosystem
Nawab Muhammad Faseeh Qureshi,Isma Farah Siddiqui,Asad Abbas,Ali Kashif Bashir,Kee-Hyun Choi,Jaehyoun Kim,Dong-Ryeol Shin +6 more
TL;DR: This paper proposes dynamic container-based resource management framework, that shifts coupled associations of job profiles to dynamically available resource containers and relieves static container allocations and presumes them as a fresh piece of resource allocation for new job profile.
Journal ArticleDOI
Coffea -- Columnar Object Framework For Effective Analysis
Nicholas Smith,Lindsey Gray,Matteo Cremonesi,B. Jayatilaka,Oliver Gutsche,Allison Reinsvold Hall,Kevin Pedro,Maria Acosta,Andrew Melo,S. Belforte,Jim Pivarski +10 more
TL;DR: This work will discuss the experience in implementing analysis of CMS data using the coffea framework, and a discussion of the user experience and future directions.
Proceedings ArticleDOI
beHEALTHIER: A Microservices Platform for Analyzing and Exploiting Healthcare Data
Argyro Mavrogiorgou,Spyridon Kleftakis,Konstantinos Mavrogiorgos,Nikolaos Zafeiropoulos,Andreas Menychtas,Athanasios Kiourtis,Ilias Maglogiannis,Dimosthenis Kyriazis +7 more
TL;DR: In this paper, the authors present a platform based on Microservice Architecture (MSA), which is able to efficiently manage and analyze these vast amounts of data, by utilizing a newly proposed kind of electronic health records (i.e., eXtended Health Records) and their corresponding networks.
Journal ArticleDOI
BioWorkbench: a high-performance framework for managing and analyzing bioinformatics experiments.
Maria Luiza Mondelli,Thiago Tavares Magalhães,Guilherme Loss,Michael Wilde,Ian Foster,Marta Mattoso,Daniel S. Katz,Helio J. C. Barbosa,Ana Tereza Ribeiro de Vasconcelos,Kary A. C. S. Ocaña,Luiz M. R. Gadelha +10 more
TL;DR: This work presents BioWorkbench, a framework for managing and analyzing bioinformatics experiments, and shows that the framework is scalable and achieves high-performance, reducing up to 98% of the case studies execution time.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.