Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Journal ArticleDOI
Fast and Precise: Parallel Processing of Vehicle Traffic Videos Using Big Data Analytics
TL;DR: In this paper , a low-cost distributed infrastructure based on Hadoop and Spark frameworks for data processing is proposed, where videos are equally divided and distributed to multicore CPU nodes for analysis.
Proceedings ArticleDOI
Resilient Neural Forecasting Systems
TL;DR: This paper describes how the deep learning model deals with missing values natively without requiring imputation and results in a fully autonomous forecasting system that compares favourably to a hybrid system consisting of the algorithm and human overrides.
Proceedings ArticleDOI
Orchestral: A Lightweight Framework for Parallel Simulations of Cell-Cell Communication
Adrien Coulier,Andreas Hellander +1 more
TL;DR: In this article, the authors develop a modeling and simulation framework capable of massively parallel simulation of multicellular systems with spatially resolved stochastic kinetics in individual cells, using operator splitting to decouple the simulation of reaction-diffusion kinetics inside the cells from the simulations of molecular cell-cell interactions occurring on the boundaries between cells.
Proceedings ArticleDOI
Parallel String Graph Construction and Transitive Reduction for De Novo Genome Assembly
TL;DR: DiBELLA 2D as mentioned in this paper uses linear algebra operations over semirings using 2D distributed sparse matrices for both overlap detection and layout simplification, and achieves near linear scaling with over 80% parallel efficiency.
Book ChapterDOI
Bias and Discrimination in Artificial Intelligence: Emergence and Impact in E-Business
Jan C. Weyerer,Paul F. Langer +1 more
TL;DR: The foundations of bias and discrimination in AI are described, highlighting its scientific and practical relevance, as well as describing its meaning, emergence, functioning, and impact in the context of e-business.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.