Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Journal ArticleDOI
Distributed Privacy-Aware Fast Selection Algorithm for Large-Scale Data
Hao Liu,Jiming Chen +1 more
TL;DR: A Distributed Privacy-Aware Median (DPAM) selection algorithm for median selection in distributed large-scale data while preserving local statistics privacy is proposed, and it is extended to arbitrary $k$
Book ChapterDOI
Using Deep Learning to Classify Class Imbalanced Gene-Expression Microarrays Datasets
A. Reyes-Nava,H. Cruz-Reyes,R. Alejo,E. Rendón-Lara,A. A. Flores-Fuentes,E. E. Granda-Gutiérrez +5 more
TL;DR: Performance of deep learning neural networks to classify class imbalanced gene-expression microarrays datasets is studied and shows that the noise or separability of the dataset is more determinant than its dimensionality in the classifier performance.
Journal ArticleDOI
A comparison of forecasting models for the resource usage of MapReduce applications
TL;DR: It is shown that an LSTM model trained in a specific machine may be used to predict the resource usage in another machine and that multivariate long short-term memory recurrent neural network models are more sensitive to sample size than multiple linear regression models.
Book ChapterDOI
“I Really Don’t Know What ‘Thumbs Up’ Means”: Algorithmic Experience in Movie Recommender Algorithms
TL;DR: The present study combines a semiotic inspection analysis of the Netflix interface with sensitized design workshops and semi-structured interviews to explore AX requirements for movie recommender algorithms, and extends the current AX design framework with two new design areas: algorithmic usefulness and algorithmic social practices.
Journal ArticleDOI
Spark implementation of the enhanced Scatter Search metaheuristic: Methodology and assessment
TL;DR: A parallel implementation of the enhanced Scatter Search metaheuristic using Spark is presented and helpful guidance to readers interested in applying, or developing their own, parallel metaheuristics to solve challenging problems in the Cloud is provided.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.