Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Journal ArticleDOI
A Holistic View on Resource Management in Serverless Computing Environments: Taxonomy and Future Directions
TL;DR: In this article , the authors identify the major aspects covering the broader concept of resource management in serverless environments and propose a taxonomy of elements that influence these aspects, encompassing characteristics of system design, workload attributes, and stakeholder expectations.
Journal ArticleDOI
Heracles: A framework for developing and evaluating text mining algorithms
TL;DR: A practical use case shows the versatility and ease-of-use of the proposed Heracles framework in the domain of aspect-based sentiment analysis, a framework for developing and evaluating text mining algorithms, with a broad range of applications in industry.
Journal ArticleDOI
Fair multi-agent task allocation for large datasets analysis
TL;DR: In this paper, the reducer agents interact during the job and the task reallocation is based on negotiation in order to decrease the workload of the most loaded reducer and so the runtime.
Proceedings ArticleDOI
A Performance Study of Big Spatial Data Systems
TL;DR: The present status of the Big Spatial Data systems is investigated by conducting a comprehensive feature analysis and performance evaluation of a few representative systems with a benchmark, and shows that SpatialIgnite performs better than Hadoop and Spark based systems that are evaluated.
Journal ArticleDOI
Analysis of Stellar Spectra from LAMOST DR5 with Generative Spectrum Networks.
Wang Rui,Luo A-li,Zhang Shuo,Hou Wen,Du Bing,Song Yi-han,Wu Ke-Fei,Chen Jian-Jun,Zuo Fang,Qin Li,Chen Xiang-Lei,Lu Yan +11 more
TL;DR: In this article, the fundamental stellar atmospheric parameters (Teff, log g, [Fe/H] and [{\alpha}/Fe]) were derived for low-resolution spectroscopy from LAMOST DR5 with Generative Spectrum Networks (GSN).
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.