Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Journal ArticleDOI
Properties of Radial Velocities Measurement Based on LAMOST-II Medium-resolution Spectroscopic Observations
Ran Wang,A-Li Luo,J.-J. Chen,Z-R. Bai,Li Chen,Xuefei Chen,Subo Dong,Bing Du,Jian-Ning Fu,Zhanwen Han,J.-L. Hou,Yining Hou,Wen Hou,Dengkai Jiang,Xu Kong,L. Li,Chengze Liu,Jun-Ming Liu,Li Qin,Jianrong Shi,Hai-Jun Tian,Hong Wu,Chao-Jian Wu,Ji-Wei Xie,Hong Zhang,S. N. Zhang,Guo-Ming Zhao,Y.-H. Zhao,Jiayong Zhong,Weikai Zong,Fang Zuo +30 more
TL;DR: In this article, the radial velocities of the Large Sky Area Multi-Object Fibre Spectroscopic Telescope-II (LAMOST-II) medium-resolution (R similar to 7500) spectra are measured for 1,594,956 spectra through matching with templates.
Proceedings ArticleDOI
A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log
TL;DR: Three in-progress data analytics projects that leverage a multi-user Big Data analytics framework for HPC log data at Oak Ridge National Laboratory to assess system status, mine event patterns, and study correlations between user applications and system events are introduced.
Journal ArticleDOI
Visible-near infrared spectrum-based classification of apple chilling injury on cloud computing platform
TL;DR: The experimental results showed that, by using the cloud computing platform, an efficient spectrum classification model of apple chilling injury was established; the ANN model had slightly higher accuracy than the SVM model (not including the second-derivative spectra), but the S VM model was more efficient.
Proceedings ArticleDOI
Artificial Intelligence with Big Data
TL;DR: Big Data has become a new source of opportunity among applications in Artificial Intelligence and by embracing this new paradigm, parallel processing can be effectively leveraged to support development at a level of scale and performance that was not possible earlier.
Journal ArticleDOI
PyGMQL: scalable data extraction and analysis for heterogeneous genomic datasets
TL;DR: The expressiveness and performance of PyGMQL is demonstrated through a sequence of biological data analysis scenarios of increasing complexity, which highlight reproducibility, expressive power and scalability.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.