Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Journal ArticleDOI
Container orchestration on HPC systems through Kubernetes
Naweiluo Zhou,Yiannis Georgiou,Marcin Pospieszny,Li Zhong,Huan Zhou,Christoph Niethammer,Branislav Pejak,Oskar Marko,Dennis Hoppe +8 more
TL;DR: Torque-Operator as discussed by the authors is a hybrid architecture that integrates HPC and Cloud clusters seamlessly with little interference to HPC systems where container orchestration is performed on two levels.
Journal ArticleDOI
An exploratory teaching program in big data analysis for undergraduate students
TL;DR: According to students’ feedback, the exploratory teaching program is useful for learning how to analyze large datasets and identify patterns that will improve any company’s and organization decision-making process.
Journal ArticleDOI
Inter-technology relationship networks : Arranging technologies through text mining
TL;DR: This work develops an analytical method to generate technology-related network data that retraces elapsed patterns of technological change and processes textual data from different information sources into an analyzable and readable inter-technology relationship network.
Journal ArticleDOI
Spark-based parallel dynamic programming and particle swarm optimization via cloud computing for a large-scale reservoir system
TL;DR: This study proposes the spark-based parallel dynamic programming (SPDP) and spark- based parallel particle swarm optimization (SPPSO) methods via parallel cloud computing, which ensures the global search capability of the algorithm.
Journal ArticleDOI
Spine Toolbox: A flexible open-source workflow management system with scenario and data management
Juha Kiviluoma,Fabiano Pallonetto,Manuel Marin,Pekka Savolainen,Antti Soininen,Per Vennström,Erkka Rinne,Jiangyi Huang,Iasonas Kouveliotis-Lysikatos,Maren Ihlemann,Erik Delarue,Ciara O'Dwyer,Terence O’Donnel,Mikael Amelin,Lennart Söder,Joseph Dillon +15 more
TL;DR: The Spine Toolbox as mentioned in this paper is an open-source software for defining, managing, simulating and optimising energy system models, which gives the user the ability to collect, create, organize, and validate model input data, execute a model with selected data and finally archive and visualise results/output data.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.