Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Journal ArticleDOI
An energy-aware scheduling algorithm for big data applications in Spark
TL;DR: An energy-aware scheduling algorithm for Spark (EASAS) to reduce energy consumption while satisfying the service level agreement (SLA) and a new energy consumption model based on Spark framework is presented.
Journal ArticleDOI
BDWatchdog: Real-time monitoring and profiling of Big Data applications and frameworks
TL;DR: BDWatchdog is a novel framework that allows real-time and scalable analysis of Big Data applications by combining time series for resource monitorization and flame graphs for code profiling, focusing on the processes that make up the workload rather than the underlying instances on which they are executed.
Journal ArticleDOI
An Online Learning Collaborative Method for Traffic Forecasting and Routing Optimization
TL;DR: An online learning data-driven model is developed to extract prior knowledge and enhance collaboration between road segments and vehicles by combining short-term traffic forecasting and real-time routing optimization.
Posted Content
Alert Classification for the ALeRCE Broker System: The Real-time Stamp Classifier
Rodrigo Carrasco-Davis,Esteban Reyes,C. Valenzuela,F. Forster,Pablo A. Estevez,Giuliano Pignata,Franz E. Bauer,I. Reyes,P. Sanchez-Saez,G. Cabrera-Vives,Susana Eyheramendy,Márcio Catelan,Javier Arredondo,Ernesto Castillo-Navarrete,Diego Rodríguez-Mancini,Daniela Ruz-Mieres,Alberto Moya,Luis Sabatini-Gacitúa,Cristóbal Sepúlveda-Cobo,Ashish Mahabal,Javier Silva-Farfán,Ernesto Camacho-Iñiquez,Lluís Galbany +22 more
TL;DR: This work presents a real-time stamp classifier of astronomical events for the Automatic Learning for the Rapid Classification of Events broker, ALeRCE, based on a convolutional neural network trained on alerts ingested from the Zwicky Transient Facility, with high accuracy.
Journal ArticleDOI
A Rhombic Dodecahedron Topology for Human-Centric Banking Big Data
TL;DR: A new design rule for topologies including: 1) low coordination number; 2) high packing density; and 3) having a 3-D structure is proposed, and a rhombic dodecahedron topology is proposed.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.