Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Dissertation
Data-Driven Anomaly Detection in Industrial Networks
TL;DR: A visual flow monitoring system and a multivariate ADS that is able to tackle data heterogeneity and to scale efficiently are presented, and a Big Data, MSPCinspired ADS that monitors field and network data to detect anomalies is presented.
Book ChapterDOI
Incremental Learning for Large Scale Classification Systems
Athanasios Alexopoulos,Andreas Kanavos,Andreas Kanavos,Konstantinos C. Giotopoulos,Alaa Mohasseb,Mohamed Bader-El-Den,Athanasios K. Tsakalidis +6 more
TL;DR: This paper performs classification analysis using Apache Spark in one real dataset, and the effect of the dataset size and input features on the classification results is examined.
Journal ArticleDOI
Application of Big Data Technology in the Impact of Tourism E-Commerce on Tourism Planning
TL;DR: Wang et al. as discussed by the authors proposed a research strategy on the impact of tourism e-commerce on customized tourism in the era of big data (EBD), including related theoretical research methods, random forest algorithms, support vector machine classification algorithms, and Bayesian estimation algorithms, which are used to customize tourism ecommerce in the EBD.
Proceedings ArticleDOI
Designing a Feature Selection Technique for Analyzing Mixed Data
TL;DR: A new technique is introduced to boost model performances by determining optimal features in noisy mixed data by performing a continuous evaluation to determine the best possible features that suit to a chosen data analysis algorithm.
Proceedings ArticleDOI
Graphical Spark Programming in IoT Mashup Tools
TL;DR: This study focuses on the tight integration of data analytics capabilities of Spark in IoT mashup tools and devising a novel, generic approach for programming Spark from graphical flows that comprises early-stage validation and code generation of Java Spark programs.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.