Journal ArticleDOI
Apache Spark: a unified engine for big data processing
Matei Zaharia,Reynold Xin,Patrick Wendell,Tathagata Das,Michael Armbrust,Ankur Dave,Xiangrui Meng,Josh Rosen,Shivaram Venkataraman,Michael J. Franklin,Ali Ghodsi,Joseph E. Gonzalez,Scott Shenker,Ion Stoica +13 more
Reads0
Chats0
TLDR
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.Abstract:
This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applicationsread more
Citations
More filters
Journal ArticleDOI
Fuzzy and Real-Coded Chemical Reaction Optimization for Intrusion Detection in Industrial Big Data Environment
TL;DR: A fuzzy and real coded chemical reaction optimization-based cluster analysis approach with feature selection is proposed for the intrusion detection system in a Big Data platform and uses the use of the Flexible Mutual Information Feature Selection approach to avoid the processing of a large number of features.
Journal ArticleDOI
Minimizing cost by reducing scaling operations in distributed stream processing
TL;DR: In this article, the authors use advanced filtering techniques from the field of signal processing to pre-process raw system measurements, thus mitigating superfluous scaling operations and overcompensating reactions to short-term changes in the workload.
Proceedings ArticleDOI
Towards a real-time unsupervised estimation of predictive model degradation
TL;DR: A novel unsupervised methodology to automatically detect prediction-quality degradation of machine learning models and its scalability performance is suitable for soft real-time applications such as predictive maintenance, Industry 4.0, and text mining.
Journal ArticleDOI
CrossRec: Cross-Domain Recommendations Based on Social Big Data and Cognitive Computing
TL;DR: This work proposes a cross-domain recommender system, including three approaches, based on multi-source social big data, and shows that the accuracies of the three proposed approaches are significantly improved compared with the conventional recommender approaches, such as collaborative filtering and matrix factorization.
Proceedings ArticleDOI
Big Data Platform for Smart Grids Power Consumption Anomaly Detection
TL;DR: This paper presents a big data platform for anomaly detection of power consumption data, based on an ingestion layer with data densification options, Apache Flink as part of the speed layer and HDFS/KairosDB as data storage layers.
References
More filters
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
A bridging model for parallel computation
TL;DR: The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
TL;DR: A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Proceedings ArticleDOI
Dryad: distributed data-parallel programs from sequential building blocks
TL;DR: The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.