Proceedings ArticleDOI
Pregel: a system for large-scale graph processing
Grzegorz Malewicz,Matthew H. Austern,Aart J. C. Bik,James C. Dehnert,Ilan Horn,Naty Leiser,Grzegorz Czajkowski +6 more
- pp 135-146
Reads0
Chats0
TLDR
A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.Abstract:
Many practical computing problems concern large graphs. Standard examples include the Web graph and various social networks. The scale of these graphs - in some cases billions of vertices, trillions of edges - poses challenges to their efficient processing. In this paper we present a computational model suitable for this task. Programs are expressed as a sequence of iterations, in each of which a vertex can receive messages sent in the previous iteration, send messages to other vertices, and modify its own state and that of its outgoing edges or mutate graph topology. This vertex-centric approach is flexible enough to express a broad set of algorithms. The model has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier. Distribution-related details are hidden behind an abstract API. The result is a framework for processing large graphs that is expressive and easy to program.read more
Citations
More filters
Book
Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers
TL;DR: It is argued that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas.
Proceedings Article
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Matei Zaharia,Mosharaf Chowdhury,Tathagata Das,Ankur Dave,Justin Ma,Murphy McCauley,Michael J. Franklin,Scott Shenker,Ion Stoica +8 more
TL;DR: Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.
Journal ArticleDOI
Big Data: A Survey
Min Chen,Shiwen Mao,Yunhao Liu +2 more
TL;DR: The background and state-of-the-art of big data are reviewed, including enterprise management, Internet of Things, online social networks, medial applications, collective intelligence, and smart grid, as well as related technologies.
Proceedings ArticleDOI
Apache Hadoop YARN: yet another resource negotiator
Vinod Kumar Vavilapalli,Arun C. Murthy,Chris Douglas,Sharad Agarwal,Mahadev Konar,Robert Evans,Thomas Graves,Jason Lowe,Hitesh Shah,Siddharth Seth,Bikas Saha,Carlo Curino,Owen O'Malley,Sanjay Radia,Benjamin Reed,Eric Baldeschwieler +15 more
TL;DR: The design, development, and current state of deployment of the next generation of Hadoop's compute platform: YARN is summarized, which decouples the programming model from the resource management infrastructure, and delegates many scheduling functions to per-application components.
Book
Mining of Massive Datasets
TL;DR: This book focuses on practical algorithms that have been used to solve key problems in data mining and which can be used on even the largest datasets, and explains the tricks of locality-sensitive hashing and stream processing algorithms for mining data that arrives too fast for exhaustive processing.
References
More filters
BookDOI
Fault-Tolerant Parallel Computation
TL;DR: This book presents models for Robust Computation with Shared Memory Randomized Algorithms and Distributed Models and Algorithm and explains how these models can be modified for distributed systems.
Book
Graph Theory and Its Applications, Second Edition (Discrete Mathematics and Its Applications)
Jonathan L. Gross,Jay Yellen +1 more
Journal ArticleDOI
Inductive graphs and functional graph algorithms
TL;DR: A new style of writing graph algorithms in functional languages which is based on an alternative view of graphs as inductively defined data types is proposed, and it is demonstrated how graph algorithms can be succinctly given by recursive function definitions based on the inductive graph view.
Proceedings ArticleDOI
Lifting sequential graph algorithms for distributed-memory parallel computation
Douglas Gregor,Andrew Lumsdaine +1 more
TL;DR: This paper revisits the abstractions comprising the Boost Graph Library in the context of distributed-memory parallelism, lifting away the implicit requirements of sequential execution and a single shared address space and develops general principles and patterns for using (and reusing) generic, object-oriented parallel software libraries.