Many practical computing problems concern large graphs. Standard examples include the Web graph and various social networks. The scale of these graphs - in some cases billions of vertices, trillions of edges - poses challenges to their efficient processing. In this paper we present a computational model suitable for this task. Programs are expressed as a sequence of iterations, in each of which a vertex can receive messages sent in the previous iteration, send messages to other vertices, and modify its own state and that of its outgoing edges or mutate graph topology. This vertex-centric approach is flexible enough to express a broad set of algorithms. The model has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier. Distribution-related details are hidden behind an abstract API. The result is a framework for processing large graphs that is expressive and easy to program.

/pdf/pregel-a-system-for-large-scale-graph-processing-30lgjenxxg.pdf

Pregel: a system for large-scale graph processing

From the Publisher:
The Message Passing Interface (MPI) specification is widely used for solving significant scientific and engineering problems on parallel computers There exist more than a dozen implementations on computer platforms ranging from IBM SP-2 supercomputers to clusters of PCs running Windows NT or Linux ("Beowulf" machines) The initial MPI Standard document, MPI-1, was recently updated by the MPI Forum The new version, MPI-2, contains both significant enhancements to the existing MPI core and new features
Using MPI is a completely up-to-date version of the authors' 1994 introduction to the core functions of MPI It adds material on the new C++ and Fortran 90 bindings for MPI throughout the book It contains greater discussion of datatype extents, the most frequently misunderstood feature of MPI-1, as well as material on the new extensions to basic MPI functionality added by the MPI-2 Forum in the area of MPI datatypes and collective operations
Using MPI-2 covers the new extensions to basic MPI These include parallel I/O, remote memory access operations, and dynamic process management The volume also includes material on tuning MPI applications for high performance on modern MPI implementations

Using MPI-2: Advanced Features of the Message Passing Interface

I. Parallelism 1. Introduction 2. Parallel Computer Architectures 3. Parallel Programming Considerations II. Applications 4. General Application Issues 5. Parallel Computing in CFD 6. Parallel Computing in Environment and Energy 7. Parallel Computational Chemistry 8. Application Overviews III. Software technologies 9. Software Technologies 10. Message Passing and Threads 11. Parallel I/O 12. Languages and Compilers 13. Parallel Object-Oriented Libraries 14. Problem-Solving Environments 15. Tools for Performance Tuning and Debugging 16. The 2-D Poisson Problem IV. Enabling Technologies and Algorithms 17. Reusable Software and Algorithms 18. Graph Partitioning for Scientific Simulations 19. Mesh Generation 20. Templates and Numerical Linear Algebra 21. Software for the Scalable Solutions of PDEs 22. Parallel Continuous Optimization 23. Path Following in Scientific Computing 24. Automatic Differentiation V. Conclusion 25. Wrap-up and Features

Sourcebook of parallel computing

From the Publisher:
Use of Beowulf clusters (collections of off-the-shelf commodity computers programmed to act in concert, resulting in supercomputer performance at a fraction of the cost) has spread far and wide in the computational science community. Many application groups are assembling and operating their own "private supercomputers" rather than relying on centralized computing centers. Such clusters are used in climate modeling, computational biology, astrophysics, and materials science, as well as non-traditional areas such as financial modeling and entertainment. Much of this new popularity can be attributed to the growth of the open-source movement.

The second edition of Beowulf Cluster Computing with Linux has been completely updated; all three stand-alone sections have important new material. The introductory material in the first part now includes a new chapter giving an overview of the book and background on cluster-specific issues, including why and how to choose a cluster, as well as new chapters on cluster initialization systems (including ROCKS and OSCAR) and on network setup and tuning. The information on parallel programming in the second part now includes chapters on basic parallel programming and available libraries and programs for clusters. The third and largest part of the book, which describes software infrastructure and tools for managing cluster resources, has new material on cluster management and on the Scyld system.

/pdf/beowulf-cluster-computing-with-linux-1kl0l45e1m.pdf

Beowulf Cluster Computing with Linux

Bulk Synchronous Parallelism (BSP) is a parallel programming model that abstracts from low-level program structures in favour of supersteps. A superstep consists of a set of independent local computations, followed by a global communication phase and a barrier synchronisation. Structuring programs in this way enables their costs to be accurately determined from a few simple architectural parameters, namely the permeability of the communication network to uniformly-random traffic and the time to synchronise. Although permutation routing and barrier synch ronisations are widely regarded as inherently expensive, this is not the case. As a result, the structure imposed by BSP does not reduce performance, while bringing considerable benefits for application building. This paper answers the most common questions we are asked about BSP and justifies its claim to be a major step forward in parallel programming.

/pdf/questions-and-answers-about-bsp-53o470u5gv.pdf

Questions and Answers about BSP

BSPlib is a small communications library for bulk synchronous parallel (BSP) programming which consists of only 20 basic operations. This paper presents the full definition of BSPlib in C, motivates the design of its basic operations, and gives examples of their use. The library enables programming in two distinct styles: direct remote memory access (DRMA) using put or get operations, and bulk synchronous message passing (BSMP). Currently, implementations of BSPlib exist for a variety of modern architectures, including massively parallel computers with distributed memory, shared memory multiprocessors, and networks of workstations. BSPlib has been used in several scientific and industrial applications; this paper briefly describes applications in benchmarking, Fast Fourier Transforms (FFTs), sorting, and molecular dynamics.

/pdf/bsplib-the-bsp-programming-library-4s91qsdote.pdf

BSPlib: The BSP programming library

The Bulk-Synchronous Parallel (BSP) model was proposed by Valiant as a model for general-purpose parallel computation. The objective of the model is to allow the design of parallel programs that can be executed efficiently on a variety of architectures. While many theoretical arguments in support of the BSP model have been presented, the degree to which the model can be efficiently utilized on existing parallel machines remains unclear. To explore this question, we implemented s small library of BSP functions, called the Green BSP library, on several parallel platfotms. We also created a number of parallel applications based on this library. Here, we report on the performance of six of these applications on three different parallel platforms. Our preliminary results suggest that the BSP model can be used to develop efficient and portable programs for a range of machines and applications.

Towards efficiency and portability: programming with the BSP model

A central question in parallel computing is to determine the extent to which one can write parallel programs using a high-level, general-purpose, and architecture-independent programming language and have them executed on a variety of parallel and distributed architectures without sacrificing efficiency. A large body of research suggests that, at least in theory, general-purpose parallel computing is indeed possible provided certain conditions are met: an excess of logical parallelism in the program, and the ability of the target architecture to efficiently realize balanced communication patterns. The canonical example of a balanced communication pattern is an h-relation, in which each processor is the origin and destination of at most h messages. A plethora of protocols has been designed for routing h-relations in a variety of networks. The goal has been to minimize the value of h while guaranteeing delivery of the messages within a time constant factor from optimal. In this paper we describe protocols that meet the most stringent efficiency requirement, namely delivery of messages within time that is a lower order additive term from the best achievable. Such protocols are called 1-optimal. While these protocols achieve 1-optimality only for heavily loaded networks, that is, for large values of h, they are remarkable for their simplicity in that they only use the total-exchange communication primitive. The total-exchange can be realized in many networks using very simple, contention-free, and extremely efficient schemes. The technical contribution of this paper is a protocol to route random h-relations in an N-processor network using /sup h///sub N/(1+o(1))+O(log log N) total-exchange rounds with high probability. Using message duplication, we can improve the bound to /sup h///sub N/(1+o(1))+O(log*N). This improves upon the /sup h///sub N/(1+o(1))+O(log N) bound of Gerbessiotis and Valiant. While our theoretical improvements are modest, our experimental results show an improvement over the protocol of A. Gerebessiotis and L.G. Valiant. >

Efficient communication using total-exchange

Using recurrent neural networks to learn the structure of interconnection networks

The selection of appropriate communication mechanisms is a key issue in parallel computing. We argue that the current emphasis on single-message communication has led to inefficient systems and unnecessarily confusing code. In contrast, batch communication has substantial implementation advantages, is suitable for almost all parallel applications, and encourages a programming paradigm that is easy to reason about.

Mark W. Goudreau

Papers

BSPlib: The BSP programming library

Towards efficiency and portability: programming with the BSP model

Efficient communication using total-exchange

Using recurrent neural networks to learn the structure of interconnection networks

Single-Message vs. Batch Communication