A vast body of theoretical research has focused either on overly simplistic models of parallel computation, notably the PRAM, or overly specific models that have few representatives in the real world. Both kinds of models encourage exploitation of formal loopholes, rather than rewarding development of techniques that yield performance across a range of current and future parallel machines. This paper offers a new parallel machine model, called LogP, that reflects the critical technology trends underlying parallel computers. it is intended to serve as a basis for developing fast, portable parallel algorithms and to offer guidelines to machine designers. Such a model must strike a balance between detail and simplicity in order to reveal important bottlenecks without making analysis of interesting problems intractable. The model is based on four parameters that specify abstractly the computing bandwidth, the communication bandwidth, the communication delay, and the efficiency of coupling communication and computation. Portable parallel algorithms typically adapt to the machine configuration, in terms of these parameters. The utility of the model is demonstrated through examples that are implemented on the CM-5.

/pdf/logp-towards-a-realistic-model-of-parallel-computation-2hpmqi5j0f.pdf

LogP: towards a realistic model of parallel computation

We present a new model of parallel computation---the LogGP model---and use it to analyze a number of algorithms, most notably, the single node scatter (one-to-all personalized broadcast). The LogGP model is an extension of the LogP model for parallel computation which abstracts the communication of fixed-sized short messages through the use of four parameters: the communication latency (L), overhead (o), bandwidth (g), and the number of processors (P). As evidenced by experimental data, the LogP model can accurately predict communication performance when only short messages are sent (as on the CM-5). However, many existing parallel machines have special support for long messages and achieve a much higher bandwidth for long messages compared to short messages (e.g., IBM SP-2, Paragon, Meiko CS-2, Ncube/2). We extend the basic LogP model with a linear model for long messages. This combination, which we call the LogGP model of parallel computation, has one additional parameter, G, which captures the bandwidth obtained for long messages. Experimental data collected on the Meiko CS-2 shows that this simple extension of the LogP model can quite accurately predict communication performance for both short and long messages. This paper discusses algorithm design and analysis under the new model, examining the all-to-all remap, FFT, and radix sort. We also examine, in more detail, the single node scatter problem. We derive solutions for this problem and prove their optimality under the LogGP model. These solutions are qualitatively different from those obtained under the simpler LogP model, reflecting the importance of capturing long messages in a model.

LogGP: incorporating long messages into the LogP model—one step closer towards a realistic model for parallel computation

This paper introduces the Asynchronous PRAM model of computation, a variant of the PRAM in which the processors run asy ~chronously and there is an explicit charge for synchronization. A fanfily of Asynchrooous PRAM's are defined, varying in the types of synchronization steps permitted and the costs for accessing the shared memory. Algorithms, lower bounds, and simulation results are presented for an interesting member of the family. 1 I n t r o d u c t i o n The PRAM model of computation consists of a collection ofp sequential processors, each with its own private local memory, communicating with one another through a shared global memory. The processors execute in lockstep, although each processor does have its own local program. A PRAM computation is a sequence of time steps, alternating between three types of instructions: read, compute, and write. In a read step, each processor can read one global memory location into a local memory location. In a compute step, each processor can execute a single RAM operation whose operands are in local memory, storing the result in a local memory location. In a write step, each processor can write the contents of one local memory location into a global memory location. All three steps are assumed to take unit time in the model. Although an idealized model, the PRAM has proven to be a useful model for studying parallel computation (see [KRS8] for a survey of results). The model is simple and relatively easy to use: its shared memory abstraction hides the details of the interprocessor communication and synchronization. This work was supported in part by the International Computer Science Institute, Berkeley, CA and by the IBM Almaden Research Center, San Jose, CA. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a %e and/or specific permission. ~) 1989 ACM 0-89791-323-X/89 /0006 /0158 $1.50 158 There are several difficulties that arise in mapping PRAM algorithms onto existing shared memory MIMD machines, such as the Sequent Balance, the BBN Butterfly, the NYU Ultracomputer, and the IBM RP3. First, realistic MIMD machines have more limited communication capabilities than the PRAM. The PRAM assumes that each processor can access any memory location in one step. Realistic machines are more limited in at least three respects: * the shared memory locations are partitioned into a smaller number of memory banks, each of which can support only a constant number of accesses per

/pdf/a-more-practical-pram-model-18o75amdk1.pdf

A more practical PRAM model

We are interested in implementing data structures on shared memory multiprocessors. A natural model for these machines is an asynchronous parallel machine, in which the processors are subject to arbitrary delays. On such machines, it is desuable for algorithms to be loclc-free, that is, they must allow concurrent access to data without using mutual exclusion. Eflicient lock-free implementations are known ror some specific data structures, but these algorithms do not generalize weIl to other structures. For most data structures, the only previously known lock-free algorithm is due to Herlihy [12]. Herlihy presents a simple methodology to create a lock-free implementation of a general data structure, but his approach can be very expensive. We present a technique that provides the semantics of exclusive access to data without using mutual exclusion. Using this technique, we devise the caching method, a general method of implementing lock-free data structures that is provably bett er than Herlihy's methodology for many weIl-known data structures. The cost of one operation using the caching method is proportional to TlogT, where T is the sequential cost of the operation. Under Herlihy's methodology, the cost is proportional to T + C, where C is the time needed to make a logical copy of the data structure. For many data struetures, such as arrays and weil connected pointer-based structures (e.g., a doubly linked list), the best known value for Cis proportional to the size ofthe structure, making the copying time much larger than the sequential cost of an operation. The new method can also allow concurrent updates to the data structurej Herlihy's methodology cannot. A correct lock-free implementation can be derived from a correct sequential implementation in a straightforward manner using this method. The method is also fiexiblej a programmer can change many of the details of the default implementation to optimize for a particular pattern of data structure use.

/pdf/a-method-for-implementing-lock-free-shared-data-structures-5baiu7mt4p.pdf

A method for implementing lock-free shared-data structures

A wad-free implementation of a data object in shared memory is one that guarantees that any process can complete any operation in a finite number of steps, regardless of the execution speeds of the other processes. Much of the literature on wait-free synchronization has focused on the construction of atomic registers, which are memory locations that can be read 01 written instantaneously by concurrent processes. This model, in which a set of asynchronous processes communicate through shared atomic registers, is sometimes known as asynchronous PRAM. It is known, however, that the asynchronous PRAM model is not sufficiently powerful to construct wait-free implementations of many simple data types such as lists, queues, stacks, test-and-set registers, and others. In this paper, we give an algebraic characterization of a large class of objects that do have wait-free implementations in asynchronous PRAM, as well as a general algorithm for implementing them.

http://cs.brown.edu/people/mph/AspnesH90/p340-aspnes.pdf

Wait-free data structures in the asynchronous PRAM model

The PRAM model is a machine that comprises p processors and m memory cells; each processor can access each memory cell in constant time. The PRAM has proved a popular model for parallel algorithm design. For the task of designing efficient, highly parallel algorithms is quite difficult, in general. The PRAM model provides an abstraction that strips away problems of synchronization, reliability and communication delays, thereby permitting algorithm designers to focus first and foremost on the structure of the computational problem at hand, rather than the architecture of a currently available machine. As a consequence, a considerable body of PRAM algorithms has been discovered in the past several years, and a number of powerful techniques for designing such algorithms have been identified (see, for instance, the survey articles [KR88, EG88]).

O. Zajicek

Papers

The APRAM: incorporating asynchrony into the PRAM model