MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google's clusters every day.

/pdf/mapreduce-simplified-data-processing-on-large-clusters-16emdgg5h5.pdf

MapReduce: simplified data processing on large clusters

MapReduce is a programming model and an associated implementation for processing and generating large datasets that is amenable to a broad variety of real-world tasks. Users specify the computation in terms of a map and a reduce function, and the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks. Programmers find the system easy to use: more than ten thousand distinct MapReduce programs have been implemented internally at Google over the past four years, and an average of one hundred thousand MapReduce jobs are executed on Google's clusters every day, processing a total of more than twenty petabytes of data per day.

/pdf/mapreduce-simplified-data-processing-on-large-clusters-1zth35tzla.pdf

This paper evaluates the suitability of the MapReduce model for multi-core and multi-processor systems. MapReduce was created by Google for application development on data-centers with thousands of servers. It allows programmers to write functional-style code that is automatically parallelized and scheduled in a distributed system. We describe Phoenix, an implementation of MapReduce for shared-memory systems that includes a programming API and an efficient runtime system. The Phoenix runtime automatically manages thread creation, dynamic task scheduling, data partitioning, and fault tolerance across processor nodes. We study Phoenix with multi-core and symmetric multiprocessor systems and evaluate its performance potential and error recovery features. We also compare MapReduce code to code written in lower-level APIs such as P-threads. Overall, we establish that, given a careful implementation, MapReduce is a promising model for scalable performance on shared-memory systems with simple parallel code

/pdf/evaluating-mapreduce-for-multi-core-and-multiprocessor-1xfheav61u.pdf

Evaluating MapReduce for Multi-core and Multiprocessor Systems

We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attention-kernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can be also used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.

Rethinking Attention with Performers

Increasing integrated-circuit pin bandwidth has motivateda corresponding increase in the degree or radix of interconnection networksand their routers. This paper introduces the flattened butterfly, a cost-efficient topology for high-radix networks. On benign (load-balanced) traffic, the flattened butterfly approaches the cost/performance of a butterfly network and has roughly half the cost of a comparable performance Clos network.The advantage over the Clos is achieved by eliminating redundant hopswhen they are not needed for load balance. On adversarial traffic, the flattened butterfly matches the cost/performance of a folded-Clos network and provides an order of magnitude better performance than a conventional butterfly.In this case, global adaptive routing is used to switchthe flattened butterfly from minimal to non-minimal routing - usingredundant hops only when they are needed. Minimal and non-minimal, oblivious and adaptive routing algorithms are evaluated on the flattened butterfly.We show that load-balancing adversarial traffic requires non-minimalglobally-adaptive routing and show that sequential allocators are required to avoid transient load imbalance when using adaptive routing algorithms.We also compare the cost of the flattened butterfly to folded-Clos, hypercube,and butterfly networks with identical capacityand show that the flattened butterfly is more cost-efficient thanfolded-Clos and hypercube topologies.

/pdf/flattened-butterfly-a-cost-efficient-topology-for-high-radix-46tduypx6i.pdf

Flattened butterfly: a cost-efficient topology for high-radix networks

We prove that prefix sums of n integers of at most b bits can be found on a COMMON CRCW PRAM in time with a linear time-processor product. The algorithm is optimally fast, for any polynomial number of processors. In particular, if the time taken is . This is a generalisation of previous result. The previous time algorithm was valid only for O(log n)-bit numbers. Application of this algorithm to r-way parallel merge sort algorithm is also considered. We also consider a more realistic PRAM variant, in which the word size, m, may be smaller than b (m≥log n). On this model, prefix sums can be found in optimal time.

On parallel prefix computation

We consider the problem of deterministic sorting of integers on a parallel RAM (PRAM). The best previous result ( T. Hagerup, 1987 , Inform. and Comput.75, 39–51) states that n integers of size polynomial in n can be sorted in time O(log n) on a Priority CRCW PRAM with O( n log log n log n ) processors. We prove that n integers drawn from a set {0, …, m−1} can be sorted on an Arbitrary CRCW PRAM in time O( log n log log n + log log m) with a time-processor product of O(n log log m). In particular, if m = n(log n)O(1), the time and number of processors used are O( log n log log n ) and O( n( log log n) 2 log n ) , respectively. This improves the previous result in several respects: The new algorithm is faster, it works on a weaker PRAM model, and it is closer to optimality for input numbers of superpolynomial size. If log log m = O( log n log log n ) , the new algorithm is optimally fast, for any polynomial number of processors, and if log log m = (1 + Ω(1)) log log n and log log m = 0 ( log n ), it has optimal speedup relative to the fastest known sequential algorithm. The space needed is O(nme), for arbitrary but fixed e > 0. The sorting algorithm derives its speed from a fast solution to a special list ranking problem of possible independent interest, the monotonic list ranking problem. In monotonic list ranking, each list element has an associated key, and the keys are known to increase monotonically along the list. We show that monotonic list ranking problems of size n can be solved optimally in time O( log n log log n ) . We also discuss and attempt to solve some of the problems arising in the precise description and implementation of parallel recursive algorithms. As part of this effort, we introduce a new PRAM variant, the allocated PRAM.

Improved deterministic parallel integer sorting

An algorithm with worst case time complexity O(log/sup 2/N) in two dimensions and O(m/sup 1/2/log N) in three dimensions with N input points and m as the number of tetrahedra in triangulation is given. Its AT/sup 2/ VLSI complexity on Thompson's logarithmic delay model, (1983) is O(N/sup 2/log/sup 6/N) in two dimensions and O(m/sup 2/Nlog/sup 4/ N) in three dimensions. >

Efficient VLSI parallel algorithm for Delaunay triangulation on orthogonal tree network in two and three dimensions

A modified interval line representation is proposed. This representation makes it possible to speed up some heuristics and algorithms for planar routing problems. This technique is illustrated by speeding up the heuristic proposed by Tsukiyama and Kuh for the double row routing problem. The heuristic suggested by Tsukiyama and Kuh for double row routing problem, can be implemented in O(n/sup 2/rlg r) time algorithm, whereas the implementation suggested by Tsukiyama and Kuh requires O(n/sup 3/r) time. It is also shown that the heuristic can be implemented in parallel in O(lg/sup 2/n+lg r) time using O(N/sup 4/+n/sup 2/r) processors on the concurrent read exclusive write (CREW) model. >

http://ieeexplore.ieee.org/iel2/328/4744/00185112.pdf

V.C. Prasad

Papers

On parallel prefix computation

Improved deterministic parallel integer sorting

Efficient VLSI parallel algorithm for Delaunay triangulation on orthogonal tree network in two and three dimensions

Modified interval line representation and its applications to planar routing problems