Previous studies of application usage show that the performance of collective communications are critical for high-performance computing. Despite active research in the field, both general and feasible solution to the optimization of collective communication problem is still missing. 

In this paper, we analyze and attempt to improve intra-cluster collective communication in the context of the widely deployed MPI programming paradigm by extending accepted models of point-to-point communication, such as Hockney, LogP/LogGP, and PLogP, to collective operations. We compare the predictions from models against the experimentally gathered data and using these results, construct optimal decision function for broadcast collective. We quantitatively compare the quality of the model-based decision functions to the experimentally-optimal one. Additionally, in this work, we also introduce a new form of an optimized tree-based broadcast algorithm, splitted-binary. 

Our results show that all of the models can provide useful insights into various aspects of the different algorithms as well as their relative performance. Still, based on our findings, we believe that the complete reliance on models would not yield optimal results. In addition, our experimental results have identified the gap parameter as being the most critical for accurate modeling of both the classical point-to-point-based pipeline and our extensions to fan-out topologies.

/pdf/performance-analysis-of-mpi-collective-operations-3bs7qwviid.pdf

Performance analysis of MPI collective operations

Modern compilers are responsible for translating the idealistic operational semantics of the source program into a form that makes efficient use of a highly complex heterogeneous machine. Since optimization problems are associated with huge and unstructured search spaces, this combinational task is poorly achieved in general, resulting in weak scalability and disappointing sustained performance. We address this challenge by working on the program representation itself, using a semi-automatic optimization approach to demonstrate that current compilers offen suffer from unnecessary constraints and intricacies that can be avoided in a semantically richer transformation framework. Technically, the purpose of this paper is threefold: (1) to show that syntactic code representations close to the operational semantics lead to rigid phase ordering and cumbersome expression of architecture-aware loop transformations, (2) to illustrate how complex transformation sequences may be needed to achieve significant performance benefits, (3) to facilitate the automatic search for program transformation sequences, improving on classical polyhedral representations to better support operation research strategies in a simpler, structured search space. The proposed framework relies on a unified polyhedral representation of loops and statements, using normalization rules to allow flexible and expressive transformation sequencing. This representation allows to extend the scalability of polyhedral dependence analysis, and to delay the (automatic) legality checks until the end of a transformation sequence. Our work leverages on algorithmic advances in polyhedral code generation and has been implemented in a modern research compiler.

/pdf/semi-automatic-composition-of-loop-transformations-for-deep-ktssmubyud.pdf

Semi-automatic composition of loop transformations for deep parallelism and memory hierarchies

Fourier and related transforms are a family of algorithms widely employed in diverse areas of computational science, notoriously difficult to scale on high-performance parallel computers with a large number of processing elements (cores). This paper introduces a popular software package called P3DFFT which implements fast Fourier transforms (FFTs) in three dimensions in a highly efficient and scalable way. It overcomes a well-known scalability bottleneck of three-dimensional (3D) FFT implementations by using two-dimensional domain decomposition. Designed for portable performance, P3DFFT achieves excellent timings for a number of systems and problem sizes. On a Cray XT5 system P3DFFT attains 45% efficiency in weak scaling from 128 to 65,536 computational cores. Library features include Fourier and Chebyshev transforms, Fortran and C interfaces, in- and out-of-place transforms, uneven data grids, and single and double precision. P3DFFT is available as open source at http://code.google.com/p/p3dfft/. This pa...

P3DFFT: A Framework for Parallel Computations of Fourier Transforms in Three Dimensions

Partitioned Global Address Space (PGAS) languages combine the programming convenience of shared memory with the locality and performance control of message passing. One such language, Unified Parallel C (UPC) is an extension of ISO C defined by a consortium that boasts multiple proprietary and open source compilers. Another PGAS language, Titanium, is a dialect of JavaTM designed for high performance scientific computation. In this paper we describe some of the highlights of two related projects, the Titanium project centered at U.C. Berkeley and the UPC project centered at Lawrence Berkeley National Laboratory. Both compilers use a source-to-source strategy that trans-lates the parallel languages to C with calls to a communication layer called GASNet. The result is portable high-performance compilers that run on a large variety of shared and distributed memory multiprocessors. Both projects combine compiler, runtime, and application efforts to demonstrate some of the performance and productivity advantages to these languages.

/pdf/productivity-and-performance-using-partitioned-global-369b23aboz.pdf

Productivity and performance using partitioned global address space languages

Preface. 1. Introductory Tutorial. 1.1 Getting Started. 1.2 Private and Shared Data. 1.3 Shared Arrays and Affinity of Shared Data. 1.4 Synchronization and Memory Consistency. 1.5 Work Sharing. 1.6 UPC Pointers. 1.7 Summary. Exercises. 2. Programming View and UPC Data Types. 2.1 Programming Models. 2.2 UPC Programming Model. 2.3 Shared and Private Variables. 2.4 Shared and Private Arrays. 2.5 Blocked Shared Arrays. 2.6 Compiling Environments and Shared Arrays. 2.7 Summary. Exercises. 3. Pointers and Arrays. 3.1 UPC Pointers. 3.2 Pointer Arithmetic. 3.3 Pointer Casting and Usage Practices. 3.4 Pointer Information and Manipulation Functions. 3.5 More Pointer Examples. 3.6 Summary. Exercises. 4. Work Sharing and Domain Decomposition. 4.1 Basic Work Distribution. 4.2 Parallel Iterations. 4.3 Multidimensional Data. 4.4 Distributing Trees. 4.5 Summary. Exercises. 5. Dynamic Shared Memory Allocation. 5.1 Allocating a Global Shared Memory Space Collectively. 5.2 Allocating Multiple Global Spaces. 5.3 Allocating Local Shared Spaces. 5.4 Freeing Allocated Spaces. 5.5 Summary. Exercises. 6. Synchronization and Memory Consistency. 6.1 Barriers. 6.2 Split-Phase Barriers. 6.3 Locks. 6.4 Memory Consistency. 6.5 Summary. Exercises. 7. Performance Tuning and Optimization. 7.1 Parallel System Architectures. 7.2 Performance Issues in Parallel Programming. 7.3 Role of Compilers and Run-Time Systems. 7.4 UPC Hand Optimization. 7.5 Case Studies. 7.6 Summary. Exercises. 8. UPC Libraries. 8.1 UPC Collective Library. 8.2 UPC-IO Library. 8.3 Summary. References. Appendix A: UPC Language Specifications, v1.1.1. Appendix B: UPC Collective Operations Specifications, v1.0. Appendix C: UPC-IO Specifications, v1.0. Appendix D: How to Compile and Run UPC Programs. Appendix E: Quick UPC Reference. Index.

UPC: Distributed Shared-Memory Programming

Partitioned Global Address Space languages like Unified Parallel C (UPC) are typically valued for their expressiveness, especially for computations with fine-grained random accesses. In this paper we show that the one-sided communication model used in these languages also has a significant performance advantage for bandwidth-limited applications. We demonstrate this benefit through communication microbenchmarks and a case-study that compares UPC and MPI implementations of the NAS Fourier Transform (FT) benchmark. Our optimizations rely on aggressively overlapping communication with computation but spreading communication events throughout the course of the local computation. This alleviates the potential communication bottleneck that occurs when the communication is packed into a single phase (e.g., the large all-to-all in a multidimensional FFT). Even though the new algorithms require more messages for the same total volume of data, the resulting overlap leads to speedups of over 1.75x and 1.9x for the two-sided and one-sided implementations, respectively, when compared to the default NAS Fortran/MPI release. Our best one-sided implementations show an average improvement of 15 percent over our best two-sided implementations. We attribute this difference to the lower software overhead of one-sided communication, which is partly fundamental to the semantic difference between one-sided and two-sided communication. Our UPC results use the Berkeley UPC compiler with the GASNet communication system, and demonstrate the portability and scalability of that language and implementation, with performance approaching 0.5TFlop/s on the FT benchmark running on 512 processors.

/pdf/optimizing-bandwidth-limited-problems-using-one-2jvs14m5a0.pdf

Optimizing Bandwidth Limited Problems Using One-SidedCommunication and Overlap

This paper demonstrates the one-sided communication used in languages like UPC can provide a significant performance advantage for bandwidth-limited applications. This is shown through communication microbenchmarks and a case-study of UPC and MPI implementations of the NAS FT benchmark. Our optimizations rely on aggressively overlapping communication with computation, alleviating bottlenecks that typically occur when communication is isolated in a single phase. The new algorithms send more and smaller messages, yet the one-sided versions achieve > 1.9/spl times/ speedup over the base Fortran/MPI. Our one-sided versions show an average 15% improvement over the two-sided versions, due to the lower software overhead of onesided communication, whose semantics are fundamentally lighter-weight than message passing. Our UPC results use Berkeley UPC with GASNet and demonstrate the scalability of that system, with performance approaching 0.5 TFlop/s on the FT benchmark with 512 processors.

/pdf/optimizing-bandwidth-limited-problems-using-one-sided-4hqmgygr56.pdf

Optimizing bandwidth limited problems using one-sided communication and overlap

High-end supercomputers are increasingly built out of commodity components, and lack tight integration between the processor and network. This often results in inefficiencies in the communication subsystem, such as high software overheads and/or message latencies. In this paper we use a set of microbenchmarks to quantify the cost of this commoditization, measuring software overhead, latency, and bandwidth on five contemporary supercomputing networks. We compare the performance of the ubiquitous MPI layer to that of lower-level communication layers, and quantify the advantages of the latter for small message performance. We also provide data on the potential for various communication-related optimizations, such as overlapping communication with computation or other communication. Finally, we determine the minimum size needed for a message to be considered 'large' (i.e., bandwidth-bound) on these platforms, and provide historical data on the software overheads of a number of supercomputers over the past decade.

/pdf/an-evaluation-of-current-high-performance-networks-45w6oli70k.pdf

An evaluation of current high-performance networks

This paper proposes anew memory registration strategy for supporting Remote DMA (RDMA) operations over pinning-based networks, as existing approaches are insufficient for efficiently implementing Global Address Space (GAS) languages. Although existing approaches often maximize bandwidth, they require levels of synchronization that discourage one-sided communication, and can have significant latency costs for small messages. The proposed Firehose algorithm attempts to expose one-sided, zero-copy communication as a common case, while minimizing the number of host-level synchronizations required to support remote memory operations. The basic idea is to reap the performance benefits of a pin-everything approach in the common case (without the drawbacks) and revert to a rendezvous-based approach to handle the uncommon case. In all cases, the algorithm attempts to amortize the cost of synchronization and pinning over multiple remote memory operations, improving performance over rendezvous by avoiding many handshaking messages and the cost of re-pinning recently used pages. Performance results are presented which demonstrate that the cost of two-sided handshaking and memory registration is negligible when the set of remotely referenced memory pages on a given node is smaller than the physical memory (where the entire working set can remain pinned), and for applications with larger working sets the performance degrades gracefully and consistently outperforms conventional approaches.

/pdf/a-new-dma-registration-strategy-for-pinning-based-high-yxmi5hkp8q.pdf

A new DMA registration strategy for pinning-based high performance networks

The Cray X1 was recently introduced as the first in a new line of parallel systems to combine high-bandwidth vector processing with an MPP system architecture. Alongside capabilities such as automatic fine-grained data parallelism through the use of vector instructions, the X1 offers hardware support for a transparent global-address space (GAS), which makes it an interesting target for GAS languages. In this paper, we describe our experience with developing a portable, open-source and high performance compiler for Unified Parallel C (UPC), a SPMD global-address space language extension of ISO C. As part of our implementation effort, we evaluate the X1's hardware support for GAS languages and provide empirical performance characterizations in the context of leveraging features such as vectorization and global pointers for the Berkeley UPC compiler. We discuss several difficulties encountered in the Cray C compiler which are likely to present challenges for many users, especially implementors of libraries and source-to-source translators. Finally, we analyze the performance of our compiler on some benchmark programs and show that, while there are some limitations of the current compilation approach, the Berkeley UPC compiler uses the X1 network more effectively than MPI or SHMEM, and generates serial code whose vectorizability is comparable to the original C code.

/pdf/evaluating-support-for-global-address-space-languages-on-the-2fflawkz9b.pdf

Christian Bell

Papers

Optimizing Bandwidth Limited Problems Using One-SidedCommunication and Overlap

Optimizing bandwidth limited problems using one-sided communication and overlap

An evaluation of current high-performance networks

A new DMA registration strategy for pinning-based high performance networks

Evaluating support for global address space languages on the Cray X1