The most exciting development in parallel computer architecture is the convergence of traditionally disparate approaches on a common machine structure. This book explains the forces behind this convergence of shared-memory, message-passing, data parallel, and data-driven computing architectures. It then examines the design issues that are critical to all parallel architecture across the full range of modern design, covering data access, communication performance, coordination of cooperative work, and correct implementation of useful semantics. It not only describes the hardware and software techniques for addressing each of these issues but also explores how these techniques interact in the same system. Examining architecture from an application-driven perspective, it provides comprehensive discussions of parallel programming for high performance and of workload-driven evaluation, based on understanding hardware-software interactions.


* synthesizes a decade of research and development for practicing engineers, graduate students, and researchers in parallel computer architecture, system software, and applications development

* presents in-depth application case studies from computer graphics, computational science and engineering, and data mining to demonstrate sound quantitative evaluation of design trade-offs 

* describes the process of programming for performance, including both the architecture-independent and architecture-dependent aspects, with examples and case-studies

* illustrates bus-based and network-based parallel systems with case studies of more than a dozen important commercial designs

Table of Contents

1 Introduction 
2 Parallel Programs 
3 Programming for Performance 
4 Workload-Driven Evaluation 
5 Shared Memory Multiprocessors 
6 Snoop-based Multiprocessor Design 
7 Scalable Multiprocessors
8 Directory-based Cache Coherence
9 Hardware-Software Tradeoffs 
10 Interconnection Network Design 
11 Latency Tolerance 
12 Future Directions 
APPENDIX A Parallel Benchmark Suites

Parallel Computer Architecture: A Hardware/Software Approach

Aimed at the working researcher or scientific C/C++ or Fortran programmer, this text introduces the competent research programmer to a new vocabulary of idioms and techniques for parallelizing software using OpenMP.

http://www.dfmf.uned.es/~osotolongo/pub/mpi/Parallel%20programming%20in%20OpenMP.pdf

Parallel Programming in OpenMP

The SGI Origin 2000 is a cache-coherent non-uniform memory access (ccNUMA) multiprocessor designed and manufactured by Silicon Graphics, Inc. The Origin system was designed from the ground up as a multiprocessor capable of scaling to both small and large processor counts without any bandwidth, latency, or cost cliffs. The Origin system consists of up to 512 nodes interconnected by a scalable Craylink network. Each node consists of one or two R10000 processors, up to 4 GB of coherent memory, and a connection to a portion of the XIO IO subsystem. This paper discusses the motivation for building the Origin 2000 and then describes its architecture and implementation. In addition, performance results are presented for the NAS Parallel Benchmarks V2.2 and the SPLASH2 applications. Finally, the Origin system is compared to other contemporary commercial ccNUMA systems.

/pdf/the-sgi-origin-a-ccnuma-highly-scalable-server-1ccbawslrg.pdf

The SGI Origin: a ccNUMA highly scalable server

In this article we examine the problem of extending modern operating systems to run efficiently on large-scale shared-memory multiprocessors without a large implementation effort. Our approach brings back an idea popular in the 1970s: virtual machine monitors. We use virtual machines to run multiple commodity operating systems on a scalable multiprocessor. This solution addresses many of the challenges facing the system software for these machines. We demonstrate our approach with a prototype called Disco that runs multiple copies of Silicon Graphics' IRIX operating system on a multiprocessor. Our experience shows that the overheads of the monitor are small and that the approach provides scalability as well as the ability to deal with the nonuniform memory access time of these systems. To reduce the memory overheads associated with running multiple operating systems, virtual machines transparently share major data structures such as the program code and the file system buffer cache. We use the distributed-system support of modern operating systems to export a partial single system image to the users. The overall solution achieves most of the benefits of operating systems customized for scalable multiprocessors, yet it can be achieved with a significantly smaller implementation effort.

/pdf/disco-running-commodity-operating-systems-on-scalable-2hxj56mgei.pdf

Disco: running commodity operating systems on scalable multiprocessors

Commercial applications such as databases and Web servers constitute the largest and fastest-growing segment of the market for multiprocessor servers. Ongoing innovations in disk subsystems, along with the ever increasing gap between processor and memory speeds, have elevated memory system design as the critical performance factor for such workloads. However, most current server designs have been optimized to perform well on scientific and engineering workloads, potentially leading to design decisions that are non-ideal for commercial applications. The above problem is exacerbated by the lack of information on the performance requirements of commercial workloads, the lack of available applications for widespread study, and the fact that most representative applications are too large and complex to serve as suitable benchmarks for evaluating trade-offs in the design of processors and servers.This paper presents a detailed performance study of three important classes of commercial workloads: online transaction processing (OLTP), decision support systems (DSS), and Web index search. We use the Oracle commercial database engine for our OLTP and DSS workloads, and the AltaVista search engine for our Web index search workload. This study characterizes the memory system behavior of these workloads through a large number of architectural experiments on Alpha multiprocessors augmented with full system simulations to determine the impact of architectural trends. We also identify a set of simplifications that make these workloads more amenable to monitoring and simulation without affecting representative memory system behavior. We observe that systems optimized for OLTP versus DSS and index search workloads may lead to diverging designs, specifically in the size and speed requirements for off-chip caches.

/pdf/memory-system-characterization-of-commercial-workloads-2y6dcbve62.pdf

Memory system characterization of commercial workloads

"STiNG" is a Cache Coherent Non-Uniform Memory Access (CC-NUMA) Multiprocessor designed and built by Sequent Computer Systems, Inc. It combines four processor Symmetric Multi-processor (SMP) nodes (called Quads), using a Scalable Coherent Interface (SCI) based coherent interconnect. The Quads are based on the Intel P6 processor and the external bus it defines. In addition to 4 P6 processors, each Quad may contain up to 4 GBytes of system memory, 2 Peripheral Component Interface (PCI) busses for I/O, and a Lynx board. The Lynx board provides the datapath to the SCI-based interconnect and ensures system-wide cache coherency. STiNG is one of the first commercial CC-NUMA systems to be built. This paper describes the motivation for building STiNG as well as its architecture and implementation. In addition, performance analysis is provided for On-Line Transaction Processing (OLTP) and Decision Support System (DSS) workloads. Finally, the status of the current implementation is reviewed.

STiNG: A CC-NUMA Computer System for the Commercial Marketplace

A Cost-Reduced Enterprise Server (CRES) system includes a flexible resource-efficient server having a plurality of Processor Memory Boards (PMBs) coupled to an Input/Output Module (IOM). The IOM provides all networking and storage interfaces for the server. The IOM is implemented as a field-replaceable pluggable module, and thus all Input/Output (I/O) capabilities or resources of a CRES system may be upgraded via replacement of the IOM. Each PMB is dividable into a pair of Symmetric MultiProcessor (SMP) complexes, and each complex is coupled to a respective portion of the I/O resources provided by the IOM. Each portion of the IOM provides a pair of I/O daughter-module connectors compatible with standard I/O interfaces, such as Peripheral Component Interconnect (PCI)-X and PCI-Express. One or more CRES systems may be coupled to one or more Enterprise Server (ES) systems to form a multi-chassis server managed collectively as one or more provisioned servers.

Fabric-Backplane Enterprise Servers with Pluggable I/O Sub-System

An implementation-efficient, multiple-counter value hardware performance counter is disclosed. A hardware counter of one embodiment includes a memory array and a hardware incrementer. The array stores counter values that are indexable by an index constructed based at least on the number of events to which the counter values correspond. The index may be constructed as a concatenation of a number of bits binarily representing the number of events, and a number of bits binarily representing the number of qualifiers to the events. The incrementer reads the counter values from the array, increments the counter values, and writes the resulting counter values back into the array. The array may be divided into banks over which the counter values are stored, where each bank has a separate instance of the incrementer. Each bank may have a separate instance of the index that indexes only those counters stored in the bank.

Implementation-efficient multiple-counter value hardware performance counter

In recent years, DRAM technology improvements have scaled at a much slower pace than processors. While server processor core counts grow from 33% to 50% on a yearly cadence, DDR 3/4 memory channel bandwidth has grown at a slower rate, and memory latency has remained relatively flat for some time. Combined with new computing paradigms such as big data analytics, which involves analyzing massive volumes of data in real time, there is a trend of increasing pressure on the memory subsystem. This makes it important for computer architects to understand the sensitivity of the performance of big data workloads to memory bandwidth and latency, and how these workloads compare to more conventional workloads. To address this, we present straightforward analytic equations to quantify the impact of memory bandwidth and latency on workload performance, leveraging measured data from performance counters on real systems. We demonstrate how the values of the components of these equations can be used to classify different workloads according to their inherent bandwidth requirement and latency sensitivity. Using this performance model, we show the relative sensitivities of big data, high-performance computing, and enterprise workload classes to changes in memory bandwidth and latency.

Quantifying the Performance Impact of Memory Latency and Bandwidth for Big Data Workloads

Caching memory contents into cache partitions based on their locations is disclosed. A location of a line of memory to be cached in a cache is determined. The cache is partitioned into a number of cache sections. The section for the line of memory is determined based on the location of the line of memory as applied against a memory line location-dependent allocation policy. The line of memory is then stored in the section of the cache determined.

Russell M. Clapp

Papers

STiNG: A CC-NUMA Computer System for the Commercial Marketplace

Fabric-Backplane Enterprise Servers with Pluggable I/O Sub-System

Implementation-efficient multiple-counter value hardware performance counter

Quantifying the Performance Impact of Memory Latency and Bandwidth for Big Data Workloads

Caching memory contents into cache partitions baesed on memory locations