Dynamic page placement policies for NUMA (nonuniform memory access time) shared-memory architectures are explored using two approaches that complement each other in important ways. The authors measure the performance of parallel programs running on the experimental DUnX operating system kernel for the BBN GP1000, which supports a highly parameterized dynamic page placement policy. They also develop and apply an analytic model of memory system performance of a local/remote NUMA architecture based on approximate mean-value analysis techniques. The model is validated against experimental data obtained with DUnX while running a synthetic workload. The results of this validation show that, in general, model predictions are quite good. Experiments investigating the effectiveness of dynamic page-placement and, in particular, dynamic multiple-copy page placement the cost of replication/coherency fault errors, and the cost of errors in deciding whether a page should move or be remotely referenced are described. >

Evaluation of NUMA memory management through modeling and measurements

Motivated by experience gained during the validation of a recent Approximate Mean Value Analysis (AMVA) model of modern shared memory architectures, this paper re-examines the “standard” AMVA approximation for non-exponential FCFS queues. We find that this approximation is often inaccurate for FCFS queues with high service time variability. For such queues, we propose and evaluate: (1) AMVA estimates of the mean residual service time at an arrival instant that are much more accurate than the standard AMVA estimate, (2) a new AMVA technique that provides a much more accurate estimate of mean center residence time than the standard AMVA estimate, and (3) a new AMVA technique for computing the mean residence time at a “downstream” queue which has a more bursty arrival process than is assumed in the standard AMVA equations. Together, these new techniques increase the range of applications to which AMVA may be fruitfully applied, so that for example, the memory system architecture of shared memory systems with complex modern processors can be analyzed with these computationally efficient methods.

/pdf/amva-techniques-for-high-service-time-variability-wfhhrpcc3f.pdf

AMVA techniques for high service time variability

This paper investigates the performance of shared-memory cluster-based architectures where each cluster is a shared-bus multiprocessor augmented with a protocol processor maintaining cache coherence across clusters. For a given number of processors, sixteen in this study, we evaluate the performance of various cluster configurations. We also consider the impact of adding a remote shared cache in each cluster. We use Mean Value Analysis to estimate the cache miss latencies of various types and the overall execution time. The service demands of shared resources are characterized in detail by examining the sub-requests issued in resolving cache misses. In addition to the architectural system parameters and the service demands on resources, the analytical model needs parameters pertinent to applications. The latter, in particular cache miss profiles, are obtained by trace-driven simulation of three benchmarks.Our results show that without remote caches the performance of cluster-based architectures is mixed. In some configurations, the negative effects of the longer latency of inter-cluster misses and of the contention on the protocol processor are too large to counter-balance the lower contention on the data buses. For two out of the three applications best results are obtained when the system has clusters of size 2 or 4. The cluster-based architectures with remote caches consistently outperform the single bus system for all 3 applications. We also exercise the model with parameters reflecting the current trend in technology making the processor relatively faster than the bus and memory. Under these new conditions, our results show a clear performance advantage for the cluster-based architectures, with or without remote caches, over single bus systems.

A performance evaluation of cluster architectures

In this work, we extend and evaluate a simple performance model to account for NUMA and bandwidth effects for single and multi-threaded calculations within the Gaussian 03 computational chemistry code on a contemporary multi-core, NUMA platform. By using the thread and memory placement APIs in Solaris, we present results for a set of calculations from which we analyze on-chip interconnect and intra-core bandwidth contention and show the importance of load-balancing between threads. The extended model predicts single threaded performance to within 1% errors and most multi-threaded experiments within 15% errors. Our results and modeling shows that accounting for bandwidth constraints within user-space code is beneficial.

A Simple Performance Model for Multithreaded Applications Executing on Non-uniform Memory Access Computers

The class of NUMA (nonuniform memory access time) shared memory architectures is becoming increasingly important with the desire for larger scale multiprocessors. In such machines, the placement and movement of code and data are crucial to performance. The operating system can play a role in managing placement through the policies and mechanisms of the virtual memory subsystem. In this paper, we develop an analytic model of memory system performance of a Local/Remote NUMA architecture based on approximate mean-value analysis techniques. The model assumes that a simple workload model based on a few parameters can often provide insight into the general behavior of real applications. The model is validated against experimental data obtained with the DUnX operating system kernel for the BBN GP1000 while running a synthetic workload. The results of this validation show that in general, model predictions are quite good, though in some cases the model fails to include the effect of unexpected behaviors in the implementation. Experiments investigate the effectiveness of dynamic multiple-copy page placement. We investigate the cost of incorrect policy decisions by introducing different percentages of policy error and measuring their effect on performance.

An analysis of dynamic page placement on a NUMA multiprocessor

Scalable shared-memory multiprocessors are the subject of much current research, but little is known about the performance behavior of these machines. This paper studies the performance effects of two machine characteristics and two program characteristics that seem to be major factors in determining the performance of a hierarchical shared-memory machine. We develop an analytical model of the traffic in a machine loosely based on Stanford's DASH multiprocessor and use program parameters extracted from multiprocessor traces to study its performance. It is shown that both locality in the data reference stream and the amount of data sharing in a program have an important impact on performance. Although less obvious, the bandwidth within each cluster in the hierarchy also has a significant performance effect. Optimizations that improve the intracluster cache coherence protocol or increase the bandwidth within a cluster can be quite effective.

Joseph Torrellas

Papers

Analysis of critical architectural and programming parameters in a hierarchical