Introduction to stochastic programming

The ability of performance technology to keep pace with the growing complexity of parallel and distributed systems depends on robust performance frameworks that can at once provide system-specific performance capabilities and support high-level performance problem solving Flexibility and portability in empirical methods and processes are influenced primarily by the strategies available for instrmentation and measurement, and how effectively they are integrated and composed This paper presents the TAU (Tuning and Analysis Utilities) parallel performance sytem and describe how it addresses diverse requirements for performance observation and analysis

/pdf/the-tau-parallel-performance-system-3fr3lax14z.pdf

The Tau Parallel Performance System

SUMMARY HPCTOOLKIT is an integrated suite of tools that supports measurement, analysis, attribution, and presentation of application performance for both sequential and parallel programs. HPCTOOLKIT can pinpoint and quantify scalability bottlenecks in fully-optimized parallel programs with a measurement overhead of only a few percent. Recently, new capabilities were added to HPCTOOLKIT for collecting call path profiles for fully-optimized codes without any compiler support, pinpointing and quantifying bottlenecks in multithreaded programs, exploring performance information and source code using a new user interface, and displaying hierarchical space-time diagrams based on traces of asynchronous call stack samples. This paper provides an overview of HPCTOOLKIT and illustrates its utility for performance analysis of parallel applications.

/pdf/hpctoolkit-tools-for-performance-analysis-of-optimized-1yvjndp7yn.pdf

HPCTOOLKIT: tools for performance analysis of optimized parallel programs

Tapenade is an Automatic Differentiation (AD) tool which, given a Fortran or C code that computes a function, creates a new code that computes its tangent or adjoint derivatives. Tapenade puts particular emphasis on adjoint differentiation, which computes gradients at a remarkably low cost. This article describes the principles of Tapenade, a subset of the general principles of AD. We motivate and illustrate with examples the AD model of Tapenade, that is, the structure of differentiated codes and the strategies used to make them more efficient. Along with this informal description, we formally specify this model by means of data-flow equations and rules of Operational Semantics, making this the reference specification of the tangent and adjoint modes of Tapenade. One benefit we expect from this formal specification is the capacity to formally study the AD model itself, especially for the adjoint mode and its sophisticated strategies. This article also describes the architectural choices of the implementation of Tapenade. We describe the current performance of Tapenade on a set of codes that include industrial-size applications. We present the extensions of the tool that are planned in a foreseeable future, deriving from our ongoing research on AD.

/pdf/the-tapenade-automatic-differentiation-tool-principles-model-2h9b5qhsm2.pdf

The Tapenade automatic differentiation tool: Principles, model, and specification

Ice flow models used to project the mass balance of ice sheets in Greenland and Antarctica usually rely on the Shallow Ice Approximation (SIA) and the Shallow-Shelf Approximation (SSA), sometimes combined into so-called hybrid models. Such models, while computationally efficient, are based on a simplified set of physical assumptions about the mechanical regime of the ice flow, which does not uniformly apply everywhere on the ice sheet/ice shelf system, especially near grounding lines, where rapid changes are taking place at present. Here, we present a new thermomechanical finite element model of ice flow named ISSM (Ice Sheet System Model) that includes higher-order stresses, high spatial resolution capability and data assimilation techniques to better capture ice dynamics and produce realistic simulations of ice sheet flow at the continental scale. ISSM includes several approximations of the momentum balance equations, ranging from the two-dimensional SSA to the three-dimensional full-Stokes formulation. It also relies on a massively parallelized architecture and state-of-the-art scalable tools. ISSM employs data assimilation techniques, at all levels of approximation of the momentum balance equations, to infer basal drag at the ice-bed interface from satellite radar interferometry-derived observations of ice motion. Following a validation of ISSM with standard benchmarks, we present a demonstration of its capability in the case of the Greenland Ice Sheet. We show ISSM is able to simulate the ice flow of an entire ice sheet realistically at a high spatial resolution, with higher-order physics, thereby providing a pathway for improving projections of ice sheet evolution in a warming climate. Copyright 2012 by the American Geophysical Union.

/pdf/continental-scale-high-order-high-spatial-resolution-ice-3d784xhe9r.pdf

Continental scale, high order, high spatial resolution, ice sheet modeling using the Ice Sheet System Model (ISSM)

The Open/ADF tool allows the evaluation of derivatives of functions defined by a Fortran program. The derivative evaluation is performed by a Fortran code resulting from the analysis and transformation of the original program that defines the function of interest. Open/ADF has been designed with a particular emphasis on modularity, flexibility, and the use of open source components. While the code transformation follows the basic principles of automatic differentiation, the tool implements new algorithmic approaches at various levels, for example, for basic block preaccumulation and call graph reversal. Unlike most other automatic differentiation tools, Open/ADF uses components provided by the Open/AD framework, which supports a comparatively easy extension of the code transformations in a language-independent fashion. It uses code analysis results implemented in the OpenAnalysis component. The interface to the language-independent transformation engine is an XML-based format, specified through an XML schema. The implemented transformation algorithms allow efficient derivative computations using locally optimized cross-country sequences of vertex, edge, and face elimination steps. Specifically, for the generation of adjoint codes, Open/ADF supports various code reversal schemes with hierarchical checkpointing at the subroutine level. As an example from geophysical fluid dynamics, a nonlinear time-dependent scalable, yet simple, barotropic ocean model is considered. OpenAD/F's reverse mode is applied to compute sensitivities of some of the model's transport properties with respect to gridded fields such as bottom topography as independent (control) variables.

https://www.mcs.anl.gov/uploads/cels/papers/P1230.pdf

OpenAD/F: A Modular Open-Source Tool for Automatic Differentiation of Fortran Codes

Understanding why the performance of a multithreaded program does not improve linearly with the number of cores in a shared-memory node populated with one or more multicore processors is a problem of growing practical importance This paper makes three contributions to performance analysis of multithreaded programs First, we describe how to measure and attribute parallel idleness, namely, where threads are stalled and unable to work This technique applies broadly to programming models ranging from explicit threading (eg, Pthreads) to higher-level models such as Cilk and OpenMP Second, we describe how to measure and attribute parallel overhead -- when a thread is performing miscellaneous work other than executing the user's computation By employing a combination of compiler support and post-mortem analysis, we incur no measurement cost beyond normal profiling to glean this information Using idleness and overhead metrics enables one to pinpoint areas of an application where concurrency should be increased (to reduce idleness), decreased (to reduce overhead), or where the present parallelization is hopeless (where idleness and overhead are both high) Third, we describe how to measure and attribute arbitrary performance metrics for high-level multithreaded programming models, such as Cilk This requires bridging the gap between the expression of logical concurrency in programs and its realization at run-time as it is adaptively partitioned and scheduled onto a pool of threads We have prototyped these ideas in the context of Rice University's HPCToolkit performance tools We describe our approach, implementation, and experiences applying this approach to measure and attribute work, idleness, and overhead in executions of Cilk programs

/pdf/effective-performance-measurement-and-analysis-of-1yjlec3pch.pdf

Effective performance measurement and analysis of multithreaded applications

It is increasingly difficult for complex scientific programs to attain a significant fraction of peak performance on systems that are based on microprocessors with substantial instruction-level parallelism and deep memory hierarchies. Despite this trend, performance analysis and tuning tools are still not used regularly by algorithm and application designers. To a large extent, existing performance tools fail to meet many user needs and are cumbersome to use. To address these issues, we developed HPCVIEW—a toolkit for combining multiple sets of program profile data, correlating the data with source code, and generating a database that can be analyzed anywhere with a commodity Web browser. We argue that HPCVIEW addresses many of the issues that have limited the usability and the utility of most existing tools. We originally built HPCVIEW to facilitate our own work on data layout and optimizing compilers. Now, in addition to daily use within our group, HPCVIEW is being used by several code development teams in DoD and DoE laboratories as well as at NCSA.

http://www.cs.rice.edu/~johnmc/papers/hpcview-lacsi01.pdf

HPCVIEW: A Tool for Top-down Analysis of Node Performance

High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale simulations. However, the lack of deep understanding on how modern GPUs can be connected and the real impact of state-of-the-art interconnect technology on multi-GPU application performance become a hurdle. In this paper, we fill the gap by conducting a thorough evaluation on five latest types of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NVLink-SLI and NVSwitch, from six high-end servers and HPC platforms: NVIDIA P100-DGX-1, V100-DGX-1, DGX-2, OLCF's SummitDev and Summit supercomputers, as well as an SLI-linked system with two NVIDIA Turing RTX-2080 GPUs. Based on the empirical evaluation, we have observed four new types of GPU communication network NUMA effects: three are triggered by NVLink's topology, connectivity and routing, while one is caused by PCIe chipset design issue. These observations indicate that, for an application running in a multi-GPU node, choosing the right GPU combination can impose considerable impact on GPU communication efficiency, as well as the application's overall performance. Our evaluation can be leveraged in building practical multi-GPU performance models, which are vital for GPU task allocation, scheduling and migration in a shared environment (e.g., AI cloud and HPC centers), as well as communication-oriented performance tuning.

Nathan R. Tallent

Papers

HPCTOOLKIT: tools for performance analysis of optimized parallel programs

OpenAD/F: A Modular Open-Source Tool for Automatic Differentiation of Fortran Codes

Effective performance measurement and analysis of multithreaded applications

HPCVIEW: A Tool for Top-down Analysis of Node Performance

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect