Modern large-scale deep learning workloads highlight the need for parallel execution across many devices in order to fit model data into hardware accelerator memories. In these settings, array redistribution may be required during a computation, but can also become a bottleneck if not done efficiently. In this paper we address the problem of redistributing multi-dimensional array data in SPMD computations, the most prevalent form of parallelism in deep learning. We present a type-directed approach to synthesizing array redistributions as sequences of MPI-style collective operations. We prove formally that our synthesized redistributions are memory-efficient and perform no excessive data transfers. Array redistribution for SPMD computations using collective operations has also been implemented in the context of the XLA SPMD partitioner, a production-grade tool for partitioning programs across accelerator systems. We evaluate our approach against the XLA implementation and find that our approach delivers a geometric mean speedup of $1.22\times$, with maximum speedups as a high as $5.7\times$, while offering provable memory guarantees, making our system particularly appealing for large-scale models.

Memory-efficient array redistribution through portable collective communication

X10 is a partitioned global address space programming language that supports the notion of places; a place consists of some data and some lightweight tasks called activities. Each activity runs at a place and may invoke a place-change operation (using the at-construct) to synchronously perform some computation at another place. These place-change operations can be very expensive, as they need to copy all the required data from the current place to the remote place. However, identifying the necessary number of place-change operations and the required data during each place-change operation are non-trivial tasks, especially in the context of irregular applications (like graph applications) that contain complex code with large amounts of cross-referencing objects—not all of those objects may be actually required, at the remote place. In this article, we present AT-Com, a scheme to optimize X10 code with place-change operations.AT-Com consists of two inter-related new optimizations: (i) AT-Opt, which minimizes the amount of data serialized and communicated during place-change operations, and (ii) AT-Pruning, which identifies/elides redundant place-change operations and does parallel execution of place-change operations. AT-Opt uses a novel abstraction, called abstract-place-tree, to capture place-change operations in the program. For each place-change operation, AT-Opt uses a novel inter-procedural analysis to precisely identify the data required at the remote place in terms of the variables in the current scope. AT-Opt then emits the appropriate code to copy the identified data-items to the remote place. AT-Pruning introduces a set of program transformation techniques to emit optimized code such that it avoids the redundant place-change operations. We have implemented AT-Com in the x10v2.6.0 compiler and tested it over the IMSuite benchmark kernels. Compared to the current X10 compiler, the AT-Com optimized code achieved a geometric mean speedup of 18.72× and 17.83× on a four-node (32 cores per node) Intel and two-node (16 cores per node) AMD system, respectively.

https://dl.acm.org/doi/pdf/10.1145/3345558

Optimizing Remote Communication in X10

X10 is a partitioned global address space (PGAS) programming language that supports the notion of places; a place consists of some data and some lightweight tasks called activities. Each activity runs at a place and may invoke a place-change operation (using the at-construct) to synchronously perform some computation at another place. These place-change operations need to copy all the required data from the current place to the remote place. However, identifying the required data during each place-change operation is a non-trivial task, especially in the context of irregular applications (like graph applications) that contain large amounts of cross-referencing objects - not all of those objects may be actually required, at the remote place. In this paper, we present a new optimization AT-Opt that minimizes the amount of data serialized and communicated during place-change operations. AT-Opt uses a novel abstraction called abstract-place-tree to capture place-change operations in the program. For each place-change operation, AT-Opt uses a novel inter-procedural analysis to precisely identify the data required at the remote place, in terms of the variables in the current scope. AT-Opt then emits the appropriate code to copy the identified data-items to the remote place. We have implemented AT-Opt in the x10v2.6.0 compiler and tested it over the IMSuite benchmark kernels. Compared to the current X10 compiler, the AT-Opt optimized code achieved a geometric mean speedup of 8.61× and 5.57×, on a two-node (32 cores each) Intel and two-node (16 cores each) AMD system, respectively.

Optimizing remote data transfers in X10

A measurement system which uses custom-made compact wireless units to measure and display parameters related to automotive thermal comfort is proposed. This system consists of a central Data Acquisition (DAQ) Unit and an interface for display of measured data. The DAQ acts as a typical gateway system by collecting information via Bluetooth from different sensors, processing them and transferring the entire data to the cloud for visualization and storage. The system is capable of measuring ambient, globe and surface temperatures. Ambient and surface temperatures are measured through Bluetooth enabled devices and transferred via the globe sensor module to the central DAQ unit. This data is visualized on a user-friendly interface – either on a personal computer (dashboard view) or on a mobile / tablet (web page view). The developed sensor kit is tested and validated in laboratory and vehicle conditions. The sensors are initially tested for accuracy by placing them in hot ovens in laboratory conditions. This is followed by tests for precision in which the sensors are placed in an automobile and their outputs are compared with outputs of standard commercial sensors. It is observed that the measurement error of the wireless kit is less than 1% for temperature measurements and less than 2% for humidity measurements. These accurate and precise measurements indicate that the proposed wireless kit and methodology is ideal for evaluation of cabin thermal environment.

Custom Wireless Sensing System for Evaluation of Automotive Cabin Thermal Comfort Metrics

The study of numerical models for the human body has become a major focus of the research community in biology and medicine. For instance, numerical ionic models of a complex organ, such as the heart, must be able to represent individual cells and their interconnections through ionic channels, forming a system with billions of cells, and requiring efficient code to handle such a large system. The modeling of the electrical system of the heart combines a compute-intensive kernel that calculates the intensity of current flowing through cell membranes, and feeds a linear solver for computing the electrical potential of each cell. Considering this context, we propose limpetMLIR, a code generator and compiler transformer to accelerate the kernel phase of ionic models and bridge the gap between compiler technology and electrophysiology simulation. LimpetMLIR makes use of the MLIR infrastructure, its dialects, and transformations to drive forward the study of ionic models, and accelerate the execution of multi-cell systems. Experiments conducted in 43 ionic models show that our limpetMLIR based code generation greatly outperforms current state-of-the-art simulation systems by an average of 2.9x, reaching peak speedups of more than 15x in some cases. To our knowledge, this is the first work that deeply connects an optimizing compiler infrastructure to electrophysiology models of the human body, showing the potential benefits of using compiler technology in the simulation of human cell interactions.

/pdf/lifting-code-generation-of-cardiac-physiology-simulation-to-2owlome1.pdf

Arun Thangamani

Papers

Optimizing remote data transfers in X10

Custom Wireless Sensing System for Evaluation of Automotive Cabin Thermal Comfort Metrics

Lifting Code Generation of Cardiac Physiology Simulation to Novel Compiler Technology

Optimizing Remote Communication in X10

Intelligent buildings with IoT systems using ML and HITL for indoor environmental control: an investigation of occupants’ adoption intent