Stephen J. Hartley Oxford University Press, New York, 1998, 260 pp. ISBN 0-19-511315-2, $45.00 Concurrent Programming  is a thorough treatment of Java multi-threaded programming for both a stand-alone and distributed environment. Designed mostly for students in concurrent or parallel programming classes, the text is also an excellent reference for the practicing professional developing multi-threaded programs or applets. Hartley first provides a complete explanation of the features of Java necessary to write concurrent programs, including topics such as exception handling, interfaces, and packages. He then gives the reader a solid background to write multi-threaded programs and also presents the problems introduced when writing concurrent programs—namely race conditions, mutual exclusion, and deadlock. Hartley also provides several software solutions that do not require the use of common process and thread mechanisms. Once the groundwork is laid for writing concurrent programs, Hartley then takes a different approach than most Java references. Rather than presenting how Java handles mutual exclusion with the  synchronized  keyword (although it is covered later), he first looks at semaphore-based solutions to classic concurrent problems such as bounded-buffer, readers-writers, and the dining philosophers. Hartley also uses the same approach to develop Java classes for monitors and message passing. This unique approach to introducing concurrency allows the readers to both understand how Java threads are synchronized and how the basic synchronization mechanism can be used to construct more abstract tools such as semaphores. If there is a shortcoming with the text it is with the lack of sufficient coverage of remote method invocation (RMI), although there is a section covering RMI. This is quite understandable as RMI is a fairly recent phenomenon with the Java community. Also, the classes that Hartley provides could easily implement RMI rather than sockets to handle communication. The strengths of the book include its ease in reading, several examples at the end of chapters, a package similar to Xtango that provides algorithm animation, and a supportive web site by the author (see  www.mcs.drexel.edu/~shartley/ConcProgJava/index.html ) including compressed source code. As Java becomes more dominant on the server side of multi-tier applications, writing thread-safe concurrent applications becomes even more important.  Concurrent Programming  is a strong step towards teaching students and professionals such skills. Greg Gagne, Westminster College of Salt Lake City Salt Lake City, Utah

Distributed Computing: Fundamentals, Simulations and Advanced Topics

The next generation of supercomputers will break the exascale barrier. Soon we will have systems capable of at least one quintillion (billion billion) floating-point operations per second (1018 FLOPS). Tremendous amounts of work have been invested into identifying and overcoming the challenges of the exascale era. In this work, we present an overview of these efforts and provide insight into the important trends, developments, and exciting research opportunities in exascale computing. We use a three-stage approach in which we (1) discuss various exascale landmark studies, (2) use data-driven techniques to analyze the large collection of related literature, and (3) discuss eight research areas in depth based on influential articles. Overall, we observe that great advancements have been made in tackling the two primary exascale challenges: energy efficiency and fault tolerance. However, as we look forward, we still foresee two major concerns: the lack of suitable programming tools and the growing gap between processor performance and data bandwidth (i.e., memory, storage, networks). Although we will certainly reach exascale soon, without additional research, these issues could potentially limit the applicability of exascale computing.

https://dl.acm.org/doi/pdf/10.1145/3372390

The Landscape of Exascale Research: A Data-Driven Literature Analysis

The Sunway TaihuLight supercomputer is the world’s first system with a peak performance of 125 PFlops. In this report, we provide a detailed introduction to the TaihuLight system. In contrast with other existing heterogeneous supercomputers, which include both CPU processors and PCIe-connected many-core accelerators (NVIDIA GPU or Intel Xeon Phi), the computing power of TaihuLight is provided by a homegrown many-core SW26010 CPU that includes both the management processing elements (MPEs) and computing processing elements (CPEs) in one chip. With 260 processing elements in one CPU, a single SW26010 provides a peak performance of over three TFlops. To alleviate the memory bandwidth bottleneck in most applications, each CPE comes with a scratch pad memory, which serves as a user-controlled cache. To support the parallelization of programs on the new many-core architecture, in addition to the basic C/C++ and Fortran compilers, the system provides a customized Sunway OpenACC tool that supports the OpenACC 2.0 syntax. We also reports our preliminary efforts on developing and optimizing applications on the TaihuLight system, focusing on key application domains, such as earth system modeling, ocean surface wave modeling, atomistic simulation, and virtual drug screening,etc.

The Sunway TaihuLight supercomputer：system and applications

In HPC, two trends have led to the emergence and popularity of an operating-system approach in which multiple kernels are run simultaneously on each compute node. The first trend has been the increase in complexity of the HPC software environment, which has placed the traditional HPC kernel approaches under stress. Meanwhile, microprocessors with more and more cores are being produced, allowing specialization within a node. As is typical in an emerging field, different groups are considering many different approaches to deploying multi-kernels.In this paper we identify and describe a number of ongoing HPC multi-kernel efforts. Given the increasing number of choices for implementing and providing compute node kernel functionality, users and system designers will find value in understanding the differences among the kernels (and among the perspectives) of the different multi-kernel efforts. To that end, we provide a survey of approaches and qualitatively compare and contrast the alternatives. We identify a series of criteria that characterize the salient differences among the approaches, providing users and system designers with a common language for discussing the features of a design that are relevant for them. In addition to the set of criteria for characterizing multi-kernel architectures, the paper contributes a classification of current multi-kernel projects according to those criteria.

A Multi-Kernel Survey for High-Performance Computing

The Message Passing Interface has been the dominating programming model for developing scalable and high-performance parallel applications. Collective operations empower group communication operations in a portable, and efficient manner and are used by a large number of applications across different domains. Optimization of collective operations is the key to achieve good performance speed-ups and portability. Broadcast or One-to-all communication is one of the most commonly used collectives in MPI applications. However, the existing algorithms for broadcast do not effectively utilize the high degree of parallelism and increased message rate capabilities offered by modern architectures. In this paper, we address these challenges and propose a Scalable Multi-Endpoint broadcast algorithm that combines hierarchical communication with multiple endpoints per node for high performance and scalability. We evaluate the proposed algorithm against state-of-the-art designs in other MPI libraries, including MVAPICH2, Intel MPI, and Spectrum MPI. We demonstrate the benefits of the proposed algorithm at benchmark and application level at scale on four different hardware architectures, including Intel Cascade Lake, Intel Skylake, AMD EPYC, and IBM POWER9, and with InfiniBand and Omni-Path interconnects. Compared to other state-of-the-art designs, our proposed design shows up to 2.5 times performance improvements at a microbenchmark level with 128 Nodes. We also observe up to 37% improvement in broadcast communication latency for the SPECMPI scientific applications

Leveraging Network-level parallelism with Multiple Process-Endpoints for MPI Broadcast

In this paper we describe the hardware and application-inherent challenges that future exascale systems pose to high-performance computing (HPC) and propose a system architecture that addresses them. This architecture is based on proven building blocks and few principles: (1) a fast light-weight kernel that is supported by a virtualized Linux for tasks that are not performance critical, (2) decentralized load and health management using fault-tolerant gossip-based information dissemination, (3) a maximally-parallel checkpoint store for cheap checkpoint/restart in the presence of frequent component failures, and (4) a runtime that enables applications to interact with the underlying system platform through new interfaces. The paper discusses the vision behind FFMK and the current state of a prototype implementation of the system, which is based on a microkernel and an adapted MPI runtime.

/pdf/ffmk-a-fast-and-fault-tolerant-microkernel-based-system-for-3m5icj3pgi.pdf

FFMK: A Fast and Fault-tolerant Microkernel-based System for Exascale Computing

Major data centre providers are introducing RDMA-based networks for their tenants, as well as for operating the underlying infrastructure. In comparison to traditional socket-based network stacks, RDMA-based networks offer higher throughput, lower latency and reduced CPU overhead. However, transparent checkpoint and migration operations become much more difficult. The key reason is that the OS is removed from the critical path of communication. As a result, some of the communication state itself resides in the NIC hardware and is no more under the direct control of the OS. This control includes especially the support for virtualisation of communication which is needed for live migration of communication partners. In this paper, we propose the basic principles required to implement a migration-capable RDMA-based network. We recommend some changes at the software level and small changes at the hardware level. As a proof of concept, we integrate the proposed changes into SoftRoCE, an open-source kernel-level implementation of the RoCE protocol. We claim that these changes introduce no runtime overhead when migration does not happen. Finally, we develop a proof-of-concept implementation for migrating containerised applications that use RDMA-based networks.

MigrOS: Transparent Operating Systems Live Migration Support for Containerised RDMA-applications

Driven by ever increasing performance demands of compute-intensive applications, supercomputing systems comprise more and more nodes. This growth is a significant burden for fast group communication primitives and also makes those systems more susceptible to failures of individual nodes. In this paper we present a two-phase fault-tolerant scheme for group communication. Using broadcast as an example, we provide a full-spectrum discussion of our approach --- from a formal analysis to LogP-based simulations to a message-passing-based implementation running on a large cluster. Ultimately, we are able to reduce the complex problem of reliable and fault-tolerant collective group communication to a graph theoretical renumbering problem. Both, simulations and measurements, show our solution to achieve a latency reduction of 50% with up to six times fewer messages sent in comparison to existing schemes.

Corrected trees for reliable group communication

Contemporary HPC systems use batch scheduling of compute jobs running on exclusively assigned hardware resources. During communication, polling for progress is the state of the art as it promises minimal latency. Previous work on oversubscription and event-based communication, i.e. vacating the CPU while waiting for communication to finish, shows that these techniques can improve the overall system utilisation and reduce the energy consumption. Despite these findings, neither of the two techniques is commonly used in HPC systems today. We believe that the current lack of detailed studies of the low-level effects of event-based communication, a key enabler of efficient oversubscription for classical MPI-based applications, is a major obstacle to a wider adoption. We demonstrate that the sched_yield system call, which is often used for oversubscription scenarios, is not best suited for this purpose on modern Linux systems. Furthermore, we incorporate event-based communication into Open MPI and evaluate the effects on latency and energy consumption using an MPI micro-benchmark. Our results indicate that event-base communication incurs significant latency overhead but also saves energy. Both effects grow with the imbalance of the application using MPI.

Jan Bierbaum

Papers

FFMK: A Fast and Fault-tolerant Microkernel-based System for Exascale Computing

MigrOS: Transparent Operating Systems Live Migration Support for Containerised RDMA-applications

Corrected trees for reliable group communication

Towards Efficient Oversubscription: On the Cost and Benefit of Event-Based Communication in MPI

TardiS: Migrating Containers with RDMA Networks.