This paper presents and characterizes Rodinia, a benchmark suite for heterogeneous computing. To help architects study emerging platforms such as GPUs (Graphics Processing Units), Rodinia includes applications and kernels which target multi-core CPU and GPU platforms. The choice of applications is inspired by Berkeley's dwarf taxonomy. Our characterization shows that the Rodinia benchmarks cover a wide range of parallel communication patterns, synchronization techniques and power consumption, and has led to some important architectural insight, such as the growing importance of memory-bandwidth limitations and the consequent importance of data layout.

/pdf/rodinia-a-benchmark-suite-for-heterogeneous-computing-gib3ik78jh.pdf

Rodinia: A benchmark suite for heterogeneous computing

The advent of multicore CPUs and manycore GPUs means that mainstream processor chips are now parallel systems. Furthermore, their parallelism continues to scale with Moore's law. The challenge is to develop mainstream application software that transparently scales its parallelism to leverage the increasing number of processor cores, much as 3D graphics applications transparently scale their parallelism to manycore GPUs with widely varying numbers of cores.

/pdf/scalable-parallel-programming-with-cuda-fx7qj9io5g.pdf

Scalable parallel programming with CUDA

The rapid increase in the performance of graphics hardware, coupled with recent improvements in its programmability, have made graphics hardware a compelling platform for computationally demanding tasks in a wide variety of application domains. In this report, we describe, summarize, and analyze the latest research in mapping general-purpose computation to graphics hardware. We begin with the technical motivations that underlie general-purpose computation on graphics processors (GPGPU) and describe the hardware and software developments that have led to the recent interest in this field. We then aim the main body of this report at two separate audiences. First, we describe the techniques used in mapping general-purpose computation to graphics hardware. We believe these techniques will be generally useful for researchers who plan to develop the next generation of GPGPU algorithms and techniques. Second, we survey and categorize the latest developments in general-purpose application development on graphics hardware. This survey should be of particular interest to researchers who are interested in using the latest GPGPU applications in their systems of interest.

/pdf/a-survey-of-general-purpose-computation-on-graphics-hardware-2phx1q9xxr.pdf

A Survey of General-Purpose Computation on Graphics Hardware

A Survey of General-Purpose Computation on Graphics Hardware.

The advent of multicore CPUs and manycore GPUs means that mainstream processor chips are now parallel systems. Furthermore, their parallelism continues to scale with Moore’s law. The challenge is to develop mainstream application software that transparently scales its parallelism to leverage the increasing number of processor cores, much as 3D graphics applications transparently scale their parallelism to manycore GPUs with widely varying numbers of cores.

https://dl.acm.org/doi/pdf/10.1145/1365490.1365500

Scalable Parallel Programming with CUDA: Is CUDA the parallel programming model that application developers have been waiting for?

Systems and methods for a multi-core optimized Recurrent Neural Network (RNN) architecture are disclosed. The various architectures affect communication and synchronization operations according to the Multi-Bulk-Synchronous-Parallel (MBSP) model for a given processor. The resulting family of network architectures, referred to as MBSP-RNNs, perform similarly to a conventional RNNs having the same number of parameters, but are substantially more efficient when mapped onto a modern general purpose processor. Due to the large gain in computational efficiency, for a fixed computational budget, MBSP-RNNs outperform RNNs at applications such as end-to-end speech recognition.

Systems and methods for a multi-core optimized recurrent neural network

A system, method, and computer program product are provided for grouping linearly ordered primitives. In operation, a plurality of primitives are linearly ordered. Additionally, the primitives are grouped. Furthermore, at least one intersection query is performed, utilizing the grouping.

System, method, and computer program product for grouping linearly ordered primitives

A system, method, and computer program product are provided for converting a scan algorithm to a segmented scan algorithm in an operator independent manner. In operation, a scan algorithm and a limit index data structure are identified. Utilizing the limit index data structure, the scan algorithm is converted to a segmented scan algorithm in an operator-independent manner. Additionally, the segmented scan algorithm is performed to produce an output.

System, method, and computer program product for converting a scan algorithm to a segmented scan algorithm in an operator-independent manner

A system, method, and computer program product are provided for converting a reduction algorithm to a segmented reduction algorithm. In operation, a reduction algorithm is identified. Additionally, the reduction algorithm is converted to a segmented reduction algorithm. Furthermore, the segmented reduction algorithm is performed to produce an output.

System, method, and computer program product for converting a reduction algorithm to a segmented reduction algorithm

Graphics Processing Units (GPUs) are a fast evolving architecture. Over the last decade their programmability has been harnessed to solve non-graphics tasks—in many cases at a huge performance advantage to CPUs. Unlike CPUs, GPUs have always been a highly parallel architecture—thousands of lightweight execution contexts are necessary to keep the chip busy. CPUs too have become parallel over the same time period. But having started from the opposite end of the spectrum they expose parallelism of a different nature, where only tens of heavyweight execution contexts are active at any point in time. As parallel architectures become increasingly common, GPUs provide a research vehicle for developing parallel algorithms, programming models, and languages. This dissertation describes a library, CUDA Data Parallel Primitives Library (CUDPP), of building blocks called primitives for efficiently solving a broad range of problems on the GPU. CUDPP has become one of the most used libraries for data-parallel programming on GPUs thus showing the applicability of these building blocks. The most basic of the primitives is scan and its segmented variant, both of which have a rich history in the literature of data-parallel programming. I describe multiple efficient implementations for both variants. Subsequently I look at a class of problems exhibiting nested parallelism that have traditionally proved challenging for many-core processors. For each instance, segmented scan based primitives efficiently solve such problems for the first time on the GPU. In terms of complexity, sort and hash, fundamental algorithms in computer science, prove to be quite challenging to efficiently implement on the GPU. I take a detailed look at both radix sort and a class of comparison sorts. I compare between two different merge strategies that were used on parallel processors in the past and discuss their efficiency on GPUs. I show how sort is used to develop the first efficient algorithm for building spatial hierarchies on the GPU, thus showing how building spatial hierarchies is, in essence, sorting in three dimensions. The penultimate chapter describes the first efficient hash algorithm for GPUs based on cuckoo hashing. I close by looking at two implementations of the set data structure.

Shubhabrata Sengupta

Papers

Systems and methods for a multi-core optimized recurrent neural network

System, method, and computer program product for grouping linearly ordered primitives

System, method, and computer program product for converting a scan algorithm to a segmented scan algorithm in an operator-independent manner

System, method, and computer program product for converting a reduction algorithm to a segmented reduction algorithm

Efficient primitives and algorithms for many-core architectures