With the SIMT execution model, GPUs can hidememory latency through massive multithreading for many applications that have regular memory access patterns. To support applications with irregular memory access patterns, cache hierarchies have been introduced to GPU architectures to capture temporal and spatial locality and mitigate the effect of irregular accesses. However, GPU caches exhibit poor efficiency due to the mismatch of the throughput-oriented execution model and its cache hierarchy design, which limits system performance and energy-efficiency. The massive amount of memory requests generated by GPU scause cache contention and resource congestion. Existing CPUcache management policies that are designed for multicoresystems, can be suboptimal when directly applied to GPUcaches. We propose a specialized cache management policy for GPGPUs. The cache hierarchy is protected from contention by the bypass policy based on reuse distance. Contention and resource congestion are detected at runtime. To avoid oversaturatingon-chip resources, the bypass policy is coordinated with warp throttling to dynamically control the active number of warps. We also propose a simple predictor to dynamically estimate the optimal number of active warps that can take full advantage of the cache space and on-chip resources. Experimental results show that cache efficiency is significantly improved and on-chip resources are better utilized for cache sensitive benchmarks. This results in a harmonic mean IPCimprovement of 74% and 17% (maximum 661% and 44% IPCimprovement), compared to the baseline GPU architecture and optimal static warp throttling, respectively.

/pdf/adaptive-cache-management-for-energy-efficient-gpu-computing-2g1vt0rp2l.pdf

Adaptive Cache Management for Energy-Efficient GPU Computing

An operator definition file 102 and the like included in a source program 101 and a compiler 100 that translates the source program 101 into a machine language program 105 are provided. The operator definition file 102 includes definitions of various fixed point type operators by class definitions. The compiler 100 can generate effectively advanced and specific instructions that a processor executes and make improvements through expanding functions and the like without repeating frequently upgrading of the version of the compiler itself. The compiler 100 is made up of an intermediate code generation unit 121 that generates intermediate codes, a machine language instruction substitution unit 122 that substitutes the intermediate codes referring to classes defined by the operator definition file 102 with machine language instructions and an optimization unit 130 that performs optimization targeting the intermediate codes including the substituted machine language instructions.

Compiler, compiler apparatus and compilation method

Exploiting data locality in GPUs is critical to making more efficient use of the existing caches and the NUMA-based memory hierarchy expected in future GPUs. While modern GPU programming models are designed to explicitly express parallelism, there is no clear explicit way to express data locality---i.e., reuse-based locality to make efficient use of the caches, or NUMA locality to efficiently utilize a NUMA system. On the one hand, this lack of expressiveness makes it a very challenging task for the programmer to write code to get the best performance out of the memory hierarchy. On the other hand, hardware-only architectural techniques are often suboptimal as they miss key higher-level program semantics that are essential to effectively exploit data locality. In this work, we propose the Locality Descriptor, a cross-layer abstraction to explicitly express and exploit data locality in GPUs. The Locality Descriptor (i) provides the software a flexible and portable interface to optimize for data locality, requiring no knowledge of the underlying memory techniques and resources, and (ii) enables the architecture to leverage key program semantics and effectively coordinate a range of techniques (e.g., CTA scheduling, cache management, memory placement) to exploit locality in a programmer-transparent manner. We demonstrate that the Locality Descriptor improves performance by 26.6% on average (up to 46.6%) when exploiting reuse-based locality in the cache hierarchy, and by 53.7% (up to 2.8X) when exploiting NUMA locality in a NUMA memory system.

The locality descriptor: a holistic cross-layer abstraction to express data locality in GPUs

Long latency of memory operation is a prominent performance bottleneck in graphics processing units (GPUs). The small data cache that must be shared across dozens of warps (a collection of threads) creates significant cache contention and premature data eviction. Prior works have recognized this problem and proposed warp throttling which reduces the number of active warps contending for cache space. In this paper we discover that individual load instructions in a warp exhibit four different types of data locality behavior: (1) data brought by a warp load instruction is used only once, which is classified as streaming data (2) data brought by a warp load is reused multiple times within the same warp, called intra-warp locality (3) data brought by a warp is reused multiple times but across different warps, called inter-warp locality (4) and some data exhibit both a mix of intra- and inter-warp locality. Furthermore, each load instruction exhibits consistently the same locality type across all warps within a GPU kernel. Based on this discovery we argue that cache management must be done using per-load locality type information, rather than applying warp-wide cache management policies. We propose Access Pattern-aware Cache Management (APCM), which dynamically detects the locality type of each load instruction by monitoring the accesses from one exemplary warp. APCM then uses the detected locality type to selectively apply cache bypassing and cache pinning of data based on load locality characterization. Using an extensive set of simulations we show that APCM improves performance of GPUs by 34% for cache sensitive applications while saving 27% of energy consumption over baseline GPU.

https://dl.acm.org/doi/pdf/10.1145/3079856.3080239

Access Pattern-Aware Cache Management for Improving Data Utilization in GPU

With increasing core-count, the cache demand of modern processors has also increased. However, due to strict area/power budgets and presence of poor data-locality workloads, blindly scaling cache capacity is both infeasible and ineffective. Cache bypassing is a promising technique to increase effective cache capacity without incurring power/area costs of a larger sized cache. However, injudicious use of cache bypassing can lead to bandwidth congestion and increased miss-rate and hence, intelligent techniques are required to harness its full potential. This paper presents a survey of cache bypassing techniques for CPUs, GPUs and CPU-GPU heterogeneous systems, and for caches designed with SRAM, non-volatile memory (NVM) and die-stacked DRAM. By classifying the techniques based on key parameters, it underscores their differences and similarities. We hope that this paper will provide insights into cache bypassing techniques and associated tradeoffs and will be useful for computer architects, system designers and other researchers.

A Survey of Cache Bypassing Techniques

The performance of General Purpose Graphics Processing Units (GPGPUs) is frequently limited by the off-chip memory bandwidth To mitigate this bandwidth wall problem, recent GPUs are equipped with on-chip L1 and L2 caches However, there has been little work for better utilizing on-chip shared caches in GPGPUs In this paper, we propose two cache management schemes: write-buffering and read-bypassing The write buffering technique tries to utilize the shared cache for inter-block communication, and thereby reduces the DRAM accesses as much as the capacity of the cache The read-bypassing scheme prevents the shared cache from being polluted by streamed data that are consumed only within a thread-block The proposed schemes can be selectively applied to global memory instructions using newly defined cache operators We evaluate the effects of the proposed schemes for a few GPGPU applications by simulations We have shown that the off-chip memory accesses can be successfully reduced by the proposed techniques We also analyze the effectiveness of these methods when the throughput gap between cores and off-chip memory becomes wider

Reducing off-chip memory traffic by selective cache management scheme in GPGPUs

We develop real-time image processing programs for a digital copier using a general-purpose microprocessor. To exploit the inherent data parallelism in many image processing algorithms, we use Intel's Pentium processor with multimedia extension (MMX). Each step of the digital copier process including the X-Zoom and the error diffusion halftoning is aggressively optimized for the Pentium MMX processor. The X-Zoom process that is based on the linear interpolation method is optimized using the software pipelining technique. For the error diffusion halftoning which requires nonlinear feedback, we exploit both the control-level and data-level parallelism. For the latter approach, a speculative quantization method is developed to break the dependency relation due to feedback and quantization operations. Our implementation acquires the maximum throughput of 30 ppm for A4-size paper using one 166 MHz Pentium MMX CPU, which is approximately five times faster than the code without MMX optimization.

Pentium-MMX-based implementation of a digital copier

Implementation of recursive equations using parallel computer architecture has long been of interest because the dependency problem makes it difficult to achieve significant speed-up. In this paper, efficient implementation of recursive filtering equations on partitioned data-path SIMD (Single Instruction Multiple Data) processors is studied. Especially, three parallel computation techniques, which are the block filtering, recursive doubling, and multi-block filtering methods, are implemented and their performances are compared using a Pentium CPU based system. The performance evaluation result of the multi-block processing method on a scalable SIMD processor is also presented.

/pdf/simd-processor-based-implementation-of-recursive-filtering-3kzjm17k74.pdf

SIMD processor based implementation of recursive filtering equations

As the complexity of multimedia applications increases, the need for efficient and compiler-friendly processor architectures also grows. In this paper, a new multimedia processor architecture is proposed. This processor has a 2-issue VLIW architecture with 64-bit SIMD arithmetic functional units to exploit the instruction-level and subword data parallelism found in multimedia applications. Moreover, densely encoded instructions supporting memory operands, DSP-like addressing modes, and SIMD capability boost the performance while keeping the code size and hardware cost small. To maximally utilize this architecture, a software environment including a code converter, a VLIW compiler system, and a compiled simulator has also been developed. The processor core has been synthesized for LSI logic 0.25 /spl mu/m library, which results in the total gate count of 102 K. In spite of the relatively smaller issue rate, the proposed processor shows a comparable or higher performance in terms of both the cycle count and the code size when compared to the 8-issue TMS320C62xx, for DSP benchmark kernels and an H.263 video encoder.

A 2 way VLIW processor architecture for embedded multimedia applications

General purpose graphics processing units (GPGPUs) contain many execution units, thus they are very attractive for high speed image processing. However, the error diffusion halftoning algorithm can hardly exploit the benefit of massively parallel processing architecture because this algorithm uses feedback of the output error as well as the results of neighboring pixels. In this study, pixels that can be processed without dependency are found by examining the dependency graph. Also, a parallel processing method requiring less synchronization overhead is developed by considering the characteristics of GPGPUs.

Jaewoo Ahn

Papers

Reducing off-chip memory traffic by selective cache management scheme in GPGPUs

Pentium-MMX-based implementation of a digital copier

SIMD processor based implementation of recursive filtering equations

A 2 way VLIW processor architecture for embedded multimedia applications

Parallel implementation of an error diffusion halftoning algorithm with a general purpose graphics processing unit