This paper investigates the problem of partitioning a shared cache between multiple concurrently executing applications. The commonly used LRU policy implicitly partitions a shared cache on a demand basis, giving more cache resources to the application that has a high demand and fewer cache resources to the application that has a low demand. However, a higher demand for cache resources does not always correlate with a higher performance from additional cache resources. It is beneficial for performance to invest cache resources in the application that benefits more from the cache resources rather than in the application that has more demand for the cache resources. This paper proposes utility-based cache partitioning (UCP), a low-overhead, runtime mechanism that partitions a shared cache between multiple applications depending on the reduction in cache misses that each application is likely to obtain for a given amount of cache resources. The proposed mechanism monitors each application at runtime using a novel, cost-effective, hardware circuit that requires less than 2kB of storage. The information collected by the monitoring circuits is used by a partitioning algorithm to decide the amount of cache resources allocated to each application. Our evaluation, with 20 multiprogrammed workloads, shows that UCP improves performance of a dual-core system by up to 23% and on average 11% over LRU-based cache partitioning.

/pdf/utility-based-cache-partitioning-a-low-overhead-high-4t50s5xud2.pdf

Utility-Based Cache Partitioning: A Low-Overhead, High-Performance, Runtime Mechanism to Partition Shared Caches

As much of the world's computing continues to move into the cloud, the overprovisioning of computing resources to ensure the performance isolation of latency-sensitive tasks, such as web search, in modern datacenters is a major contributor to low machine utilization. Being unable to accurately predict performance degradation due to contention for shared resources on multicore systems has led to the heavy handed approach of simply disallowing the co-location of high-priority, latency-sensitive tasks with other tasks. Performing this precise prediction has been a challenging and unsolved problem. In this paper, we present Bubble-Up, a characterization methodology that enables the accurate prediction of the performance degradation that results from contention for shared resources in the memory subsystem. By using a bubble to apply a tunable amount of “pressure” to the memory subsystem on processors in production datacenters, our methodology can predict the performance interference between co-locate applications with an accuracy within 1% to 2% of the actual performance degradation. Using this methodology to arrive at “sensible” co-locations in Google's production datacenters with real-world large-scale applications, we can improve the utilization of a 500-machine cluster by 50% to 90% while guaranteeing a high quality of service of latency-sensitive applications.

/pdf/bubble-up-increasing-utilization-in-modern-warehouse-scale-4sumeu2h5q.pdf

Bubble-Up: increasing utilization in modern warehouse scale computers via sensible co-locations

DRAM memory is a major resource shared among cores in a chip multiprocessor (CMP) system. Memory requests from different threads can interfere with each other. Existing memory access scheduling techniques try to optimize the overall data throughput obtained from the DRAM and thus do not take into account inter-thread interference. Therefore, different threads running together on the same chip can ex- perience extremely different memory system performance: one thread can experience a severe slowdown or starvation while another is un- fairly prioritized by the memory scheduler. This paper proposes a new memory access scheduler, called the Stall-Time Fair Memory scheduler (STFM), that provides quality of service to different threads sharing the DRAM memory system. The goal of the proposed scheduler is to "equalize" the DRAM-related slowdown experienced by each thread due to interference from other threads, without hurting overall system performance. As such, STFM takes into account inherent memory characteristics of each thread and does not unfairly penalize threads that use the DRAM system without interfering with other threads. We show that STFM significantly reduces the unfairness in the DRAM system while also improving system throughput (i.e., weighted speedup of threads) on a wide variety of workloads and systems. For example, averaged over 32 different workloads running on an 8-core CMP, the ratio between the highest DRAM-related slowdown and the lowest DRAM-related slowdown reduces from 5.26X to 1.4X, while the average system throughput improves by 7.6%. We qualitatively and quantitatively compare STFM to one new and three previously- proposed memory access scheduling algorithms, including network fair queueing. Our results show that STFM provides the best fairness, system throughput, and scalability.

/pdf/stall-time-fair-memory-access-scheduling-for-chip-kj13rlydtp.pdf

Stall-Time Fair Memory Access Scheduling for Chip Multiprocessors

In a chip-multiprocessor (CMP) system, the DRAM system isshared among cores. In a shared DRAM system, requests from athread can not only delay requests from other threads by causingbank/bus/row-buffer conflicts but they can also destroy other threads’DRAM-bank-level parallelism. Requests whose latencies would otherwisehave been overlapped could effectively become serialized. As aresult both fairness and system throughput degrade, and some threadscan starve for long time periods.This paper proposes a fundamentally new approach to designinga shared DRAM controller that provides quality of service to threads,while also improving system throughput. Our parallelism-aware batchscheduler (PAR-BS) design is based on two key ideas. First, PARBSprocesses DRAM requests in batches to provide fairness and toavoid starvation of requests. Second, to optimize system throughput,PAR-BS employs a parallelism-aware DRAM scheduling policythat aims to process requests from a thread in parallel in the DRAMbanks, thereby reducing the memory-related stall-time experienced bythe thread. PAR-BS seamlessly incorporates support for system-levelthread priorities and can provide different service levels, includingpurely opportunistic service, to threads with different priorities.We evaluate the design trade-offs involved in PAR-BS and compareit to four previously proposed DRAM scheduler designs on 4-, 8-, and16-core systems. Our evaluations show that, averaged over 100 4-coreworkloads, PAR-BS improves fairness by 1.11X and system throughputby 8.3% compared to the best previous scheduling technique, Stall-Time Fair Memory (STFM) scheduling. Based on simple request prioritizationrules, PAR-BS is also simpler to implement than STFM.

http://users.ece.cmu.edu/~omutlu/pub/parbs_isca08.pdf

Parallelism-Aware Batch Scheduling: Enhancing both Performance and Fairness of Shared DRAM Systems

This paper studies the impact of L2 cache sharing on threads that simultaneously share the cache, on a chip multi-processor (CMP) architecture. Cache sharing impacts threads nonuniformly, where some threads may be slowed down significantly, while others are not. This may cause severe performance problems such as sub-optimal throughput, cache thrashing, and thread starvation for threads that fail to occupy sufficient cache space to make good progress. Unfortunately, there is no existing model that allows extensive investigation of the impact of cache sharing. To allow such a study, we propose three performance models that predict the impact of cache sharing on co-scheduled threads. The input to our models is the isolated L2 cache stack distance or circular sequence profile of each thread, which can be easily obtained on-line or off-line. The output of the models is the number of extra L2 cache misses for each thread due to cache sharing. The models differ by their complexity and prediction accuracy. We validate the models against a cycle-accurate simulation that implements a dual-core CMP architecture, on fourteen pairs of mostly SPEC benchmarks. The most accurate model, the inductive probability model, achieves an average error of only 3.9%. Finally, to demonstrate the usefulness and practicality of the model, a case study that details the relationship between an application's temporal reuse behavior and its cache sharing impact is presented.

/pdf/predicting-inter-thread-cache-contention-on-a-chip-multi-t4ae0ol8pw.pdf

Predicting inter-thread cache contention on a chip multi-processor architecture

This paper presents a detailed study of fairness in cache sharing between threads in a chip multiprocessor (CMP) architecture. Prior work in CMP architectures has only studied throughput optimization techniques for a shared cache. The issue of fairness in cache sharing, and its relation to throughput, has not been studied. Fairness is a critical issue because the operating system (OS) thread scheduler's effectiveness depends on the hardware to provide fair cache sharing to co-scheduled threads. Without such hardware, serious problems, such as thread starvation and priority inversion, can arise and render the OS scheduler ineffective. This paper makes several contributions. First, it proposes and evaluates five cache fairness metrics that measure the degree of fairness in cache sharing, and shows that two of them correlate very strongly with the execution-time fairness. Execution-time fairness is defined as how uniform the execution times of co-scheduled threads are changed, where each change is relative to the execution time of the same thread running alone. Secondly, using the metrics, the paper proposes static and dynamic L2 cache partitioning algorithms that optimize fairness. The dynamic partitioning algorithm is easy to implement, requires little or no profiling, has low overhead, and does not restrict the cache replacement algorithm to LRU. The static algorithm, although requiring the cache to maintain LRU stack information, can help the OS thread scheduler to avoid cache thrashing. Finally, this paper studies the relationship between fairness and throughput in detail. We found that optimizing fairness usually increases throughput, while maximizing throughput does not necessarily improve fairness. Using a set of co-scheduled pairs of benchmarks, on average our algorithms improve fairness by a factor of 4/spl times/, while increasing the throughput by 15%, compared to a nonpartitioned shared cache.

/pdf/fair-cache-sharing-and-partitioning-in-a-chip-multiprocessor-16bmv7sfbi.pdf

Fair Cache Sharing and Partitioning in a Chip Multiprocessor Architecture

This paper describes the hardware architecture for a flexible probability density estimation unit to be used in a large vocabulary speech recognition system, and targeted for mobile platforms. The speech recognition system is based on hidden Markov models and consists of two computationally intensive parts - the probability density estimation using Gaussian distributions, and the Viterbi decoding. The power hungry nature of these computations prevents porting the application successfully to mobile devices. We have designed a flexible probability estimation unit that is both power efficient and meets real time requirements while being flexible enough to handle emerging speech recognition techniques. The flexible nature of the design allows it to utilize emerging power and computation reduction techniques (at the algorithm level) to achieve up to an 80% power reduction as compared to conventional designs

Flexible Low Power Probability Density Estimation Unit For Speech Recognition

This paper proposes an architecture for real-time large vocabulary speech recognition on a mobile embedded device. The speech recognition system is based on Hidden Markov Model (HMM), which involves complex mathematical operations such as probability estimation and Viterbi decoding. This computational nature makes it power hungry and realtime recognition is not achieved by porting software solutions on embedded device. Our system architecture has a low power embedded processor and dedicated ASIC units for complex computations. These units operate at a low frequency of 50 MHz thus consuming low power. The system uses RAM for the intermediate values and flash memory to store acoustic and language models for speech recognition.

Architecture for Low Power Large Vocabulary Speech Recognition

With computing trend moving towards ubiquitous computing propelled by the advances in embedded mobile processors and battery technology, speech recognition is becoming an essential part of embedded processor I/O device. Speech recognition is also used in command and control and automated customer service. Real time speech recognition application is both computation and memory intensive and it overwhelms even a high end multi-gigahertz processor to achieve real time performance. An embedded mobile device cannot support real time large vocabulary speech recognition application as the processors are less aggressive because of tighter power budget. Hardware solution to speech recognition, in the past, have mainly concentrated on building specialized hardware or ASIC accelerators to run software speech application faster but have largely ignored design for large vocabulary and power reduction. In this work, we propose a hardware-software co-design for real time large vocabulary speech recognition. Our design has custom ASIC blocks and RAM memories and a low power processor. The processor maintains a high level control over the blocks and processes parts of speech recognition application which is not computation and memory intensive. The custom ASIC computes the Gaussian probability and performs word search in the dictionary. The RAMs are used for storing the intermediate values and states. The design can handle large vocabulary speech recognition in real time on a mobile embedded device. Our word search uses innovative dictionary word layout in memory which reduces bandwidth by a factor of 11 compared to software implementation and by a factor of 4 compared to other ASIC implementation. One unit of our proposed design can perform 4x and 20x better than other proposed design of specialized hardware design for software speech application in computing the Gaussian probability and word search, respectively.

Dhruba Chandra

Papers

Fair Cache Sharing and Partitioning in a Chip Multiprocessor Architecture

Predicting inter-thread cache contention on a chip multi-processor architecture

Flexible Low Power Probability Density Estimation Unit For Speech Recognition

Architecture for Low Power Large Vocabulary Speech Recognition

Speech recognition co-processor