GPUShare: Fair-Sharing Middleware for GPU Clouds

TLDR: GPUShare is presented, a software-based mechanism that can yield a kernel before all of its threads have run, thus giving finer control over the time slice for which the GPU is allocated to a process and improves fair GPU sharing across tenants.

Abstract: Many new cloud-focused applications such as deeplearning and graph analytics have started to rely on the highcomputing throughput of GPUs, but cloud providers cannotcurrently support fine-grained time-sharing on GPUs to enablemulti-tenancy for these types of applications. Currently, schedulingis performed by the GPU driver in combination with ahardware thread dispatcher to maximize utilization. However, when multiple applications with contrasting kernel running timesand high-utilization of the GPU need to be co-located, thisapproach unduly favors one or more of the applications at theexpense of others. This paper presents GPUShare, a middleware solution forGPU fair sharing among high-utilization, long-running applications. It begins by analyzing the scenarios under which thecurrent driver-based multi-process scheduling fails, noting thatsuch scenarios are quite common. It then describes a softwarebasedmechanism that can yield a kernel before all of its threadshave run, thus giving finer control over the time slice for whichthe GPU is allocated to a process. In controlling time slices onthe GPU by yielding kernels, GPUShare improves fair GPUsharing across tenants and outperforms the CUDA driver byup to 45% for two tenants and by up to 89% for more thantwo tenants, while incurring a maximum overhead of only 12%.Additional improvements are obtained from having a centralscheduler that further smooths out disparities across tenants'GPU shares improving fair sharing by up to 92% for two tenantsand by up to 76% for more than two tenants.