Performance evaluation is the basis for choosing appropriate system‐level configurations for large‐scale data centers. While the change of a system‐level configuration would impact lots of jobs in the data centers, traditional load‐testing benchmarks are not sufficient to support the decision‐making because they cannot accurately reproduce the complex behaviors of a large number of jobs. Therefore, we expect to further evaluate the system configuration based on the production environment. However, there are technical challenges, namely, the lack of a holistic evaluation method that can unite the evaluation results of various jobs, and the uninterruptable production environment that should not be affected by the evaluation procedure. To address these challenges, we propose a holistic performance evaluation methodology and design its implementation platform. We introduce a simple but powerful performance metric, ERU (effectiveness of resource usage), and combine the ERU of involved jobs into a summarized value to measure the effect of a configuration change. We validate our ERU metric by comparing it with the CPI (Cycle per Instruction) and QPS (query per second) metrics, deploy the platform to production data centers and demonstrate the effectiveness for measuring system‐level configurations of both software (JVM compiler update) and hardware (NUMA on/off) to save 14.44% and 11% resources respectively in advance.

A holistic evaluation methodology for configuring production data centers

Non-uniform memory access (NUMA) has been extensively studied at the machine level but few studies have examined NUMA optimizations at the cluster level. This paper introduces a holistic NUMA-aware scheduling policy that combines both machine-level and cluster-level NUMA-aware optimizations. We evaluate our holistic NUMA-aware scheduling policy on Google's production cluster trace with a cluster scheduling simulator that measures the impact of NUMAaware scheduling under two scheduling algorithms, Best Fit and Enhanced PVM (E-PVM). While our results highlight that a holistic NUMA-aware scheduling policy substantially increases the proportion of NUMA-fit tasks by 22.0% and 25.6% for both the Best Fit and E-PVM scheduling algorithms, respectively, there is a non-trivial tradeoff between cluster job packing efficiency and NUMA-fitness for the E-PVM algorithm under certain circumstances.

Xiangling Kong

Papers

Evaluation of NUMA-Aware Scheduling in Warehouse-Scale Clusters