Datacenter applications demand microsecond-scale tail latencies and high request rates from operating systems, and most applications handle loads that have high variance over multiple timescales. Achieving these goals in a CPU-efficient way is an open problem. Because of the high overheads of today’s kernels, the best available solution to achieve microsecond-scale latencies is kernel-bypass networking, which dedicates CPU cores to applications for spin-polling the network card. But this approach wastes CPU: even at modest average loads, one must dedicate enough cores for the peak expected load. Shenango achieves comparable latencies but at far greater CPU efficiency. It reallocates cores across applications at very fine granularity—every 5 μs—enabling cycles unused by latency-sensitive applications to be used productively by batch processing applications. It achieves such fast reallocation rates with (1) an efficient algorithm that detects when applications would benefit from more cores, and (2) a privileged component called the IOKernel that runs on a dedicated core, steering packets from the NIC and orchestrating core reallocations. When handling latency-sensitive applications, such as memcached, we found that Shenango achieves tail latency and throughput comparable to ZygOS, a state-of-the-art, kernel-bypass network stack, but can linearly trade latency-sensitive application throughput for batch processing application throughput, vastly increasing CPU efficiency.

/pdf/shenango-achieving-high-cpu-efficiency-for-latency-sensitive-3leziece6v.pdf

Shenango: Achieving High {CPU} Efficiency for Latency-sensitive Datacenter Workloads

This paper presents our design and experience with a microkernel-inspired approach to host networking called Snap. Snap is a userspace networking system that supports Google's rapidly evolving needs with flexible modules that implement a range of network functions, including edge packet switching, virtualization for our cloud platform, traffic shaping policy enforcement, and a high-performance reliable messaging and RDMA-like service. Snap has been running in production for over three years, supporting the extensible communication needs of several large and critical systems. Snap enables fast development and deployment of new networking features, leveraging the benefits of address space isolation and the productivity of userspace software development together with support for transparently upgrading networking services without migrating applications off of a machine. At the same time, Snap achieves compelling performance through a modular architecture that promotes principled synchronization with minimal state sharing, and supports real-time scheduling with dynamic scaling of CPU resources through a novel kernel/userspace CPU scheduler co-design. Our evaluation demonstrates over 3x Gbps/core improvement compared to a kernel networking stack for RPC workloads, software-based RDMA-like performance of up to 5M IOPS/core, and transparent upgrades that are largely imperceptible to user applications. Snap is deployed to over half of our fleet of machines and supports the needs of numerous teams.

Snap: a microkernel approach to host networking

Emerging Multicore SoC SmartNICs, enclosing rich computing resources (e.g., a multicore processor, onboard DRAM, accelerators, programmable DMA engines), hold the potential to offload generic datacenter server tasks. However, it is unclear how to use a SmartNIC efficiently and maximize the offloading benefits, especially for distributed applications. Towards this end, we characterize four commodity SmartNICs and summarize the offloading performance implications from four perspectives: traffic control, computing capability, onboard memory, and host communication. Based on our characterization, we build iPipe, an actor-based framework for offloading distributed applications onto SmartNICs. At the core of iPipe is a hybrid scheduler, combining FCFS and DRR-based processor sharing, which can tolerate tasks with variable execution costs and maximize NIC compute utilization. Using iPipe, we build a real-time data analytics engine, a distributed transaction system, and a replicated key-value store, and evaluate them on commodity SmartNICs. Our evaluations show that when processing 10/25Gbps of application bandwidth, NIC-side offloading can save up to 3.1/2.2 beefy Intel cores and lower application latencies by 23.0/28.0 μs.

Offloading distributed applications onto smartNICs using iPipe

Intraoperative fluoroscopy, portable X-ray, and CT: patient and operating room personnel radiation exposure in spinal surgery

In recent years, many applications have started using serverless computing platforms primarily due to the ease of deployment and cost efficiency they offer. However, the existing scheduling mechanisms of serverless platforms fall short in catering to the unique characteristics of such applications: burstiness, short and variable execution times, statelessness and use of a single core. Specifically, the existing mechanisms fall short in meeting the requirements generated due to the combined effect of these characteristics: scheduling at a scale of millions of function invocations per second while achieving predictable performance. In this paper, we argue for a cluster-level centralized and core-granular scheduler for serverless functions. By maintaining a global view of the cluster resources, the centralized approach eliminates queue imbalances while the core granularity reduces interference; together these properties enable reduced performance variability. We expect such a scheduler to increase the adoption of serverless computing platforms by various latency and throughput sensitive applications.

Centralized Core-granular Scheduling for Serverless Functions

The recently proposed dataplanes for microsecond scale applications, such as IX and ZygOS, use nonpreemptive policies to schedule requests to cores. For the many real-world scenarios where request service times follow distributions with high dispersion or a heavy tail, they allow short requests to be blocked behind long requests, which leads to poor tail latency.

Shinjuku is a single-address space operating system that uses hardware support for virtualization to make preemption practical at the microsecond scale. This allows Shinjuku to implement centralized scheduling policies that preempt requests as often as every 5µsec and work well for both light and heavy tailed request service time distributions. We demonstrate that Shinjuku provides significant tail latency and throughput improvements over IX and ZygOS for a wide range of workload scenarios. For the case of a RocksDB server processing both point and range queries, Shinjuku achieves up to 6.6× higher throughput and 88% lower tail latency.

/pdf/shinjuku-preemptive-scheduling-for-usecond-scale-tail-lsc64dm16t.pdf

Shinjuku: preemptive scheduling for µsecond-scale tail latency

Background The impact of patient body mass index (BMI) on image-guided spine interventions remains unknown. Higher BMI is known to complicate the acquisition of radiographic images. Therefore it can be hypothesized that the patient's body habitus can influence the delivery of a spinal injection. Objective To quantify the impact of patient BMI on the length of fluoroscopy and procedure times during spine interventions. Design Secondary analysis of 2 prospective observational studies. Setting All injections were performed in an outpatient university setting. Participants A total of 209 patients in whom spine injections were performed (99 women), with a mean age of 54.6 years. Methods The fluoroscopy times for 202 participants and total procedure times for 137 participants were recorded. Additional participant characteristics, including age, gender, BMI, and actual procedures performed, also were collected. Analysis of covariance and linear and nonlinear model analysis were performed to assess the effect of BMI on fluoroscopy and procedure times. Main Outcome Measurements Fluoroscopy time and procedure duration times. Results Participants had a mean age of 54.6 years, 51% were men, and 77% (n = 155) were overweight (BMI ≥25). Participants received the following interventions: 40 zygapophyseal joint injections, 33 medial branch nerve blocks, 113 transforaminal epidural injections, and 16 combined zygapophyseal joint injections and epidural injections. Gender, procedure number, and procedure type did not differ between groups. The overweight group demonstrated a 30% increase in mean fluoroscopy time and a 35% increase in mean procedure time. Controlling for other variables, we found that differences in fluoroscopy time and procedure time were significant ( P = .032 and P = .031, respectively) between the 2 groups. Conclusions Significantly prolonged procedure time and fluoroscopy time in overweight patients increase the risks associated with spine interventions, not only to the patients but also to the operating room staff exposed to ionizing radiation.

Duration of Fluoroscopic-Guided Spine Interventions and Radiation Exposure Is Increased in Overweight Patients

An 80-year-old female with a history of osteoporosis was evaluated for sudden onset axial low back pain with bilateral lower extremity weakness, hyperreflexia, pain, urinary retention, and decreased rectal tone. Computed tomography of the lumbar spine revealed L1 compression fracture, retropulsion of bone causing spinal canal compromise with associated severe central canal stenosis. Following cement kyphoplasty of L1 with polymethyl methacrylate, the patient developed tachycardia and dyspnea. Chest radiograph and computed tomographic pulmonary angiogram revealed a large collection of hyperdense material within the right lower lobe pulmonary artery, consistent with pulmonary cement emboli. Management and imaging are discussed.

Pulmonary cement embolism after kyphoplasty.

Objective To compare the sensitivity of physical examination (internal rotation of the hip) with radiographs (using the Kellgren-Lawrence grading scale) in the diagnosis of clinically significant hip osteoarthritis. Design Case Series, Retrospective chart review of hip pain patients that underwent fluoroscopically guided hip steroid and anesthetic injections. Participants 10 patients with hip pain patients seen at an academic outpatient center over a 2 year period were analyzed. Interventions Fluoroscopically guided hip steroid and anesthetic injection. Main outcome measure Pain relief and change in VAS pain score after intra-articular hip steroid and lidocaine injection was the main outcome measure. Results Based on Fisher's exact test, there was no association between severity of radiographic hip arthritis and pain relief with intra-articular anesthetic/steroid injection (p=0.45). Physical examination (provocative hip internal rotation) however was associated with a significant decrease in VAS pain score after intra-articular lidocaine and corticosteroid hip injection (p=0.022). Conclusion Simple hip radiographs alone are not sufficient to diagnose clinically significant hip osteoarthritis. Physical examination (hip internal rotation) was found to be more accurate than simple radiographs in the diagnosis of clinically significant hip osteoarthritis. Radiographs seem to best utilized when they are an extension of the physical examination and patient history.

Timothy Chong

Papers

Shinjuku: preemptive scheduling for µsecond-scale tail latency

Duration of Fluoroscopic-Guided Spine Interventions and Radiation Exposure Is Increased in Overweight Patients

Pulmonary cement embolism after kyphoplasty.

The value of physical examination in the diagnosis of hip osteoarthritis