Described in detail herein is a method of copying data of one or more virtual machines being hosted by one or more non-virtual machines. The method includes receiving an indication that specifies how to perform a copy of data of one or more virtual machines hosted by one or more virtual machine hosts. The method may include determining whether the one or more virtual machines are managed by a virtual machine manager that manages or facilitates management of the virtual machines. If so, the virtual machine manager is dynamically queried to automatically determine the virtual machines that it manages or that it facilitates management of. If not, a virtual machine host is dynamically queried to automatically determine the virtual machines that it hosts. The data of each virtual machine is then copied according to the specifications of the received indication.

Systems and methods for management of virtualization data

Virtualization is used in data center and cloud environments to decouple applications from the hardware they run on. Hardware virtualization and operating system level virtualization are two prominent technologies that enable this. Containers, which use OS virtualization, have recently surged in interest and deployment. In this paper, we study the differences between the two virtualization technologies. We compare containers and virtual machines in large data center environments along the dimensions of performance, manageability and software development.We evaluate the performance differences caused by the different virtualization technologies in data center environments where multiple applications are running on the same servers (multi-tenancy). Our results show that co-located applications can cause performance interference, and the degree of interference is higher in the case of containers for certain types of workloads. We also evaluate differences in the management frameworks which control deployment and orchestration of containers and VMs. We show how the different capabilities exposed by the two virtualization technologies can affect the management and development of applications. Lastly, we evaluate novel approaches which combine hardware and OS virtualization.

Containers and Virtual Machines at Scale: A Comparative Study

Gang migration refers to the simultaneous live migration of multiple Virtual Machines (VMs) from one set of physical machines to another in response to events such as load spikes and imminent failures. Gang migration generates a large volume of network traffic and can overload the core network links and switches in a data center. In this paper, we present an approach to reduce the network overhead of gang migration using global deduplication (GMGD). GMGD identifies and eliminates the retransmission of duplicate memory pages among VMs running on multiple physical machines in the cluster. We describe the design, implementation and evaluation of a GMGD prototype using QEMU/KVM VMs. Evaluations on a 30-node Gigabit Ethernet cluster having 10GigE core links shows that GMGD can reduce the network traffic on core links by up to 65% and the total migration time of VMs by up to 42% when compared to the default migration technique in QEMU/KVM. Furthermore, GMGD has a smaller adverse performance impact on network-bound applications.

/pdf/gang-migration-of-virtual-machines-using-cluster-wide-55krh066a5.pdf

Gang migration of virtual machines using cluster-wide deduplication

Software, firmware, and systems are described herein that create and use a non-production copy of a virtual machine for reverting or restoring the virtual machine. The virtual machine is associated with an external storage device via a logical mapping. A snapshot is taken of a virtual disk associated with the virtual machine to create a snapshot copy of the virtual disk. A snapshot is taken of at least a portion of the mapped external storage device to create a snapshot copy of the mapped external storage device. The snapshot copy of the virtual disk is associated with the snapshot copy of the mapped external storage device. The snapshot copies can then be used to either revert or restore the virtual machine.

Information management of virtual machines having mapped storage devices

The advent of new memory technologies that are denser and cheaper than commodity DRAM has renewed interest in two-tiered main memory schemes. Infrequently accessed application data can be stored in such memories to achieve significant memory cost savings. Past research on two-tiered main memory has assumed a 4KB page size. However, 2MB huge pages are performance critical in cloud applications with large memory footprints, especially in virtualized cloud environments, where nested paging drastically increases the cost of 4KB page management. We present Thermostat, an application-transparent huge-page-aware mechanism to place pages in a dual-technology hybrid memory system while achieving both the cost advantages of two-tiered memory and performance advantages of transparent huge pages. We present an online page classification mechanism that accurately classifies both 4KB and 2MB pages as hot or cold while incurring no observable performance overhead across several representative cloud applications. We implement Thermostat in Linux kernel version 4.5 and evaluate its effectiveness on representative cloud computing workloads running under KVM virtualization. We emulate slow memory with performance characteristics approximating near-future high-density memory technology and show that Thermostat migrates up to 50% of application footprint to slow memory while limiting performance degradation to 3%, thereby reducing memory cost up to 30%.

Thermostat: Application-transparent Page Management for Two-tiered Main Memory

In order to make save and restore features practical, saved virtual machines (VMs) must be able to quickly restore to normal operation. Unfortunately, fetching a saved memory image from persistent storage can be slow, especially as VMs grow in memory size. One possible solution for reducing this time is to lazily restore memory after the VM starts. However, accesses to unrestored memory after the VM starts can degrade performance, sometimes rendering the VM unusable for even longer. Existing performance metrics do not account for performance degradation after the VM starts, making it difficult to compare lazily restoring memory against other approaches. In this paper, we propose both a better metric for evaluating the performance of different restore techniques and a better scheme for restoring saved VMs.Existing performance metrics do not reflect what is really important to the user -- the time until the VM returns to normal operation. We introduce the time-to-responsiveness metric, which better characterizes user experience while restoring a saved VM by measuring the time until there is no longer a noticeable performance impact on the restoring VM. We propose a new lazy restore technique, called working set restore, that minimizes performance degradation after the VM starts by prefetching the working set. We also introduce a novel working set estimator based on memory tracing that we use to test working set restore, along with an estimator that uses access-bit scanning. We show that working set restore can improve the performance of restoring a saved VM by more than 89% for some workloads.

/pdf/fast-restore-of-checkpointed-memory-using-working-set-sbxql990nk.pdf

Fast restore of checkpointed memory using working set estimation

Page data of a virtual machine is represented for efficient save and restore operations. One form of representation applies to each page with an easily identifiable pattern. The page is described, saved, and restored in terms of metadata reflective of the pattern rather than a complete page of data reflecting the pattern. During a save or restore operation, however, the metadata of the page is represented, but not the page data. Another form of representation applies to each page sharing a canonical instance of a complex pattern that is instantiated in memory during execution, and explicitly saved and restored. Each page sharing the canonical page is saved and restored as a metadata reference, without the need to actually save redundant copies of the page data.

Method for improving save and restore performance in virtual machine systems

VMware ESXi leverages hardware support for MMU virtualization available in modern Intel/AMD CPUs. To optimize address translation performance when running on such CPUs, ESXi preferably uses host large pages (2MB in x86-64 systems) to back VM's guest memory. While using host large pages provides best performance when host has sufficient free memory, it increases host memory pressure and effectively defeats page sharing. Hence, the host is more likely to hit the point where ESXi has to reclaim VM memory through much more expensive techniques such as ballooning or host swapping. As a result, using host large pages may significantly hurt consolidation ratio. To deal with this problem, we propose a new host large page management policy that allows to: a) identify 'cold' large pages and break them even when host has plenty of free memory; b) break all large pages proactively when host free memory becomes scarce, but before the host starts ballooning or swapping; c) reclaim the small pages within the broken large pages through page sharing. With the new policy, the shareable small pages can be shared much earlier and the amount of memory that needs to be ballooned or swapped can be largely reduced when host memory pressure is high. We also propose an algorithm to dynamically adjust the page sharing rate when proactively breaking large pages using a VM large page shareability estimator for higher efficiency. Experimental results show that the proposed large page management policy can improve the performance of various workloads up to 2.1x by significantly reducing the amount of ballooned or swapped memory when host memory pressure is high. Applications still fully benefit from host large pages when memory pressure is low.

Proactively Breaking Large Pages to Improve Memory Overcommitment Performance in VMware ESXi

Cloud providers are increasingly looking to use virtual machine checkpointing for new applications beyond fault tolerance. Existing checkpointing systems designed for fault tolerance only optimize for saving checkpointed state, so they cannot support these new applications, which require better restore performance. Improving restore performance requires a predictive technique to reduce the number of disk accesses to bring in the VM's memory on restore. However, complex VM workloads can diverge at any time due to external inputs, background processes, and timing variation, so predicting which pages the VM will access on restore to reduce faults to disk is impossible. Instead, we focus on predicting which pages the VM will access together on restore to improve the efficiency of disk accesses.

To reduce the number of faults to disk on restore, we group memory pages likely to be accessed together into locality blocks. On each fault, we can load a block of pages that are likely to be accessed with the faulting page, eliminating future faults and increasing disk efficiency. We implement support for locality blocks, along with several other optimizations, in a new checkpointing system for VMware ESXi Server called Halite. Our experiments show that Halite reduces restore overhead by up to 94% for a range of workloads.

/pdf/optimizing-vm-checkpointing-for-restore-performance-in-zloafp7uus.pdf

Optimizing VM checkpointing for restore performance in VMware ESXi

Methods, systems, and computer programs for managing physical memory in a host of a virtual infrastructure are presented. One method includes an operation for detecting that a guest physical page (GPP) of a virtual machine (VM) is a balloon page. The GPP, previously mapped to a machine page (MP), is re-mapped to a shared page of memory in order to free the MP, such that a read to the GPP causes a read to the shared page of memory. Further, the method includes an operation for detecting a write to the shared page of memory after the re-mapping of the GPP to the shared page, where the write to the shared page is caused by a write to the GPP. After detecting the write, all balloon pages created by the VM are reset in order to reset the balloon application in the VM.

Yury Baskakov

Papers

Fast restore of checkpointed memory using working set estimation

Method for improving save and restore performance in virtual machine systems

Proactively Breaking Large Pages to Improve Memory Overcommitment Performance in VMware ESXi

Optimizing VM checkpointing for restore performance in VMware ESXi

Efficient readable ballooning of guest memory by backing balloon pages with a shared page