A service provider may apply customer-selected or customer-defined auto-scaling policies to a cluster of resources (e.g., virtualized computing resource instances or storage resource instances in a MapReduce cluster). Different policies may be applied to different subsets of cluster resources (e.g., different instance groups containing nodes of different types or having different roles). Each policy may define an expression to be evaluated during execution of a distributed application, a scaling action to take if the expression evaluates true, and an amount by which capacity should be increased or decreased. The expression may be dependent on metrics emitted by the application, cluster, or resource instances by default, metrics defined by the client and emitted by the application, or metrics created through aggregation. Metric collection, aggregation and rules evaluation may be performed by a separate service or by cluster components. An API may support auto-scaling policy definition.

Automatic scaling of resource instance groups within compute clusters

According to certain aspects, a system includes a client device that includes a virtual machine (VM) executed by a hypervisor, a driver located within the hypervisor, and a data agent. The VM may include a virtual hard disk file and a change block bitmap file. The driver may intercept a first write operation generated by the VM to store data in a first sector, determine an identity of the first sector based on the intercepted write operation, determine an entry in the change block bitmap file that corresponds with the first sector, and modify the entry in the change block bitmap file to indicate that data in the first sector has changed. The data agent may generate an incremental backup of the VM based on the change block bitmap file in response to an instruction from a storage manager, where the incremental backup includes the data in the first sector.

Virtual machine change block tracking

Systems and methods enable a virtual machine, including any applications executing thereon, to quickly start executing and servicing users based on pre-staged data blocks supplied from a backup copy in secondary storage. An enhanced media agent may pre-stage certain backed up data blocks which may be needed to launch the virtual machine, based on predictive analysis pertaining to the virtual machine's operational profile. The enhanced media agent may also pre-stage backed up data blocks for a virtual-machine-file-relocation operation, based on the operation's relocation scheme. Servicing read requests to the virtual machine may take priority over ongoing pre-staging of backed up data. Read requests may be tracked so that the media agent may properly maintain the contents of an associated read cache. Some embodiments of the illustrative storage management system may lack, or may simply not require, the relocation operation, and may operate in a “live mount” configuration.

Efficient live-mount of a backed up virtual machine in a storage management system

Various embodiments predict performance of a system including a plurality of server tiers. In one embodiment, a first set of performance information is collected for a base allocation of computing resources across multiple server tiers in the plurality of sever tiers for a set of workloads. A set of experimental allocations of the computing resources is generated on a tier-by-tier basis. Each of the set of experimental allocations varies the computing resources allocated by the base allocation for a single server tier of the multiple server tiers. A second set of performance information associated with the single server tier for each of the set of experimental allocations is collected for a plurality of workloads. At least one performance characteristic of at least one candidate allocation of computing resources across the multiple server tiers is predicted for a given workload based on the first and second sets of performance information.

Prediction-based provisioning planning for cloud environments

The data storage system according to certain aspects can manage the archiving of virtual machines to (and restoring of virtual machines from) secondary storage. The system can determine whether to archive virtual machines based on usage data or information. The usage information may include storage usage, CPU usage, memory usage, network usage, events defined by a virtual machine software or application, etc. The system may archive virtual machines that are determined to have a low level of utilization. For example, a virtual machine can be archived when its usage level falls below a threshold level. The system may create a virtual machine placeholder for an archived virtual machine, which may be a “light” or minimal version of the virtual machine that acts as if it is the actual virtual machine. By using a virtual machine placeholder, a virtual machine may appear to be active and selectable by the user.

Archiving virtual machines in a data storage system

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Data and compute nodes are separated into different virtual machines (VM). Compute VMs are used to launch containers from different tenants. Compute VMs are organized in pools of hot spare VMs that are immediately available for launching a container and executing a task, and pools of cold spare VMs. Each compute VM may include a mounted network filesystem provided by a node manager to share intermediate outputs across VMs executing on the same host.

Container virtual machines for hadoop

Embodiments support instant forking of virtual machines (VMs) and state customization A computing device initiates execution of a first group of services (eg, identity-independent) in a first VM A second VM is instantiated from the first VM The second VM shares memory and storage with the first VM The computing device customizes the second VM based on configuration data associated with the second VM A second group of services (eg, identity-dependent) starts executing on the second VM after configuring the identity of the second VM Customizing the second VM includes configuring one or more identities of the second VM In some embodiments, a domain identity is selected from a pool of previously-created identities and applied to the second VM, before bootup completes on the second VM

State customization of forked virtual machines

Application resource scheduler module is provided to achieve cooperative application workload scheduling for a consolidated virtual environment. The application resource scheduler aids an application workload scheduler that is part of a distributed computing application, such as Hadoop, to achieve a specified relative priority of the application workload virtual machines to other virtual machines in the virtual environment. The application resource scheduler assists in achieving cooperative workload scheduling by revising the amount of resources that the application workload scheduler sees as available and by setting resource controls for the virtual machines of the distributed computing application to influence the resources the virtual machines receive from the underlying consolidated virtual environment.

Cooperative Application Workload Scheduling for a Consolidated Virtual Environment

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Multiple instances of a distributed computing framework, such as Hadoop, may be executed concurrently. A centralized manager detects when contention for computing resources, such as memory and CPU, causes tasks to run slower on VMs executing on a given host, and scales up or scales down a cluster based on the detected resource contention.

Virtual hadoop manager

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Deployments of a distributed computing application, such as Hadoop, may be executed concurrently with a distributed database application, such as HBase, using a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Computing resources allocated to region server nodes executing as VMs may be isolated from compute VMs of the distributed computing application, as well as from data nodes executing as VMs of the distributed filesystem.

Jayanth Gummaraju

Papers

Container virtual machines for hadoop

State customization of forked virtual machines

Cooperative Application Workload Scheduling for a Consolidated Virtual Environment

Virtual hadoop manager

Multi-tenant distributed computing and database