A management server includes a configuration and management module processing server configuration information, including a VPN peer list and VLAN/subnet settings. The management server automatically calculates the VPN configuration information, including the VPN peer subnet route information identifying which of the subnets participating in the VPN are behind which of the routers and keys to establish VPN tunnels between those routers participating in the VPN. Each of the routers participating in the VPN includes a VPN tunnel with the other routers participating in the VPN, a set of data structures storing data identifying contact information for each of the subnets participating in the VPN, a combination of an IP address and port to reach one of routers that that subnet is behind, and a forwarding module to forward traffic between the subnets.

System and method for managing site-to-site vpns of a cloud managed network

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Data and compute nodes are separated into different virtual machines (VM). Compute VMs are used to launch containers from different tenants. Compute VMs are organized in pools of hot spare VMs that are immediately available for launching a container and executing a task, and pools of cold spare VMs. Each compute VM may include a mounted network filesystem provided by a node manager to share intermediate outputs across VMs executing on the same host.

Container virtual machines for hadoop

A service provider may apply customer-selected or customer-defined auto-scaling policies to a cluster of resources (e.g., virtualized computing resource instances or storage resource instances in a MapReduce cluster). Different policies may be applied to different subsets of cluster resources (e.g., different instance groups containing nodes of different types or having different roles). Each policy may define an expression to be evaluated during execution of a distributed application, a scaling action to take if the expression evaluates true, and an amount by which capacity should be increased or decreased. The expression may be dependent on metrics emitted by the application, cluster, or resource instances by default, metrics defined by the client and emitted by the application, or metrics created through aggregation. Metric collection, aggregation and rules evaluation may be performed by a separate service or by cluster components. An API may support auto-scaling policy definition.

Automatic scaling of resource instance groups within compute clusters

According to certain aspects, a system includes a client device that includes a virtual machine (VM) executed by a hypervisor, a driver located within the hypervisor, and a data agent. The VM may include a virtual hard disk file and a change block bitmap file. The driver may intercept a first write operation generated by the VM to store data in a first sector, determine an identity of the first sector based on the intercepted write operation, determine an entry in the change block bitmap file that corresponds with the first sector, and modify the entry in the change block bitmap file to indicate that data in the first sector has changed. The data agent may generate an incremental backup of the VM based on the change block bitmap file in response to an instruction from a storage manager, where the incremental backup includes the data in the first sector.

Virtual machine change block tracking

A cloud computing system including a plurality of computing devices configured to host virtual machine instances, each computing device in the plurality of computing devices including a monitor operable to collect metadata about the associated computing device. The system also includes a registry operable to receive and store the metadata from the plurality of computing devices and a scheduler operable to select a host computing device out of the plurality of computing devices on which to instantiate a virtual machine instance based on the metadata stored in the registry.

System and Method for Metadata Discovery and Metadata-Aware Scheduling

Embodiments perform automated provisioning of a cluster for a distributed computing platform. Target host computing devices are selected from a plurality of host computing devices based on configuration information, such as a desired cluster size, a data set, code for processing the data set and, optionally, a placement strategy. One or more virtual machines (VMs) are instantiated on each target host computing device. Each VM is configured to access a virtual disk that is preconfigured with code for executing functionality of the distributed computing platform and serves as a node of the cluster. The data set is stored in a distributed file system accessible by at least a subset of the VMs. The code for processing the data set is provided to at least a subset of the VMs, and execution of the code is initiated to obtain processing results.

Provisioning of distributed computing clusters

A virtualized computing system for executing a distributed computing application, such as Hadoop, is discussed. The virtualized computing system stores data in a distributed filesystem, such as Hadoop Distributed File System, and processes data using a topology awareness that takes into account the virtualization layer of the virtualized computing system. The virtualized computing system employs locality-related policies, including replica placement policies, replica choosing policies, balancer policies, and task scheduling policies that take advantage of the awareness of the virtualization topology.

Virtualization-aware data locality in distributed data processing

A rapid virtual machine (VM) cloning technique is provided that creates cloned VMs on hosts from multiple source VMs, rather than a single source VM that may otherwise be a bottleneck. The described technique selects particular hosts, disposed in particular racks, on which to create VM clones in a dynamic manner that reduces total deployment time for the plurality of VMs. A rapid VM reconfiguration technique is also provided that reduces the time spent reconfiguring the provisioned VMs for use in a distributed computing application.

Junping Du

Papers

Container virtual machines for hadoop

Provisioning of distributed computing clusters

Virtualization-aware data locality in distributed data processing

Rapid creation and reconfiguration of virtual machines on hosts

Provisioning a cluster of distributed computing platform based on placement strategy