/pdf/a-break-in-the-clouds-towards-a-cloud-definition-svovsw1r5a.pdf

A Break in the Clouds: Towards a Cloud Definition

The Grid 2: Blueprint for a New Computing Infrastructure

Google's Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior. We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it.

/pdf/large-scale-cluster-management-at-google-with-borg-4i7d9qnw5x.pdf

Large-scale cluster management at Google with Borg

To better understand the challenges in developing effective cloud-based resource schedulers, we analyze the first publicly available trace data from a sizable multi-purpose cluster. The most notable workload characteristic is heterogeneity: in resource types (e.g., cores:RAM per machine) and their usage (e.g., duration and resources needed). Such heterogeneity reduces the effectiveness of traditional slot- and core-based scheduling. Furthermore, some tasks are constrained as to the kind of machine types they can use, increasing the complexity of resource assignment and complicating task migration. The workload is also highly dynamic, varying over time and most workload features, and is driven by many short jobs that demand quick scheduling decisions. While few simplifying assumptions apply, we find that many longer-running jobs have relatively stable resource utilizations, which can help adaptive resource schedulers.

http://www.istc-cc.cmu.edu/publications/papers/2012/googletrace-socc2012.pdf

Heterogeneity and dynamicity of clouds at scale: Google trace analysis

Network-based cloud computing is rapidly expanding as an alternative to conventional office-based computing. As cloud computing becomes more widespread, the energy consumption of the network and computing resources that underpin the cloud will grow. This is happening at a time when there is increasing attention being paid to the need to manage energy consumption across the entire information and communications technology (ICT) sector. While data center energy use has received much attention recently, there has been less attention paid to the energy consumption of the transmission and switching networks that are key to connecting users to the cloud. In this paper, we present an analysis of energy consumption in cloud computing. The analysis considers both public and private clouds, and includes energy consumption in switching and transmission as well as data processing and data storage. We show that energy consumption in transport and switching can be a significant percentage of total energy consumption in cloud computing. Cloud computing can enable more energy-efficient use of computing power, especially when the computing tasks are of low intensity or infrequent. However, under some circum- stances cloud computing can consume more energy than conventional computing where each user performs all com- puting on their own personal computer (PC).

/pdf/green-cloud-computing-balancing-energy-in-processing-storage-2z2x5rjq9b.pdf

Green Cloud Computing: Balancing Energy in Processing, Storage, and Transport For processing large amounts of data, management and switching of communications may contribute significantly to energy consumption and cloud computing seems to be an alternative to office-based computing.

Many large-scale distributed computing applications demand real-time responses by soft deadlines. To enable such real-time task distribution and execution on the volunteer resources, we previously proposed the design of the real-time volunteer computing platform called RT-BOINC. The system gives low O(1) worst-case execution time for task management operations, such as task scheduling, state transitioning, and validation. In this work, we present a full implementation RT-BOINC, adding new features including deadline timer and parameter-based admission control. We evaluate RT-BOINC at large scale using two real-time applications, namely, the games Go and Chess. The results of our case study show that RT-BOINC provides much better performance than the original BOINC in terms of average and worst-case response time, scalability and efficiency.

/pdf/towards-real-time-volunteer-distributed-computing-9qlguiotjj.pdf

Towards Real-Time, Volunteer Distributed Computing

Understanding and modelling resources of Internet end hosts is essential for the design of desktop software and Internet-distributed applications. In this paper we develop a correlated resource model of Internet end hosts based on real trace data taken from the SETI@home project. This data covers a 5-year period with statistics for 2.7 million hosts. The resource model is based on statistical analysis of host computational power, memory, and storage as well as how these resources change over time and the correlations between them. We find that resources with few discrete values (core count, memory) are well modeled by exponential laws governing the change of relative resource quantities over time. Resources with a continuous range of values are well modeled with either correlated normal distributions (processor speed for integer operations and floating point operations) or log-normal distributions (available disk space). We validate and show the utility of the models by applying them to a resource allocation problem for Internet-distributed applications, and demonstrate their value over other models. We also make our trace data and tool for automatically generating realistic Internet end hosts publicly available.

/pdf/correlated-resource-models-of-internet-end-hosts-1f2u7fr95p.pdf

Correlated Resource Models of Internet End Hosts

Understanding and modeling resources of Internet end hosts are essential for the design of desktop software and Internet-distributed applications. In this paper, we develop a correlated resource model of Internet end hosts based on real-trace data taken from several volunteer computing projects, including SETI@home. This data cover a five-year period with statistics for 6.7 million hosts. Our resource model is based on statistical analysis of host computational power, memory, and storage as well as how these resources change over time and the correlations among them. We find that resources with few discrete values (core count, memory) are well modeled by approximations governing the change of relative resource quantities over time. Resources with a continuous range of values are well modeled by correlated log-normal distributions (cache, processor speed, and available disk space). We validate and show the utility of the model by applying it to a resource allocation problem for Internet-distributed applications, and compare it to other models. We also make our trace data and tool for automatically generating realistic Internet end hosts publicly available.

A Correlated Resource Model of Internet End Hosts

The many interactions of tRNA with the ribosome are fundamental to protein synthesis. During the peptidyl transferase reaction, the acceptor ends of the aminoacyl and peptidyl tRNAs must be in close proximity to allow peptide bond formation, and their respective anticodons must base pair simultaneously with adjacent trinucleotide codons on the mRNA. The two tRNAs in this state can be arranged in two nonequivalent general configurations called the R and S orientations, many versions of which have been proposed for the geometry of tRNAs in the ribosome. Here, we report the combined use of computational analysis and tethered hydroxyl-radical probing to constrain their arrangement. We used Fe(II) tethered to the 5' end of anticodon stem-loop analogs (ASLs) of tRNA and to the 5' end of deacylated tRNA(Phe) to generate hydroxyl radicals that probe proximal positions in the backbone of adjacent tRNAs in the 70S ribosome. We inferred probe-target distances from the resulting RNA strand cleavage intensities and used these to calculate the mutual arrangement of A-site and P-site tRNAs in the ribosome, using three different structure estimation algorithms. The two tRNAs are constrained to the S configuration with an angle of about 45 degrees between the respective planes of the molecules. The terminal phosphates of 3'CCA are separated by 23 A when using the tRNA crystal conformations, and the anticodon arms of the two tRNAs are sufficiently close to interact with adjacent codons in mRNA.

/pdf/calculation-of-the-relative-geometry-of-trnas-in-the-4o7p9cwg54.pdf

Calculation of the relative geometry of tRNAs in the ribosome from directed hydroxyl-radical probing data.

Desktop grids, which use the idle cycles of many desktop PC?s, are currently one of the largest distributed systems in the world. Despite the popularity and success of many desktop grid projects, the volatility of hosts within desktop grids has been poorly understood. Yet, such host characterization is essential for accurate simulation and modelling of such platforms. In this paper, we present application-level traces of four enterprise desktop grids with a wide range of user bases. We then describe aggregate and per host statistics that reflect the volatility of desktop grid resources. Further, we determine the correlation of volatility between resources, and investigate the correlation of volatility and other host characteristics. Finally, we detail a number of implications of these findings with respect to application performance.

Derrick Kondo

Papers

Towards Real-Time, Volunteer Distributed Computing

Correlated Resource Models of Internet End Hosts

A Correlated Resource Model of Internet End Hosts

Calculation of the relative geometry of tRNAs in the ribosome from directed hydroxyl-radical probing data.

On Resource Volatility in Enterprise Desktop Grids