/pdf/a-break-in-the-clouds-towards-a-cloud-definition-svovsw1r5a.pdf

A Break in the Clouds: Towards a Cloud Definition

Facebook recently deployed Facebook Messages, its first ever user-facing application built on the Apache Hadoop platform. Apache HBase is a database-like layer built on Hadoop designed to support billions of messages per day. This paper describes the reasons why Facebook chose Hadoop and HBase over other systems such as Apache Cassandra and Voldemort and discusses the applicationBs requirements for consistency, availability, partition tolerance, data model and scalability. I explore the enhancements made to Hadoop to make it a more effective realtime system, the tradeoffs we made while configuring the system, and how this solution has significant advantages over the sharded MySQL database scheme used in other applications at Facebook and many other web-scalecompanies. I discuss the motivations behind my design choices, the challenges that we face in day-to-day operations, and future capabilities and improvements still under development.I offer these observations on the deployment as a model for other companies who are contemplating a Hadoop-based solution over traditional sharded RDBMS deployments.

/pdf/apache-hadoop-goes-realtime-at-facebook-25mf7o9ess.pdf

Apache Hadoop Goes Realtime at Facebook

The size and energy consumption of datacenters have been increasing significantly over the past years. As a result, datacenters’ increasing electricity monetary cost, energy consumption and energy harmful gas emissions have become a severe problem. Renewable energy supply is widely seen as a promising solution. However, the instability of renewable energy brings about a new challenge since insufficient energy supply may lead to job running interruptions or failures. Though previous works attempt to more accurately predict the amount of produced renewable energy, due to the instability of its influencing factors (e.g., wind, temperature), sufficient renewable energy supply cannot be always guaranteed. To handle this problem, in this paper, we propose allocating jobs with the same service-level-objective (SLO) level to the same physical machine (PM) group, and power each PM group with renewable energy generators that have probability no less than its SLO to produce the amount no less than its energy demand. It ensures that insufficient renewable energy supply will not lead to SLO violations. We use a deep learning technique to predict the probability of producing amount no less than each value of each renewable energy source and predict the energy demands of each PM area. We formulate an optimization problem: how to match renewable energy resources with different instabilities to different PM groups as energy supply in order to minimize the number of SLO violations (due to interruption from insufficient renewable energy supply), total energy monetary cost and total carbon emission. We then use reinforcement learning method and linear programming method to solve the optimization problem. The real trace driven experiments show that our method can achieve much lower SLO violations, total energy monetary cost and total carbon emission compared to other methods.

Smartly Handling Renewable Energy Instability in Supporting A Cloud Datacenter

As a widely used IT service, more and more companies shift their services to cloud datacenters. It is important for cloud service providers (CSPs) to provide cloud service resources with high elasticity and cost-effectiveness and then achieve good quality of service (QoS) for their clients. However, meeting QoS with cost-effective resource is a challenging problem for CSPs because the workloads of Virtual Machines (VMs) experience variation over time. It is highly necessary to provide an accurate VMs workload prediction method for resource provisioning to efficiently manage cloud resources. In this paper, we first compare the performance of representative state-of-the-art workload prediction methods. We suggest a method to conduct the prediction a certain time before the predicted time point in order to allow sufficient time for task scheduling based on predicted workload. To further improve the prediction accuracy, we introduce a clustering based workload prediction method, which first clusters all the tasks into several categories and then trains a prediction model for each category respectively. The trace-driven experiments based on Google cluster trace demonstrates that our clustering based workload prediction methods outperform other comparison methods and improve the prediction accuracy to around 90% both in CPU and memory.

Machine Learning Based Workload Prediction in Cloud Computing

A large-scale cloud data center needs to provide high service reliability and availability with low failure occurrence probability. However, current large-scale cloud data centers still face high failure rates due to many reasons such as hardware and software failures, which often result in task and job failures. Such failures can severely reduce the reliability of cloud services and also occupy huge amount of resources to recover the service from failures. Therefore, it is important to predict task or job failures before occurrence with high accuracy to avoid unexpected wastage. Many machine learning and deep learning based methods have been proposed for the task or job failure prediction by analyzing past system message logs and identifying the relationship between the data and the failures. In order to further improve the failure prediction accuracy of the previous machine learning and deep learning based methods, in this paper, we propose a failure prediction algorithm based on multi-layer Bidirectional Long Short Term Memory (Bi-LSTM) to identify task and job failures in the cloud. The goal of Bi-LSTM prediction algorithm is to predict whether the tasks and jobs are failed or completed. The trace-driven experiments show that our algorithm outperforms other state-of-art prediction methods with 93% accuracy and 87% for task failure and job failures respectively.

Task Failure Prediction in Cloud Data Centers Using Deep Learning

A large-scale cloud data center needs to provide high service reliability and availability with low failure occurrence probability. However, current large-scale cloud data centers still face high failure rates due to many reasons such as hardware and software failures, which often result in task and job failures. Such failures can severely reduce the reliability of cloud services and also occupy huge amount of resources to recover the service from failures. Therefore, it is important to predict task or job failures before occurrence with high accuracy to avoid unexpected wastage. Many machine learning and deep learning based methods have been proposed for the task or job failure prediction by analyzing past system message logs and identifying the relationship between the data and the failures. In order to further improve the failure prediction accuracy of the previous machine learning and deep learning based methods, in this paper, we propose a failure prediction algorithm based on multi-layer Bidirectional Long Short Term Memory (Bi-LSTM) to identify task and job failures in the cloud. The trace-driven experiments show that our algorithm outperforms other state-of-art prediction methods with 93% accuracy and 87% for task failure and job failures respectively.

With the rapid development of web applications in datacenters, network latency becomes more important to user experience. The network latency will be greatly increased by incast congestion, in which a huge number of requests arrive at the front-end server simultaneously. Previous incast problem solutions usually handle the data transmission between the data servers and the front-end server directly, and they are not sufficiently effective in proactively avoiding incast congestion. To further improve the effectiveness, in this paper, we propose a Proactive Incast Congestion Control system (PICC). Since each connection has bandwidth limit, PICC novelly limits the number of data servers concurrently connected to the front-end server to avoid the incast congestion through data placement. Specifically, the front-end server gathers popular data objects (i.e., frequently requested data objects) into as few data servers as possible, but without overloading them. It also re-allocates the data objects that are likely to be concurrently or sequentially requested into the same server. As a result, PICC reduces the number of data servers concurrently connected to the front-end server (which avoids the incast congestion), and also the number of connection establishments (which reduces the network latency). Since the selected data servers tend to have long queues to send out data, to reduce the queuing latency, PICC incorporates a queuing delay reduction algorithm that assigns higher transmission priorities to data objects with smaller sizes and longer queuing times. The experimental results on simulation and a real cluster based on a benchmark show the superior performance of PICC over previous incast congestion problem solutions.

Haoyu Wang

Papers

Smartly Handling Renewable Energy Instability in Supporting A Cloud Datacenter

Machine Learning Based Workload Prediction in Cloud Computing

Task Failure Prediction in Cloud Data Centers Using Deep Learning

Task Failure Prediction in Cloud Data Centers Using Deep Learning

Proactive Incast Congestion Control in a Datacenter Serving Web Applications