A super-peer is a node in a peer-to-peer network that operates both as a server to a set of clients, and as an equal in a network of super-peers. Super-peer networks strike a balance between the efficiency of centralized search, and the autonomy, load balancing and robustness to attacks provided by distributed search. Furthermore, they take advantage of the heterogeneity of capabilities (e.g., bandwidth, processing power) across peers, which recent studies have shown to be enormous. Hence, new and old P2P systems like KaZaA and Gnutella are adopting super-peers in their design. Despite their growing popularity, the behavior of super-peer networks is not well understood. For example, what are the potential drawbacks of super-peer networks? How can super-peers be made more reliable? How many clients should a super-peer take on to maximize efficiency? we examine super-peer networks in detail, gaming an understanding of their fundamental characteristics and performance tradeoffs. We also present practical guidelines and a general procedure for the design of an efficient super-peer network.

Designing an Super-Peer Network

As preservers of history and the objects that others leave behind, it is easy to get distracted by a desire for those objects to be “perfect.” We worry about every little tear, stain or blemish, and we want nothing more than to return objects to their "original state." However, so many times those rough spots are part of the item’s history. It is important to understand the difference between the types of condition issues on objects that cause structural issues versus those that are purely cosmetic. Knowing and identifying these differences can help when determining whether or not an item requires conservation or deciding simple factors like the best way to store or exhibit an item.

Facebook“卖流量”新招

The emerging Internet-of-Things (IoT) are vulnerable to Sybil attacks where attackers can manipulate fake identities or abuse pseudoidentities to compromise the effectiveness of the IoT and even disseminate spam. In this paper, we survey Sybil attacks and defense schemes in IoT. Specifically, we first define three types Sybil attacks: SA-1, SA-2, and SA-3 according to the Sybil attacker’s capabilities. We then present some Sybil defense schemes, including social graph-based Sybil detection (SGSD), behavior classification-based Sybil detection (BCSD), and mobile Sybil detection with the comprehensive comparisons. Finally, we discuss the challenging research issues and future directions for Sybil defense in IoT.

Sybil Attacks and Their Defenses in the Internet of Things

Facebook recently deployed Facebook Messages, its first ever user-facing application built on the Apache Hadoop platform. Apache HBase is a database-like layer built on Hadoop designed to support billions of messages per day. This paper describes the reasons why Facebook chose Hadoop and HBase over other systems such as Apache Cassandra and Voldemort and discusses the applicationBs requirements for consistency, availability, partition tolerance, data model and scalability. I explore the enhancements made to Hadoop to make it a more effective realtime system, the tradeoffs we made while configuring the system, and how this solution has significant advantages over the sharded MySQL database scheme used in other applications at Facebook and many other web-scalecompanies. I discuss the motivations behind my design choices, the challenges that we face in day-to-day operations, and future capabilities and improvements still under development.I offer these observations on the deployment as a model for other companies who are contemplating a Hadoop-based solution over traditional sharded RDBMS deployments.

/pdf/apache-hadoop-goes-realtime-at-facebook-25mf7o9ess.pdf

Apache Hadoop Goes Realtime at Facebook

The size and energy consumption of datacenters have been increasing significantly over the past years. As a result, datacenters’ increasing electricity monetary cost, energy consumption and energy harmful gas emissions have become a severe problem. Renewable energy supply is widely seen as a promising solution. However, the instability of renewable energy brings about a new challenge since insufficient energy supply may lead to job running interruptions or failures. Though previous works attempt to more accurately predict the amount of produced renewable energy, due to the instability of its influencing factors (e.g., wind, temperature), sufficient renewable energy supply cannot be always guaranteed. To handle this problem, in this paper, we propose allocating jobs with the same service-level-objective (SLO) level to the same physical machine (PM) group, and power each PM group with renewable energy generators that have probability no less than its SLO to produce the amount no less than its energy demand. It ensures that insufficient renewable energy supply will not lead to SLO violations. We use a deep learning technique to predict the probability of producing amount no less than each value of each renewable energy source and predict the energy demands of each PM area. We formulate an optimization problem: how to match renewable energy resources with different instabilities to different PM groups as energy supply in order to minimize the number of SLO violations (due to interruption from insufficient renewable energy supply), total energy monetary cost and total carbon emission. We then use reinforcement learning method and linear programming method to solve the optimization problem. The real trace driven experiments show that our method can achieve much lower SLO violations, total energy monetary cost and total carbon emission compared to other methods.

Smartly Handling Renewable Energy Instability in Supporting A Cloud Datacenter

Advancements in cloud computing are leading to a promising future for collaborative cloud computing (CCC), where globally-scattered distributed cloud resources belonging to different organizations or individuals (i.e., entities) are collectively used in a cooperative manner to provide services. Due to the autonomous features of entities in CCC, the issues of resource management and reputation management must be jointly addressed in order to ensure the successful deployment of CCC. However, these two issues have typically been addressed separately in previous research efforts, and simply combining the two systems generates double overhead. Also, previous resource and reputation management methods are not sufficiently efficient or effective. By providing a single reputation value for each node, the methods cannot reflect the reputation of a node in providing individual types of resources. By always selecting the highest-reputed nodes, the methods fail to exploit node reputation in resource selection to fully and fairly utilize resources in the system and to meet users' diverse QoS demands. We propose a CCC platform, called Harmony, which integrates resource management and reputation management in a harmonious manner. Harmony incorporates three key innovations: integrated multi-faceted resource/reputation management, multi-QoS-oriented resource selection, and price-assisted resource/reputation control. The trace data we collected from an online trading platform implies the importance of multi-faceted reputation and the drawbacks of highest-reputed node selection. Simulations and trace-driven experiments on the real-world PlanetLab testbed show that Harmony outperforms existing resource management and reputation management systems in terms of QoS, efficiency and effectiveness.

An Efficient and Trustworthy Resource Sharing Platform for Collaborative Cloud Computing

Video sharing has been an increasingly popular application in online social networks (OSNs). However, its sustainable development is severely hindered by the intrinsic limit of the client/server architecture deployed in current OSN video systems, which is not only costly in terms of server bandwidth and storage but also not scalable. The peer-assisted Video-on-Demand (VOD) technique, in which participating peers assist the server in delivering video content has been proposed recently. Unfortunately, videos can only be disseminated through friends in OSNs. Therefore, current VOD works that explore clustering nodes with similar interests or close location for high performance are suboptimal, if not entirely inapplicable, in OSNs. Based on our long-term real-world measurement of over 1,000,000 users and 2,500 videos in Facebook, we propose SocialTube, a novel peer-assisted video sharing system that explores social relationship, interest similarity, and physical location between peers in OSNs. Specifically, SocialTube incorporates three algorithms: a social network (SN)-based P2P overlay construction algorithm, a SN-based chunk prefetch algorithm, and a buffer management algorithm. The trace driven based simulation results show that SocialTube can improve the quality of user experience and system scalability over current P2P VOD techniques.

SocialTube: P2P-assisted video sharing in online social networks

Though the new OSN model with many worldwide distributed small datacenters helps reduce service latency, it brings a problem of higher inter-datacenter communication load. In Facebook, each datacenter has a full copy of all data and the master datacenter updates all other datacenters, which obviously generates tremendous load in this new model. Distributed data storage that only stores a user's data to his/her geographically-closest datacenters mitigates the problem. However, frequent interactions between far-away users lead to frequent inter-datacenter communication and hence long service latency. In this paper, we aim to reduce inter-datacenter communications while still achieve low service latency. We first verify the benefits of the new model and present OSN typical properties that lay the basis of our design. We then propose Selective Data replication mechanism in Distributed Datacenters (SD3). In SD3, a datacenter jointly considers update rate and visit rate to select user data for replication, and further atomizes a user's different types of data (e.g., status update, friend post) for replication, making sure that a replica always reduces inter-datacenter communication. The results of trace-driven experiments on the real-world PlanetLab testbed demonstrate the higher efficiency and effectiveness of SD3 in comparison to other replication methods.

Selective Data replication for Online Social Networks with Distributed Datacenters

Though the new OSN model, which deploys datacenters globally, helps reduce service latency, it causes higher inter-datacenter communication load. In Facebook, each datacenter has a full copy of all data, and the master datacenter updates all other datacenters, generating tremendous load in this new model. Distributed data storage, which only stores a user's data to his/her geographically closest datacenters mitigates the problem. However, frequent interactions between distant users lead to frequent inter-datacenter communication and hence long service latencies. In this paper, we aim to reduce inter-datacenter communications while still achieving low service latency. We first verify the benefits of the new model and present OSN typical properties that underlie the basis of our design. We then propose Selective Data replication mechanism in Distributed Datacenters ( $SD^3$ ). Since replicas need inter-datacenter data updates, datacenters in $SD^3$ jointly consider update rates and visit rates to select user data for replication; furthermore, $SD^3$ atomizes users’ different types of data (e.g., status update, friend post, music) for replication, ensuring that a replica always reduces inter-datacenter communication. $SD^3$ also incorporates three strategies to further enhance its performance: locality-aware multicast update tree, replica deactivation, and datacenter congestion control. The results of trace-driven experiments on the real-world PlanetLab testbed demonstrate the higher efficiency and effectiveness of $SD^3$ in comparison to other replication methods and the effectiveness of its three schemes.

Selective Data Replication for Online Social Networks with Distributed Datacenters

Many cloud service providers (CSPs) provide data storage services with datacenters distributed worldwide. These datacenters provide different get/put latencies and unit prices for resource utilization and reservation. Thus, when selecting different CSPs’ datacenters, cloud customers of globally distributed applications (e.g., online social networks) face two challenges: 1) how to allocate data to worldwide datacenters to satisfy application service level objective (SLO) requirements, including both data retrieval latency and availability and2) how to allocate data and reserve resources in datacenters belonging to different CSPs to minimize the payment cost. To handle these challenges, we first model the cost minimization problem under SLO constraints using the integer programming. Due to its NP-hardness, we then introduce our heuristic solution, including a dominant-cost-based data allocation algorithm and an optimal resource reservation algorithm. We further propose three enhancement methods to reduce the payment cost and service latency: 1) coefficient-based data reallocation; 2) multicast-based data transferring; and 3) request redirection-based congestion control. We finally introduce an infrastructure to enable the conduction of the algorithms. Our trace-driven experiments on a supercomputing cluster and on real clouds (i.e., Amazon S3, Windows Azure Storage, and Google Cloud Storage) show the effectiveness of our algorithms for SLO guaranteed services and customer cost minimization.

Guoxin Liu

Papers

An Efficient and Trustworthy Resource Sharing Platform for Collaborative Cloud Computing

SocialTube: P2P-assisted video sharing in online social networks

Selective Data replication for Online Social Networks with Distributed Datacenters

Selective Data Replication for Online Social Networks with Distributed Datacenters

Minimum-Cost Cloud Storage Service Across Multiple Cloud Providers