There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

We will review some of the major results in random graphs and some of the more challenging open problems. We will cover algorithmic and structural questions. We will touch on newer models, including those related to the WWW.

Random graphs

"Grid" computing has emerged as an important new field, distinguished from conventional distributed computing by its focus on large-scale resource sharing, innovative applications, and, in some cases, high performance orientation. In this article, the authors define this new field. First, they review the "Grid problem," which is defined as flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources--what is referred to as virtual organizations. In such settings, unique authentication, authorization, resource access, resource discovery, and other challenges are encountered. It is this class of problem that is addressed by Grid technologies. Next, the authors present an extensible and open Grid architecture, in which protocols, services, application programming interfaces, and software development kits are categorized according to their roles in enabling resource sharing. The authors describe requirements that they believe any such mechanisms must satisfy and discuss the importance of defining a compact set of intergrid protocols to enable interoperability among different Grid systems. Finally, the authors discuss how Grid technologies relate to other contemporary technologies, including enterprise integration, application service provider, storage service provider, and peer-to-peer computing. They maintain that Grid concepts and technologies complement and have much to contribute to these other approaches.

The Anatomy of the Grid: Enabling Scalable Virtual Organizations

Cloud Computing, the long-held dream of computing as a utility, has the potential to transform a large part of the IT industry, making software even more attractive as a service and shaping the way IT hardware is designed and purchased. Developers with innovative ideas for new Internet services no longer require the large capital outlays in hardware to deploy their service or the human expense to operate it. They need not be concerned about overprovisioning for a service whose popularity does not meet their predictions, thus wasting costly resources, or underprovisioning for one that becomes wildly popular, thus missing potential customers and revenue. Moreover, companies with large batch-oriented tasks can get results as quickly as their programs can scale, since using 1000 servers for one hour costs no more than using one server for 1000 hours. This elasticity of resources, without paying a premium for large scale, is unprecedented in the history of IT. Cloud Computing refers to both the applications delivered as services over the Internet and the hardware and systems software in the datacenters that provide those services. The services themselves have long been referred to as Software as a Service (SaaS). The datacenter hardware and software is what we will call a Cloud. When a Cloud is made available in a pay-as-you-go manner to the general public, we call it a Public Cloud; the service being sold is Utility Computing. We use the term Private Cloud to refer to internal datacenters of a business or other organization, not made available to the general public. Thus, Cloud Computing is the sum of SaaS and Utility Computing, but does not include Private Clouds. People can be users or providers of SaaS, or users or providers of Utility Computing. We focus on SaaS Providers (Cloud Users) and Cloud Providers, which have received less attention than SaaS Users. From a hardware point of view, three aspects are new in Cloud Computing.

/pdf/above-the-clouds-a-berkeley-view-of-cloud-computing-tkj3etim13.pdf

Above the Clouds: A Berkeley View of Cloud Computing

A 2001 IBM manifesto observed that a looming software complexity crisis -caused by applications and environments that number into the tens of millions of lines of code - threatened to halt progress in computing. The manifesto noted the almost impossible difficulty of managing current and planned computing systems, which require integrating several heterogeneous environments into corporate-wide computing systems that extend into the Internet. Autonomic computing, perhaps the most attractive approach to solving this problem, creates systems that can manage themselves when given high-level objectives from administrators. Systems manage themselves according to an administrator's goals. New components integrate as effortlessly as a new cell establishes itself in the human body. These ideas are not science fiction, but elements of the grand challenge to create self-managing computing systems.

The vision of autonomic computing

My thesis is that the systematic application of simple adaptive methods to file system design can produce systems that are significantly more robust to changing hardware and diverse workloads than existing systems. I present modifications to the log-structured file system that allow it to provide robust write performance in a wide range of environments. I also present a dynamic reorganization algorithm that makes disk layout responsive to read patterns. I evaluate these adaptive algorithms with trace driven simulation on a combination of synthetic and measured traces. I find that simple adaptive algorithms can dramatically improve worst case performance and can allow average case performance to scale with improvements in disk technology.

Improving file system performance with adaptive methods

Many data center traffic patterns exhibit abundant concurrent connections and high churn. In the face of these characteristics, server-centric congestion control is a poor fit—each connection, no matter how small, must start from scratch when testing when and how much to send along a given path. This is despite the fact that there are a large number of flows that may have already probed the same exact path, not just at a server level, but also at a rack level. Thus, we argue for rack-level congestion control in which all connections are tunneled through rack-to-rack JumboFlows. This design allows an entire rack’s connections to cooperate with one another for better fairness and performance, particularly for short flows. In this paper, we examine situations in which JumboFlows might be useful and present a preliminary design of a system (RackCC) that implements JumboFlows.

https://dl.acm.org/doi/pdf/10.1145/3005745.3005772

Rack-level Congestion Control

While there are many proposals to reduce the cost of data center networks (DCN), little attention has been paid to the role played by the physical links that carry packets. By studying over 300K optical links across many production DCNs, we show that these links are operating quite conservatively relative to the requirements in the IEEE standards. Motivated by this observation, to reduce DCN costs, we propose using transceivers—a key contributor to DCN cost—beyond their currently specified limit. Our experiments with multiple commodity transceivers show that their reach can be “stretched” 1.6 to 4 times their specification. However, with stretching, the performance of 1–5% of the DCN paths can fall below the IEEE standard. We develop RAIL, a system to ensure that in such a network, applications only use paths that meet their performance needs. Our proposal can reduce the network cost by up to 10% for 10Gbps networks and 44% for 40Gbps networks, without affecting the applications’ performance.

RAIL: A Case for Redundant Arrays of Inexpensive Links in Data Center Networks.

Our community has made significant progress in developing programmable network infrastructure, starting from the control plane and expanding to the data plane. As a latest trend, network devices are becoming runtime programmable while serving live traffic. This allows for reprogramming of individual device programs at fine-grained timescales to add or remove network functions. Many applications and services, however, need control over a combination of devices, including end host stacks, NICs, and switches, to accomplish their goals. We lay out our vision for runtime programmable networks, building upon device-level features to provide live, network-wide, runtime reprogramming. A whole-stack approach is needed with new programming models, compiler support, and network management abstractions. We outline a research agenda as a call to arms to the community.

A Vision for Runtime Programmable Networks

Over the last few years, several new applications have emerged on the Internet that are distributed at a scale previously unseen. Examples include peer-to-peer filesharing, content distribution networks, and voice-over-IP. This new class of distributed applications can make intelligent choices among the several paths available to them to optimize their performance. However, as a result of the best-effort packet forwarding interface exported by the Internet, applications need to explicitly measure the network to discover information about any path. Not only does measurement at the time of communication impose a significant overhead on most applications, but it is also redundant to have every application reimplement an Internet measurement component. 
In this dissertation, I design, build, and evaluate an information plane for the Internet, called iPlane, that enables distributed applications to discover information about Internet paths without explicit measurement. iPlane efficiently performs measurements from end-hosts under its control to predict path properties on the Internet between arbitrary end-hosts. I pursue a structural approach in issuing and synthesizing measurements—instead of using only end-to-end measurements and thus treating the Internet as a blackbox, I discover the Internet's routing topology and then compose measurements of links and path segments. This structural approach enables iPlane to predict multiple path properties, such as latency and loss rate. 
My evaluation of iPlane shows that iPlane's predictions of paths and path properties are accurate, significantly better than previous approaches for some of the sub-problems that iPlane tackles. Also, I used information from iPlane to drive path selection in three representative distributed applications—content distribution, peer-to-peer filesharing, and voice-over-IP. In each case, the use of iPlane helped significantly improve application performance.

Thomas Anderson

Papers

Improving file system performance with adaptive methods

Rack-level Congestion Control

RAIL: A Case for Redundant Arrays of Inexpensive Links in Data Center Networks.

A Vision for Runtime Programmable Networks

An information plane for internet applications