Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this article, we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.

Bigtable: A Distributed Storage System for Structured Data (Awarded Best Paper!).

Bigtable: A Distributed Storage System for Structured Data

Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.

/pdf/bigtable-a-distributed-storage-system-for-structured-data-2jvp27zmn1.pdf

Bigtable: a distributed storage system for structured data

The Internet architecture has proven its worth by the vast array of applications it now supports and the wide variety of network technologies over which it currently runs. Most current Internet research involves either empirical measurement studies or incremental modifications that can be deployed without major architectural changes. Easy access to virtual testbeds could foster a renaissance in applied architectural research that extends beyond these incrementally deployable designs.

/pdf/overcoming-the-internet-impasse-through-virtualization-1siq4ru2na.pdf

Overcoming the Internet impasse through virtualization

Software Defined Networking (SDN) is an exciting technology that enables innovation in how we design and manage networks. Although this technology seems to have appeared suddenly, SDN is part of a long history of efforts to make computer networks more programmable. In this paper, we trace the intellectual history of programmable networks, including active networks, early efforts to separate the control and data plane, and more recent work on OpenFlow and network operating systems. We highlight key concepts, as well as the technology pushes and application pulls that spurred each innovation. Along the way, we debunk common myths and misconceptions about the technologies and clarify the relationship between SDN and related technologies such as network virtualization.

The road to SDN: an intellectual history of programmable networks

PlanetLab is a geographically distributed overlay network designed to support the deployment and evaluation of planetary-scale network services Two high-level goals shape its design First, to enable a large research community to share the infrastructure, PlanetLab provides distributed virtualization, whereby each service runs in an isolated slice of PlanetLab's global resources Second, to support competition among multiple network services, PlanetLab decouples the operating system running on each node from the network-wide services that define PlanetLab, a principle referred to as unbundled management This paper describes how Planet-Lab realizes the goals of distributed virtualization and unbundled management, with a focus on the OS running on each node

/pdf/operating-system-support-for-planetary-scale-network-3aok8vo2rf.pdf

Operating system support for planetary-scale network services

Recent efforts to add new services to the Internet have increased interest in software-based routers that are easy to extend and evolve. This paper describes our experiences using emerging network processors---in particular, the Intel IXP1200---to implement a router. We show it is possible to combine an IXP1200 development board and a PC to build an inexpensive router that forwards minimum-sized packets at a rate of 3.47Mpps. This is nearly an order of magnitude faster than existing pure PC-based routers, and sufficient to support 1.77Gbps of aggregate link bandwidth. At lesser aggregate line speeds, our design also allows the excess resources available on the IXP1200 to be used robustly for extra packet processing. For example, with 8 × 100Mbps links, 240 register operations and 96 bytes of state storage are available for each 64-byte packet. Using a hierarchical architecture we can guarantee line-speed forwarding rates for simple packets with the IXP1200, and still have extra capacity to process exceptional packets with the Pentium. Up to 310Kpps of the traffic can be routed through the Pentium to receive 1510 cycles of extra per-packet processing.

/pdf/building-a-robust-software-based-router-using-network-1bd832dzia.pdf

Building a robust software-based router using network processors

This paper describes our effort to build an extensible router in support of active networks. Our work is driven by two goals: (1) supporting the injection of new functionality into a router, and (2) exploiting commercially available hardware. Our approach is a hierarchical architecture, in which packet flows traverse a range of processing/forwarding paths. This paper both presents the architecture, and describes our experiences implementing the architecture across a combination of general-purpose and network processors.

Extensible routers for active networks

/pdf/slice-creation-and-management-2sv96kus3u.pdf

Slice Creation and Management

Deterministic performance is desirable for many distributed applications, from vehicle control systems to financial networks. The trouble is that infrastructure for these applications must incorporate multiple independent timing sources, because uniform distribution of timing signals is only possible at small scales, such as for integrated circuits. To formally reason about the behavior of concurrent computations in large distributed systems, the nondeterminism created by independent timing must be eliminated. This dissertation proposes metasynchronization, a technique to uniformly time division all resources in distributed systems that span multiple timing domains. This allows for deterministic execution of and interaction between distributed computations, analogous to the deterministic behavior of components in synchronous integrated circuits. Such determinism allows formal correctness verification of computations with strict performance requirements. It also allows perfect virtualization of distributed resources, meaning a system where computations are unable to determine if they are executing on raw physical resources or within a virtualized environment. Nondeterminism makes perfect virtualization impossible for conventional systems. Metasynchronization creates the necessary determinism, and this dissertation proposes an execution model called hierarchical provisioning, which incorporates perfect virtualization, and thereby allows distributed computations to share resources deterministically. Importantly, metasynchronization creates uniform timing without distributing a centralized timing signal. Instead, all timing domains reach agreement on shared time in a fully decentralized self-stabilizing manner that requires no communication overhead, but does depend on small buffers and simple ongoing numerical computations for each communication link. Because of its decentralization, metasynchronization is highly robust, tolerating multiple simultaneous malicious (Byzantine) failures under normal circumstances.

Tammo Spalink

Papers

Operating system support for planetary-scale network services

Building a robust software-based router using network processors

Extensible routers for active networks

Slice Creation and Management

Deterministic sharing of distributed resources