The NMRPipe system is a UNIX software environment of processing, graphics, and analysis tools designed to meet current routine and research-oriented multidimensional processing requirements, and to anticipate and accommodate future demands and developments. The system is based on UNIX pipes, which allow programs running simultaneously to exchange streams of data under user control. In an NMRPipe processing scheme, a stream of spectral data flows through a pipeline of processing programs, each of which performs one component of the overall scheme, such as Fourier transformation or linear prediction. Complete multidimensional processing schemes are constructed as simple UNIX shell scripts. The processing modules themselves maintain and exploit accurate records of data sizes, detection modes, and calibration information in all dimensions, so that schemes can be constructed without the need to explicitly define or anticipate data sizes or storage details of real and imaginary channels during processing. The asynchronous pipeline scheme provides other substantial advantages, including high flexibility, favorable processing speeds, choice of both all-in-memory and disk-bound processing, easy adaptation to different data formats, simpler software development and maintenance, and the ability to distribute processing tasks on multi-CPU computers and computer networks.

NMRPipe: a multidimensional spectral processing system based on UNIX pipes

The Hadoop Distributed File System (HDFS) is designed to store very large data sets reliably, and to stream those data sets at high bandwidth to user applications. In a large cluster, thousands of servers both host directly attached storage and execute user application tasks. By distributing storage and computation across many servers, the resource can grow with demand while remaining economical at every size. We describe the architecture of HDFS and report on experience using HDFS to manage 25 petabytes of enterprise data at Yahoo!.

/pdf/the-hadoop-distributed-file-system-14rwlvr6z7.pdf

The Hadoop Distributed File System

Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this article, we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.

Bigtable: A Distributed Storage System for Structured Data (Awarded Best Paper!).

SPIN is an efficient verification system for models of distributed software systems. It has been used to detect design errors in applications ranging from high-level descriptions of distributed algorithms to detailed code for controlling telephone exchanges. The paper gives an overview of the design and structure of the verifier, reviews its theoretical foundation, and gives an overview of significant practical applications.

/pdf/the-model-checker-spin-1bgo4jtefp.pdf

The model checker SPIN

Many practical computing problems concern large graphs. Standard examples include the Web graph and various social networks. The scale of these graphs - in some cases billions of vertices, trillions of edges - poses challenges to their efficient processing. In this paper we present a computational model suitable for this task. Programs are expressed as a sequence of iterations, in each of which a vertex can receive messages sent in the previous iteration, send messages to other vertices, and modify its own state and that of its outgoing edges or mutate graph topology. This vertex-centric approach is flexible enough to express a broad set of algorithms. The model has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier. Distribution-related details are hidden behind an abstract API. The result is a framework for processing large graphs that is expressive and easy to program.

/pdf/pregel-a-system-for-large-scale-graph-processing-30lgjenxxg.pdf

Pregel: a system for large-scale graph processing

Very large data sets often have a flat but regular structure and span multiple disks and machines. Examples include telephone call records, network logs, and web document repositories. These large data sets are not amenable to study using traditional database techniques, if only because they can be too large to fit in a single relational database. On the other hand, many of the analyses done on them can be expressed using simple, easily distributed computations: filtering, aggregation, extraction of statistics, and so on. 

We present a system for automating such analyses. A filtering phase, in which a query is expressed using a new procedural programming language, emits data to an aggregation phase. Both phases are distributed over hundreds or even thousands of computers. The results are then collated and saved to a file. The design -- including the separation into two phases, the form of the programming language, and the properties of the aggregators -- exploits the parallelism inherent in having data and computation distributed across many machines.

/pdf/interpreting-the-data-parallel-analysis-with-sawzall-4jfu5dwt7q.pdf

Interpreting the data: Parallel analysis with Sawzall

From the Publisher:
Designed for first-time and experienced users, this book describes the UNIX® programming environment and philosophy in detail. Readers will gain an understanding not only of how to use the system, its components, and the programs, but also how these fit into the total environment.

The UNIX programming environment

By the mid 1980 s, the trend in computing was away from large centralized time-shared computers towards networks of smaller, personal machines, typically UNIX worksta­ tions . People had grown weary of overloaded, bureaucratic timesharing machines and were eager to move to small, self-maintained systems, even if that meant a net loss in computing power. As microcomputers became faster, even that loss was recovered, and this style of computing remains popular today.

/pdf/plan-9-from-bell-labs-2piygteggc.pdf

Plan 9 from Bell Labs

A graphic terminal is disclosed using bitmaps to represent plural overlapping displays. Graphics software is also disclosed in which the overlapping asynchronous windows or layers are manipulated by manipulating the bitmaps. With this software, the physical screen becomes several logical screens (layers) all running simultaneously, any one of which may be interacted with at any time.

Dynamic generation and overlaying of graphic windows for multiple active program storage areas

1. Style. Names. Expressions and Statements. Consistency and Idioms. Function Macros. Magic Numbers. Comments. Why Bother? 2. Algorithms and Data Structures. Searching. Sorting. Libraries. A Java Quicksort. O-Notation. Growing Arrays. Lists. Trees. Hash Tables. Summary. 3. Design and Implementation. The Markov Chain Algorithm. Data Structure Alternatives. Building the Data Structure in C. Generating Output. Java. C++. Awk and Perl. Performance. Lessons. 4. Interfaces. Comma-Separated Values. A Prototype Library. A Library for Others. A C++ Implementation. Interface Principles. Resource Management. Abort, Retry, Fail? User Interfaces. 5. Debugging. Debuggers. Good Clues, Easy Bugs. No Clues, Hard Bugs. Last Resorts. Non-reproducible Bugs. Debugging Tools. Other People's Bugs. Summary. 6. Testing. Test as You Write the Code. Systematic Testing. Test Automation. Test Scaffolds. Stress Tests. Tips for Testing. Who Does the Testing? Testing the Markov Program. Summary. 7. Performance. A Bottleneck. Timing and Profiling. Strategies for Speed. Tuning the Code. Space Efficiency. Estimation. Summary. 8. Portability. Language. Headers and Libraries. Program Organization. Isolation. Data Exchange. Byte Order. Portability and Upgrade. Internationalization. Summary. 9. Notation. Formatting Data. Regular Expressions. Programmable Tools. Interpreters, Compilers, and Virtual Machines. Programs that Write Programs. Using Macros to Generate Code. Compiling on the Fly. Epilogue. Appendix: Collected Rules. Index. 020161586XT04062001

Rob Pike

Papers

Interpreting the data: Parallel analysis with Sawzall

The UNIX programming environment

Plan 9 from Bell Labs

Dynamic generation and overlaying of graphic windows for multiple active program storage areas

The Practice of Programming