MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google's clusters every day.

/pdf/mapreduce-simplified-data-processing-on-large-clusters-16emdgg5h5.pdf

MapReduce: simplified data processing on large clusters

MapReduce is a programming model and an associated implementation for processing and generating large datasets that is amenable to a broad variety of real-world tasks. Users specify the computation in terms of a map and a reduce function, and the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks. Programmers find the system easy to use: more than ten thousand distinct MapReduce programs have been implemented internally at Google over the past four years, and an average of one hundred thousand MapReduce jobs are executed on Google's clusters every day, processing a total of more than twenty petabytes of data per day.

/pdf/mapreduce-simplified-data-processing-on-large-clusters-1zth35tzla.pdf

Course Description In this course, we will explore the question of the actual and potential connections between democracy and education. Our focus of attention will be placed on a critical examination of democratic theory and its implications for the civic education roles and contributions of teachers, adult educators, community development practitioners, and community organizers. We will survey and deal critically with a range of competing conceptions of democracy, variously described as classical, republican, liberal, radical, marxist, neomarxist, pragmatist, feminist, populist, pluralist, postmodern, and/or participatory. Using narrative inquiry as a means for illuminating and interpreting contemporary practice, we will analyze the implications of different conceptions of democracy for the practical work of civic education.

Democracy and education.

SUMMARY Since 1984, the Condor project has enabled ordinary users to do extraordinary computing. Today, the project continues to explore the social and technical problems of cooperative computing on scales ranging from the desktop to the world-wide computational Grid. In this paper, we provide the history and philosophy of the Condor project and describe how it has interacted with other projects and evolved along with the field of distributed computing. We outline the core components of the Condor system and describe how the technology of computing must correspond to social structures. Throughout, we reflect on the lessons of experience and chart the course travelled by research ideas as they grow into production systems. Copyright c � 2005 John Wiley & Sons, Ltd.

/pdf/distributed-computing-in-practice-the-condor-experience-1xldnwf3zo.pdf

Distributed computing in practice: the Condor experience

VMware ESX Server is a thin software layer designed to multiplex hardware resources efficiently among virtual machines running unmodified commodity operating systems. This paper introduces several novel ESX Server mechanisms and policies for managing memory. A ballooning technique reclaims the pages considered least valuable by the operating system running in a virtual machine. An idle memory tax achieves efficient memory utilization while maintaining performance isolation guarantees. Content-based page sharing and hot I/O page remapping exploit transparent page remapping to eliminate redundancy and reduce copying overheads. These techniques are combined to efficiently support virtual machine workloads that overcommit memory.

/pdf/memory-resource-management-in-vmware-esx-server-3qar0gwl8r.pdf

Memory resource management in VMware ESX server

An important threat to reliable storage of data is silent data corruption. In order to develop suitable protection mechanisms against data corruption, it is essential to understand its characteristics. In this article, we present the first large-scale study of data corruption. We analyze corruption instances recorded in production storage systems containing a total of 1.53 million disk drives, over a period of 41 months. We study three classes of corruption: checksum mismatches, identity discrepancies, and parity inconsistencies. We focus on checksum mismatches since they occur the most.We find more than 400,000 instances of checksum mismatches over the 41-month period. We find many interesting trends among these instances, including: (i) nearline disks (and their adapters) develop checksum mismatches an order of magnitude more often than enterprise-class disk drives, (ii) checksum mismatches within the same disk are not independent events and they show high spatial and temporal locality, and (iii) checksum mismatches across different disks in the same storage system are not independent. We use our observations to derive lessons for corruption-proof system design.

/pdf/an-analysis-of-data-corruption-in-the-storage-stack-3d524b8191.pdf

An analysis of data corruption in the storage stack

In a virtualized environment, the VMM is the system's primary resource manager. Some services usually implemented at the OS layer, like I/O scheduling or certain kinds of security monitoring, are therefore more naturally implemented inside the VMM. Implementing such services at the VMM layer can be complicated by the lack of OS and application-level knowledge within a VMM. This paper describes techniques that can be used by a VMM to independently overcome part of the "semantic gap" separating it from the guest operating systems it supports. These techniques enable the VMM to track the existence and activities of operating system processes. Antfarm is an implementation of these techniques that works without detailed knowledge of a guest's internal architecture or implementation. An evaluation of Antfarm for two virtualization environments and two operating systems shows that it can accurately infer process events while incurring only a small 2.5% runtime overhead in the worst case. To demonstrate the practical benefits of process information in a VMM we implement an anticipatory disk scheduler at the VMM level. This case study shows that significant disk throughput improvements are possible in a virtualized environment by exploiting process information within a VMM.

/pdf/antfarm-tracking-processes-in-a-virtual-machine-environment-270yyxnsuj.pdf

Antfarm: tracking processes in a virtual machine environment

We present WiscKey, a persistent LSM-tree-based key-value store with a performance-oriented data layout that separates keys from values to minimize I/O amplification. The design of WiscKey is highly SSD optimized, leveraging both the sequential and random performance characteristics of the device. We demonstrate the advantages of WiscKey with both microbenchmarks and YCSB workloads. Microbenchmark results show that WiscKey is 2.5×-111× faster than LevelDB for loading a database and 1.6×-14× faster for random lookups. WiscKey is faster than both LevelDB and RocksDB in all six YCSB workloads.

WiscKey: separating keys from values in SSD-conscious storage

Serverless Computation with OpenLambda.

We present OpenLambda, a new, open-source platform for building next-generation web services and applications in the burgeoningmodel of serverless computation. We describe the key aspects of serverless computation, and present numerous research challenges that must be addressed in the design and implementation of such systems. We also include a brief study of current web applications, so as to better motivate some aspects of serverless application construction.

/pdf/serverless-computation-with-openlambda-54n749qiwi.pdf

Andrea C. Arpaci-Dusseau

Papers

An analysis of data corruption in the storage stack

Antfarm: tracking processes in a virtual machine environment

WiscKey: separating keys from values in SSD-conscious storage

Serverless Computation with OpenLambda.

Serverless computation with openLambda