We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to reexamine traditional choices and explore radically different design points. The file system has successfully met our storage needs. It is widely deployed within Google as the storage platform for the generation and processing of data used by our service as well as research and development efforts that require large data sets. The largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients. In this paper, we present file system interface extensions designed to support distributed applications, discuss many aspects of our design, and report measurements from both micro-benchmarks and real world use.

/pdf/the-google-file-system-2ng4dgvddm.pdf

The Google file system

Today's data centers may contain tens of thousands of computers with significant aggregate bandwidth requirements. The network architecture typically consists of a tree of routing and switching elements with progressively more specialized and expensive equipment moving up the network hierarchy. Unfortunately, even when deploying the highest-end IP switches/routers, resulting topologies may only support 50% of the aggregate bandwidth available at the edge of the network, while still incurring tremendous cost. Non-uniform bandwidth among data center nodes complicates application design and limits overall system performance.In this paper, we show how to leverage largely commodity Ethernet switches to support the full aggregate bandwidth of clusters consisting of tens of thousands of elements. Similar to how clusters of commodity computers have largely replaced more specialized SMPs and MPPs, we argue that appropriately architected and interconnected commodity switches may deliver more performance at less cost than available from today's higher-end solutions. Our approach requires no modifications to the end host network interface, operating system, or applications; critically, it is fully backward compatible with Ethernet, IP, and TCP.

/pdf/a-scalable-commodity-data-center-network-architecture-5evikm9tu4.pdf

A scalable, commodity data center network architecture

We have developed Ceph, a distributed file system that provides excellent performance, reliability, and scalability. Ceph maximizes the separation between data and metadata management by replacing allocation tables with a pseudo-random data distribution function (CRUSH) designed for heterogeneous and dynamic clusters of unreliable object storage devices (OSDs). We leverage device intelligence by distributing data replication, failure detection and recovery to semi-autonomous OSDs running a specialized local object file system. A dynamic distributed metadata cluster provides extremely efficient metadata management and seamlessly adapts to a wide range of general purpose and scientific computing file system workloads. Performance measurements under a variety of workloads show that Ceph has excellent I/O performance and scalable metadata management, supporting more than 250,000 metadata operations per second.

/pdf/ceph-a-scalable-high-performance-distributed-file-system-5clmvj19bs.pdf

Ceph: a scalable, high-performance distributed file system

An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

GPFS is IBM's parallel, shared-disk file system for cluster computers, available on the RS/6000 SP parallel supercomputer and on Linux clusters. GPFS is used on many of the largest supercomputers in the world. GPFS was built on many of the ideas that were developed in the academic community over the last several years, particularly distributed locking and recovery technology. To date it has been a matter of conjecture how well these ideas scale. We have had the opportunity to test those limits in the context of a product that runs on the largest systems in existence. While in many cases existing ideas scaled well, new approaches were necessary in many key areas. This paper describes GPFS, and discusses how distributed locking and recovery techniques were extended to scale to large clusters.

/pdf/gpfs-a-shared-disk-file-system-for-large-computing-clusters-35v610gyrf.pdf

GPFS: A Shared-Disk File System for Large Computing Clusters

PROBLEM TO BE SOLVED: To allow both limited file access and complete file access under a file system restoration operation SOLUTION: File system restoration is made flexible by the use of indicators in individual file restoration statuses through the whole restoration processing The indicators are useful, for example, in the restoration of such a large-scale file system that the number of files comes to a range of about several hundred-millions, in particular The present invention provides various degrees of individual file accesses even when the restoration processing in the progress of execution, because the restoration of the large-scale file system requires a relatively long time The file access is made complete or limited with respect to an attribute of the file during the file system restoration The invention provides most advantageously the complete access driven dynamically (of on-demand) even when the file system restoration is in the course of operation COPYRIGHT: (C)2005,JPO&NCIPI

Quick restoration for use of file system in ultra-large-scale file system

Embodiments relate to policy-based, multi-scheme data reduction for a computer memory. An aspect includes receiving a plurality of policy rules by a policy engine of the computer memory, wherein a first policy rule specifies applying a first data reduction scheme to data in the computer memory based on the data matching first characteristics, wherein a second policy rule specifies applying a second data reduction scheme to data in the computer memory based on the data matching second characteristics, wherein the first data reduction scheme is different from the second data reduction scheme. Another aspect includes determining, by the policy engine, that first data in the computer memory matches the first characteristics, and that second data in the computer memory matches the second characteristics. Yet another aspect includes applying the first data reduction scheme to the first data, and applying the second data reduction scheme to the second data.

Policy-based, multi-scheme data reduction for computer memory

Embodiments of the invention relate to small write performance enhancements for parallel file systems. One embodiment includes flushing a received write transaction from a first memory device to a second memory device on a node in a file system based on one or more byte-ranges within a file system block. It is determined to flush the received write transaction to a recovery log that is stored in a non-volatile storage layer of the file system. If it is determined to flush the received write transaction to the recovery log: appending data associated with the received write transaction in the recovery log of the file system, replicating the data associated with the received write transaction in another non-volatile storage layer of another node if required, and marking the one or more byte-ranges as committed.

Buffering and replicating data written to a distributed storage system

A method includes flushing a write transaction based on byte-ranges within a file system block. It is determined if the write transaction is a first write transaction to the file system block and whether to flush the received write transaction to a recovery log. If it is determined to flush the write transaction to the recovery log and the received write transaction is the first write transaction: the data associated with the write transaction is appended in the recovery log and byte-ranges remaining in the file system block are recorded, and also recording an associated particular bit value or particular bit pattern that the remaining byte-ranges must be set to before the remaining byte-ranges are accessed or when a recovery sequence for recovering data in the recovery log executes due to failure. The write transaction is replicated. The entire file system block byte-range is marked as committed.

Quick initialization of data regions in a distributed storage system

Embodiments of the invention relate to supporting coordinated access to a file system's shared disk storage subsystem using an automatic alignment of a parallel file access protocol and metadata management on a per object basis. Active file servers in the cluster have equivalent access to shared data of the file system. A file server (among servers in the cluster) receives a client's request to access a file stored by the file system in the storage subsystem using a parallel file access protocol. If the file system has selected a file server to serve as a metadata manager of said file, the file server uses the selected file server to coordinate access to the file's metadata. If the file system has not selected a file server to serve as the metadata manager of the file, the file system selects the file server that received said request as the metadata manager for said file until a file close request operation is received. The file server then performs a file open request operation for the file, to the file server selected to serve as the metadata manager. The file server then opens the file and creates a file handle of the file. The file server then encodes a file server ID, of the file server selected as the metadata manager of said file, in the file handle such that the file system's file servers may identify which file server is the file's metadata manager. The file server then returns the file handle to the client for performing I/O to said file.

Frank B. Schmuck

Papers

Quick restoration for use of file system in ultra-large-scale file system

Policy-based, multi-scheme data reduction for computer memory

Buffering and replicating data written to a distributed storage system

Quick initialization of data regions in a distributed storage system

Supporting coordinated access to a file system's shared storage using automatic alignment of a parallel file access protocol and metadata management