We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to reexamine traditional choices and explore radically different design points. The file system has successfully met our storage needs. It is widely deployed within Google as the storage platform for the generation and processing of data used by our service as well as research and development efforts that require large data sets. The largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients. In this paper, we present file system interface extensions designed to support distributed applications, discuss many aspects of our design, and report measurements from both micro-benchmarks and real world use.

/pdf/the-google-file-system-2ng4dgvddm.pdf

The Google file system

Today's data centers may contain tens of thousands of computers with significant aggregate bandwidth requirements. The network architecture typically consists of a tree of routing and switching elements with progressively more specialized and expensive equipment moving up the network hierarchy. Unfortunately, even when deploying the highest-end IP switches/routers, resulting topologies may only support 50% of the aggregate bandwidth available at the edge of the network, while still incurring tremendous cost. Non-uniform bandwidth among data center nodes complicates application design and limits overall system performance.In this paper, we show how to leverage largely commodity Ethernet switches to support the full aggregate bandwidth of clusters consisting of tens of thousands of elements. Similar to how clusters of commodity computers have largely replaced more specialized SMPs and MPPs, we argue that appropriately architected and interconnected commodity switches may deliver more performance at less cost than available from today's higher-end solutions. Our approach requires no modifications to the end host network interface, operating system, or applications; critically, it is fully backward compatible with Ethernet, IP, and TCP.

/pdf/a-scalable-commodity-data-center-network-architecture-5evikm9tu4.pdf

A scalable, commodity data center network architecture

We have developed Ceph, a distributed file system that provides excellent performance, reliability, and scalability. Ceph maximizes the separation between data and metadata management by replacing allocation tables with a pseudo-random data distribution function (CRUSH) designed for heterogeneous and dynamic clusters of unreliable object storage devices (OSDs). We leverage device intelligence by distributing data replication, failure detection and recovery to semi-autonomous OSDs running a specialized local object file system. A dynamic distributed metadata cluster provides extremely efficient metadata management and seamlessly adapts to a wide range of general purpose and scientific computing file system workloads. Performance measurements under a variety of workloads show that Ceph has excellent I/O performance and scalable metadata management, supporting more than 250,000 metadata operations per second.

/pdf/ceph-a-scalable-high-performance-distributed-file-system-5clmvj19bs.pdf

Ceph: a scalable, high-performance distributed file system

An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Intelligent Automated Assistant

GPFS is IBM's parallel, shared-disk file system for cluster computers, available on the RS/6000 SP parallel supercomputer and on Linux clusters. GPFS is used on many of the largest supercomputers in the world. GPFS was built on many of the ideas that were developed in the academic community over the last several years, particularly distributed locking and recovery technology. To date it has been a matter of conjecture how well these ideas scale. We have had the opportunity to test those limits in the context of a product that runs on the largest systems in existence. While in many cases existing ideas scaled well, new approaches were necessary in many key areas. This paper describes GPFS, and discusses how distributed locking and recovery techniques were extended to scale to large clusters.

/pdf/gpfs-a-shared-disk-file-system-for-large-computing-clusters-35v610gyrf.pdf

GPFS: A Shared-Disk File System for Large Computing Clusters

A method is provided for managing the storage of a file that has been selected for migration from a first storage level within a file system to a second storage level. The method comprises copying each of one or more data blocks of the selected file that are stored in the first storage level at a physical disk address maintained in a corresponding reference of an inode for the selected file from the first storage level to the second storage level; maintaining a logical ditto address in each reference of an inode for each snapshot file in the file system that refers to one of the one or more data blocks of the selected file that were copied to the second storage level; and updating the file system to include a managed region for controlling access to the one or more data blocks through the inode for the selected file.

Efficient hierarchical storage management of a file system with snapshots

A method effectively preventing a requesting node from unfencing and mounting a file system subsequent to a failure in a cluster file system having a plurality of active nodes. The method comprising first upgrading one active node in the cluster to function as a cluster manager node. The cluster manager is in communication with all nodes. The cluster manager is assigned manager responsibilities, in part, comprising first receiving an active status request from the node requesting to mount a file system. The cluster manager first queries the quorum nodes to determine whether each node considers the cluster manager to still have cluster management responsibilities for the file system. If a majority of quorum nodes consider the cluster manager to still have cluster management responsibilities for the file system then the cluster manager responds to the requesting node's active status request. Thereafter, the requesting node proceeds with mounting the file system.

File system mounting in a clustered file system

A method is provided for limiting access to disks and files in the event of a system partitioning occurring as the result of a node failure in a data distributed processing system in which files are shared amongst a plurality of independently running nodes. The nodes in the system are provided with limited duration leases which permit data access and which have to be periodically renewed. Each partition is also associated with a quorum which is also a necessary element in providing access at proper times. The lease and quorum concepts work together to insure that recovery operations do not occur until recovery is possible. The lease concept thus is also seen as further operating in close cooperation with the quorum concept to prevent long delays in the recovery process that could otherwise occur.

Program support for disk fencing in a shared disk parallel file system across storage area network

A computer system having a shared disk file system running on multiple computers each having their own instance of an operating system and being coupled for parallel data sharing access to files residing on network attached shared disks. Methods are provided for use as a parallel file system in a shared disk environment by use of a scaleable directory service for the system with a stable cursor, and a segmented allocation map. Dynamic prefetch and cached balance pools for multiple accesses improve the system. Extended file attributes are used for implementation of Access Control Lists in a parallel file system. A metadata node manages file metadata for parallel read and write actions. For our system, tokens are used for metadata node selection and identification. We have provided recoverable local shares for quota Management. A quota server and a quota client are provided for each file system, and a quotacheck utility allows parallel environments with minimal interference to other applications which require the use of quotas.

Parallel file system with a quota check utility

Concurrent access to data is managed through concurrency control techniques. Various types of techniques are employed to manage the access, including locking-based techniques and non-locking-based techniques. A dynamic switch from one type of concurrency control technique (e.g., a locking-based technique) to a different type of concurrency control technique (e.g., a non-locking-based technique) is enabled. This switching is based on access patterns and/or application requirements for each file. The switching allows enhanced performance for both coarse-grain sharing and fine-grain sharing of data.

Frank B. Schmuck

Papers

Efficient hierarchical storage management of a file system with snapshots

File system mounting in a clustered file system

Program support for disk fencing in a shared disk parallel file system across storage area network

Parallel file system with a quota check utility

Dynamically switching between different types of concurrency control techniques to provide an adaptive access strategy for a parallel file system