Distributed storage systems provide reliable access to data through redundancy spread over individually unreliable nodes. Application scenarios include data centers, peer-to-peer storage systems, and storage in wireless networks. Storing data using an erasure code, in fragments spread across nodes, requires less redundancy than simple replication for the same level of reliability. However, since fragments must be periodically replaced as nodes fail, a key question is how to generate encoded fragments in a distributed way while transferring as little data as possible across the network. For an erasure coded system, a common practice to repair from a single node failure is for a new node to reconstruct the whole encoded data object to generate just one encoded block. We show that this procedure is sub-optimal. We introduce the notion of regenerating codes, which allow a new node to communicate functions of the stored data from the surviving nodes. We show that regenerating codes can significantly reduce the repair bandwidth. Further, we show that there is a fundamental tradeoff between storage and repair bandwidth which we theoretically characterize using flow arguments on an appropriately constructed graph. By invoking constructive results in network coding, we introduce regenerating codes that can achieve any point in this optimal tradeoff.

Network Coding for Distributed Storage Systems

Disclosed is an Internet-based referral system that enables individuals and other business entities ("associates") to market products, in return for a commission, that are sold from a merchant's Web site. The system includes automated registration software that runs on the merchant's Web site to allow entities to register as associates. Following registration, the associate sets up a Web site (or other information dissemination system) to distribute hypertextual catalog documents that includes marketing information (product reviews, recommendations, etc.) about selected products of the merchant. In association with each such product, the catalog document includes a hypertextual "referral link" that allows a user ("customer") to link to the merchant's site and purchase the product. When a customer selects a referral link, the customer's computer transmits unique IDs of the selected product and of the associate to the merchant's site, allowing the merchant to identify the product and the referring associate. If the customer subsequently purchases the product from the merchant's site, a commission is automatically credited to an account of the referring associate. The merchant site also implements an electronic shopping cart that allows the customer to select products from multiple different Web sites, and then perform a single "check out" from the merchant's site.

Internet-based customer referral system

Peer-to-peer distributed storage systems provide reliable access to data through redundancy spread over nodes across the Internet. A key goal is to minimize the amount of bandwidth used to maintain that redundancy. Storing a file using an erasure code, in fragments spread across nodes, promises to require less redundancy and hence less maintenance bandwidth than simple replication to provide the same level of reliability. However, since fragments must be periodically replaced as nodes fail, a key question is how to generate a new fragment in a distributed way while transferring as little data as possible across the network. 
In this paper, we introduce a general technique to analyze storage architectures that combine any form of coding and replication, as well as presenting two new schemes for maintaining redundancy using erasure codes. First, we show how to optimally generate MDS fragments directly from existing fragments in the system. Second, we introduce a new scheme called Regenerating Codes which use slightly larger fragments than MDS but have lower overall bandwidth use. We also show through simulation that in realistic environments, Regenerating Codes can reduce maintenance bandwidth use by 25 percent or more compared with the best previous design--a hybrid of replication and erasure codes--while simplifying system architecture.

A disk drive system and method capable of dynamically allocating data is provided. The disk drive system may include a RAID subsystem having a pool of storage, for example a page pool of storage that maintains a free list of RAIDs, or a matrix of disk storage blocks that maintain a null list of RAIDs, and a disk manager having at least one disk storage system controller. The RAID subsystem and disk manager dynamically allocate data across the pool of storage and a plurality of disk drives based on RAID-to-disk mapping. The RAID subsystem and disk manager determine whether additional disk drives are required, and a notification is sent if the additional disk drives are required. Dynamic data allocation and data progression allow a user to acquire a disk drive later in time when it is needed. Dynamic data allocation also allows efficient data storage of snapshots/point-in-time copies of virtual volume pool of storage, instant data replay and data instant fusion for data backup, recovery etc., remote data storage, and data progression, etc.

Virtual disk drive system and method

A system which uses redundant storage and redundant communication to provide a robust distributed server system.

A reliable array of distributed computing nodes

We present a novel method, that we call EVENODD, for tolerating up to two disk failures in RAID architectures. EVENODD employs the addition of only two redundant disks and consists of simple exclusive-OR computations. This redundant storage is optimal, in the sense that two failed disks cannot be retrieved with less than two redundant disks. A major advantage of EVENODD is that it only requires parity hardware, which is typically present in standard RAID-5 controllers. Hence, EVENODD can be implemented on standard RAID-5 controllers without any hardware changes. The most commonly used scheme that employes optimal redundant storage (i.e., two extra disks) is based on Reed-Solomon (RS) error-correcting codes. This scheme requires computation over finite fields and results in a more complex implementation. For example, we show that the complexity of implementing EVENODD in a disk array with 15 disks is about 50% of the one required when using the RS scheme. The new scheme is not limited to RAID architectures: it can be used in any system requiring large symbols and relatively short codes, for instance, in multitrack magnetic recording. To this end, we also present a decoding algorithm for one column (track) in error. >

/pdf/evenodd-an-efficient-scheme-for-tolerating-double-disk-bdrj6a96kf.pdf

EVENODD: an efficient scheme for tolerating double disk failures in RAID architectures

We present a novel method, that we call EVENODD, for tolerating up to two disk failures in RAID architectures. EVENODD is the first known scheme for tolerating double disk failures that is optimal with regard to both storage and performance. EVENODD employs the addition of only two redundant disks and consists of simple exclusive-OR computations. A major advantage of EVENODD is that it only requires parity hardware, which is typically present in standard RAID-5 controllers. Hence, EVENODD can be implemented on standard RAID-5 controllers without any hardware changes. The only previously known scheme that employes optimal redundant storage (i.e. two extra disks) is based on Reed-Solomon (RS) error-correcting codes, requires computation over finite fields and results in a more complex implementation. For example, we show that the number of exclusive-OR operations involved in implementing EVENODD in a disk array with 15 disks is about 50% of the number required when using the RS scheme.

/pdf/evenodd-an-optimal-scheme-for-tolerating-double-disk-1ro5ut4cdi.pdf

EVENODD: an optimal scheme for tolerating double disk failures in RAID architectures

An apparatus for dynamically allocating memory includes a processor, a free buffer pool memory and a control memory which stores control block data structures. The control block data structures enable a segmentation of the free buffer pool memory into a series of free buffer pools, each free buffer pool comprising plural identical size buffers, each succeeding free buffer pool including a larger buffer size than a preceding free buffer pool. A selection size parameter for a given free buffer pool is a value that is larger than the buffer size comprising the given free buffer pool, but less than a next larger buffer size in the next of the series of free buffer pools. A memory allocation procedure responds to a request from an executing procedure for allocation of buffer space by: (i) allocating a buffer from a free buffer pool memory whose associated selection size parameter is a next larger value than the buffer space that was requested; (ii) determining a difference between the allocated buffer size and the requested buffer space to find an unfulfilled amount of the requested buffer space; (iii) allocating a buffer from a free buffer pool memory whose selection size parameter is a next larger value, among selection size parameters, than the unfulfilled amount; and (iv) repeating ii and iii until the memory allocation procedure determines that there is no unfulfilled amount of the requested buffer space. The apparatus further includes "quickcell" memory which is allocated without use of control block data structures.

Dynamic memory allocation that enalbes efficient use of buffer pool memory segments

The data contents of up to two concurrently failed or erased DASDs can be reconstituted where the data is distributed across M DASDs as an (M-1)*M block array and where (1) the (M-1)st DASD contains the simple parity taken over each of the array diagonals in diagonal major order in the same mode (odd/even) as that exhibited by the major diagonal of the array and (2) where the M-th DASD contains the simple even parity over each of the rows in row major order. Relatedly, short write updates require fewer operations for data blocks located off the major data array diagonal.

Method and means for encoding and rebuilding the data contents of up to two unavailable DASDS in a DASD array using simple non-recursive diagonal and row parity

In a data processing system employing a disk array, prediction of a possible failure of a disk drive initiates copying of the data away from the potentially failing disk drive to a spare disk drive before the failing drive actually fails. If the disk drive does fail before the copying of the contents to a spare disk drive is completed, rebuilding of the remaining contents within the failing disk drive is performed.

James T. Brady

Papers

EVENODD: an efficient scheme for tolerating double disk failures in RAID architectures

EVENODD: an optimal scheme for tolerating double disk failures in RAID architectures

Dynamic memory allocation that enalbes efficient use of buffer pool memory segments

Method and means for encoding and rebuilding the data contents of up to two unavailable DASDS in a DASD array using simple non-recursive diagonal and row parity

Failure prediction for disk arrays