A data layout mechanism is described for allocating metadata within a storage system employing data striping. The data layout mechanism includes a number of storage devices, each of the storage devices having storage spaces allocated to store individual data stripe units associated with a number of stripes. The data layout mechanism further includes a plurality of metadata chunks allocated within the storage devices such that (1) metadata associated with at least two data stripe units of the same stripe is stored within a single metadata chunk, and (2) the metadata chunks are evenly distributed across the storage devices.

Method and system for striping data to accommodate integrity metadata

Briefly, the present invention relates to an information dispersal sytem in which original data to be stored is separated into a number of data 'slices' in such a manner that the data in each subset is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other data subsets. These data subsets are stored on separate storage devices as a way of increasing privacy and security. In accordance with an important aspect of the invention, a metadata management system stores and indexes user files across all of the storage nodes. A number of applications run on the servers supporting these storage nodes and are responsible for controlling the metadata. Metadata is the information about the data, the data slices or data subsets and the way in which these data subsets are dispersed among different storage nodes running over the network. As used herein, metadata includes data source names, their size, last modification date, authentication information etc. This information is required to keep track of dispersed data subsets among all the nodes in the system. Every time new data subsets are stored and old ones are removed from the storage nodes, the metadata is updated. In accordance with an important aspect of the invention, the metadata management system stores metadata for dispersed data where: the dispersed data is in several pieces; the metadata is in a separate dataspace from the dispersed data. Accordingly, the metadata management system is able to manage the metadata in a manner that is computationally efficient relative to known systems in order to enable broad use of the invention using the types of computers generally used by businesses, consumers and other organizations currently.

Metadata management system for an information dispersed storage system

An improved system for accessing data within a distributed data storage network (“DDSN”) is disclosed. In a system implementing the disclosed invention, traffic is routed to individual slice servers within the DDSN in accordance with objective criteria as well as user-defined policies. In accordance with one aspect of the disclosed invention, when a data segment is written to a DDSN, the segment is divided into multiple data slices, which are simultaneously transmitted to different slice servers. In accordance with another aspect of the disclosed invention, when a data segment is read from a DDSN, a list of slice servers, each containing a data slice that could be used to reconstruct the requested data segment, is assembled, and sorted in accordance with a preference rating assigned to each of the slice servers. Sufficient data slices to reconstruct the data segment are then read in accordance with the preference ranking of the slice servers.

Smart access to a dispersed data storage network

A digital data file storage system is disclosed in which original data files to be stored are dispersed using some form of information dispersal algorithm into a number of file “slices” or subsets in such a manner that the data in each file share is less usable or less recognizable or completely unusable or completely unrecognizable by itself except when combined with some or all of the other file shares. These file shares are stored on separate digital data storage devices as a way of increasing privacy and security. As dispersed file shares are being transferred to or stored on a grid of distributed storage locations, various grid resources may become non-operational or may operate below at a less than optimal level. When dispersed file shares are being written to a dispersed storage grid which not available, the grid clients designates the dispersed data shares that could not be written at that time on a Rebuild List. In addition when grid resources already storing dispersed data become non-available, a process within the dispersed storage grid designates the dispersed data shares that need to be recreated on the Rebuild List. At other points in time a separate process reads the set of Rebuild Lists used to create the corresponding dispersed data and stores that data on available grid resources.

System for rebuilding dispersed data

A billing process is disclosed for an information dispersal system or digital data storage system. The original data to be stored is separated into a number of data 'slices' or shares (22, 24, 26, 28, 30, and 32). These data subsets are stored on separate digital data storage devices (34, 36, 38, 40, 42, and 44) as a way of increasing privacy and security. A set of metadata tables are created, separate from the dispersed file share storage, to maintain information about the original data size of each block, file or set of file shares dispersed on the grid.

Billing system for information dispersal system

A semi-static distribution technique distributes parity across disks of an array. According to the technique, parity is distributed (assigned) across the disks of the array in a manner that maintains a fixed pattern of parity blocks among the stripes of the disks. When one or more disks are added to the array, the semi-static technique redistributes parity in a way that does not require recalculation of parity or moving of any data blocks. Notably, the parity information is not actually moved; the technique merely involves a change in the assignment (or reservation) for some of the parity blocks of each pre-existing disk to the newly added disk.

Semi-static distribution technique

Row-Diagonal Parity for Double Disk Failure Correction (Awarded Best Paper!).

A hybrid media storage architecture has a log-structured file system configured to control a plurality of different storage media organized as hybrid storage media that cooperate to provide a total storage space of a storage system. The log-structured file system is configured to perform initial placement and migration of data, as well as fine-grain write allocation of the data, among storage space locations of the hybrid storage media to thereby improve the performance characteristics of the media. By defining and implementing heuristics and policies directed to, e.g., types of data, the file system may initially place data on any of the different media and thereafter migrate data between the media at fine granularity and without the need for manual enforcement.

Hybrid media storage system architecture

According to one or more embodiments of the present invention, a network cache intercepts data requested by a client from a remote server interconnected with the cache through one or more wide area network (WAN) links (e.g., for Wide Area File Services, or “WAFS”). The network cache stores the data and sends the data to the client. The cache may then intercept a first write request for the data from the client to the remote server, and determine one or more portions of the data in the write request that changed from the data stored at the cache (e.g., according to one or more hashes created based on the data). The network cache then sends a second write request for only the changed portions of the data to the remote server.

System and method for bandwidth optimization in a network storage environment

A “row-diagonal” (R-D) parity technique reduces overhead of computing diagonal parity for a storage array adapted to enable efficient recovery from the concurrent failure of two storage devices in the array. The diagonal parity is computed along diagonal parity sets that collectively span all data disks and a row parity disk of the array. The parity for all of the diagonal parity sets except one is stored on the diagonal parity disk. The R-D parity technique provides a uniform stripe depth and an optimal amount of parity information.

Steven R. Kleiman

Papers

Semi-static distribution technique

Row-Diagonal Parity for Double Disk Failure Correction (Awarded Best Paper!).

Hybrid media storage system architecture

System and method for bandwidth optimization in a network storage environment

Row-diagonal parity technique for enabling efficient recovery from double failures in a storage array