Multiple applications request data from multiple storage units over a computer network. The data is divided into segments and each segment is distributed randomly on one of several storage units, independent of the storage units on which other segments of the media data are stored. At least one additional copy of each segment also is distributed randomly over the storage units, such that each segment is stored on at least two storage units. This random distribution of multiple copies of segments of data improves both scalability and reliability. When an application requests a selected segment of data, the request is processed by the storage unit with the shortest queue of requests. Random fluctuations in the load applied by multiple applications on multiple storage units are balanced nearly equally over all of the storage units. This combination of techniques results in a system which can transfer multiple, independent high-bandwidth streams of data in a scalable manner in both directions between multiple applications and multiple storage units.

Computer system and process for transferring multiple high bandwidth streams of data between multiple storage units and multiple applications in a scalable and reliable manner

A disk drive system and method capable of dynamically allocating data is provided. The disk drive system may include a RAID subsystem having a pool of storage, for example a page pool of storage that maintains a free list of RAIDs, or a matrix of disk storage blocks that maintain a null list of RAIDs, and a disk manager having at least one disk storage system controller. The RAID subsystem and disk manager dynamically allocate data across the pool of storage and a plurality of disk drives based on RAID-to-disk mapping. The RAID subsystem and disk manager determine whether additional disk drives are required, and a notification is sent if the additional disk drives are required. Dynamic data allocation and data progression allow a user to acquire a disk drive later in time when it is needed. Dynamic data allocation also allows efficient data storage of snapshots/point-in-time copies of virtual volume pool of storage, instant data replay and data instant fusion for data backup, recovery etc., remote data storage, and data progression, etc.

Virtual disk drive system and method

Multiple applications request data from multiple storage units over a computer network. The data is divided into segments and each segment is distributed randomly on one of several storage units, independent of the storage units on which other segments of the media data are stored. Redundancy information corresponding to each segment also is distributed randomly over the storage units. The redundancy information for a segment may be a copy of the segment, such that each segment is stored on at least two storage units. The redundancy information also may be based on two or more segments. This random distribution of segments of data and corresponding redundancy information improves both scalability and reliability. When a storage unit fails, its load is distributed evenly over to remaining storage units and its lost data may be recovered because of the redundancy information. When an application requests a selected segment of data, the request may be processed by the storage unit with the shortest queue of requests. Random fluctuations in the load applied by multiple applications on multiple storage units are balanced nearly equally over all of the storage units. Small data files also may be stored on storage units that combine small files into larger segments of data using a log structured file system. This combination of techniques results in a system which can transfer both multiple, independent high-bandwidth streams of data and small data files in a scalable manner in both directions between multiple applications and multiple storage units.

Computer system and process for transferring streams of data between multiple storage units and multiple applications in a scalable and reliable manner

A data storage system which automatically provides and maintains identical secondary data on a preferably geographically remote secondary data storage device. The system includes a primary host computer located in the first geographic location which is coupled to a primary data storage system also located in the first geographic location. The primary data storage system includes at least one primary data storage device on which primary data is to be stored, and a primary data storage system controller which receives data from the primary host computer and controls the storing of the primary data on the primary data storage device. The primary data storage controller is coupled by high speed communication link to a secondary data storage system controller of a secondary data storage system, and coordinates the copying of the primary data to the secondary data storing system, and is responsive to an acknowledgement from the secondary data storage system controller of at least successful receipt of the primary data for updating the maintained list of the primary data to indicate that the copied primary data has been received by the secondary data storage system to be copied to the secondary data storage device.

Remote data mirroring

A data layout mechanism is described for allocating metadata within a storage system employing data striping. The data layout mechanism includes a number of storage devices, each of the storage devices having storage spaces allocated to store individual data stripe units associated with a number of stripes. The data layout mechanism further includes a plurality of metadata chunks allocated within the storage devices such that (1) metadata associated with at least two data stripe units of the same stripe is stored within a single metadata chunk, and (2) the metadata chunks are evenly distributed across the storage devices.

Method and system for striping data to accommodate integrity metadata

An apparatus and method for disk mapping and data retrieval includes a data storage medium on which has been stored a plurality of data records. Each record includes at least a record identification portion, for uniquely identifying each record from among the plurality of data records. The apparatus builds a record locator table and high speed semiconductor memory which comprises the unique record identifiers for the records on the storage medium as well as a record locator index generated by the apparatus, which indicates the address of the data record on the storage medium. Data retrieval is facilitated by first searching the record locator table in high speed semiconductor memory for a requested data record. Utilizing the record locator index associated with the reqested data record, the system directly accesses the requested data record on the storage medium thereby minimizing storage medium search time. Also disclosed is an apparatus and method for converting CKD formatted data records to FBA formatted disk drives and for building and compressing the "count" portion of the CKD data formatted record into a record locator table.

System and method for disk mapping and data retrieval

Data storage system controlled remote data mirroring with respectively maintained data indices

A method of controlling distribution of processing system including a plurality of host data processors (12) connected to a data storage system (14) which includes a digital storage that is partitioned into a plurality of volumes (20). The method includes assigning ownership of the volumes (80) to the host processors (12) such that each of the host processors owns a different subset of the plurality of volumes, wherein a host processor is prohibited from sending I/O requests to any of the volumes which it does not own. The method further includes monitoring the I/O requests that are sent each of the volumes by each of the host processors; from information obtained through monitoring, generating workload statistics indicative of the distribution of workload among the host processors; detecting a workload imbalance in the workload statistics; and in response to detecting a workload imbalance, reassigning ownership of the volumes to the host processors (12) so as to change the distribution of workload among the host processors.

Dynamic load balancing

A video file server includes an integrated cached disk array storage subsystem and a plurality of stream server computers linking the cached disk storage subsystem to the data network for the transfer of video data streams. The video file server further includes a controller server for applying an admission control policy to client requests and assigning stream servers to service the client requests. The stream servers include a real-time scheduler for scheduling isochronous tasks, and supports at least one industry standard network file access protocol and one file access protocol for continuous media file access. The cached disk storage subsystem is responsive to video prefetch commands, and the data specified for a prefetch command for a process are retained in an allocated portion of the cache memory from the time that the cached disk storage subsystem has responded to the prefetch command to the time that the cached disk storage subsystem responds to a fetch command specifying the data for the process. The time between prefetching and fetching is selected based on available disk and cache resources. The video file server provides video-on-demand service by maintaining and dynamically allocating sliding windows of video data in the random access memories of the stream server computers.

Natan Vishlitzky

Papers

Remote data mirroring

System and method for disk mapping and data retrieval

Data storage system controlled remote data mirroring with respectively maintained data indices

Dynamic load balancing

Prefetching to service multiple video streams from an integrated cached disk array