A method and system for reducing storage requirements and speeding up storage operations by reducing the storage of redundant data includes receiving a request that identifies one or more data objects to which to apply a storage operation. For each data object, the storage system determines if the data object contains data that matches another data object to which the storage operation was previously applied. If the data objects do not match, then the storage system performs the storage operation in a usual manner. However, if the data objects do match, then the storage system may avoid performing the storage operation.

System and method for storing redundant information

A method and system for reducing storage requirements and speeding up storage operations by reducing the storage of redundant data includes receiving a request that identifies one or more files or data objects to which to apply a storage operation. For each file or data object, the storage system determines if the file or data object contains data that matches another file or data object to which the storage operation was previously applied, based on awareness of the application that created the data object. If the data objects do not match, then the storage system performs the storage operation in a usual manner. However, if the data objects do match, then the storage system may avoid performing the storage operation with respect to the particular file or data object.

Application-aware and remote single instance data management

A system and method of creating archive copies of data sets is described. In some examples, the system creates an archive copy from an original data set. In some examples, the system creates an archive copy when creating a recovery copy for a data set. In some examples, the system creates a copy without redundant data, and then encrypts the data set.

Systems and methods for creating copies of data, such as archive copies

A distributed, deduplicated storage system according to certain embodiments is arranged in a parallel configuration including multiple deduplication nodes. Deduplicated data is distributed across the deduplication nodes. The deduplication nodes can be networked together and communicate with one another according using a light-weight, customized communication scheme (e.g., a scheme based on FTP or HTTP). In some cases, deduplication management information including deduplication signatures and/or other metadata is stored separately from the deduplicated data in deduplication management nodes, improving performance and scalability.

Distributed deduplicated storage system

Described in detail herein are systems and methods for single instancing blocks of data in a data storage system. For example, the data storage system may include multiple computing devices (e.g., client computing devices) that store primary data. The data storage system may also include a secondary storage computing device, a single instance database, and one or more storage devices that store copies of the primary data (e.g., secondary copies, tertiary copies, etc.). The secondary storage computing device receives blocks of data from the computing devices and accesses the single instance database to determine whether the blocks of data are unique (meaning that no instances of the blocks of data are stored on the storage devices). If a block of data is unique, the single instance database stores it on a storage device. If not, the secondary storage computing device can avoid storing the block of data on the storage devices.

Block-level single instancing

A system and method for 'on-the-fly' de-duplication of data before storing the data in a storage system is provided. A data de-duplication module illustratively cooperates with protocol servers and a file system of a storage operating system executing on the storage system to implement the novel de-duplication technique. The deduplication module illustratively generates a block store, an index file and a hash table on storage space provided by the storage system. The hash table is utilized for tracking fingerprints and locations of blocks within the block store. The index file is utilized for storing directory information identifying the contents of data containers stored on the storage system, while the block store is utilized to store raw data blocks that comprise the data containers.

System and method for on-the-fly elimination of redundant data

According to one or more embodiments of the present invention, a network cache intercepts data requested by a client from a remote server interconnected with the cache through one or more wide area network (WAN) links (e.g., for Wide Area File Services, or “WAFS”). The network cache stores the data and sends the data to the client. The cache may then intercept a first write request for the data from the client to the remote server, and determine one or more portions of the data in the write request that changed from the data stored at the cache (e.g., according to one or more hashes created based on the data). The network cache then sends a second write request for only the changed portions of the data to the remote server.

System and method for bandwidth optimization in a network storage environment

A technique for eliminating duplicate data is provided. Upon receipt of a new data set, one or more anchor points are identified within the data set. A bit-by-bit data comparison is then performed of the region surrounding the anchor point in the received data set with the region surrounding an anchor point stored within a pattern database to identify forward/backward delta values. The duplicate data identified by the anchor point, forward and backward delta values is then replaced in the received data set with a storage indicator.

System and method for sampling based elimination of duplicate data

Embodiments of the present invention provide mechanisms for improving storage consumption on a sequential access medium, such as a physical tape, by preferably storing one instance of a data block of a backup data set on the tape media. When another instance of a data block is received having the same pattern as the stored data block, rather than storing the data block itself, a reference to the data block is stored on the sequential access medium. When data are restored, data blocks are cached at a block store on a storage device(s) having a faster seek time than the tape. When a reference to a previously stored data block is read from the tape, rather than re-winding the tape to search for the data block on the tape (which might take a long time to locate), the referenced data block can be found on the storage device having a faster seek time than the tape media.

De-duplication of data stored on tape media

A system and method for signature based data container recognition is provided When a new data container, such as a lun, is created, a security appliance generates a signature of the data container, by, eg, examining the contents of one or more data blocks of the data container The generated signature is then associated with the appropriate encryption key for the data container and is stored either within a configuration database of the security appliance or on a key management system operating within a security appliance environment To identify the encryption key associated with a data container, the security appliance generates a signature of the data container and compares the generated signature with the stored signatures Should there be a matching signature, the security appliance utilizes the encryption key associated with the matching signature to process data access requests to/from the data container

Yuval Frandzel

Papers

System and method for on-the-fly elimination of redundant data

System and method for bandwidth optimization in a network storage environment

System and method for sampling based elimination of duplicate data

De-duplication of data stored on tape media

System and method for signature based data container recognition