Systems and methods are disclosed for copying a data object to a target storage pool using a hybrid of storage pools, in which at least one of the storage pools is particularly efficient at identifying data that should be used for copying the data object to the target storage pool, and at least one of the storage pools is particularly efficient at retrieving the data that should be sent to the target storage pool. The system comprises a performance storage pool for storing data and having relatively high performance for retrieving stored data; a deduplicating storage pool for storing deduplicated data and storing metadata about data objects in the system and which has relatively high performance for identifying and specifying differences in a data object over time; and a controller for causing the performance storage pool to retrieve differences and provide the data to the target storage pool.

Incremental copy performance between data stores

Systems and methods are disclosed for checking the data integrity of a data object copied between storage pools by comparing data fingerprints of data objects, comprising scheduling a series of successive copy operations over time for copying a data object from a source data store to a target data store; generating a partial fingerprint of the data object at the source data store that creates a fingerprint from a subset of the data object; sending the partial fingerprint of the data object to the target data store; sending any new data contents to the target data store; and creating a partial fingerprint of the data object at the target data store and comparing it to the received partial fingerprint to determine if they differ, thereby allowing incremental verification that the copy of the data object at the target data store is the same as at the source data store.

Successive data fingerprinting for copy accuracy assurance

Computerized methods and systems are disclosed for creating an incremental backup of application data by creating a snapshot associated with a current incremental backup of a data file using a change tracking bitmap such that a data file associated with the current incremental backup can be restored from just the snapshot associated with the current incremental backup and an initial backup without needing to access one or more previously generated incremental backups of the data file.

Computerized methods and apparatus for incremental database backup using change tracking

Systems and techniques for managing data storage are disclosed. In some aspects, a front-end node responds to a request to write an object by dividing the object into multiple source data segments. The front-end node generates redundancy data for the multiple source data segments using a rateless erasure encoding. The front-end node associates a respective subset of the redundancy data with each of the multiple source data segments, wherein each subset of redundancy data and associated source data segment form an encoded segment. The rateless erasure encoding further includes defining multiple segment-level fragments within each of the encoded segments. The front-end node transmits each of the encoded segments to a selected one of multiple storage nodes, wherein each of the selected storage nodes are selected based on a determined storage layout of the encoded segments across the multiple storage nodes. For each of the received encoded segments, the storage node generates one or more protection fragments based on redundancy data generated from the segment-level fragments and stores the segment-level fragments and corresponding protection fragments across multiple storage media devices managed by the selected storage node.

Hierarchical wide spreading of distributed storage

Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration In the single-tier configuration, a data object is encoded, eg, based on an erasure coding method, to generate many data fragments, which are stored across many storage devices In the multi-tier configuration, a data object is encoded, eg, based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes and at least one latent storage Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device

Resiliency fragment tiering

Metadata generate for incremental backup is disclosed. A subset of blocks used to store file system metadata are identified in a set of blocks changed since a last backup. File system metadata stored in the subset of blocks is used to obtain file system metadata associated with file system objects that have been created and/or modified since the last backup. The file system metadata associated with file system objects that have been created and/or modified since the last backup is used to generate file system metadata files for the incremental backup.

Ramesh A. Babu

Papers

Metadata generation for incremental backup