In one embodiment, a layered file system of a storage input/output (I/O) stack executes on one or more nodes of a cluster. The layered file system includes a flash-optimized, log-structured layer configured to provide sequential storage of data and metadata (i.e., a log-structured layout) on solid state drives (SSDs) of storage arrays in the cluster to reduce write amplification, while leveraging a data de-duplication feature of the storage I/O stack. An extent store layer of the file system performs and maintains mappings of the extent keys to SSD storage locations, while a volume layer of the file system performs and maintains mappings of the LUN offset ranges to the extent keys. Separation of the mapping functions between the volume and extent store layers enables different volumes with different offset ranges to reference a same extent key (and thus a same extent).

Global in-line extent-based deduplication

A system, method, and computer-readable storage medium for protecting a set of storage devices using a secret sharing scheme in combination with an external secret. An initial master secret is generated and then transformed into a final master secret using an external secret. A plurality of shares are generated from the initial master secret and distributed to the storage devices. The data of each storage device is encrypted with a device-specific key, and this key is encrypted using the final master secret. In order to read the data on a given storage device, the initial master secret reconstructed from a threshold number of shares and the external secret is retrieved. Next, the initial master secret is transformed into the final master secret using the external secret, and then the final master secret is used to decrypt the encrypted key of a given storage device.

Data protection in a storage system using external secrets

A system and method for managing multiple fingerprint tables in a deduplicating storage system. A computer system includes a storage medium, a first fingerprint table comprising a first plurality of entries, and a second fingerprint table comprising a second plurality of entries. Each of the first plurality of entries and the second plurality of entries are configured to store fingerprint related data corresponding to data stored in the storage medium. A storage controller is configured to select the first fingerprint table for storage of entries corresponding to data stored in the data storage medium that has been deemed more likely to be successfully deduplicated than other data stored in the data storage medium; and select the second fingerprint table for storage of entries corresponding to data stored in the data storage medium that has been deemed less likely to be successfully deduplicated than other data stored in the storage medium.

Method for maintaining multiple fingerprint tables in a deduplicating storage system

Systems, methods, and computer readable storage mediums for maintaining the history of remapped data in a storage system. A first portion of a first medium may be remapped to a second medium as part of read optimization operations, wherein the second medium is younger than the first medium. A tag associated with the first portion of the first medium may be retained when the first portion is remapped to the second medium. When the second medium is replicated to another storage array, the first portion of the first medium may be prevented from being replicated based on the tag identifying the first portion as being part of a previous replication operation.

Optimizing replication by distinguishing user and system write activity

A method for achieving optimal use of compute, storage and network resources in a distributed computing system is provided. In one embodiment, a job request associated with a file is submitted. A Job Scheduler in conjunction with a workflow manager determines the best fit for dynamically chunking size of the file. The distributed storage subsystem provides a dynamic view to the compute resources where the job will be run. This knowledge is also used to pre-staging the data by the distributed storage subsystem. Furthermore, keeping a history of the job and their data allows for further improvements in resources and job execution times by being able to skip processing on data that has been done previously. Given the ability to chunk the file dynamically, it is possible to re-balance resources for running the job dynamically to meet its service level agreement.

Method and apparatus for achieving optimal resource allocation dynamically in a distributed computing environment

A system and non-transitory computer program product for preserving data redundancy in a data deduplication system in a computing environment is provided. A selected data segment, to be written through the data deduplication system, is encrypted such that the selected data segment is not subject to a deduplication operation. Copies of the data segment that are to be precluded from data deduplication are determined and identified. A unique encryption key is used to encrypt the selected data segment to be written through the data deduplication system such that the selected data segment is not subject to a deduplication operation. The data deduplication system is tricked to recognize the encrypted, selected data segment as new, undeduplicated data by the encrypting thereby skipping steps of the deduplication operation that includes fingerprint generation and matching. The encrypted, selected data segment is directly written to a new physical storage location.

Preserving redundancy in data deduplication systems by encryption

Mechanisms identify one or more first storage devices in a first tier of the tiered storage system that may be placed in a minimal power consumption state and identify one or more data segments stored on the one or more first storage devices that are most likely to be accessed during a period of time in which the one or more first storage devices are in the minimal power consumption state. The mechanisms migrate the one or more data segments to one or more second storage devices in one of the first tier or a second tier of the storage system and place the one or more first storage devices in the minimal power consumption state. Access requests to the one or more data segments are serviced by the one or more second storage devices while the one or more first storage devices are in the minimal power consumption state.

Reducing power consumption by migration of data within a tiered storage system

A mechanism is provided in a storage system for efficient inline data de-duplication. The mechanism receives a write command and a hash key for a portion of data to be written from an application host to a write address. The write command indicates whether the application host is tolerant or intolerant to data loss. Responsive to the write command indicating the application host is tolerant to data loss, the mechanism performs a hash key lookup in a hash index. The mechanism determines whether the portion of data has previously been written to the storage system. Responsive to determining the portion of data has previously been written to the storage system, the mechanism stores a pointer to the previously written data at the write address.

Efficient inline data de-duplication on a storage system

Provided are a computer program product, system, and method for converting a first address mapping function for mapping addresses to storage locations to a second address mapping function. For each of a plurality of addresses allocated in the storage using the first address mapping function, a node is generated in the second address mapping function. Each node in the second address mapping function associates a logical address with a physical location for the logical address. A determination is made of addresses having unused space and storage space is freed for the determined addresses having the unused space. Indication is made in the second address mapping function that the storage space for the determined addresses has been freed.

Converting a first address mapping function for mapping addresses to storage locations to a second address mapping function

A mechanism is provided for optimizing and enhancing performance for parity based storage, particularly redundant array of independent disk (RAID) storage. The mechanism optimizes a repetitive pattern write command for performance for storage configurations that require parity calculations. The mechanism eliminates the need for laborious parity calculations that are resource intensive and add to IO latency. For repetitive write commands that span across the full stripe of a RAID5 or similar volume, the mechanism calculates parity by looking at the pattern and the number of columns in the volume. The mechanism may avoid the XOR operation altogether for repetitive pattern write commands. The mechanism may enhance secure delete operations that use repetitive pattern write commands by eliminating data reliability operations like parity generation and writing altogether.

Rahul M. Fiske

Papers

Preserving redundancy in data deduplication systems by encryption

Reducing power consumption by migration of data within a tiered storage system

Efficient inline data de-duplication on a storage system

Converting a first address mapping function for mapping addresses to storage locations to a second address mapping function

Optimizing and Enhancing Performance for Parity Based Storage