A data processing system includes one or more nodes, each node including a memory sub-system. The sub-system includes a fine-grained, memory, and a less-fine-grained (e.g., page-based) memory. The fine-grained memory optionally serves as a cache and/or as a write buffer for the page-based memory. Software executing on the system uses a node address space which enables access to the page-based memories of all nodes. Each node optionally provides ACID memory properties for at least a portion of the space. In at least a portion of the space, memory elements are mapped to locations in the page-based memory. In various embodiments, some of the elements are compressed, the compressed elements are packed into pages, the pages are written into available locations in the page-based memory, and a map maintains an association between the some of the elements and the locations.

System including a fine-grained memory and a less-fine-grained memory

Resources of an address space are managed in dynamically sized ranges, extents, sets, and/or blocks. The address space may be divided into regions, each corresponding to a different, respective allocation granularity. Allocating a block within a first region of the address space may comprise allocating a particular number of logical addresses (e.g., a particular range, set, and/or block of addresses), and allocating a block within a different region may comprise allocating a different number of logical addresses. The regions may be configured to reduce the metadata overhead needed to identify free address blocks (and/or maintain address block allocations), while facilitating efficient use of the address space for differently sized data structures.

Systems and methods for storage allocation

A de-duplication is configured to cache data for access by a plurality of different storage clients, such as virtual machines. A virtual machine may comprise a virtual machine de-duplication module configured to identify data for admission into the de-duplication cache. Data admitted into the de-duplication cache may be accessible by two or more storage clients. Metadata pertaining to the contents of the de-duplication cache may be persisted and/or transferred with respective storage clients such that the storage clients may access the contents of the de-duplication cache after rebooting, being power cycled, and/or being transferred between hosts.

Systems and methods for a de-duplication cache

An apparatus, system, and method are disclosed for data block usage information synchronization for a non-volatile storage volume. The method includes referencing first data block usage information for data blocks of a non-volatile storage volume managed by a storage manager. The first data block usage information is maintained by the storage manager. The method also includes synchronizing second data block usage information managed by a storage controller with the first data block usage information maintained by the storage manager. The storage manager maintains the first data block usage information separate from second data block usage information managed by the storage controller.

Apparatus, system, and method for data block usage information synchronization for a non-volatile storage volume

Data replication in a distributed node system including one or more nodes. A consensus protocol for failure recovery is implemented. Data items and information relating to consensus protocol roles of participant nodes are stored in at least some of the plurality of nodes. Logical logs stored in at least some of the plurality of nodes are created. The logical logs contain additional consensus protocol information including container metadata and replicated data.

Failure recovery using consensus replication in a distributed flash memory system

A system is described that includes a network attached persistent memory unit. The system includes a processor node for initiating persistent memory operations (e.g., read/write). The processor unit references its address operations relative to a persistent memory virtual address space that corresponds to a persistent memory physical address space. A network interface is used to communicate with the persistent memory unit wherein the persistent memory unit has its own network interface. The processor node and the persistent memory unit communicate over a communication link such as a network (e.g., SAN). The persistent memory unit is configured to translate between the persistent memory virtual address space known to the processor nodes and a persistent memory physical address space known only to the persistent memory unit. In other embodiments, multiple address spaces are provided wherein the persistent memory unit provides translation from these spaces to a persistent memory physical address space.

Communication-link-attached persistent memory device

A method of updating database volumes by writing database updates to a smart storage controller, and having the smart storage controller generate the log update records for updating local and remote log volumes, and data volumes.

Log Driven Storage Controller with Network Persistent Memory

In at least one embodiment, transaction processing systems comprise non-disk persistent memory that is utilized to commit transactions.

Transaction processing systems and methods utilizing non-disk persistent memory

A system is described that includes a network interface attached to a persistent memory unit. The persistent memory unit is configured to receive checkpoint data from a primary process, and to provide access to the checkpoint data for use in a backup process, which provides recovery capability in the event of a failure of the primary process. The network interface is configured to provide address translation information between virtual and physical addresses in the persistent memory unit. In other embodiments, the persistent memory unit is capable of storing multiple updates to the checkpoint state. The checkpoint state and the updates to the checkpoint state, if any, can be retrieved by the backup process periodically, or all at once upon failure of the primary process.

Persistent memory device for backup process checkpoint states

Migrating resources is a useful tool for balancing load in a distributed system, but it is difficult to determine when to move resources, where to move resources, and how much of them to move. We look at resource migration for file system metadata and show how CephFS's dynamic subtree partitioning approach can exploit varying degrees of locality and balance because it can partition the namespace into variable sized units. Unfortunately, the current metadata balancer is complicated and difficult to control because it struggles to address many of the general resource migration challenges inherent to the metadata management problem. To help decouple policy from mechanism, we introduce a programmable storage system that lets the designer inject custom balancing logic. We show the flexibility and transparency of this approach by replicating the strategy of a state-of-the-art metadata balancer and conclude by comparing this strategy to other custom balancers on the same system.

Sam Fineberg

Papers

Communication-link-attached persistent memory device

Log Driven Storage Controller with Network Persistent Memory

Transaction processing systems and methods utilizing non-disk persistent memory

Persistent memory device for backup process checkpoint states

Mantle: a programmable metadata load balancer for the ceph file system