A data processing system includes one or more nodes, each node including a memory sub-system. The sub-system includes a fine-grained, memory, and a less-fine-grained (e.g., page-based) memory. The fine-grained memory optionally serves as a cache and/or as a write buffer for the page-based memory. Software executing on the system uses a node address space which enables access to the page-based memories of all nodes. Each node optionally provides ACID memory properties for at least a portion of the space. In at least a portion of the space, memory elements are mapped to locations in the page-based memory. In various embodiments, some of the elements are compressed, the compressed elements are packed into pages, the pages are written into available locations in the page-based memory, and a map maintains an association between the some of the elements and the locations.

System including a fine-grained memory and a less-fine-grained memory

Systems and methods for restriping files distributed among a set of smart storage units 114, wherein data blocks for a particular stripe do not typically need to be located at any particular location on the storage units, wherein data can be typically restriped among the smart storage units with minimal data movement, and wherein data is typically protected and recoverable even if a system failure occurs during the restriping process.One embodiment relates to a method of selecting a storage unit 114 for a block of data. The method comprises identifying storage units 114 on which the block of data is currently stored; identifying an optimal location on first storage unit; determining a set of optimal storage units; if the first storage unit is one of the storage units on which the block of data is currently stored, selecting the first storage unit; and if the first storage unit is not in the list of storage units on which the block of data is currently stored, selecting one of the set of optimal storage units.Another aspect of the present invention relates to a method of restriping data stored in a plurality of smart storage devices 114 in an intelligent distributed file system 110. The method comprises identifying at least one smart storage device on which clusters of data from an existing file may be stored; storing the clusters on the identified smart storage device; determining that the clusters were successfully stored; and updating metadata related to the clusters of data to reflect new assignments.

Systems and methods for restriping files in a distributed file system

Data replication in a distributed node system including one or more nodes. A consensus protocol for failure recovery is implemented. Data items and information relating to consensus protocol roles of participant nodes are stored in at least some of the plurality of nodes. Logical logs stored in at least some of the plurality of nodes are created. The logical logs contain additional consensus protocol information including container metadata and replicated data.

Failure recovery using consensus replication in a distributed flash memory system

In one embodiment, a user or client device is connected to a distributed file system comprised of one or more physical nodes. The data on each of the physical nodes store metadata about files and directories within the file system. Some of the embodiments permit a user to take a snapshot of data stored on the file system. The snapshot may include a single file, a single directory, a plurality of files within a directory, a plurality of directories, a path on the file system that includes nested files and subdirectories, or more than one path on the file system that each includes nested files and directories.

Systems and methods for a snapshot of data

A distributed database system has multiple compute nodes each running an instance of a database management system (DBMS) program that accesses database records in a local buffer cache. Records are persistently stored in distributed flash memory on multiple storage nodes. A Sharing Data Fabric (SDF) is a middleware layer between the DBMS programs and the storage nodes and has API functions called by the DBMS programs when a requested record is not present in the local buffer cache. The SDF fetches the requested record from flash memory and loads a copy into the local buffer cache. The SDF has threads on a home storage node that locate database records using a node map. A global cache directory locks and pins records to local buffer caches for updating by a node's DBMS program. DBMS operations are grouped into transactions that are committed or aborted together as a unit.

Scalable database management software on a cluster of nodes using a shared-distributed flash memory

A plurality of mid-tier databases form a single, consistent cache grid for data in a one or more backend data sources, such as a database system. The mid-tier databases may be standard relational databases. Cache agents at each mid-tier database swap in data from the backend database as needed. Consistency in the cache grid is maintained by ownership locks. Cache agents prevent database operations that will modify cached data in a mid-tier database unless and until ownership of the cached data can be acquired for the mid-tier database. Cache groups define what backend data may be cached, as well as a general structure in which the backend data is to be cached. Metadata for cache groups is shared to ensure that data is cached in the same form throughout the entire grid. Ownership of cached data can then be tracked through a mapping of cached instances of data to particular mid-tier databases.

Distributed consistent grid of in-memory database caches

A two copy versioning scheme is proposed. The scheme provides efficient implementation for a high read load database and eliminates read dependencies. The versioning scheme provides efficient implementation for transactions executing under ANSI READ COMMITTED isolation, and is extensible to serializable isolation. The two-copy versioning scheme is garbage free meaning that all versions of data records are used during execution and is adapted to index scanning operations. A 'deferred delete' scheme allows read transactions to complete data deallocation operations. Statements that require 'snapshots' are efficiently supported.

A database management system with efficient version control

A database is stored as a plurality of database shards in a distributed database grid comprising a plurality of grid elements, each including a mid-tier database system. A first grid element receives, from an application executing in the same memory as a mid-tier database system of the first grid element, a first database transaction including at least one database operation on specific data stored in a first database shard that belongs to the first grid element. The first grid element performs and commits the first database transaction without participation of another grid element of the plurality of grid elements. The first grid element receives a second database transaction that requires access to another database shard that does not belong to the first grid element. Multiple grid elements of the plurality of grid elements perform the second database transaction and commit the second database transaction using a two-phase commit protocol.

Direct-connect functionality in a distributed database grid

New lock modes are based on the concept of neighborhoods and are applied to spaces in indexes. The new lock modes include a Read, or shared, neighborhood (Snei) lock mode and a Write neighborhood (Xnei) that enhance concurrency among non-serializable transactions.

Neighboring locking technique for increasing concurrency among transactions

In column domain dictionary compression, column values in one or more columns are tokenized by a single dictionary. The domain of the dictionary is the entire set of columns. A dictionary may not only map a token to a tokenized value, but also to a count ("token count") of the number of occurrences of the token and corresponding tokenized value in the dictionary's domain. Such information may be used to compute queries on the base table.

Chi-Kim Hoang

Papers

Distributed consistent grid of in-memory database caches

A database management system with efficient version control

Direct-connect functionality in a distributed database grid

Neighboring locking technique for increasing concurrency among transactions

Column Domain Dictionary Compression