Innovations for adaptive compression and decompression for dictionaries of a column-store database can reduce the amount of memory used for columns of the database, allowing a system to keep column data in memory for more columns, while delays for access operations remain acceptable. For example, dictionary compression variants use different compression techniques and implementation options. Some dictionary compression variants provide more aggressive compression (reduced memory consumption) but result in slower run-time performance. Other dictionary compression variants provide less aggressive compression (higher memory consumption) but support faster run-time performance. As another example, a compression manager can automatically select a dictionary compression variant for a given column in a column-store database. For different dictionary compression variants, the compression manager predicts run-time performance and compressed dictionary size, given the values of the column, and selects one of the dictionary compression variants.

Adaptive dictionary compression/decompression for column-store databases

For remote direct memory access (RDMA) by a client to a data record stored in a cache on a server, a hash map is received by a client from a server. The hash map includes one or more entries associated with a key for the data record stored in the cache on the server that stores a server-side remote pointer referencing the data record stored in the cache on the server. The client, using the key, looks up the server-side remote pointer for the data record from the hash map, and then performs one or more RDMA operations using the server-side remote pointer that allow the client to directly access the data record stored in the cache on the server.

RDMA-optimized high-performance distributed cache

Techniques for handling large string values in a string dictionary within an in-memory database system are presented. The string dictionary can be a paged-column main dictionary. For example, a value block of a dictionary may be loaded into memory, where the dictionary stores values for dictionary compression. The value block may include a first portion of a large string value and one or more logical pointers to one or more large string pages containing a remainder of the large string value. The large string value may be materialized into a contiguous location in memory, and a pointer to the materialized large string value may be created.

Large string access and storage

A server system may include a cluster of multiple computers that are networked for high-speed data communications. Each of the computers has a remote direct memory access (RDMA) network interface to allow high-speed memory sharing between computers. A relational database engine of each computer is configured to utilize a hierarchy of memory for temporary storage of working data, including in order of decreasing access speed (a) local main memory, (b) remote memory accessed via RDMS, and (c) mass storage. The database engine uses the local main memory for working data, and additionally uses the RDMA accessible memory for working data when the local main memory becomes depleted. The server system may include a memory broker to which individual computers report their available or unused memory, and which leases shared memory to requesting computers.

Memory sharing for working data using rdma

A server and/or a client stores a metadata hash map that includes one or more entries associated with keys for data records stored in a cache on a server, wherein the data records comprise Java Virtual Machine (JVM) artifacts or monitoring information. Each of the entries stores metadata for a corresponding data record, wherein the metadata comprises a server-side remote pointer that references the corresponding data record stored in the cache, as well as a version identifier for the key. A selected data record is accessed using a provided key by: (1) identifying potentially matching entries in the metadata hash map using the provided key; (2) accessing data records stored in the cache using the server-side remote pointers from the potentially matching entries; and (3) determining whether the accessed data records match the selected data record using the provided key and the version identifiers from the potentially matching entries.

High performance, distributed, shared, data grid for distributed java virtual machine runtime artifacts

Data structures can provide for a column store of a database table. Methods can use the data structures for efficiently responding to a query. Unique field values of a column of a database table can be identified. The unique values can be stored in a dictionary table along with reference keys that point to a row of the database table. A reference store column can replace the original column, where the reference store column stores index values of the dictionary table. A hash table can be used in accessing the database. A hash function can provide a hash value of a query term, and the hash value can be used to access a hash table to obtain a stored value of an index value of the dictionary table. The index value can be used to access the dictionary table to obtain reference keys corresponding to rows of the database table.

Optimizing wide data-type storage and analysis of data in a column store database

Techniques to process a query and perform a join of tables that are distributed across nodes of a network. The join can be performed by analyzing a Where clause. An active flag structure can have flag values that identify table entries satisfying criteria of the Where clause. Keys of surviving entries of a first table can be used to generate a request for a second table to be joined. The request can be for second flags for the second table when the Where clause has criteria for the second table. A response can be used to update the first flags to change a first flag to False. After updating, data can be retrieved for first flags that are True. Requests can use identifiers associated with the first table that identify a location for sending the request, e.g., using RDMA or MPI.

Chris Baskett

Papers

Optimizing wide data-type storage and analysis of data in a column store database

Joining large database tables