This paper presents the design of a read-optimized relational DBMS that contrasts sharply with most current systems, which are write-optimized. Among the many differences in its design are: storage of data by column rather than by row, careful coding and packing of objects into storage including main memory during query processing, storing an overlapping collection of column-oriented projections, rather than the current fare of tables and indexes, a non-traditional implementation of transactions which includes high availability and snapshot isolation for read-only transactions, and the extensive use of bitmap indexes to complement B-tree structures.We present preliminary performance data on a subset of TPC-H and show that the system we are building, C-Store, is substantially faster than popular commercial products. Hence, the architecture looks very encouraging.

/pdf/c-store-a-column-oriented-dbms-3oddwwyeo0.pdf

C-store: a column-oriented DBMS

In previous papers [SC05, SBC+07], some of us predicted the end of "one size fits all" as a commercial relational DBMS paradigm. These papers presented reasons and experimental evidence that showed that the major RDBMS vendors can be outperformed by 1--2 orders of magnitude by specialized engines in the data warehouse, stream processing, text, and scientific database markets.Assuming that specialized engines dominate these markets over time, the current relational DBMS code lines will be left with the business data processing (OLTP) market and hybrid markets where more than one kind of capability is required. In this paper we show that current RDBMSs can be beaten by nearly two orders of magnitude in the OLTP market as well. The experimental evidence comes from comparing a new OLTP prototype, H-Store, which we have built at M.I.T. to a popular RDBMS on the standard transactional benchmark, TPC-C.We conclude that the current RDBMS code lines, while attempting to be a "one size fits all" solution, infact, excel at nothing. Hence, they are 25 year old legacy code lines that should be retired in favor of a collection of "from scratch" specialized engines. The DBMS vendors (and the research community) should start with a clean sheet of paper and design systems for yesterday's needs.

/pdf/the-end-of-an-architectural-era-it-s-time-for-a-complete-4nfadcfvob.pdf

The end of an architectural era: it's time for a complete rewrite

Silo is a new in-memory database that achieves excellent performance and scalability on modern multicore machines. Silo was designed from the ground up to use system memory and caches efficiently. For instance, it avoids all centralized contention points, including that of centralized transaction ID assignment. Silo's key contribution is a commit protocol based on optimistic concurrency control that provides serializability while avoiding all shared-memory writes for records that were only read. Though this might seem to complicate the enforcement of a serial order, correct logging and recovery is provided by linking periodically-updated epochs with the commit protocol. Silo provides the same guarantees as any serializable database without unnecessary scalability bottlenecks or much additional latency. Silo achieves almost 700,000 transactions per second on a standard TPC-C workload mix on a 32-core machine, as well as near-linear scalability. Considered per core, this is several times higher than previously reported results.

/pdf/speedy-transactions-in-multicore-in-memory-databases-1l3hl0zrmo.pdf

Speedy transactions in multicore in-memory databases

Growing main memory capacity has fueled the development of in-memory big data management and processing. By eliminating disk I/O bottleneck, it is now possible to support interactive data analytics. However, in-memory systems are much more sensitive to other sources of overhead that do not matter in traditional I/O-bounded disk-based systems. Some issues such as fault-tolerance and consistency are also more challenging to handle in in-memory environment. We are witnessing a revolution in the design of database systems that exploits main memory as its data storage layer. Many of these researches have focused along several dimensions: modern CPU and memory hierarchy utilization, time/space efficiency, parallelism, and concurrency control. In this survey, we aim to provide a thorough review of a wide range of in-memory data management and processing proposals and systems, including both data storage systems and data processing frameworks. We also give a comprehensive presentation of important technology in memory management, and some key factors that need to be considered in order to achieve efficient in-memory data management and processing.

/pdf/in-memory-big-data-management-and-processing-a-survey-1k2b0h2jut.pdf

In-Memory Big Data Management and Processing: A Survey

Computer systems in the near future are expected to have Non-Volatile Main Memory (NVMM), enabled by a new generation of Non-Volatile Memory (NVM) technologies, such as Phase Change Memory (PCM), STT-MRAM, and Memristor The non-volatility property has the promise to persist in-memory data structures for instantaneous failure recovery However, realizing such promise requires a careful design to ensure that in-memory data structures are in known consistent states after failuresThis paper studies persistent in-memory B+-Trees as B+-Trees are widely used in database and data-intensive systems While traditional techniques, such as undo-redo logging and shadowing, support persistent B+-Trees, we find that they incur drastic performance overhead because of extensive NVM writes and CPU cache flush operations PCM-friendly B+-Trees with unsorted leaf nodes help mediate this issue, but the remaining overhead is still large In this paper, we propose write atomic B+-Trees (wB+-Trees), a new type of main-memory B+-Trees, that aim to reduce such overhead as much as possible wB+-Tree nodes employ a small indirect slot array and/or a bitmap so that most insertions and deletions do not require the movement of index entries In this way, wB+-Trees can achieve node consistency either through atomic writes in the nodes or by redo-only logging We model fast NVM using DRAM on a real machine and model PCM using a cycle-accurate simulator Experimental results show that compared with previous persistent B+-Tree solutions, wB+-Trees achieve up to 88x speedups on DRAM-like fast NVM and up to 271x speedups on PCM for insertions and deletions while maintaining good search performance Moreover, we replaced Memcached's internal hash index with tree indices Our real machine Memcached experiments show that wB+-Trees achieve up to 38X improvements over previous persistent tree structures with undo-redo logging or shadowing

/pdf/persistent-b-trees-in-non-volatile-main-memory-55z1x7coa8.pdf

Persistent B+-trees in non-volatile main memory

Hekaton is a new database engine optimized for memory resident data and OLTP workloads. Hekaton is fully integrated into SQL Server; it is not a separate system. To take advantage of Hekaton, a user simply declares a table memory optimized. Hekaton tables are fully transactional and durable and accessed using T-SQL in the same way as regular SQL Server tables. A query can reference both Hekaton tables and regular tables and a transaction can update data in both types of tables. T-SQL stored procedures that reference only Hekaton tables can be compiled into machine code for further performance improvements. The engine is designed for high con-currency. To achieve this it uses only latch-free data structures and a new optimistic, multiversion concurrency control technique. This paper gives an overview of the design of the Hekaton engine and reports some experimental results.

/pdf/hekaton-sql-server-s-memory-optimized-oltp-engine-5bo22ki6nm.pdf

Hekaton: SQL server's memory-optimized OLTP engine

Two features allow for increased performance in running database consistency checks. First, a database consistency check runs directly on a database, as opposed to running on a saved database snapshot. This saves memory since there is no need to save a database snapshot. To handle any pages modified since a start of the database consistency check, a supplemental check runs on modified pages. Any facts generated by either check run through an early aggregation that attempts to aggregate facts during the database consistency check.

Optimizations for a background database consistency check

The present invention extends to methods, systems, and computer program products for main-memory database checkpointing. Embodiments of the invention use a transaction log as an interface between online threads and a checkpoint subsystem. Using the transaction log as an interface reduces synchronization overhead between threads and the checkpoint subsystem. Transactions can be assigned to files and storage space can be reserved in a lock free manner to reduce overhead of checkpointing online transactions. Meta-data independent data files and delta files can be collapsed and merged to reduce storage overhead. Checkpoints can be updated incrementally such that changes made since the last checkpoint (and not all data) are flushed to disk. Checkpoint I/O is sequential, helping ensure higher performance of physical I/O layers. During recovery checkpoint files can be loaded into memory in parallel for multiple devices.

Main-memory database checkpointing

The integrity of B-tree structures can become compromised for many reasons. Since these inconsistencies manifest themselves in unpredictable ways, all commercial database management systems include mechanisms to verify the integrity and trustworthiness of an individual index and of a set of related indexes, and all vendors recommend index verification as part of regular database maintenance. This paper introduces algorithms for B-tree validation, reviews the algorithms’ strengths and weaknesses, and proposes a simple yet effective improvement for key verification across multiple B-tree levels. The performance is such that B-tree verification can become part of scans or backups. Our experimental comparisons include algorithm performance and scalability measured using a shipping product.

Efficient Verification of B-tree Integrity.

Methods, systems, and computer-readable media of data storage that include storing page identities of individual pages and logical relationships between pages are disclosed A particular system includes a plurality of data storage devices A storage manager is configured to store data as pages at the data storage devices Each page includes a page payload and a page identity The storage manager is also configured to store one or more relationships indicating logical order between pages

Ryan L. Stonecipher

Papers

Hekaton: SQL server's memory-optimized OLTP engine

Optimizations for a background database consistency check

Main-memory database checkpointing

Efficient Verification of B-tree Integrity.

Data storage including storing of page identity and logical relationships between pages