This paper presents the design of a read-optimized relational DBMS that contrasts sharply with most current systems, which are write-optimized. Among the many differences in its design are: storage of data by column rather than by row, careful coding and packing of objects into storage including main memory during query processing, storing an overlapping collection of column-oriented projections, rather than the current fare of tables and indexes, a non-traditional implementation of transactions which includes high availability and snapshot isolation for read-only transactions, and the extensive use of bitmap indexes to complement B-tree structures.We present preliminary performance data on a subset of TPC-H and show that the system we are building, C-Store, is substantially faster than popular commercial products. Hence, the architecture looks very encouraging.

/pdf/c-store-a-column-oriented-dbms-3oddwwyeo0.pdf

C-store: a column-oriented DBMS

In previous papers [SC05, SBC+07], some of us predicted the end of "one size fits all" as a commercial relational DBMS paradigm. These papers presented reasons and experimental evidence that showed that the major RDBMS vendors can be outperformed by 1--2 orders of magnitude by specialized engines in the data warehouse, stream processing, text, and scientific database markets.Assuming that specialized engines dominate these markets over time, the current relational DBMS code lines will be left with the business data processing (OLTP) market and hybrid markets where more than one kind of capability is required. In this paper we show that current RDBMSs can be beaten by nearly two orders of magnitude in the OLTP market as well. The experimental evidence comes from comparing a new OLTP prototype, H-Store, which we have built at M.I.T. to a popular RDBMS on the standard transactional benchmark, TPC-C.We conclude that the current RDBMS code lines, while attempting to be a "one size fits all" solution, infact, excel at nothing. Hence, they are 25 year old legacy code lines that should be retired in favor of a collection of "from scratch" specialized engines. The DBMS vendors (and the research community) should start with a clean sheet of paper and design systems for yesterday's needs.

/pdf/the-end-of-an-architectural-era-it-s-time-for-a-complete-4nfadcfvob.pdf

The end of an architectural era: it's time for a complete rewrite

Silo is a new in-memory database that achieves excellent performance and scalability on modern multicore machines. Silo was designed from the ground up to use system memory and caches efficiently. For instance, it avoids all centralized contention points, including that of centralized transaction ID assignment. Silo's key contribution is a commit protocol based on optimistic concurrency control that provides serializability while avoiding all shared-memory writes for records that were only read. Though this might seem to complicate the enforcement of a serial order, correct logging and recovery is provided by linking periodically-updated epochs with the commit protocol. Silo provides the same guarantees as any serializable database without unnecessary scalability bottlenecks or much additional latency. Silo achieves almost 700,000 transactions per second on a standard TPC-C workload mix on a 32-core machine, as well as near-linear scalability. Considered per core, this is several times higher than previously reported results.

/pdf/speedy-transactions-in-multicore-in-memory-databases-1l3hl0zrmo.pdf

Speedy transactions in multicore in-memory databases

Growing main memory capacity has fueled the development of in-memory big data management and processing. By eliminating disk I/O bottleneck, it is now possible to support interactive data analytics. However, in-memory systems are much more sensitive to other sources of overhead that do not matter in traditional I/O-bounded disk-based systems. Some issues such as fault-tolerance and consistency are also more challenging to handle in in-memory environment. We are witnessing a revolution in the design of database systems that exploits main memory as its data storage layer. Many of these researches have focused along several dimensions: modern CPU and memory hierarchy utilization, time/space efficiency, parallelism, and concurrency control. In this survey, we aim to provide a thorough review of a wide range of in-memory data management and processing proposals and systems, including both data storage systems and data processing frameworks. We also give a comprehensive presentation of important technology in memory management, and some key factors that need to be considered in order to achieve efficient in-memory data management and processing.

/pdf/in-memory-big-data-management-and-processing-a-survey-1k2b0h2jut.pdf

In-Memory Big Data Management and Processing: A Survey

Computer systems in the near future are expected to have Non-Volatile Main Memory (NVMM), enabled by a new generation of Non-Volatile Memory (NVM) technologies, such as Phase Change Memory (PCM), STT-MRAM, and Memristor The non-volatility property has the promise to persist in-memory data structures for instantaneous failure recovery However, realizing such promise requires a careful design to ensure that in-memory data structures are in known consistent states after failuresThis paper studies persistent in-memory B+-Trees as B+-Trees are widely used in database and data-intensive systems While traditional techniques, such as undo-redo logging and shadowing, support persistent B+-Trees, we find that they incur drastic performance overhead because of extensive NVM writes and CPU cache flush operations PCM-friendly B+-Trees with unsorted leaf nodes help mediate this issue, but the remaining overhead is still large In this paper, we propose write atomic B+-Trees (wB+-Trees), a new type of main-memory B+-Trees, that aim to reduce such overhead as much as possible wB+-Tree nodes employ a small indirect slot array and/or a bitmap so that most insertions and deletions do not require the movement of index entries In this way, wB+-Trees can achieve node consistency either through atomic writes in the nodes or by redo-only logging We model fast NVM using DRAM on a real machine and model PCM using a cycle-accurate simulator Experimental results show that compared with previous persistent B+-Tree solutions, wB+-Trees achieve up to 88x speedups on DRAM-like fast NVM and up to 271x speedups on PCM for insertions and deletions while maintaining good search performance Moreover, we replaced Memcached's internal hash index with tree indices Our real machine Memcached experiments show that wB+-Trees achieve up to 38X improvements over previous persistent tree structures with undo-redo logging or shadowing

/pdf/persistent-b-trees-in-non-volatile-main-memory-55z1x7coa8.pdf

Persistent B+-trees in non-volatile main memory

Hekaton is a new database engine optimized for memory resident data and OLTP workloads. Hekaton is fully integrated into SQL Server; it is not a separate system. To take advantage of Hekaton, a user simply declares a table memory optimized. Hekaton tables are fully transactional and durable and accessed using T-SQL in the same way as regular SQL Server tables. A query can reference both Hekaton tables and regular tables and a transaction can update data in both types of tables. T-SQL stored procedures that reference only Hekaton tables can be compiled into machine code for further performance improvements. The engine is designed for high con-currency. To achieve this it uses only latch-free data structures and a new optimistic, multiversion concurrency control technique. This paper gives an overview of the design of the Hekaton engine and reports some experimental results.

/pdf/hekaton-sql-server-s-memory-optimized-oltp-engine-5bo22ki6nm.pdf

Hekaton: SQL server's memory-optimized OLTP engine

Performing database queries. A method includes receiving a particular database query. The method further includes accessing a query plan based on the particular database query. The query plan has operators and specific operational parameters associated with each of the operators. The association of operators and specific operational parameters is specific to the particular database query. From the query plan, the method further includes instantiating a plurality of compiled code templates. Each code template includes executable code that when executed performs functionality of one of the operators from the query plan with the specific operational parameters applied in the compilation. The method further includes binding the code templates together using programmatic control flow to create a functioning program.

Method of converting query plans to native code

A system and method for creation, modification and/or use of plan guide(s) is provided. Plan guide(s) can be employed to optimize the performance of query(ies) when it is not possible and/or desirable to change the text of the query directly. Plan guides can be useful, for example, when a small subset of queries in a database application deployed from a third-party vendor is not performing as expected. Plan guides influence optimization of queries by attaching query hints to them. In accordance with an aspect of the subject invention, a query plan execution system is provided. The system includes an execution component, a query optimizer, and, a plan cache. Plan guides and other associated metadata are stored in a plan guide metadata store.

Query plan selection control using run-time association mechanism

Systems and methods that generate specialized plans for compiling SQL queries. A plan generator component scans the query representation for parameter sensitive predicates and evaluates each predicate individually based on the parameter values. Accordingly, queries can be identified not only based on their structures, but also based on their parameter conditions. The specialized plans are more efficient for particular values, wherein queries that employ such values are optimally executed.

Parameter-sensitive plans for structural scenarios

Hekaton is a new database engine optimized for memory resident data and OLTP workloads that is fully integrated into Microsoft SQL Server. A key innovation that enables high performance in Hekaton is compilation of SQL stored procedures into machine code.

/pdf/compilation-in-the-microsoft-sql-server-hekaton-engine-3ms5mzetly.pdf

Erik Ismert

Papers

Hekaton: SQL server's memory-optimized OLTP engine

Method of converting query plans to native code

Query plan selection control using run-time association mechanism

Parameter-sensitive plans for structural scenarios

Compilation in the Microsoft SQL Server Hekaton Engine.