FaSST is an RDMA-based system that provides distributed in-memory transactions with serializability and durability. Existing RDMA-based transaction processing systems use one-sided RDMA primitives for their ability to bypass the remote CPU. This design choice brings several drawbacks. First, the limited flexibility of one-sided RDMA reduces performance and increases software complexity when designing distributed data stores. Second, deep-rooted technical limitations of RDMA hardware limit scalability in large clusters. FaSST eschews one-sided RDMA for fast RPCs using two-sided unreliable datagrams, which we show drop packets extremely rarely on modern RDMA networks. This approach provides better performance, scalability, and simplicity, without requiring expensive reliability mechanisms in software. In comparison with published numbers, FaSST outperforms FaRM on the TATP benchmark by almost 2x while using close to half the hardware resources, and it outperforms DrTM+R on the SmallBank benchmark by around 1.7x without making data locality assumptions.

/pdf/fasst-fast-scalable-and-simple-distributed-transactions-with-4l7fa7oqnk.pdf

FaSST: fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs

A method and system for enabling Non-Volatile Memory express (NVMe) for accessing remote solid state drives (SSDs) (or other types of remote non-volatile memory) over the Ethernet or other networks. An extended NVMe controller is provided for enabling CPU to access remote non-volatile memory using NVMe protocol. The extended NVMe controller is implemented on one server for communication with other servers or non-volatile memory via Ethernet switch. The NVMe protocol is used over the Ethernet or similar networks by modifying it to provide a special NVM-over-Ethernet frame.

NVM express controller for remote access of memory and I/O over ethernet-type networks

There is currently an active debate on which RDMA primitive (i.e., one-sided or two-sided) is optimal for distributed transactions. Such a debate has led to a number of optimizations based on one RDMA primitive, which was shown with better performance than the other.In this paper, we perform a systematic comparison between different RDMA primitives with a combination of various optimizations using representative OLTP workloads. More specifically, we first implement and compare different RDMA primitives with existing and our new optimizations upon a single well-tuned execution framework. This gives us insights into the performance characteristics of different RDMA primitives. Then we investigate the implementation of optimistic concurrency control (OCC) by comparing different RDMA primitives using a phase-by-phase approach with various transactions from TPC-C, SmallBank, and TPC-E. Our results show that no single primitive (one-sided or two-sided) wins over the other on all phases. We further conduct an end-to-end comparison of prior designs on the same codebase and find none of them is optimal.Based on the above studies, we build DrTM+H, a new hybrid distributed transaction system that always embraces the optimal RDMA primitives at each phase of transactional execution. Evaluations using popular OLTP workloads including TPC-C and SmallBank show that DrTM+H achieves over 7.3 and 90.4 million transactions per second on a 16-node RDMA-capable cluster (ConnectX-4) respectively, without locality assumption. This number outperforms the pure one-sided and two-sided systems by up to 1.89X and 2.96X for TPC-C with over 49% and 65% latency reduction. Further, DrTM+H scales well with a large number of connections on modern RDMA network.

/pdf/deconstructing-rdma-enabled-distributed-transactions-hybrid-261fl18yka.pdf

Deconstructing RDMA-enabled distributed transactions: hybrid is better

A processing device receives a message encapsulating an input/output (I/O) command from a remote computing device. The processing device identifies one or more physical storage devices to be accessed to satisfy the I/O command. The processing device then sends, to each physical storage device of the one or more physical storage devices, one or more non-volatile memory express (NVMe) commands directed to that physical storage device.

Non-volatile memory express over ethernet

Techniques in advanced deep learning provide improvements in one or more of accuracy, performance, and energy efficiency, such as accuracy of learning, accuracy of prediction, speed of learning, performance of learning, and energy efficiency of learning An array of processing elements performs flow-based computations on wavelets of data Each processing element has a respective compute element and a respective routing element Each compute element has processing resources and memory resources Each router enables communication via wavelets with at least nearest neighbors in a 2D mesh Stochastic gradient descent, mini-batch gradient descent, and continuous propagation gradient descent are techniques usable to train weights of a neural network modeled by the processing elements Reverse checkpoint is usable to reduce memory usage during the training

Accelerated deep learning

A data storage system includes a storage server, including non-volatile memory (NVM) and a server network interface controller (NIC), which couples the storage server to a network. A host computer includes a host central processing unit (CPU), a host memory and a host NIC, which couples the host computer to the network. The host computer runs a driver program that is configured to receive, from processes running on the host computer, commands in accordance with a protocol defined for accessing local storage devices connected to a peripheral component interface bus of the host computer, and upon receiving a storage access command in accordance with the protocol, to initiate a remote direct memory access (RDMA) operation to be performed by the host and server NICs so as to execute on the storage server, via the network, a storage transaction specified by the command.

Storage system and server

A method in a system that includes first and second devices that communicate with one another over a fabric that operates in accordance with a fabric address space, and in which the second device accesses a local memory via a local connection and not over the fabric, includes sending from the first device to a translation agent (TA) a translation request that specifies an untranslated address in an address space according to which the first device operates, for directly accessing the local memory of the second device. A translation response that specifies a respective translated address in the fabric address space, which the first device is to use instead of the untranslated address is received by the first device. The local memory of the second device is directly accessed by the first device over the fabric by converting the untranslated address to the translated address.

Address translation services for direct accessing of local memory over a network fabric

A method includes communicating between at least first and second devices over a bus in accordance with a bus address space, including providing direct access over the bus to a local address space of the first device by mapping at least some of the addresses of the local address space to the bus address space. In response to indicating, by the first device or the second device, that the second device requires to access a local address in the local address space that is not currently mapped to the bus address space, the local address is mapped to the bus address space, and the local address is accessed directly, by the second device, using the mapping.

Direct access to local memory in a pci-e device

A network interface device includes a host interface for connection to a host processor and a network interface, which is configured to transmit and receive data packets over a network, and which comprises multiple distinct physical ports configured for connection to the network. Processing circuitry is configured to receive, via one of the physical ports, a data packet from the network and to decide, responsively to a destination identifier in the packet, whether to deliver a payload of the data packet to the host processor via the host interface or to forward the data packet to the network via another one of the physical ports.

Nic with switching functionality between network ports

A method for data storage includes configuring a driver program on a host computer to receive commands in accordance with a protocol defined for accessing local storage devices connected to a peripheral component interface bus of the host computer. When the driver program receives, from an application program running on the host computer a storage access command in accordance with the protocol, specifying a storage transaction, a remote direct memory access (RDMA) operation is performed by a network interface controller (NIC) connected to the host computer so as to execute the storage transaction via a network on a remote storage device.

Shlomo Raikin

Papers

Storage system and server

Address translation services for direct accessing of local memory over a network fabric

Direct access to local memory in a pci-e device

Nic with switching functionality between network ports

Accessing remote storage devices using a local bus protocol