Disaggregated rack-scale data centers have been proposed as the only promising avenue to break the barrier of the fixed CPU-to-memory proportionality caused by main-tray direct-attached conventional/traditional server-centric systems However, memory disaggregation has stringent network requirements in terms of latency, energy efficiency, bandwidth, and bandwidth density This paper identifies all the requirements and key performance indicators of a network to disaggregate IT resources while summarizing the progress and importance of optical interconnects Crucially, it proposes a rack-and-cluster scale architecture, which supports the disaggregation of CPU, memory, storage, and/or accelerator blocks Optical circuit switching forms the core of this architecture, whereas the end-points (IT resources) are equipped with on-chip programmable hybrid electrical packet/circuit switches This architecture offers dynamically reconfigurable physical topology to form virtual ones, each embedded with a set of functions It analyzes the latency overhead of disaggregated DDR4 (parallel) and the proposed hybrid memory cube (serial) memory elements on the conventional and the proposed architecture A set of resource allocation algorithms are introduced to (1) optimally select disaggregated IT resources with the lowest possible latency, (2) pool them together by means of a virtual network interconnect, and (3) compose virtual disaggregated servers Simulation findings show up to a 34% resource utilization increase over traditional data centers while highlighting the importance of the placement and locality among compute, memory, and storage resources In particular, the network-aware locality-based resource allocation algorithm achieves as low as 15 ns, 95 ns, and 315 ns memory transaction round-trip latency on 63%, 22%, and 15% of the allocated virtual machines (VMs) accordingly while utilizing 100% of the CPU resources Furthermore, a formulation to parameterize and evaluate the additional financial costs endured by disaggregation is reported It is shown that the more diverse the VM requests are, the higher the net financial gain is Finally, an experiment was carried out using silicon photonic midboard optics and an optical circuit switch, which demonstrates forward error correction free 10−12 bit error rate performance on up to five-tier scale-out networks

Optically disaggregated data centers with minimal remote memory latency: Technologies, architectures, and resource allocation [Invited]

A network data monitoring device provides for the flexible, programmable port-to-multi-port steering of data packet traffic between network port pairs, with tap data streams being directed to any of a plurality of monitor ports. The network data monitoring device is constructed utilizing one or more switching integrated circuits programmed to disable layer-2 routing and impose port-to-multiport data packet steering. Physical layer protocol encoding/decoding circuits enable connectivity to physical network media connectors though a system of fail-safe relays. A system controller, preferably implemented by a microprocessor, is connected to all switching integrated circuits and relays for configuration, status and control. Hardware-based logic selectively in complement to the switching integrated circuits provides for the programmable filtering, modification and programmable steering of data packets through the device.

Ethernet switch-based network monitoring system and methods

A network tap for enabling attached devices, such as an intrusion detection system, to transmit information into a communication line of the network without disrupting the network traffic on the communication line. When the attached device is an intrusion detection device, the network tap is connected to a firewall. An Ethernet switch or field programmable gate array (FPGA) is incorporated in the network tap to coordinate the transmission of device data to avoid data collisions with data transmissions already existing in the communication line. The network tap may be operated in various modes to accommodate different capabilities of attached devices. In addition, the network tap has various port configurations to allow a user to connect various attached devices through a single cable or dual cables and to connect various combinations of attached devices.

Network tap for use with multiple attached devices

The present invention provides for flexibly configurable network diagnostic modules that can implement any of a plurality of network diagnostic functions. A network diagnostic module receives an indication that a selected network diagnostic function (e.g., network analyzer, jammer, generator, bit error rate tester, etc.) is to be implemented. The network diagnostic module receives a bit file with instructions or data for implementing the selected network diagnostic function at one or more ports. The network diagnostic module identifies a programmable logic module (e.g., a Field ­Programmable Gate Array ('FPGA')) that controls the one or ports. The network diagnostic module loads a portion of the bit file at the identified programmable logic module to cause the programmable logic module and the one or more ports to interoperate to implement the selected network diagnostic function. A plurality of network diagnostic modules can be included in a common computer system chassis.

Multi-purpose network diagnostic modules

This work was supported by the U.K. EPSRC through the Centre for Advanced Photonics and Electronics (CAPE), and the Swedish Foundation for Strategic Research.

40 Gb/s data transmission over a 1-m-long multimode polymer spiral waveguide for board-level optical interconnects

An optical backplane ecosystem is described and demonstrated that is capable of multi-Tb/s bandwidth and is based on embedded polymer waveguides, passive optical backplane connectors, and midboard optical transceivers with bandwidth up to 28 Gb/s per lane. These systems provide the highest bandwidth-density, lowest power consumption, while maintaining the signal integrity. Ecosystems built around this architecture will provide the bandwidth-density required for next generation fabric interconnect for storage, switching, and routing applications in future high capacity generations of Data Centers and HPC systems. To demonstrate the applicability of this technology, it was used to provide embedded optical connectivity within a functional data storage enclosure.

960 Gb/s Optical Backplane Ecosystem Using Embedded Polymer Waveguides and Demonstration in a 12G SAS Storage Array

Widespread adoption of electro-optical circuit boards based on embedded glass waveguide technology would enable seamless optical connectivity from external fiber-optic networks to system embedded optical interconnect architectures. In this paper, we report on the fabrication of planar multimode waveguides within thin glass foils based on a two-step thermal ion exchange process. Novel lamination techniques were developed to allow glass waveguide panels to be reliably integrated into a conventional electronic multilayer printed circuit board. In addition, a complete suite of optical connector technologies were developed to enable both direct fiber-to-board and board-to-board connectivity. We present the design, development, and characterization of a fully integrated connection platform, comprising a $281 \times 233$
 mm2 multilayer electro-optical backplane with integrated planar glass waveguides, a pluggable connector system, and five pluggable test cards. Both on-card and externally generated 850 and 1310-nm optical test data were conveyed through the connector and waveguide system and characterised for in-system and system-to-system optical connectivity at data rates up to 32 Gb/s per channel exhibiting bit error rates of less than $10^{-12}$
 .

Pluggable Electro-Optical Circuit Board Interconnect Based on Embedded Graded-Index Planar Glass Waveguides

A network monitor that taps data from a network link has first and second interfaces that allow the network monitor to be connected in-line in the network link between two network devices. The monitor has first and second programmable logic devices. The programmable logic devices are arranged to receive parallel data output by the interfaces and to process said data for network analysis purposes. Each programmable logic device is controllable so as to selectively pass a copy of the received parallel data to the other programmable logic device so that the network monitor can operate in in-line mode and not to pass a copy of the received parallel data to the other programmable logic device so that the network monitor can operate in end station mode.

Network monitor and method

The evolution of data storage communication protocols and corresponding in-system bandwidth densities is set to impose prohibitive cost and performance constraints on future data storage system designs, fuelling proposals for hybrid electronic and optical architectures in data centers. The migration of optical interconnect into the system enclosure itself can substantially mitigate the communications bottlenecks resulting from both the increase in data rate and internal interconnect link lengths. In order to assess the viability of embedding optical links within prevailing data storage architectures, we present the design and assembly of a fully operational data storage array platform, in which all internal high speed links have been implemented optically. This required the deployment of mid-board optical transceivers, an electro-optical midplane and proprietary pluggable optical connectors for storage devices. We present the design of a high density optical layout to accommodate the midplane interconnect requirements of a data storage enclosure with support for 24 Small Form Factor (SFF) solid state or rotating disk drives and the design of a proprietary optical connector and interface cards, enabling standard drives to be plugged into an electro-optical midplane. Crucially, we have also modified the platform to accommodate longer optical interconnect lengths up to 50 meters in order to investigate future datacenter architectures based on disaggregation of modular subsystems. The optically enabled data storage system has been fully validated for both 6 Gb/s and 12 Gb/s SAS data traffic conveyed along internal optical links.

Demonstration of fully enabled data center subsystem with embedded optical interconnect

The invention provides a storage system comprising a solid state storage carrier sized to correspond to a defined storage form factor and to include a support printed circuit board (PCB) having arranged thereon a plurality of solid state storage devices. The storage system can include a storage enclosure including a midplane and at least one slot on an end of the storage enclosure, sized to receive a solid state storage carrier for connection with the midplane in the storage enclosure. The storage system can also include at least one midplane connector located on the midplane, wherein the at least one midplane connector is configured to mate with at least one form factor connector on an end of the solid state storage carrier.

Alexander Carl Worrall

Papers

960 Gb/s Optical Backplane Ecosystem Using Embedded Polymer Waveguides and Demonstration in a 12G SAS Storage Array

Pluggable Electro-Optical Circuit Board Interconnect Based on Embedded Graded-Index Planar Glass Waveguides

Network monitor and method

Demonstration of fully enabled data center subsystem with embedded optical interconnect

Solid state storage system