An optical backplane ecosystem is described and demonstrated that is capable of multi-Tb/s bandwidth and is based on embedded polymer waveguides, passive optical backplane connectors, and midboard optical transceivers with bandwidth up to 28 Gb/s per lane. These systems provide the highest bandwidth-density, lowest power consumption, while maintaining the signal integrity. Ecosystems built around this architecture will provide the bandwidth-density required for next generation fabric interconnect for storage, switching, and routing applications in future high capacity generations of Data Centers and HPC systems. To demonstrate the applicability of this technology, it was used to provide embedded optical connectivity within a functional data storage enclosure.

960 Gb/s Optical Backplane Ecosystem Using Embedded Polymer Waveguides and Demonstration in a 12G SAS Storage Array

Disaggregated rack-scale data centers have been proposed as the only promising avenue to break the barrier of the fixed CPU-to-memory proportionality caused by main-tray direct-attached conventional/traditional server-centric systems However, memory disaggregation has stringent network requirements in terms of latency, energy efficiency, bandwidth, and bandwidth density This paper identifies all the requirements and key performance indicators of a network to disaggregate IT resources while summarizing the progress and importance of optical interconnects Crucially, it proposes a rack-and-cluster scale architecture, which supports the disaggregation of CPU, memory, storage, and/or accelerator blocks Optical circuit switching forms the core of this architecture, whereas the end-points (IT resources) are equipped with on-chip programmable hybrid electrical packet/circuit switches This architecture offers dynamically reconfigurable physical topology to form virtual ones, each embedded with a set of functions It analyzes the latency overhead of disaggregated DDR4 (parallel) and the proposed hybrid memory cube (serial) memory elements on the conventional and the proposed architecture A set of resource allocation algorithms are introduced to (1) optimally select disaggregated IT resources with the lowest possible latency, (2) pool them together by means of a virtual network interconnect, and (3) compose virtual disaggregated servers Simulation findings show up to a 34% resource utilization increase over traditional data centers while highlighting the importance of the placement and locality among compute, memory, and storage resources In particular, the network-aware locality-based resource allocation algorithm achieves as low as 15 ns, 95 ns, and 315 ns memory transaction round-trip latency on 63%, 22%, and 15% of the allocated virtual machines (VMs) accordingly while utilizing 100% of the CPU resources Furthermore, a formulation to parameterize and evaluate the additional financial costs endured by disaggregation is reported It is shown that the more diverse the VM requests are, the higher the net financial gain is Finally, an experiment was carried out using silicon photonic midboard optics and an optical circuit switch, which demonstrates forward error correction free 10−12 bit error rate performance on up to five-tier scale-out networks

Optically disaggregated data centers with minimal remote memory latency: Technologies, architectures, and resource allocation [Invited]

The high-performance server compute landscape is changing. The traditional model of building general-purpose enterprise compute boxes that end-users can configure with storage and networking to assemble their desired compute environments, has evolved to purpose-built systems optimized for specific applications. This tight integration of hardware and software components together with high-density midboard optical modules and an optical backplane allows for unprecedented levels of switching and compute efficiencies and has fueled the penetration of optical interconnects deep “inside the box,” particularly for switch scale-up. We briefly review earlier 40 G/port switching systems based on active optical cables, and present our newest system: An all-optically-interconnected 100 G/port 8.2 Tb/s InfiniBand packet switch ASIC with 41 ports running 100 Gb/s per port interconnected by 12-channel midboard optical transceivers with 25 Gb/s per channel per direction of optical I/O. Using a blind-mate optical backplane, these components enable systems with up to 50 Tb/s bandwidth in a 2U standard rack mount configuration with industry-leading density, efficiency, and latency. For even tighter co-integration of optical interconnects with switch and processor ASICs, we discuss photonic multichip module and interposer packaging technologies that will further improve system energy efficiencies and overcome impending system I/O bottlenecks.

From Chip to Cloud: Optical Interconnects in Engineered Systems

This work was supported by the U.K. EPSRC through the Centre for Advanced Photonics and Electronics (CAPE), and the Swedish Foundation for Strategic Research.

40 Gb/s data transmission over a 1-m-long multimode polymer spiral waveguide for board-level optical interconnects

Optical technologies have received large interest in recent years for use in board-level interconnects. Polymer multimode waveguides in particular, constitute a promising technology for high-capacity optical backplanes as they can be cost-effectively integrated onto conventional printed circuit boards (PCBs). This paper presents the first optical backplane demonstrator based on the use of PCB-integrated polymer multimode waveguides and a regenerative shared bus architecture. The backplane demonstrator is formed with commercially-available low-cost electronic and photonic components onto conventional FR4 substrates and comprises two opto-electronic (OE) bus modules interconnected via a prototype regenerator unit. The system enables interconnection between the connected cards over four optical channels, each operating at 10 Gb/s. Bus extension is achieved by cascading OE bus modules via 3R regenerator units, overcoming therefore the inherent limitation of optical bus topologies in the maximum number of cards that can be connected to the bus. Details of the design, fabrication, and assembly of the different parts of this optical bus backplane are presented and related optical and data transmission characterisation studies are reported. The optical layer of the OE bus modules comprises a four-channel three-card waveguide layout that is compatible with VCSEL/PD arrays and ribbon fibres. All on-board optical paths exhibit insertion losses below 13 dB and intra-channel crosstalk lower than -29 dB. The robustness of the signal distribution from the bus inputs to all respective bus output ports in the presence of input misalignment is demonstrated, while 1 dB input alignment tolerances of approximately ±10 μm are obtained. The electrical layer of the OE bus modules comprises the essential driving circuitry for 1×4 VCSEL and PD arrays and the corresponding control and power regulation circuits. The interface between the optical and electrical layers of the bus modules is achieved with simple OE connectors that enable end-fired optical coupling into and out of the on-board polymer waveguides. The backplane demonstrator achieves error-free (BER <; 10-12) 10 Gb/s data transmission over each optical channel, enabling therefore, an aggregate interconnection capacity of 40 Gb/s between any connected cards.

/pdf/a-40-gb-s-optical-bus-for-optical-backplane-interconnections-100uv54kq5.pdf

A 40 Gb/s Optical Bus for Optical Backplane Interconnections

The evolution of data storage communication protocols and corresponding in-system bandwidth densities is set to impose prohibitive cost and performance constraints on future data storage system designs, fuelling proposals for hybrid electronic and optical architectures in data centers. The migration of optical interconnect into the system enclosure itself can substantially mitigate the communications bottlenecks resulting from both the increase in data rate and internal interconnect link lengths. In order to assess the viability of embedding optical links within prevailing data storage architectures, we present the design and assembly of a fully operational data storage array platform, in which all internal high speed links have been implemented optically. This required the deployment of mid-board optical transceivers, an electro-optical midplane and proprietary pluggable optical connectors for storage devices. We present the design of a high density optical layout to accommodate the midplane interconnect requirements of a data storage enclosure with support for 24 Small Form Factor (SFF) solid state or rotating disk drives and the design of a proprietary optical connector and interface cards, enabling standard drives to be plugged into an electro-optical midplane. Crucially, we have also modified the platform to accommodate longer optical interconnect lengths up to 50 meters in order to investigate future datacenter architectures based on disaggregation of modular subsystems. The optically enabled data storage system has been fully validated for both 6 Gb/s and 12 Gb/s SAS data traffic conveyed along internal optical links.

Demonstration of fully enabled data center subsystem with embedded optical interconnect

The underlying technology trends that have enabled optics to displace copper in telecommunications, datacommunications, and datacenters are discussed. Future directions are outlined that will drive optical interconnects further into the fabric of computer systems.

Trends and future directions in optical interconnects for datacenter and computer applications

An emerging eco-system is described enabling Tb/s bandwidth optical backplanes based on embedded polymer waveguides, passive optical backplane connectors and mid-board optical transceivers capable of bandwidths up to 28 Gb/s per lane.

960 Gb/s optical backplane using embedded polymer waveguides and demonstration in a 12G SAS storage array

Board-mounted optical assemblies (BOAs) enable significant bandwidth density increase relative to pluggable optics at the card edge. We discuss the challenges for the next step in this evolution as optics moves towards the chip.

Katharine Schmidtke

Papers

960 Gb/s Optical Backplane Ecosystem Using Embedded Polymer Waveguides and Demonstration in a 12G SAS Storage Array

Demonstration of fully enabled data center subsystem with embedded optical interconnect

Trends and future directions in optical interconnects for datacenter and computer applications

960 Gb/s optical backplane using embedded polymer waveguides and demonstration in a 12G SAS storage array

Taking Optics to the Chip: From Board-mounted Optical Assemblies to Chip-level Optical Interconnects