The high-performance server compute landscape is changing. The traditional model of building general-purpose enterprise compute boxes that end-users can configure with storage and networking to assemble their desired compute environments, has evolved to purpose-built systems optimized for specific applications. This tight integration of hardware and software components together with high-density midboard optical modules and an optical backplane allows for unprecedented levels of switching and compute efficiencies and has fueled the penetration of optical interconnects deep “inside the box,” particularly for switch scale-up. We briefly review earlier 40 G/port switching systems based on active optical cables, and present our newest system: An all-optically-interconnected 100 G/port 8.2 Tb/s InfiniBand packet switch ASIC with 41 ports running 100 Gb/s per port interconnected by 12-channel midboard optical transceivers with 25 Gb/s per channel per direction of optical I/O. Using a blind-mate optical backplane, these components enable systems with up to 50 Tb/s bandwidth in a 2U standard rack mount configuration with industry-leading density, efficiency, and latency. For even tighter co-integration of optical interconnects with switch and processor ASICs, we discuss photonic multichip module and interposer packaging technologies that will further improve system energy efficiencies and overcome impending system I/O bottlenecks.

From Chip to Cloud: Optical Interconnects in Engineered Systems

Co-packaged datacenter optics: Opportunities and challenges

For accurate DC IR-drop analysis of integrated circuits, electrical-thermal co-simulation is necessary to consider the effect of Joule heating. A suitable numerical method for this co-simulation is the finite element method (FEM) because of its capabilities in modeling complex geometries and materials. To deal with large-scale problems, a domain decomposition scheme is applied to the FEM to enable simulation with multiple processors in parallel and achieve significant reduction in computation time.

Electrical-Thermal Co-Simulation for DC IR-Drop Analysis of Large-Scale Power Delivery

Democratizing digital design and manufacturing using high performance cloud computing: Performance evaluation and benchmarking

The co-design of a 10Gb/s 45nm CMOS transceiver and low-loss interconnect for parallel links demonstrates 1.4–2.4pJ/bit I/O power efficiency. A C4 bump pattern with an effective 20∶3 signal to ground ratio is implemented with on-die transmission line routing to escape from the I/O circuitry to the package-die interface. Top-of-the-package connector based (TPCB) interconnects, including high density interconnect (HDI), bonded polyimide (PI) flex, liquid crystal polymer (LCP) flex and micro-twinax cables, are trace length matched in bundles of 9 or 10 I/O lanes to minimize clocking power while requiring only 2-tap TX equalization and 106–373mVpp-diff TX swing. The link measured maximum lane-to-lane phase mismatch ranged from 6ps to 22ps.

Multi-Gbit I/O and interconnect co-design for power efficient links

Design of PCB interconnects for data channels running at bitrate 50 Gbps and beyond is a very challenging problem that requires analyses and measurements over extremely broad frequency bandwidth from DC to 50 GHz and above. This paper shares our experience in building a practical methodology to make predictable 50 Gbps interconnects models. Substantial part of interconnects can be simulated with transmission line models that require identification of causal broadband dielectric and conductor roughness models. It is shown that separation of losses between the conductor roughness and dielectric models is essential element of such identification. Examples of proper and improper material models identification and consequences are provided in the paper. Accurate prediction of interconnect behavior also requires localization and 3D EM analysis for all transitions or discontinuities. Examples of optimized interconnects designed for 50 Gbps channels and the validation with measurements are also provided.

https://www.simberian.com/Presentations/14-WE2Presentation_LessonsLearnedHowToMake_v3.pdf

Lessons learned: How to Make Predictable PCB Interconnects for Data Rates of 50 Gbps and Beyond

The microelectronic packaging field is moving into the third dimension for miniaturization, low power consumption, and better performance In this paper, we present a double-sided flip-chip organic substrate with a memory controller on one side of the package, and 3D stacked disaggregated memory chips on the other side of the package This design allows the controller to interface with the DRAM stack directly through the substrate providing the shortest possible interconnect path, and thus achieving the fastest signaling speed However, this double-sided flip chip on organic substrate also causes yield, assembly, test, and reliability challenges In order to optimize the assembly process, a sequential 3D finite-element model was developed to simulate the package assembly process In these simulations, various assembly process sequences were simulated with different conditions and materials In addition, a probing test model was also built to study the connectivity of the Land Grid Array (LGA) pin array with the PCB sockets Results show that the careful selection of assembly steps and package materials are crucial for the successful package assembly and also important for the probing test

Design and assembly of a double-sided 3D package with a controller and a DRAM stack

An LCP flex interconnect is used to bypass the core vias and balls of the memory controller package, the PTH vias and traces of the FR-4 motherboard and memory module, and the traditional through-hole or SMT DIMM connector. A mating two-piece pin-grid array connector that can accommodate 36 differential pair signals is used to interface the flex to the memory controller package. On the memory side, the DRAM CSP package is directly soldered to the top layer of the flex and the bottom layer of the flex is soldered to the memory module via solder balls. Controller and memory test chips that are capable of operating in the 10 to 16 Gbps range are used for evaluating the possible data rates for 6″ and 12″ long flex links. The system margins are compared to a 3″ long motherboard-soldered memory system that was shown to operate at 16 Gbps.

Evaluation of a module based memory system with an LCP flex interconnect

The development of the next generation serial interfaces that operate between 50 Gbps and 60 Gbps is underway to deploy 400 Gb/s Ethernet systems. Design, analysis, and characterization of passive channels at these data rates are very challenging. Advanced modeling, analysis, and improved measurement techniques are required to accurately characterize high-speed links over broad frequency ranges. This paper describes the design and measurement used to characterize high-speed interconnects: boards, packages, and connectors including transition structures. Various interconnect components including several boards with various PCB laminates, backplanes with one and two connectors, straight through and orthogonal midplanes, chip-to-chip, and chip-to-module systems with transmitter and receiver packages are built and measured. Since both NRZ and PAM-4 signaling are presently under consideration for these new interfaces, the optimized interconnects are then analyzed using various equalization and these two signaling techniques at data rate of 56 Gbps. The resulting link performance is provided for the measured interconnect systems.

Measurement and characterization of backplanes for serial links operating at 56 Gbps

The design of interconnects for links operating at 50 Gbps and beyond is very challenging. The loss, dispersion, and discontinuities along the signaling path have to be minimized over a wide frequency range. Frequency dependent material properties and surface roughness has to be accurately considered. The impacts of short via stubs that are ignored at lower data rates can severely degrade the signals when operating at higher data rates. In order to provide ways to mitigate these effects and optimize the performance of the system, it is primarily essential to correctly model and characterize the passive channel. In this paper, the modeling and characterization techniques that guarantee successful designs of passive channels for data rates of 50 Gbps and beyond will be presented. Detailed studies and measurement results on the effects of short via stubs are also presented.

Don Mullen

Papers

Lessons learned: How to Make Predictable PCB Interconnects for Data Rates of 50 Gbps and Beyond

Design and assembly of a double-sided 3D package with a controller and a DRAM stack

Evaluation of a module based memory system with an LCP flex interconnect

Measurement and characterization of backplanes for serial links operating at 56 Gbps

Design, modeling, and characterization of passive channels for data rates of 50 Gbps and beyond