Data centers are experiencing an exponential increase in the amount of network traffic that they have to sustain due to cloud computing and several emerging web applications. To face this network load, large data centers are required with thousands of servers interconnected with high bandwidth switches. Current data center networks, based on electronic packet switches, consume excessive power to handle the increased communication bandwidth of emerging applications. Optical interconnects have gained attention recently as a promising solution offering high throughput, low latency and reduced energy consumption compared to current networks based on commodity switches. This paper presents a thorough survey on optical interconnects for next generation data center networks. Furthermore, the paper provides a qualitative categorization and comparison of the proposed schemes based on their main features such as connectivity and scalability. Finally, the paper discusses the cost and the power consumption of these schemes that are of primary importance in the future data center networks.

A Survey on Optical Interconnects for Data Centers

Optics offers unique opportunities for reducing energy in information processing and communications while simultaneously resolving the problem of interconnect bandwidth density inside machines. Such energy dissipation overall is now at environmentally significant levels; the source of that dissipation is progressively shifting from logic operations to interconnect energies. Without the prospect of substantial reduction in energy per bit communicated, we cannot continue the exponential growth of our use of information. The physics of optics and optoelectronics fundamentally addresses both interconnect energy and bandwidth density, and optics may be the only scalable solution to such problems. Here we summarize the corresponding background, status, opportunities, and research directions for optoelectronic technology and novel optics, including subfemtojoule devices in waveguide and novel two-dimensional (2-D) array optical systems. We compare different approaches to low-energy optoelectronic output devices and their scaling, including lasers, modulators and LEDs, optical confinement approaches (such as resonators) to enhance effects, and the benefits of different material choices, including 2-D materials and other quantum-confined structures. With such optoelectronic energy reductions, and the elimination of line charging dissipation by the use optical connections, the next major interconnect dissipations are in the electronic circuits for receiver amplifiers, timing recovery, and multiplexing. We show we can address these through the integration of photodetectors to reduce or eliminate receiver circuit energies, free-space optics to eliminate the need for timing and multiplexing circuits (while also solving bandwidth density problems), and using optics generally to save power by running large synchronous systems. One target concept is interconnects from ∼1 cm to ∼10 m that have the same energy (∼10 fJ/bit) and simplicity as local electrical wires on chip.

Attojoule Optoelectronics for Low-Energy Information Processing and Communications

Smart grid has modernized the way electricity is generated, transported, distributed, and consumed by integrating advanced sensing, communications, and control in the day-to-day operation of the grid. Electricity is a core utility for the functioning of society and for the services provided by information and communication technologies (ICTs). Several concepts of the smart grid, such as dynamic pricing, distributed generation, and demand management, have significantly impacted the operation of ICT services, in particular, communication networks and data centers. Ongoing energy-efficiency and operational expenditures reduction efforts in communication networks and data centers have gained another dimension with those smart grid concepts. In this paper, we provide a comprehensive survey on the smart grid-driven approaches in energy-efficient communications and data centers, and the interaction between smart grid and information and communication infrastructures. Although the studies on smart grid, energy-efficient communications, and green data centers have been separately surveyed in previous studies, to this end, research that falls in the intersection of those fields has not been properly classified and surveyed yet. We start our survey by providing background information on the smart grid and continue with surveying smart grid-driven approaches in energy-efficient communication systems, followed by energy, cost and emission minimizing approaches in data centers, and the corresponding cloud network infrastructure. We discuss the open issues in smart grid-driven approaches in ICTs and point some important research directions such as the distributed renewable energy generation capability-coupled communication infrastructures, optimum energy-efficient network design for the smart grid environment, the impact of green communication techniques on the reliability and latency requirements of smart grid data, workload consolidation with smart grid-awareness, and many more.

Energy-Efficient Information and Communication Infrastructures in the Smart Grid: A Survey on Interactions and Open Issues

Optics offers unique opportunities for reducing energy in information processing and communications while resolving the problem of interconnect bandwidth density inside machines. Such energy dissipation overall is now at environmentally significant levels; the source of that dissipation is progressively shifting from logic operations to interconnect energies. Without the prospect of substantial reduction in energy per bit communicated, we cannot continue the exponential growth of our use of information. The physics of optics and optoelectronics fundamentally addresses both interconnect energy and bandwidth density, and optics may be the only scalable solution to such problems. Here we summarize the corresponding background, status, opportunities, and research directions for optoelectronic technology and novel optics, including sub-femtojoule devices in waveguide and novel 2D array optical systems. We compare different approaches to low-energy optoelectronic output devices and their scaling, including lasers, modulators and LEDs, optical confinement approaches (such as resonators) to enhance effects, and the benefits of different material choices, including 2D materials and other quantum-confined structures. Beyond the elimination of line charging by the use optical connections, the next major interconnect dissipations are in the electronic circuits for receiver amplifiers, timing recovery and multiplexing. We can address these through the integration of photodetectors to reduce or eliminate receiver circuit energies, free-space optics to eliminate the need for timing and multiplexing circuits (while solving bandwidth density problems), and using optics generally to save power by running large synchronous systems. One target concept is interconnects from ~ 1 cm to ~ 10 m that have the same energy (~ 10fJ/bit) and simplicity as local electrical wires on chip.

Attojoule Optoelectronics for Low-Energy Information Processing and Communications: a Tutorial Review

Simulation techniques have become a powerful tool for deciding the best starting conditions on pay-as-you-go scenarios. This is the case of public cloud infrastructures, where a given number and type of virtual machines (in short VMs) are instantiated during a specified time, being this reflected in the final budget. With this in mind, this paper introduces and validates iCanCloud, a novel simulator of cloud infrastructures with remarkable features such as flexibility, scalability, performance and usability. Furthermore, the iCanCloud simulator has been built on the following design principles: (1) it's targeted to conduct large experiments, as opposed to others simulators from literature; (2) it provides a flexible and fully customizable global hypervisor for integrating any cloud brokering policy; (3) it reproduces the instance types provided by a given cloud infrastructure; and finally, (4) it contains a user-friendly GUI for configuring and launching simulations, that goes from a single VM to large cloud computing systems composed of thousands of machines.

/pdf/icancloud-a-flexible-and-scalable-cloud-infrastructure-2adg31izmp.pdf

iCanCloud: A Flexible and Scalable Cloud Infrastructure Simulator

Cloud computing data centers are becoming increasingly popular for the provisioning of computing resources. The cost and operating expenses of data centers have skyrocketed with the increase in computing capacity. Several governmental, industrial, and academic surveys indicate that the energy utilized by computing and communication units within a data center contributes to a considerable slice of the data center operational costs. In this paper, we present a simulation environment for energy-aware cloud computing data centers. Along with the workload distribution, the simulator is designed to capture details of the energy consumed by data center components (servers, switches, and links) as well as packet-level communication patterns in realistic setups. The simulation results obtained for two-tier, three- tier, and three-tier high-speed data center architectures demonstrate the effectiveness of the simulator in utilizing power management schema, such as voltage scaling, frequency scaling, and dynamic shutdown that are applied to the computing and networking components.

/pdf/greencloud-a-packet-level-simulator-of-energy-aware-cloud-526e1edu1n.pdf

GreenCloud: A Packet-Level Simulator of Energy-Aware Cloud Computing Data Centers

The demand-led growth of datacenter networks has meant that many constituent technologies are beyond the research community's budget. NetFPGA SUME is an FPGA-based PCI Express board with I/O capabilities for 100 Gbps operation as a network interface card, multiport switch, firewall, or test and measurement environment. NetFPGA SUME provides an accessible development environment that both reuses existing codebases and enables new designs.

NetFPGA SUME: Toward 100 Gbps as Research Commodity

In recent years, spurred on by the development and availability of programmable NICs, end hosts have increasingly become the enforcement point for core network functions such as load balancing, congestion control, and application specific network offloads. However, implementing custom designs on programmable NICs is not easy: many potential bottlenecks can impact performance. This paper focuses on the performance implication of PCIe, the de-facto I/O interconnect in contemporary servers, when interacting with the host architecture and device drivers. We present a theoretical model for PCIe and pcie-bench, an open-source suite, that allows developers to gain an accurate and deep understanding of the PCIe substrate. Using pcie-bench, we characterize the PCIe subsystem in modern servers. We highlight surprising differences in PCIe implementations, evaluate the undesirable impact of PCIe features such as IOMMUs, and show the practical limits for common network cards operating at 40Gb/s and beyond. Furthermore, through pcie-bench we gained insights which guided software and future hardware architectures for both commercial and research oriented network cards and DMA engines.

Understanding PCIe performance for end host networking

The demand-led growth of datacenter networks has meant that many constituent technologies are beyond the budget of the wider community. In order to make and validate timely and relevant new contributions, the wider community requires accessible evaluation, experimentation and demonstration environments with specification comparable to the subsystems of the most massive datacenter networks. We demonstrate NetFPGA, an open-source platform for rapid prototyping of networking devices with I/O capabilities up to 100Gbps. NetFPGA offers an integrated environment that enables networking research by users from a wide range of disciplines: from hardware-centric research to formal methods.

https://dl.acm.org/doi/pdf/10.1145/2785956.2790029

NetFPGA: Rapid Prototyping of Networking Devices in Open Source

Network Function Virtualization (NFV) allows creating specialized network appliances out of general-purpose computing equipment (servers, storage, and switches). In this paper we present a PCIe DMA engine that allows boosting the performance of virtual network appliances by using FPGA accelerators. Two key technologies are demonstrated, SR-IOV and PCI Passthrough. Using these two technologies, a single FPGA board can accelerate several virtual software appliances. The final goal is, in an NFV scenario, to substitute conventional Ethernet NICs by networking FPGA boards (such as NetFPGA SUME). The advantage of this approach is that FPGAs can very efficiently implement many networking tasks, thus boosting the performance of virtual networking appliances. The SR-IOV capable PCIe DMA engine presented in this work, as well as its associated driver, are key elements in achieving this goal of using FPGA networking boards instead of conventional NICs. Both DMA engine and driver are open source, and target the Xilinx 7-Series and UltraScale PCIe Gen3 endpoint. The design has been tested on a NetFPGA SUME board, offering transfer rates reaching 50 Gb/s for bulk transmissions. By taking advantage of SR-IOV and PCI Passthrough technologies, our DMA engine provides transfers rate well above 40 Gb/s for data transmissions from the FPGA to a virtual machine. We have also identified the bottlenecks in the use of virtualized FPGA accelerators caused by reductions in the maximum read request size and maximum payload PCIe parameters. Finally, the DMA engine presented in this paper is a very compact design, using just 2% of a Xilinx Virtex-7 XC7VX690T device.

/pdf/a-pcie-dma-engine-to-support-the-virtualization-of-40-gbps-10ch87uokq.pdf

Yury Audzevich

Papers

GreenCloud: A Packet-Level Simulator of Energy-Aware Cloud Computing Data Centers

NetFPGA SUME: Toward 100 Gbps as Research Commodity

Understanding PCIe performance for end host networking

NetFPGA: Rapid Prototyping of Networking Devices in Open Source

A PCIe DMA engine to support the virtualization of 40 Gbps FPGA-accelerated network appliances