Reliable, fast and deterministic transmission of control information in a network is a need for many distributed systems. One example is timing systems, where a reference frequency is used to accurately schedule time-critical messages. The White Rabbit (WR) project is a multi-laboratory and multi-company effort to bring together the best of the data transfer and timing worlds in a completely open design. It takes advantage of the latest developments for improving timing over Ethernet, such as IEEE 1588 (Precision Time Protocol) and Synchronous Ethernet. The presented approach aims for a general purpose, fieldbus-like transmission system, which provides deterministic data and timing (sub-ns accuracy and ps jitter) to around 1000 stations. It automatically compensates for fiber lengths in the order of 10 km. This paper describes the WR design goals and the specification used for the project. It goes on to describe the central component of the WR system structure the WR switch - with theoretical considerations about the requirements. Finally, it presents real timing measurements for the first prototypes of WR hardware.

https://cds.cern.ch/record/1743073/files/thbl2.pdf

The white rabbit project

A Short-Baseline Neutrino (SBN) physics program of three LAr-TPC detectors located along the Booster Neutrino Beam (BNB) at Fermilab is presented. This new SBN Program will deliver a rich and compelling physics opportunity, including the ability to resolve a class of experimental anomalies in neutrino physics and to perform the most sensitive search to date for sterile neutrinos at the eV mass-scale through both appearance and disappearance oscillation channels. Using data sets of 6.6e20 protons on target (P.O.T.) in the LAr1-ND and ICARUS T600 detectors plus 13.2e20 P.O.T. in the MicroBooNE detector, we estimate that a search for muon neutrino to electron neutrino appearance can be performed with ~5 sigma sensitivity for the LSND allowed (99% C.L.) parameter region. In this proposal for the SBN Program, we describe the physics analysis, the conceptual design of the LAr1-ND detector, the design and refurbishment of the T600 detector, the necessary infrastructure required to execute the program, and a possible reconfiguration of the BNB target and horn system to improve its performance for oscillation searches.

A Proposal for a Three Detector Short-Baseline Neutrino Oscillation Program in the Fermilab Booster Neutrino Beam

The increasing gap between the growth of datacenter traffic and electrical switch capacity is expected to worsen due to the slowdown of Moore's law, motivating the need for a new switching technology for the post-Moore's law era that can meet the increasingly stringent requirements of hardware-driven cloud workloads. We propose Sirius, an optically-switched network for datacenters providing the abstraction of a single, high-radix switch that can connect thousands of nodes---racks or servers---in a datacenter while achieving nanosecond-granularity reconfiguration. At its core, Sirius uses a combination of tunable lasers and simple, passive gratings that route light based on its wavelength. Sirius' switching technology and topology is tightly codesigned with its routing and scheduling and with novel congestion-control and time-synchronization mechanisms to achieve a scalable yet flat network that can offer high bandwidth and very low end-to-end latency. Through a small-scale prototype using a custom tunable laser chip that can tune in less than 912 ps, we demonstrate 3.84 ns end-to-end reconfiguration atop 50 Gbps channels. Through large-scale simulations, we show that Sirius can approximate the performance of an ideal, electrically-switched non-blocking network with up to 74-77% lower power.

Sirius: A Flat Datacenter Network with Nanosecond Optical Switching

Clock synchronization is a prerequisite for the realization of emerging applications in various domains such as industrial automation and the intelligent power grid. This paper surveys the standardized protocols and technologies for providing synchronization of devices connected by packet-switched networks. A review of synchronization impairments and the state-of-the-art mechanisms to improve the synchronization accuracy is then presented. Providing microsecond to sub-microsecond synchronization accuracy under the presence of asymmetric delays in a cost-effective manner is a challenging problem, and still an open issue in many application scenarios. Further, security is of significant importance for systems where timing is critical. The security threats and solutions to protect exchanged synchronization messages are also discussed.

/pdf/a-survey-of-clock-synchronization-over-packet-switched-482sqjvsqc.pdf

A Survey of Clock Synchronization Over Packet-Switched Networks

The application of White Rabbit precision time protocol (WR-PTP) in long-distance optical fiber links has been investigated. WR-PTP is an implementation of PTP in synchronous Ethernet optical fiber networks, originally intended for synchronization of equipment within a range of 10 km. This paper discusses the results and limitations of two implementations of WR-PTP in the existing communication fiber networks. A 950-km WR-PTP link was realized using unidirectional paths in a fiber pair between Espoo and Kajaani, Finland. The time transfer on this link was compared (after initial calibration) against a clock comparison by GPS precise point positioning (PPP). The agreement between the two methods remained within ${\pm }{2}\; \text{ns}$ over three months of measurements. Another WR-PTP implementation was realized between Delft and Amsterdam, the Netherlands, by cascading two links of 137 km each. In this case, the WR links were realized as bidirectional paths in single fibers. The measured time offset between the starting and end points of the link was within 5 ns with an uncertainty of 8 ns, mainly due to the estimated delay asymmetry caused by chromatic dispersion.

White Rabbit Precision Time Protocol on Long-Distance Fiber Links

This article describes time distribution in a White Rabbit Network. We start by presenting a short overview of the White Rabbit project explaining its requirements to highlight the importance of the timing aspects of the system. We then introduce the technologies used to achieve high clock accuracy, stability and resilience in all the components of the network. In particular, the choice of the IEEE 1588-2008 (PTP) and Synchronous Ethernet standards are explained. In order to accommodate hardware-supported mechanisms to increase PTP synchronization accuracy, we introduce the White Rabbit extension to PTP (WRPTP). The hardware used to supportWRPTP is presented.Measured results of WRPTP performance demonstrate sub-nanosecond accuracy over a 5km fiber optic link with a precision below 10ps and a reduced PTP-message exchange rate. Tests of the implementation show full compatibility with existing PTP gear.

White rabbit: a PTP application for robust sub-nanosecond synchronization

This paper describes the long-term performance of White Rabbit (WR) based time and frequency transfer in the systems deployed at CERN and Gran Sasso National Laboratory. WR is a new technology based on IEEE 1588-2008 and Synchronous Ethernet which allows for sub-nanosecond accuracy and picosecond precision of synchronization in an Ethernet-based network. The first installation of WR is used in the CERN Neutrino to Gran Sasso (CNGS) project to transfer the Coordinated Universal Time (UTC) from a Global Positioning System (GPS) receiver to the underground extraction/detection points. The data collected during the system operation is used to evaluate its performance. Additionally, the performance in varying temperature conditions is verified with tests in a climatic chamber. We evaluate time transfer by measuring the offset between the time reference and the time receiver (WR Node). The stability of the transfered frequency is evaluated by analyzing the Allan Deviation (ADEV) and the Maximum Time Interval Error (MTIE).

Performance results of the first White Rabbit installation for CNGS time transfer

This paper investigates the ultimate limits of White Rabbit (WR), an high-accuracy time distribution system based on field-programmable gate array (FPGA). The knowledge of such limits is essential for new emerging applications that are evaluating WR. In this paper, we identify and study the key elements in the WR synchronization: the digital dual mixer time difference phase detector and the Gigabit Ethernet transceiver. The benchmarks and experimental analysis of these key elements allow us to determine the WR switch (WRS) performance limits and evaluate their evolution with newer FPGAs. The identified performance limits are achievable by the present-day generation of WRS. The ultimate limits of short-term synchronization performance due to FPGA implementation have been derived through analysis and then demonstrated using the existing WRS enhanced with an additional daughterboard. This combination (WRS and daughterboard) achieves a tenfold improvement in terms of phase noise, jitter, and short-term stability with respect to the current WR performance. Both phase detectors and Gigabit transceivers have a similar phase noise contribution equal to a short-term stability of modified Allan deviation 4E−13 at $\tau = 1$ s (dominated by flicker phase modulation noise).

White Rabbit Clock Synchronization: Ultimate Limits on Close-In Phase Noise and Short-Term Stability Due to FPGA Implementation

White Rabbit (WR) extends the Precision Time Protocol (PTP) to provide synchronisation with sub-nanosecond accuracy and sub-50 picoseconds precision. The protocol aspects of the WR extension are currently studied and integrated into the upcoming revision of PTP. In the context of this PTP revision, mechanisms are added to allow the control of the Layer 1 (L1) syntonisation by the PTP protocol. This article focuses on the frequency transfer characteristics of the L1 syntonisation in WR. We first explain the interaction between L1 syntonisation and PTP synchronisation in a WR device and describe the architecture of its Phase-Locked Loop (PLL). We then characterize the frequency transfer through a WR network in two ways: measuring the characteristics of the WR switch according to the Synchronous Ethernet (SyncE) metrics defined in ITU-T G.8262, and performing phase noise analysis. The results of the measurements allow us to propose improvements that might be useful for different types of WR applications. Metrology laboratories might be interested in the optimisations made to significantly reduce the phase noise. On the other hand, the telecom industry might be interested in the modifications that make the WR switch SyncE-compliant but deteriorate its performance. Notably, the latter was achieved merely by modifying the software that implements the WR PLL.

White rabbit clock characteristics

This article provides a non-exhaustive overview of applications and enhancements to the White Rabbit (WR) extension of the IEEE 1588 Precision Time Protocol (PTP). Initially developed to serve accelerators at the European Or-ganization for Nuclear Research (CERN), WR has become a widely-used synchronization solution in scientific installations. This article classifies WR applications into five types, briefly explains each and describes example its installations. The article then summarizes WR enhancements that have been triggered by different applications and outlines WR’s integration into the PTP standard. Based on the presented variety of WR applications and enhancements, it concludes that WR will continue to proliferate in scientific applications and should soon find its way into industry.

Maciej Lipinski

Papers

White rabbit: a PTP application for robust sub-nanosecond synchronization

Performance results of the first White Rabbit installation for CNGS time transfer

White Rabbit Clock Synchronization: Ultimate Limits on Close-In Phase Noise and Short-Term Stability Due to FPGA Implementation

White rabbit clock characteristics

White Rabbit Applications and Enhancements