Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

White Rabbit
Ethernet-based solution for sub-ns synchronization and
deterministic, reliable data delivery

Maciej Lipinski
on behalf of White Rabbit Team
Hardware and Timing Section
Institute of Electronic Systems

@
@

CERN
Warsaw University of Technology

15 July 2013
`
IEEE Plenary Meeting Geneve

Maciej Lipinski

White Rabbit

1/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Outline
1

Introduction

CERN Control & Timing

WR Network

Time Distribution
Timing demo

Data Distribution
Redundancy demo

WR @ CERN

Summary

Maciej Lipinski

White Rabbit

2/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Outline
1

Introduction

CERN Control & Timing

WR Network

Time Distribution
Timing demo

Data Distribution
Redundancy demo

WR @ CERN

Summary

Maciej Lipinski

White Rabbit

3/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

CERN

Maciej Lipinski

White Rabbit

4/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

CERN

Maciej Lipinski

White Rabbit

4/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

CERN

Maciej Lipinski

White Rabbit

4/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

CERN Accelerator Complex

Maciej Lipinski

White Rabbit

5/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

CERN Accelerator Complex

Maciej Lipinski

White Rabbit

5/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

CERN Accelerator Complex

Maciej Lipinski

White Rabbit

5/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

CERN Accelerator Complex

Maciej Lipinski

White Rabbit

5/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Beams Controls Hardware & Timing

Maciej Lipinski

White Rabbit

6/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Beams Controls Hardware & Timing

Maciej Lipinski

White Rabbit

6/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Beams Controls Hardware & Timing

Maciej Lipinski

White Rabbit

6/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Beams Controls Hardware & Timing

Maciej Lipinski

White Rabbit

6/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Beams Controls Hardware & Timing

Maciej Lipinski

White Rabbit

6/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

What is White Rabbit?

Renovation of accelerators control and timing

Based on well-known technologies
Open Hardware and Open Software
International collaboration

Maciej Lipinski

White Rabbit

7/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit features

standard-compatible
sub-ns accuracy
tens-ps precision
upper-bound low-latency
white-box simulation &
analysis
high reliability
tens-km span
thousands-nodes systems

Maciej Lipinski

White Rabbit

8/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit applications

Deployed for time distribution:
CERN Neutrinos to Gran Sasso

Maciej Lipinski

White Rabbit

9/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit applications

Deployed for time distribution:
CERN Neutrinos to Gran Sasso

Future applications:
CERN and GSI

Maciej Lipinski

White Rabbit

9/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit applications

Deployed for time distribution:
CERN Neutrinos to Gran Sasso

Future applications:
CERN and GSI
HiSCORE: Gamma&Cosmic-Ray
experiment (Tunka, Siberia)

Maciej Lipinski

White Rabbit

9/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit applications

Deployed for time distribution:
CERN Neutrinos to Gran Sasso

Future applications:
CERN and GSI
HiSCORE: Gamma&Cosmic-Ray
experiment (Tunka, Siberia)
The Large High Altitude Air Shower
Observatory (China)

Maciej Lipinski

White Rabbit

9/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit applications

Deployed for time distribution:
CERN Neutrinos to Gran Sasso

Future applications:
CERN and GSI
HiSCORE: Gamma&Cosmic-Ray
experiment (Tunka, Siberia)
The Large High Altitude Air Shower
Observatory (China)

Potential applications:
Cherenkov Telescope Array

Maciej Lipinski

White Rabbit

9/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit applications

Deployed for time distribution:
CERN Neutrinos to Gran Sasso

Future applications:
CERN and GSI
HiSCORE: Gamma&Cosmic-Ray
experiment (Tunka, Siberia)
The Large High Altitude Air Shower
Observatory (China)

Potential applications:
Cherenkov Telescope Array
International Thermonuclear
Experimental Reactor (ITER)

Maciej Lipinski

White Rabbit

9/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit applications

Deployed for time distribution:
CERN Neutrinos to Gran Sasso

Future applications:
CERN and GSI
HiSCORE: Gamma&Cosmic-Ray
experiment (Tunka, Siberia)
The Large High Altitude Air Shower
Observatory (China)

Potential applications:
Cherenkov Telescope Array
International Thermonuclear
Experimental Reactor (ITER)
European deep-sea research
infrastructure (KM3NET)

Maciej Lipinski

White Rabbit

9/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Outline
1

Introduction

CERN Control & Timing

WR Network

Time Distribution
Timing demo

Data Distribution
Redundancy demo

WR @ CERN

Summary

Maciej Lipinski

White Rabbit

10/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

CERN Control and Timing System

6 accelerators
including LHC: 27km
A huge real-time
distributed system
Thousands of devices

Maciej Lipinski

White Rabbit

11/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Controlling accelerators

Maciej Lipinski

White Rabbit

12/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Controlling accelerators

Maciej Lipinski

White Rabbit

12/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

CERN Control System event distribution (1)

Events messages which trigger actions

Each event is identified by an ID

Maciej Lipinski

White Rabbit

13/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

CERN Control System event distribution (2)

Devices are subscribed to events

Each device knows what to do on a particular event

Maciej Lipinski

White Rabbit

14/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

CERN Control System event distribution (3)

Each event (ID) has a trigger time associated

A set of events is sent as a single Control Message (CM)
CM is broadcast to all the end devices (nodes)
CM is sent in advance (upper-bound message latency)

Maciej Lipinski

White Rabbit

15/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

CERN Control & Timing Network requirements

Maciej Lipinski

White Rabbit

16/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit Network Ethernet-based

Standard Ethernet network

Few thousands nodes
Bandwidth: 1 Gbps
WR Switch: 18 ports
Non-WR Devices
Ethernet features (VLAN)
& protocols (SNMP)

Maciej Lipinski

White Rabbit

17/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit Network Ethernet-based

High accuracy/precision
synchronization
Deterministic, reliable and
low-latency Control Data
delivery

Maciej Lipinski

White Rabbit

18/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

White Rabbit Switch

Central element of WR network

Original design optimized for timing, designed from scratch
18 ports
1000BASE-BX10 SFPs: up to 10 km, single-mode fiber
Open design (H/W and S/W)

Maciej Lipinski

White Rabbit

19/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

White Rabbit Node

Ethernet MAC with White Rabbit
Open IP Core
Easily integrated into custom FPGA-based designs

Maciej Lipinski

White Rabbit

20/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

White Rabbit Node

Ethernet MAC with White Rabbit
Open IP Core
Easily integrated into custom FPGA-based designs

WR Node: universal carrier board

Maciej Lipinski

White Rabbit

20/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Outline
1

Introduction

CERN Control & Timing

WR Network

Time Distribution
Timing demo

Data Distribution
Redundancy demo

WR @ CERN

Summary

Maciej Lipinski

White Rabbit

21/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Time Distribution in White Rabbit Network

Synchronization with sub-ns accuracy tens-ps precision

Combination of
Precision Time Protocol (IEEE1588) synchronization
Layer 1 syntonization
Digital Dual-Mixer Time Difference (DDMTD) phase
detection

Maciej Lipinski

White Rabbit

22/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Precision Time Protocol (IEEE1588)

Simple calculations:
3 t2 )
link delayms : ms = (t4 t1 )(t
2
clock offsetms = t2 t1 + ms

Assumes medium symmetry

Maciej Lipinski

White Rabbit

23/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Precision Time Protocol (IEEE1588)

Simple calculations:
3 t2 )
link delayms : ms = (t4 t1 )(t
2
clock offsetms = t2 t1 + ms

Assumes medium symmetry

Disadvantages
all nodes have free-running oscillators
frequency drift compensation vs.
message exchange traffic

Maciej Lipinski

White Rabbit

23/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Layer 1 Syntonization

Maciej Lipinski

White Rabbit

24/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Phase Tracking (DDMTD)

Monitor phase of bounced-back clock
Enhance PTP timestamps with phase measurement
Phase-locked loop in the slave follows the phase changes

Maciej Lipinski

White Rabbit

25/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Link Delay Model

Maciej Lipinski

White Rabbit

26/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Link Delay Model

Maciej Lipinski

White Rabbit

26/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Link Delay Model

delayms = txm + ms + rxs
delaysm = txs + sm + rxm
fixed delays

variable delays

fixed delays

master

slave
t1

txm

ms

rxs

t2

t4

rxm

sm

txs

t3

Maciej Lipinski

White Rabbit

27/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Link Delay Model

delayms = txm + ms + rxs
delaysm = txs + sm + rxm
fixed delays

variable delays

fixed delays

master

slave
t1

txm

ms

rxs

t2

t4

rxm

sm

txs

t3

Relative Delay Coefficient ()

for 1000base-X over a Single-mode
Optical Fibre

Maciej Lipinski

White Rabbit

ms = (1 + ) sm
27/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Link Delay Model: fiber optic solution

Solution for Ethernet

over a Single-mode
Optical Fiber

asymmetry = txm +rxs

Maciej Lipinski

White Rabbit

28/59

+
2+

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit extension to PTP

White Rabbit requires:
WR-specific states
Exchange of WR-specific information

Maciej Lipinski

White Rabbit

29/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit extension to PTP

White Rabbit requires:
WR-specific states
Exchange of WR-specific information

White Rabbit estimates link asymmetry

Maciej Lipinski

White Rabbit

29/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit extension to PTP

White Rabbit requires:
WR-specific states
Exchange of WR-specific information

White Rabbit estimates link asymmetry

WR PTP

Maciej Lipinski

White Rabbit

29/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit extension to PTP

White Rabbit requires:
WR-specific states
Exchange of WR-specific information

White Rabbit estimates link asymmetry

WR PTP
PTP extensions mechanisms
Enhanced precision t1 , t2 , t3 , t4
Correction for asymmetry
Interoperability with PTP gear

Maciej Lipinski

White Rabbit

29/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit extension to PTP

White Rabbit requires:
WR-specific states
Exchange of WR-specific information

White Rabbit estimates link asymmetry

WR PTP
PTP extensions mechanisms
Enhanced precision t1 , t2 , t3 , t4
Correction for asymmetry
Interoperability with PTP gear

ISPCS Plug Fest

WR: most accurate PTP implementation
in the world!

Maciej Lipinski

White Rabbit

29/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR Standardization under IEEE1588

We want to standardize!

Maciej Lipinski

White Rabbit

30/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR Standardization under IEEE1588

We want to standardize!
Intention by p1588 SG
expressed in PAR

Maciej Lipinski

White Rabbit

30/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR Standardization under IEEE1588

We want to standardize!
Intention by p1588 SG
expressed in PAR
Enhanced Accuracy
Options / Profile

Maciej Lipinski

White Rabbit

30/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR synchronization performance

Maciej Lipinski

White Rabbit

31/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR synchronization performance

Maciej Lipinski

White Rabbit

31/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR synchronization performance

Maciej Lipinski

White Rabbit

31/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR synchronization performance

Maciej Lipinski

White Rabbit

31/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR synchronization performance

Maciej Lipinski

White Rabbit

31/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR synchronization performance

Maciej Lipinski

White Rabbit

31/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR synchronization performance

Maciej Lipinski

White Rabbit

31/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR synchronization performance

Maciej Lipinski

White Rabbit

31/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR synchronization performance

Maciej Lipinski

White Rabbit

31/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Timing demo

Maciej Lipinski

White Rabbit

32/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Outline
1

Introduction

CERN Control & Timing

WR Network

Time Distribution
Timing demo

Data Distribution
Redundancy demo

WR @ CERN

Summary

Maciej Lipinski

White Rabbit

33/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Data Distribution in a White Rabbit Network

Maciej Lipinski

White Rabbit

34/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Determinism and Latency (Switch)

Maciej Lipinski

White Rabbit

35/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

High Priority
Types of data distinguished by
802.1Q tag:
High Priority (strict priority)
Standard Data (Best Effort)

High Priority characteristics:

Broadcast/Multicast
Low-latency
Deterministic
Uni-directional
Re-transmission excluded

Failure of High Priority:

Medium imperfection
Network element failure
Exceeded latency

Maciej Lipinski

White Rabbit

36/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Determinism and Latency

Deterministic Latency of High Priority

By design: < 10us
(single source of High Priority)
All size of frames
All rates
Regardless of Best Effort traffic

Preliminary tests: 3us

Maciej Lipinski

White Rabbit

37/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Data Resilience (Node)

Maciej Lipinski

White Rabbit

38/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Data Redundancy

Forward Error Correction (FEC) transparent layer:

One message encoded into N Ethernet frames
Recovery of message from any M (M<N) frames

Maciej Lipinski

White Rabbit

39/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Data Redundancy

Forward Error Correction (FEC) transparent layer:

One message encoded into N Ethernet frames
Recovery of message from any M (M<N) frames

FEC can prevent data loss due to:

Maciej Lipinski

White Rabbit

39/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Data Redundancy

Forward Error Correction (FEC) transparent layer:

One message encoded into N Ethernet frames
Recovery of message from any M (M<N) frames

FEC can prevent data loss due to:

bit error

Maciej Lipinski

White Rabbit

39/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Data Redundancy

Forward Error Correction (FEC) transparent layer:

One message encoded into N Ethernet frames
Recovery of message from any M (M<N) frames

FEC can prevent data loss due to:

bit error
network reconfiguration

Maciej Lipinski

White Rabbit

39/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Topology Redundancy (Switch)

Maciej Lipinski

White Rabbit

40/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Topology Redundancy (Switch)

Ideas:
Enhanced Link Aggregation Control Protocol (eLACP)
WR Rapid Spanning Tree Protocol (WR RSTP)
WR Shortest Path Bridging (WR SPB)

Seamless redundancy = FEC + WR RSTP/SPB/eLACP

Redundant data received in end stations
Take advantage of broadcast/multicast characteristic of
Control Data traffic (within VLAN)

Maciej Lipinski

White Rabbit

41/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Topology Redundancy: eLACP (short explanation)

Control Message encoded into 4 Ethernet Frames
(F1,F2,F3,F4). Reception of any two enables to recover Control
Message (Cesar Prados, GSI).

Courtesy of Cesar Prados

Maciej Lipinski

White Rabbit

42/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Topology Redundancy: WR RSTP

Speed up RSTP max 2 frames lost on

re-configuration
H/W switch-over to the backup link
RSTPs a priori information
(alternate/backup) used
Limited number of allowed topologies
Drop only on reception within VLAN

Maciej Lipinski

White Rabbit

43/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Topology Redundancy: WR SPB

Shortest Path Bridging VID (SPBV)

Backup ports blocking on reception
Single port forwarding from source
H/W switch-over to path equally or
more distant to the root

Maciej Lipinski

White Rabbit

44/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Topology Redundancy: WR SPB

Shortest Path Bridging VID (SPBV)

Backup ports blocking on reception
Single port forwarding from source
H/W switch-over to path equally or
more distant to the root
Not fully congruent

Maciej Lipinski

White Rabbit

44/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Topology Redundancy: WR SPB

Shortest Path Bridging VID (SPBV)

Backup ports blocking on reception
Single port forwarding from source
H/W switch-over to path equally or
more distant to the root
Not fully congruent
New link metrics: link delay

Maciej Lipinski

White Rabbit

44/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Topology Resolution Unit (TRU)

Configurable module to support various software protocols
Accepts active and backup port masks (ingress and egress)
Monitors and controls ports state
Takes actions on HW-filtered frames and link-down
Triggers hardware generation of frames

Maciej Lipinski

White Rabbit

45/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Topology Resolution Unit (TRU)

Marker-based hardware-switch-over
Hardware-generated priority-based PAUSE
Hardware-generated BPDUs
Hardware-detection of BPDUs to open
blocking (pre-configured) port

Maciej Lipinski

White Rabbit

46/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Other features/ideas

Semi-automatic reconfiguration

Maciej Lipinski

White Rabbit

47/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Other features/ideas

Semi-automatic reconfiguration
Time-triggered reconfiguration

Maciej Lipinski

White Rabbit

47/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Other features/ideas

Semi-automatic reconfiguration
Time-triggered reconfiguration
Time-aware shaper

Maciej Lipinski

White Rabbit

47/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Other features/ideas

Semi-automatic reconfiguration
Time-triggered reconfiguration
Time-aware shaper
Drop non-High Priority frames
when High Priority arrives

Maciej Lipinski

White Rabbit

47/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit and IEEE 802

We want to be standard-compatible!

Maciej Lipinski

White Rabbit

48/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

White Rabbit and IEEE 802

We want to be standard-compatible!
Ideas in line with Time Sensitive Networks

Maciej Lipinski

White Rabbit

48/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

White Rabbit and IEEE 802

We want to be standard-compatible!
Ideas in line with Time Sensitive Networks
Great potential for collaboration between CERN and IEEE

Maciej Lipinski

White Rabbit

48/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

White Rabbit and IEEE 802

We want to be standard-compatible!
Ideas in line with Time Sensitive Networks
Great potential for collaboration between CERN and IEEE
Perfect platform for prototyping

Maciej Lipinski

White Rabbit

48/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Topology reconfiguration performance

Maciej Lipinski

White Rabbit

49/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Topology reconfiguration performance

Maciej Lipinski

White Rabbit

49/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Redundancy demo

Maciej Lipinski

White Rabbit

50/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Outline
1

Introduction

CERN Control & Timing

WR Network

Time Distribution
Timing demo

Data Distribution
Redundancy demo

WR @ CERN

Summary

Maciej Lipinski

White Rabbit

51/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

WR-based Control and Timing System (concept)

4 accelerator networks
Separate Data Master (DM) for each network
LIC Data Master communicates with other DMs and
control devices in their networks
Broadcast/multicast of Control Messages

Maciej Lipinski

White Rabbit

52/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

WR-based Control and Timing System (concept)

Maciej Lipinski

White Rabbit

53/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

WR-based Control and Timing System (concept)

Maciej Lipinski

White Rabbit

53/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Accelerator Networks

Maciej Lipinski

White Rabbit

54/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Traffic distribution: VLANs + multicast

Maciej Lipinski

White Rabbit

55/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Outline
1

Introduction

CERN Control & Timing

WR Network

Time Distribution
Timing demo

Data Distribution
Redundancy demo

WR @ CERN

Summary

Maciej Lipinski

White Rabbit

56/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

White Rabbit Family

Successful international collaboration of institutes, universities
and companies

WR Users:
http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers

Maciej Lipinski

White Rabbit

57/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

White Rabbit Family

Successful international collaboration of institutes, universities
and companies

WR Users:
http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers

Maciej Lipinski

White Rabbit

57/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Pushing frontiers

Scientific, open (H/W & S/W), with companies

Maciej Lipinski

White Rabbit

58/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

Pushing frontiers

Scientific, open (H/W & S/W), with companies

More applications than ever expected

Maciej Lipinski

White Rabbit

58/59

WR @ CERN Summary

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Pushing frontiers

Scientific, open (H/W & S/W), with companies

More applications than ever expected
A versatile solution for general control and data acquisition

Maciej Lipinski

White Rabbit

58/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Pushing frontiers

Scientific, open (H/W & S/W), with companies

More applications than ever expected
A versatile solution for general control and data acquisition
Fulfilling all our needs in synchronization and determinism

Maciej Lipinski

White Rabbit

58/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Pushing frontiers

Scientific, open (H/W & S/W), with companies

More applications than ever expected
A versatile solution for general control and data acquisition
Fulfilling all our needs in synchronization and determinism
Standard-compatible and standard-extending

Maciej Lipinski

White Rabbit

58/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Pushing frontiers

Scientific, open (H/W & S/W), with companies

More applications than ever expected
A versatile solution for general control and data acquisition
Fulfilling all our needs in synchronization and determinism
Standard-compatible and standard-extending
Active participation in IEEE1588 revision process

Maciej Lipinski

White Rabbit

58/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Pushing frontiers

Scientific, open (H/W & S/W), with companies

More applications than ever expected
A versatile solution for general control and data acquisition
Fulfilling all our needs in synchronization and determinism
Standard-compatible and standard-extending
Active participation in IEEE1588 revision process
Eager to collaborate with IEEE802

Maciej Lipinski

White Rabbit

58/59

Introduction CERN Control & Timing

WR Network Time Distribution

Data Distribution

WR @ CERN Summary

Thank you

More information:
http://www.ohwr.org/projects/white-rabbit/wiki

Maciej Lipinski

White Rabbit

59/59

Backup slides

Fixed Delays Measurement

110000011111000001111100
Buffer

rx
Phase
detector
RxCLK

Rx
Tx

PHY

TxCLK

125 MHz
Phase
detector

Buffer

Maciej Lipinski

tx
White Rabbit

60/59

Backup slides

WR RSTP: adding new network element

Maciej Lipinski

White Rabbit

61/59

Backup slides

Topology Resolution Unit (TRU)

Maciej Lipinski

White Rabbit

62/59

Backup slides

WR RSTP + FEC

Maciej Lipinski

White Rabbit

63/59

Backup slides

Digital Dual Mixer Time Domain (DMTD) phase

detector

Fully digital, so fully linear

Can handle multiple channels without need for extra
hardware

Maciej Lipinski

White Rabbit

64/59

Backup slides

New time transfer with WR for CNGS

Maciej Lipinski

White Rabbit

65/59

Backup slides

WR installation for CNGS

Grandmaster WR Switch
8 km of fiber between switches
Boundary Clock WR Switch
WR Node includes Time-to-Digital
Converter (TDC):
55 ps precision (std. dev)
300 ps accuracy

Performance monitoring

Maciej Lipinski

White Rabbit

66/59

Backup slides

Temperature tests setup (1)

Measurement of WR Timebase (clock)

Skew measurement with oscilloscope

Maciej Lipinski

White Rabbit

67/59

Backup slides

Temperature tests setup (2)

Maciej Lipinski

White Rabbit

68/59

Backup slides

Temperature tests results (1)

Maciej Lipinski

White Rabbit

69/59

Backup slides

Temperature tests results (2)

The change of time offset

due to temperature changes
4 ps per 1 C

Maciej Lipinski

White Rabbit

70/59