white rabbit @ gsi (bel, ee,

12-Sep-2011 Dietrich Beck, GSI

White Rabbit @ GSI (BEL, EE, …)

•  Motivation •  Who is White Rabbit? •  Associated: BuTiS •  White Rabbit for Experiments •  Status

Acknowledgements: WR Team @ CERN, BEL Timing Team @ GSI, EE (JH, NK,…), …


Motivation

•  Timing system of GSI accelerator based on 1MBit/s MIL 1553-B bus (+ 3 extra lines) è not suitable for FAIR

•  Development/application of a new timing system is required. •  The timing control / time stamping of FAIR experiments could

benefit from a new accelerator timing system. •  For me (db): Timing systems for control systems of ion-traps:

Timing is an interesting topic. •  è Project for/with the BEL group


Who is the White Rabbit (WR)?

“ “The White Rabbit (WR) project is a multi-laboratory and multi-company effort to bring together the best of the data

transfer and timing worlds …”

"Oh dear! Oh dear! I shall be too late!“ (Proc. ICALEPCS 2009) (Alice's Adventures in Wonderland)


Requirements to WR at GSI

•  ∼ 2000 timing receiver nodes on the GSI/FAIR campus •  ∼ 100 µs reaction time “granularity window” (e.g. interlocks) •  ∼ 10 µs propagation time of signals (∼ 2km) •  ∼ 1-2 ns precision (kicker control) •  ∼ 10-12(-15) robustness (1 event per year may be lost) •  1 GBit/s throughput •  Complements other components (FESA, …) •  WR is not intended for protection of machines or personnel •  Collaborative effort (CERN, GSI…)

–  avoiding isolated solutions –  involving industry


Solution Path for WR •  1 GigE, extended by •  IEEE 1588 (PTP protocol, time stamps, ∼ 1 µs precision), extended by •  WR-PTP

–  syntonization: adjust clock frequency of WR node to master (transceiver, FPGA) –  Asymmetric model for link delays (up/down) –  ∼ 8 ns jitter due to 125 MHz carrier of GigE

•  extended by SyncE –  synchronization: adjust phase of WR node to WR master –  sub-ns precision

•  Dedicated WR switches –  clock propagation (frequency, phase) –  real-time control messages + some low priority traffic –  guaranteed maximum latency for real-time control messages

•  Guaranteed maximum propagation time by network layout •  Timing Master: Broadcasts real-time control messages •  Message encoding using Forward Error Correction (FEC) •  WR initiated at CERN: Collaborative effort – GSI joins •  Open Hardware Repository (www.ohwr.org): Sharing effort and solutions


Timing Master

Control System (application layer)

LSA (settings management)

AP

I

Equipment 1 Equipment N

FES

A

WR

Equipment 2 Equipment 3

Beam Interlock System Post Mortem System

AP

I

BuTiS another link

Equipment 4

a link

Equipment ..

Common Systems

Alarm, Archive, Logging, Oscilloscope

AP

I

BuTiS

Disclaimer: This is my personal view and may be incorrect and not up-to-date


Timing Concept

Problem: Direct triggering of devices via a timing network is imprecise due to differences in propagation time (cable length) and “unknown” latency of WR switches (only upper bound is known), Δt ∼ 10 µs (not good enough).

Solution: Timing based on UTC timestamps: Jitter defined by precision of

distributed clocks, Δt < 1 ns (Intrinsic property of WR). •  Timing Master sorts/calculates data ∼ 100 µs ahead of requested

execution time: Timing messages (node x: do action y at UTC z) •  Timing messages are broadcasted considering max. propagation time. •  Timing Receiver (node x) receives message: do action y at UTC z •  If actual time (UTC) equals requested time UTC z: WR node x executes

action y autonomously.


WR versus BuTiS

WR: •  UTC time stamp distribution (sub ns precision) •  Clock distribution (8 ns granularity, sub ns jitter) •  Cheap (< 1 kEuro per node) BuTiS: •  No time stamp distribution •  Best clock distribution (14 ps granularity, accuracy 100 ps/km,

low fs jitter) •  Expensive (∼ 10 kEuro per receiver station)


BuTiS and WR

Timing Master

BuTiS Center Rb Normal 10 MHz

P0: 100 kH

z S

1: 10 MH

z

DWDM Ref. Synth. LP0: 100 kHz LS1: 10 MHz

Clock Master DDS

WR Master

WR Receiver DDS:T0

DWDM Ref. Synth. LP0: 100 kHz LS1: 10 MHz

(T0: 100 kHz) C2: 200MHz

UTC

delay correction T0/C

2 phase adjust

timing events

UTC

via PTP

UTC

/T0 phase …

T0: 100 kHz C2: 200MHz

Equipment

GPS GPS

WR Receiver DDS:T0

Equipment

WR Switch

backplane DDS: AFG


WR Hardware

•  WR multi-purpose carrier boards for PCIe, VME, (µTCA, PXI) •  SPEC (Simple Pci Express Carrier) board

•  I/O defined by FMC mezzanine boards •  At GSI: ∼ 1600 SCU (Standard Control Units) for FAIR

Combined with FPGA:

•  WR VHDL code

•  Soft-CPU (versatility, re-use existing open source C-libraries, fast reconfig.)

•  User VHDL code


Use Cases

•  Timing System for GSI/FAIR accelerator •  DAQ for GSI/FAIR accelerator – “Distributed Oscilloscope” •  … •  Time Stamping for triggered DAQ systems

–  Latch WR node via cable –  Read timestamp via host bus

•  Timestamp distribution for free running DAQ systems –  WR node spits out 100 kHz epoch markers plus encoded

timestamps on a serial line –  Fan-out from WR node to DAQ boards


Status

•  Presented on IEEE Real Time in Stockholm, 2005 •  … •  Establishment of WR link between two SPEC boards

demonstrated. Synchronization to about 100 ps. •  Functionality of WR switch demonstrated •  10 SPEC boards arrived at GSI •  2 WR switches arrived at GSI •  … •  8 FMC mezzanine boards developed and produced (JH et al.) •  10 PEXARIA2A boards – “development board for SCU” (JH),

converts to WR node, by adding two VCOs •  … •  Specification (Draft) for interface to experiments •  …


Further Reading…

•  www.ohwr.org •  https://www-acc.gsi.de/wiki/Timing/WebHome •  Talks by Tibor Fleck and Peter Moritz @ FAIR Technikforum

http://www-bd.gsi.de/dokuwiki/doku.php?id=meetings:fair-technikforum

white rabbit @ gsi (bel, ee,

Documents