2008-07-4liu,zhen'an, triggergroup,ihep1 i would like to thank prof. y. sakai and dr. y....

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 1

I would like to thank Prof. Y. Sakai and Dr. Y. Iwasaki for their kind help in BESIII trigger design, and I am happy that we have this chance to share our experience of BESIII in KEKB Trigger and DAQ design.

High Speed Signal/Data Transmissionin BESIII Trigger and PANDA TDAQ Systems

Zhen’An LIUTrigger Group, IHEP Beijing

2nd Open meeting for the KEKB proto-collaboration July 3-4th 2008

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 3

Outline Background infor for BESIII trigger Study of High speed signal transmiss

ion in BESIII trigger BESIII trigger PANDA TDAQ Computer Node Proof of Concept Application:

HADES DAQ Upgrade Comments and conclusion

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 4

Key points in BESIII trigger Design

Optical isolation with FEE, to prevent from ground loop current interference

Most latest FPGAs, Boards with simplicity, high reliability

FPGA online downloadable via VME Generalized hardware, firmware for differen

t function, for easy maintenance Scheme optimization with simulation

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 5

Difficulties for MDC tracking

Difficulties • Bad number of wires for both axial and stero layers (trigger point of view,

Hard to define Sector/board border for signal input) • Hard to input signals via 9U front and rare pannels • Too many sharing signals for neighbor boards

Solution: • RocketIO for input signals (32bits/ch, 8b/10b)• private VME J3 Backplane for sharing signals

1 2 3 4

SL 1 40 44 48 56 /16

SL 2 64 72 80 80 /16

SL 3 76 76 88 88 /8

SL 4 100 100 112 112 /16

SL 5 128 128 128 128 /16

SL10 128/256 128/256 128/256 128/256 /16

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 6

Study of High speed signal transmission in BESIII trigger

HFBR-5921L

Transmitter1

………

Fiber

Fiber

Fiber

Transmitter2

Transmitter16

HFBR-5921L

……

HFBR-5921L

HFBR-5921L

RocketIO

……

RocketIO

RocketIO

XC2VP50

RocketIOXC2VP2

Receiver

HFBR-5921LRocketIO

HFBR-5921LRocketIO

XC2VP2

XC2VP2

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 7

MFT (MDC Fiber Transmitter)

2796 hits signals from MDC QT boards are collected in MFT, 32 channels per MFT

Virtex-II Pro FPGA: XC2VP2

Fuctions:

• Stretching to 500ns

• Synchronization + alignment

signals(private Protocol)

• Serialization(8b/10b)

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 8

TKF (TracK Finder)XC2VP40 : FF1152, 804 user IOs, 43,632 logic cells, 3,456Kbit BRAM, 12 RocketIOs, 2 PowerPCs, 192 multiplier blocks

10 layers 9UVME PCB

XC2VP40HFBR5921L

Functions:• Deserialization• Channel alignment• TSF• TF• Track information

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 9

Clock correction

K codes• COMMA• IDLE

RXRECCLK RXUSRCLK

Bytes Allowed Between Clock Correction Sequences

REFCLKStability

Remove1IDLE Sequences: Remove2IDLE Sequences: Remove3IDLE Sequences: Remove4IDLE Sequences:

100 ppm 5,000 （√） 10,000 15,000 20,000

50 ppm 10,000 20,000 30,000 40,000

20 ppm 25,000 50,000 75,000 100,000

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 10

Data alignment Continuous Parallel bits transmitted in same clock =>

recovered also in same Clock Special private protocol

RocketIO

FIFO-TX

Protocol-TX

SYSCLKLocalCLK

MUX

PRBS

MUX

…Rocket

IOTriggerSignals

Protocol-RX

FIFO-RX

SYSCLKLocalCLK

TriggerSignals

Frst

PRBS Check

FrstTransmitter Receiver

RocketIO

FIFO-TX …

RocketIO

TriggerSignals

TriggerSignals

FIFO-RX

RocketIO

FIFO-TX …

RocketIO

TriggerSignals

TriggerSignals

FIFO-RX

Delay1 Delay2

Channel1

Channel2

m m+n

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 11

Clocks Crystal

• Must follow the recommendation

Use built-in DCM Clocks

• REFCLK• USRCLK• USRCLK2

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 12

Example of RocketIO Instantiation

RocketIO_1 : GT_CUSTOM generic map( CRC_END_OF_PKT => "K29_7", CRC_FORMAT => "USER_MODE", CRC_START_OF_PKT => "K27_7", TX_CRC_USE => TRUE, TX_DATA_WIDTH => 4, … ); port map ( REFCLK=>REFCLK_IN, TXUSRCLK=>TXUSRCLK_IN, TXUSRCLK2=>TXUSRCLK2_IN, TXCHARISK(3 downto 0)=>TXCHARISK_IN(3 downto 0), TXDATA(31 downto 0)=>TXDATA_IN(31 downto 0), TXFORCECRCERR=>TXFORCECRC_IN, TXN=>TXN_OUT, TXP=>TXP_OUT, … );

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 13

Transmission protocol Transmitter side Receiver side

CommaReset FIFO

Frst

ResetAlignment

Enable FIFO Writing

Enable FIFO Reading

CRC SOP

ILDEDisable FIFO

Reading

CRC EOP

Empty

Not Empty

1 2 3

4

56

7

8

Empty

Not Empty

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 14

Good reliability with RocketIO

BER ＝ 1

3.0×10e13 × 40

errors

periods × 40 < ≈ 8.3×10-16

Tx Data

Rx data

Rx random data

# of errors

Total periods

TD+TD-

RD+RD-

HFBR5921L

TDRD

MGT

TXPTXN

RXPRXN

TD+TD-

RD+RD-

HFBR5921L

RDTD

MGT

TXPTXN

RXPRXN

MFT TKF

TXDATA(31:0)

PRBSPRBSData Data

GenerateGenerate

FPGA

RXDATA(31:0)

Data Data AlignedAligned

& Checker& Checker

FPGA

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 15

MDC Sub-Trigger

MTB

MDC FiberMDC FiberTransmitterTransmitter

MDCMDCQ&TQ&T

TrackTrackFinderFinder

Inner- layer Inner- layer Track FinderTrack Finder

Long Track Long Track CounterCounter

Short Track Short Track CounterCounter

1.75Gb/s

×96 Optical

Channels

80 ch

16 ch

2

3

4

128

128

MDC Trigger CrateMDC Electronic Crates

×8 TKF

×1 ITKF

MD

C T

rigger Con

dition

sM

DC

Trigger C

ond

itions

9

TOGTL

×1 LTKC

×1 STKC

×96 MFT

ITKF TKF 108

1

1

1

8

1

96

# of boards

6

MTB

STKC

LTKC

TKF

ITKF

MFT

Board name

1MDC Trigger Backplane

1

124Total

11TracK Counter

8

11TracK Finder

11MDC Fiber Transmitter

FPGA

firmwareType of

PCB

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 16

BESIII trigger Block diagram

TCBA

Glo

bal T

rigg

er L

ogic

6.4 s

TOF FEE

MDC FEE

EMC FEE

MU FEE

TOFPR

MFT

Mu trackFastOR

Track Counter

Etotal Energy

Hit/Seg Count

TSF + TF

BEPCII RF TTC

TC Sum

L1P

41.65 MHz

Track Match

Energy Balance

Cluster Counting

Fast Control

FC Daughters

Near Detectors Counting Room

499.8 MHz

High Lights: Optical Isolation-no ground loop current

TKF

TKF

TKF

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 17

Installation

Jumper Box

MFTs in FEE crate

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 18

TOFPR

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 19

Installation

TCBA: EMC preprocessor and transmitter

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 20

Installation cont’d TDC

EMC/TOF/GTLTrigger

MDCTrigger

Optical Fibers

OpticalCables

CLK + FC

Opt-Cable under

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 21

Status of BESIII trigger Successful in Cosmic-ray test Commissioning with BEPCII

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 22

PANDA: DAQ Requirements Interactions: 10**7 Hz Data: 200GB/s Continuously Sampling ADC No hardware trigger

Hi Speed Devices Large Buffers Large Bandwidth

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 23

PANDA DAQ & Trigger

Detector’s Front-end

Concentrator Board

Switch

L1 farm

L1out

L2out

L3out

Switch

Switch

L2 farm

L3 farm

2 Alternative DAQ Concepts Still Under Discussion

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 24

The Compute Node(CN) 5 Virtex4 FX60 FPGA

• Large Computer Power

10 GB DDR2 RAM (2GB per FPGA)• Buffering capabilities

2 Embedded PowerPC in each FPGA

Slow control

32Gbit/s Bandwidth• 13x RocketIO to

backplane• 5x Gbit Ethernet Front

Pannel• 1x Gbit Ethernet

Backplane• 8x Optical Links

ATCA Compliant• Manageability

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 25

The Compute Node

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 26

Prototype of the CN by IHEP Beijing

Backplane FPGA #O FPGA #1-4 Front Panel

• Optical Links• Ethernet Plugs

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 27

Proof of Concept Application:HADES DAQ Upgrade

Est.: 2009 Read Out – Trigger

• TRB

• Faster Readout

Trigger and Data• Optical Links

Include Tracking in Trigger

12 Compute Node• 1 Full ATCA shelf

COMPUTE NODE Online Tracking RICH & TOF

• Matching with Tracking

Event Building on FPGA

Others• Remote Upgrade• IPMI

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 28

Motivation/Aim

Produce a Configurable and Scalable Hardware Platform for Multiple Applications & Experiments

• Capable of High Performance Computing• Large Bandwidth• Real Time processing

Trigger

• Flexibility: Reusable HADES - BESIII - PANDA

• Scalability Flexible network topology

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 29

Comments and Conclusion

RocketIO can be used for HS transmission at KEKB

Awareness to KEKB TDAQ design: • System clock• Larger trigger latency

Latency by RocketIO (SEDES takes time ) Needs larger pipeline buffer in FEE

• Synch. + alignment Uncertainty in recovered clock Opt-cable length difference Opt-transceiver difference

2008-07-4 LIU,Zhen'An, TriggerGroup,IHEP 30