TDAQ news and issues

M. Sozzi NA62 TDAQ WG meeting

CERN – 20/10/2010

1. TEL622. TDCB3. Clock4. L0 processor5. Interfacing to PCs6. Simulation

Outline

TEL62 - overview• Newer FPGAs and memories• Doubled bus bandwidth with SL-FPGA• Fewer I/O pins in new chosen PP-FPGA +

constraints for compatibility with LHCb cards =

After price confirmation for 500 FPGAs:increased size of PP-FPGA.

All 5 FPGAs on the TEL62 of the same type:EP3SL110F1152C4N (4x the original)

• Additional pins used for communication bus between PP-FPGAs (useful for LKr/L0)

• Internal note NA62-10-06 distributedComments from Lausanne group received

TELL1PP

PP

PP

PP

SL GbE

TEL62PP

PP

PP

PP

SL GbE

TEL62 - status• Some trouble with power distribution

Difficult to estimate power consumption before firmware is ready

• Design finished and start layout work at CERN at beginning of November

• Components for 2 prototypes being procured

• Production tests, firmware development, etc.?

TDCB – status and plans• Some delays in mounting

2 prototypes of V4 expected next week• Firmware debugging ongoing• Detailed documentation being written• Need to prepare test-bench system

• Preliminary tests in Pisa• Collect orders for first batch• First batch production (<16):

[these might be used in a synchronization run]• Tests and validation by sub-detector groups

TDCB – validation# boards

(first batch) Test where? Test who?

CEDAR

CHANTI

LAV

RICH Perugia M. Piccini

CHOD Perugia ? M. Piccini ?

STRAWS (?) 0

IRC/SAC

MUV

ComponentsNA62 TDAQ

Fallback

CEDAR GTK CHANTI LAV STRAWS RICH CHOD IRC/SAC LKR LKR/L0 MUV TDAQ TOTAL Spares STRAWS TOTAL SparesBirmingh

am Ferrara Napoli Pisa CERN Perugia Perugia ? Bulgaria CERN Roma2 Mainz Pisa CERN

Channels 128 200 2496x2 7360 2000 128 196 0 7360Channels/station 260x2 1840 1000 64 0 1840Particle rate 10 MHz 10 MHz 0Overall total rate 256 MHz 250 MHz 20 MHz 1 MHz 256 MHzFE-DAQ distance 5m <5m <6m <5m

TEL62 3 0 1 15 3 6 1 2 2 46 2 4 85 20 102Of which: spares 1 0 0 3 1 2 0 1 1 8 1 2 20 0.24 4 23 0.23Prototypes 0 0 0 0 0 0 0 0 0 1 0 1 2 0 2First batch 0 0 0 2 0 1 0 0 1 0 1 2 7 0 7Custom crates 1 0 1 0 1 1 1 1 2 0 1 1 10 0 9Of which: spares 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00Full TEL62 crates 0 0 0 14 0 0 0 0 0 3 0 0 17 5 22Of which: spares 0 0 0 1 0 0 0 0 0 1 0 0 2 0.12 1 3 0.14VME crates 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1Of which: spares 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00TDC boards 10 0 4 50 0 20 2 4 0 0 4 10 104 80 184Of which: spares 2 0 0 7 0 4 1 2 0 0 0 6 22 0.21 16 38 0.21(first batch) 2 0 0 2 0 2 0 1 0 0 1 6 14 4 18LTU 1 3 1 2 2 1 1 1 2 3 1 2 20 2 20Of which: spares 0 0 0 0 0 0 0 0 0 0 1 1 0.05 1 2 0.10TTCex 1 3 1 2 2 1 1 1 2 4 1 2 21 2 21Of which: spares 0 0 0 0 0 0 0 0 0 0 1 1 0.05 1 2 0.10# of lasers/module 2 3 10 10 4 10 2 10 2 6 10 4 73 4 73# of lasers 2 9 10 20 8 10 2 10 4 24 10 8 117 8 117TTCoc 0 0 0 2 0 0 0 0 0 3 0 1 6 0 6Of which: spares 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0.00Optical attenuators 3 1 1 2 20 6 1 2 2 2 2 4 46 20 46Of which: spares 0 0 0 0 0 0 0 0 0 0 2 2 0.04 0 2 0.04TTCrq 0 35 0 0 0 0 0 0 0 0 0 0 35 0 35Of which: spares 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Please check and update!

Clock - modules• 2 NA62 LTU prototypes available, ready for

testing at CERN (Birmingham)Then: full production (19+ modules), paid

• 19 new TTCex modules for NA62 in production• Need to finalize number of lasers: Birmingham

will distribute a final poll• Payment to NA62 account

• 5 old TTCex available in E-POOL

Clock - infrastructure• NA48 clock generator seems OK (spare?)• Organize central purchase of splitters,

attenuators, fibres (Birmingham)• Finalize “officially supported” VME CPU• Install central crate(s) + CPU• Sub-detectors: define TTC distribution network• Lay down fibres in ECN3

Burst sequence

• Start of burst signal (via TTC special command):reset timestamp counters

• Start of burst trigger (via TTC):can send data

• Normal triggers…send data

• End of burst signal (via TTC special command):store timestamp counter value

• End of burst trigger (via TTC):send end-of-burst data, stop data-taking

Sub-detectors only see 2 hardware signals: START OF BURST and END OF BURST

Calibration triggers• Sub-detector drives sub-detector calibration

system and sends signal pulse to L0TP• L0TP delivers (if possible, if allowed) calibration

trigger (after fixed, small delay)

• In-burst vs. Out-of-burst calibrations:L0TP knows when the beam really ends (EE), and can:- Deliver special control trigger (e.g. to instruct sub-detectors to change pulser parameters)- Modify acceptance rules for calibration triggers

This scheme avoids need for additional trigger and synchronization lines

Central (L0) Trigger System(logical)

CHODMUV

LKR/L0LAV

(RICH)

All s

ub-

dete

ctor

sPulser

L0 p

rim

itiv

es

(GbE

)Ch

oke/

Erro

r (L

VDS)

All s

ub-

dete

ctor

s

Cloc

k, L

0, B

urst

(T

TC)

Gen

SPS

Farm

Farm

Burst (GbE)

Dat

a (G

bE)

Burst

40MHzTime-

match L0

primitives,decide L0

and trigger type, check

latency, downscale, monitor

Monitor & record

choke/error,

send special triggers

Re-sync,

encode

Record L0

data

Generate special triggers

Gen

SPS

Farm

FarmBu

rst

(G

bE )

Dat

a (G

bE)

Burst

40MHzCHO

DMUV

LKR/L0LAV

(RICH)

All s

ub-

dete

ctor

s

Spare

L0 p

rim

itiv

es

(GbE

)Ch

oke/

Erro

r (L

VDS)

All s

ub-

dete

ctor

s

Cloc

k, L

0, B

urst

(T

TC)

Custom FPGA-basedboard

ORReal-time capable

PC

Re-

sync

hron

izer

PC (DIM)

PC interface

Central (L0) Trigger System(functional)

TTC

mod

ules

Pulser

Gen

SPS

Farm

FarmBu

rst

(G

bE )

Dat

a (G

bE)

Burst

40MHz

All s

ub-

dete

ctor

s

Cloc

k, L

0, B

urst

(T

TC)

Re-

sync

mem

ory

PC (DIM

)

GbE

RX

Corelogic

GbE

TX

Fan-out

L0

Fan-

out

CHODMUV

LKR/L0LAV

(RICH)

All s

ub-

dete

ctor

s

Spare

L0 p

rim

itiv

es

(GbE

)Ch

oke/

Erro

r (L

VDS)

Burs

t Fa

n-ou

t

PCIePC

Ie

Central (L0) Trigger System(implementation)

PCIe custo

m board

Custom

board or PC

LTU TTCex

Pulser

L0TP: Real-time tests (Ferrara)

PCI-express

FPGA

CPU

Involvement of very experienced group working on APE.Using a custom board with 2 Intel CPUs (standard Linux, so far) and FPGAs to test real-time response of a basic L0TP algorithm

L0TP: Real-time tests (Ferrara)

First naïve results: latency FPGA → CPU → MEM → FPGAwith 0 computation timeand standard Linux(depending on output buffering)

Now making more tests with more realistic time-matching algorithm

11μs

18μs

25μs

Further tests using standard PC and Pisa Altera Stratix IV PCIe development board (can such system be a L0TP?)Expect report at next meeting

L0 Trigger Processor

Rather limited progressDetailed descriptive note in preparationTemporary solution might be quite different from final system

Interfacing to PCsStarted development in Pisa of buffer manager code for readoutUser-space independent processesFixed-format shared memory

CTRL

PROD

PROD

PROD

PROD

CONS

CONS

CONS

CONS

GbE

GbE

GbE

GbEProcs

Sharedmemory

To be used for: L1 PCsMultiple processes either

running L1 (single sub-detector) algorithm or reading sub-detector data to L2 farm(on different event sets)

L1 algo

L1 algo

L1 algo

Readout

GbE L1 Trigger Processor PC

GbE L2 farm

Sub-detector

data

Sub-detector

data

L1primitives

Maybe also: software L0 ?Multiple GPUs running L0 primitive generation

(for some sub-detector).Algorithm time and latency not an issue.Raw data-transfer time not an issue.Control time?

From CPU thread without operating system?From hardware PCIe sequencer?

CondensedSub-

detector data

L0primitives

GPU

GPU

GPU

GPU

GbE L0 Trigger Processor

Online software / run control

• Existing solutions (DIM, SMI++) seem adequate• Possible help from CERN DCS group to port some of the above solutions to NA62• Need coordination and expertise within NA62• Write note describing NA62 requirements (volunteers willing to help?)

Towards a run controlCommon interface to run control:• Single machine per sub-system• Respond to the following commands:

- Enable/disable global control- Cold-start initialization- Start run <number> <global_conf> <local_conf>- End run- Query status- Reset

• No need of start/end burst (special triggers)

Simulation - Tools

Some work started (see talks).

- Do we have everything we need for trigger simulation in the official MC?Can it be ran in a fast enough mode?- Do we need a fast parameterized MC?

Simulation - Issues- Verification of L0 rates- How much below 1 MHz can we stay?- Effect of LAV?- Which control triggers can be allowed?- CHOD/RICH vs. MUV acceptance matching- Can CHOD be used in L0? Efficiency (accidentals)?- How much can RICH add? (At L1?)- How much can STRAWS contribute at L1?- What can be done realistically with LKr tiles at L0? Can something more be done with tiles at L1? Is some (summary) single-cell information useful at L1?- Most significant correlation cut at L2?Which sub-detectors are required at L2?With which resolution?

Synchronization run?TEL62 test and firmware• No help available within NA62

- Roma Tor Vergata cannot help in common TEL62 firmware development- Still missing sub-detector involvement in firmware

• Possibility of help from a PhD student in the STRAWS group

L0 Trigger Processor• Need commitments and tests