CBM at FAIR

Walter F.J. Müller, GSI, Darmstadt

XXXVI. Treffen "Kernphysik", Schleching/Obb., 17-24 February 2005

Vortrag zum Thema: Hochleistungsdatenverarbeitung in der

Kern- und Teilchenphysik

New challenges for Front-End Electronics, Data Acquisition and Trigger Systems

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

2

Outline

CBM (very briefly) observables setup

FEE/DAQ/Trigger requirements challenges strategies

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

3

CBM at FAIRSIS 100 Tm

SIS 300 Tm

U: 35 AGeV

p: 90 GeV

Compressed Baryonic MatterExperiment

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

4

Mapping the QCD Phase Diagram

RICH/LHC: explore μB → 0 travel back to the early

universe

CBM explore μB → max travel into a neutron star 10 – 45 AGeV 2nd generation

experiment penetrating probes rare probes

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

5

CBM Physics Topics and Observables In-medium modifications of hadrons

onset of chiral symmetry restoration at high ρB

measure: , , e+e- (μ+ μ-) open charm: D0, D±

Strangeness in matter enhanced strangeness production measure: K, , , ,

Indications for deconfinement at high ρB

anomalous charmonium suppression ? measure: D0, D±

J/ e+e- (μ+ μ-) Critical point

event-by-event fluctuations

measure: π, K

Good e/π separation

Low cross sections→ High interaction rates→ Selective Triggers

Hadron identification

Vertex detector

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

6

CBM Setup

Radiation hard Silicon pixel/strip detectors in a magnetic dipole field

Electron detectors: RICH & TRD & ECAL: pion suppression up to 105

Hadron identification: RPC, RICH

Measurement of photons, π0, η, and muons: ECAL

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

7

CBM and HADESAll you want to know about CBM:Technical Status Report (400 p)

now publicly available

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

8

Meson Production in central Au+AuW. Cassing, E. Bratkovskaya, A. Sibirtsev, Nucl. Phys. A 691 (2001) 745

10 MHz interaction rateneeded for 10-15 A GeV

SIS300

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

9

Open Charm Detection Example: D0 K-+ (3.9%; c = 124.4 m) reconstruct tracks find primary vertex find displaced tracks find secondary vertex

target

first two planesof vertex detector

few 100 μm

5 cm

high selectivity because combinatorics is reduced

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

10

A Typical Au+Au Collision

Central Au+Au collision at 25 AGeV:URQMD + GEANT

160 p 170 n360 - 330 + 360 0 41 K+ 13 K- 42 K0

107 Au+Au interactions/sec

109 tracks/sec to reconstruct for first level event selection

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

11

CBM Trigger Requirements In-medium modifications of hadrons

onset of chiral symmetry restoration at high ρB

measure: , , e+e-

open charm (D0, D±) Strangeness in matter

enhanced strangeness production measure: K, , , ,

Indications for deconfinement at high ρB

anomalous charmonium suppression ? measure: D0, D± -

J/ e+e Critical point

event-by-event fluctuations

measure: π, K

offline

assume archive rate:few GB/sec20 kevents/sec

offline

offline

trigger

trigger

trigger trigger on high pt e+ - e- pair

trigger ondisplaced vertex

drives FEE/DAQarchitecture

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

12

CBM DAQ Requirements Profile

D and J/Ψ signal drives the rate capability requirements D signal drives FEE and DAQ/Trigger requirements

Problem similar to B detection, like in BTeV, LHCb Adopted approach:

displaced vertex 'trigger' in first level, like in BTeV Additional Problem:

DC beam → interactions at random times

→ time stamps with ns precision needed

→ explicit event association needed Current design for FEE and DAQ/Trigger:

Self-triggered FEE Data-push architecture

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

13

Conventional FEE-DAQ-Trigger Layout Detector

Cave

Shack

FEE

Buffer

L2 Trigger L1 Trigger

DAQ

L1 A

ccep

t

L0 Trigger

fbunch

Archive

Trigger

Primitives

Especially

instrumented

detectors

Dedicated

connections

Specialized

trigger

hardware

Limited

capacity

Limited

L1 trigger

latency

Modest

bandwidth

Standard

hardware

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

14

Limits of Conventional Architecture

Decision time for first level trigger limited.

typ. max. latency 4 μs for LHC

Only especially instrumenteddetectors can contribute to

first level trigger

Large variety of veryspecific trigger hardware

Not suitable for complexglobal triggers like secondary

vertex search

Limits future triggerdevelopment

High development cost

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

15

L1 Trigger

Special

hardware

High

bandwidth

The way out .. use Data Push ArchitectureDetector

Cave

Shack

FEE

Buffer

L2 Trigger L1 Trigger

DAQ

L1 A

ccep

t

L0 Trigger

fbunch

Archive

Trigger

Primitives

Especially

instrumented

detectors

Dedicated

connections

Specialized

trigger

hardware

Limited

capacity

Limited

L1 trigger

latency

Modest

bandwidth

Standard

hardware

fclock

L2 Trigger

Timedistribution

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

16

L1 Trigger

Special

hardware

High

bandwidth

The way out ... use Data Push ArchitectureDetector

Cave

Shack

FEE

DAQ

Archive

fclock

L2 Trigger

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

17

L1 Select

Special

hardware

High

bandwidth

The way out ... use Data Push Architecture Detector

Cave

Shack

FEE

DAQ

Archive

fclock

L2 Select

Self-triggered front-end

Autonomous hit detection

No dedicated trigger connectivity

All detectors can contribute to L1

Large buffer depth available

System is throughput-limited

and not latency-limitedModular design:

Few multi-purpose rather

many special-purpose

modulesUse term: Event Selection

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

18

CBM DAQ and Online Event Selection

More than 50% of total data volume relevant for first level event selection

Aim for simplicity

Simple two layer approach:1. event building2. event processing

neededfor D needed

for J/μ usefullfor J/μ

STS, TRD, and ECAL data usedin first level event selection

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

19

Logical Data Flow

Concentrators:multiplex channelsto high-speed links

Time distribution

Buffers

Build Network

Processing resources forfirst level event selectionstructured in small farms

Connection to'high level'

selection processing

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

20

Bandwidth Requirements

Data flow:

~ 1 TB/sec

1st level selection:

~ 1014-15 operation/sec

Data flow:

few 10 GB/sec

to archive: few 1 GB/sec

Moore helps

Gilder helps

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

21

L1 Event Selection Farm Layout

Current working hypothesis: CPU + FPGA hybrid system (proviso follows)

Use programmable logic for cores of algorithms Use CPU for the non-parallelizable parts Use serial connection fabric (links and switches) Modular design (only few board types)

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

22

FPGA – Basic Building Block

CLK

F3

F2

F1

F0XQ

X

D

C

Q

LUT

CLB

Look-up Tablejust a 4x1 RAM

D Flip-FlopUniversallogic gate

Elementarystorage unit

CLB = Configurable Logic Block

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

23

FPGA – Putting it together

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

CLB

PSM

PSM

PSM

PSM

PSM

PSM

PSM

PSM

PSM

ConfigurableLogic Block

Wiring

Programmableswitch matrix

I/O blocks

Modern FPGA's:>100.000 LUT 500 MHz

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

24

Algorithms

Performance of L1 feature extraction algorithms is essential critical in CBM: STS tracking + vertex reconstruction

TRD tracking and Pid

Look for algorithms which allow massive parallel implementation e.g. Hough Transform Tracker

needs lots of bit level operations, well suited for FPGA Caveat: simulation on normal CPU quite time consuming....

Co-develop tracking detectors and analysis algorithms L1 tracking is necessarily speed optimized

→ more detector granularity and redundancy needed Aim for CBM:

Validate final hardware design with at least 2 trackers suitable for L1

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

25

Interim Summary Event definition has changed:

now based on time stamps and time correlation

Role of DAQ has changed: DAQ is simply responsible to transport data from producers to consumers

Role of 'Trigger' has changed: filter events delivered by DAQ 'Online Event Selection' is better term

System aspects: 'online' – 'offline' boundary blurs more COTS (commercial off the shelf) components much more modular system much more adaptable system

This is emerging technology in HEP, though baseline for ILCHowever: being used since many years in nuclear structure

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

26

Moore – quo vadis ?

Will price/performance of computing continue to improve ? What are limits of CMOS technology ? Where are the markets ? What are market forces ?

Technology most of the gain comes from architecture anyway conventional designs, especially x86, reach their limits

Markets end of the metal-box PC age

→ Laptops + PDA + all kind of dedicated boxes (Video, Games) end of the binary compatibility age

→ intermediate code + 'Just in Time' Compilers (JIT)

There is life after Intel x86A lot of architectural innovation ahead

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

27

BlueGene vs Cell Processor

CPU

Cache

Mem

Mem

IO

IO

Mem SPEMemSPE

Mem SPEMemSPE

Mem SPEMemSPE

Mem SPEMemSPE

CPU CPU

Cache Cache

IO IO IO IO

Mem

BlueGene:121 mm2; 130 nm2.8/5.6 DP GFlop STI Cell:

221 mm2; 90 nm256 SP GFlop 30 DP GFlop 25 GB/sec mem 78 GB/sec IO

Finally presentedon ISSCC 2005

SPE = Synergistic Processing ElementInternational Solid-State Circuit Conf.

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

28

BlueGene vs Cell Processor

Developed bySony, Toshiba and IBMMarket: VIDEOGAMESBudget: 500 M$

Developed by IBMMarket: national security science Budget: ~100 M$

High performance computing is driven now by embedded systems(games, video, ....) → Science is a spin-off, at best ...

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

29

Game Processors as Supercomputers ?

Slide from CHEP'04 Dave McQueeney

IBM CTO US Federal

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

30

CPU and FPGA paradigms merge

Register

ALUCo

ntr

ol Wide Register

ALUCo

ntr

ol

ALU ALU ALU

Wide Register

ALU

Co

ntr

ol ALU ALU

ALU

ALU

ALU ALU ALU

ALU

ALU ALU

ALU ALU

ALU ALU

ALU ALU

ALU

Conventional CPU SIMD (single instruction – multiple data) CPU

Configurable Instruction Set CPU

arithmeticresources

configurableconnection

fabric

PSMPSMPSMPSMPSM

PSMPSM PSM PSM PSM

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

31

Configurable Instruction Set Processor

Example Stretch S5xxx Hybrid design:

conventional fixed instruction set part

plus configurable instruction set part

C/C++ compiler analyses the kernel of algorithms generates custom

instruction set generates code to use it

The promise easy of use of C/C++ performance of an FPGA

Fabric is the keyword

interconnected resources

Stretch S5 engine

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

32

CPU and FPGA paradigms merge

configurablelogic

CPU

configurablelogic

CPU CPU

FPGAindustry

world view

Processorindustry

world view

Unclear what the most suitablearchitecture will be

The general trend howeverwill produce a lot of

innovation in the years to come

Essential will be availability of efficient development tools

Moore will go on ! There are the technologies

There are the marketsArchitectural changes

ahead

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

33

Summary

Self-triggered FEE: autonomous hit detection, time-stamping with ns presision sparsification, hit buffering, high output bandwidth

High bandwidth event building network to cope with few 100 MHz interaction rate in p-p, p-A likely be done in time slices or event slices

L1 processor farm feasible with PC + FPGA + Moore (needed 2014) but look beyond todays PC's and FPGA's

Efficient algorithms (109 tracks/sec) co-design of critical detectors and tracking software

Quitedifferentfrom thecurrent

LHC styleelectronics

Substantial R&D

needed

RII3-CT-2004-506078

17-24 February 2005

XXXVI. Treffen Kernphysik, Schleching/Obb., Walter F.J. Müller, GSI

34

The End

Thanks for your attention

CBM Collaboration : 39+ institutions, 14+ countriesChina:

Hua-Zhong Univ., WuhanCroatia: RBI, ZagrebCyprus: Nikosia Univ. Czech Republic:Czech Acad. Science, RezTechn. Univ. Prague France: IReS Strasbourg

Germany: Univ. Heidelberg, Phys. Inst.Univ. HD, Kirchhoff Inst. Univ. FrankfurtUniv. KaiserslauternUniv. Mannheim Univ. MarburgUniv. MünsterFZ RossendorfGSI Darmstadt

Russia:CKBM, St. PetersburgIHEP ProtvinoINR TroitzkITEP MoscowKRI, St. PetersburgKurchatov Inst., MoscowLHE, JINR DubnaLPP, JINR DubnaLIT, JINR DubnaLTP, JINR DubnaMEPhi, MoskauObninsk State Univ.PNPI GatchinaSINP, Moscow State Univ. St. Petersburg Polytec. U.

Spain: Santiago de Compostela Uni.

Ukraine: Shevshenko Univ. , Kiev

Hungaria:KFKI BudapestEötvös Univ. Budapest

Korea:Korea Univ. SeoulPusan National Univ.

Norway:Univ. Bergen

Poland:Krakow Univ.Warsaw Univ.Silesia Univ. Katowice Portugal: LIP Coimbra

Romania: NIPNE Bucharest

membership applications in italic