sts readout chain – revisited readout-asic to roc interconnect walter f.j. müller, gsi, darmstadt...

STS Readout chain – STS Readout chain – RevisitedRevisited

Readout-ASIC to Readout-ASIC to ROC InterconnectROC Interconnect

Walter F.J. Müller, GSI, Darmstadt

CBM Collaboration Meeting13th April 2010

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 2

Slides fromSlides fromFEE/DAQFEE/DAQWorkshopWorkshop


The CBM Generic Read-out The CBM Generic Read-out ChainChainDetectorFront-End

BoardRead-OutController

Active BufferBoard

FEB ROC ABB

Data CombinerBoard

DCB

First LevelEvent Selector

FLES

TNet

Data

Control

Data &Control

Sync

Data

DigitizationCustom ASIC

FEE DAQInterfaceLocal Pre-Processing

BufferingSystem

Synchronization

DAQFLESInterface

FPGA Coprocessor

Event SelectionCPU Farm

DAQDetector Sub-System FLES


Options for CBM FLES LocationOptions for CBM FLES Location

CBM CBM ServiceServiceBuildingBuilding

~350m

linear dist.

Distance is Distance is preliminary!!preliminary!!

'Green Cube''Green Cube'


STS Hit Rates per 128 Channel STS Hit Rates per 128 Channel ChipChip

Hit rate [MHz]

Sta0

Sta 1

Sta 2

Sta 3

Sta 4

Sta 5

Sta 6

Sta 7

Total

32-64 21 14 9 1 0 0 0 0 4516-32 67 72 157 116 73 60 1 1 5478-16 208 357 477 555 325 430 157 189 26984-8 494 521 250 440 964 785 104

8753 5255

2-4 441 376 51 24 219 294 693 1079

3177

1-2 104 51 0 0 3 5 21 90 2740.5-1 1 1 0 0 0 4 0 0 6Total 1336 1392 944 1136 1584 157

8192

0211

21200

2 Very few chip in inner region have hit rate > 32 MHz >90% of chips have hit rates in 2...16 MHz range

Hit rate per chip StatisticsHit rate per chip StatisticsAu+Au @ 25 AGevAu+Au @ 25 AGev

101077 evt/sec evt/sec


STS FEB Data Aggregation STS FEB Data Aggregation CasesCases

FEBaFEBa88

FEBaFEBa44

FEBaFEBa22

FEBaFEBa11

2.5 Gbps serial link forFEB ROC connect

local chip chipdata interconnect


ProvisoProviso:link counts likely to go up to

account for more background,

overhead, head room,....

38 chips exceed 2.0 Gbps: 29 with 2.0..2.4; 9 with 2.4..3.2 # of FEBs and type distributions varies for stations # of links is quite similar for all stations

STS FEB Type and Link STS FEB Type and Link DistributionDistribution

FEBtype

Sta0

Sta 1

Sta 2

Sta 3

Sta 4

Sta 5

Sta 6

Sta 7

totFEBs

totlinks

FEBa1 16 16 19 14 5 7 0 0 77 616FEBa2 10 23 52 50 26 30 12 9 212 848FEBa4 51 60 34 67 112 87 71 45 527 105

4FEBa8 91 77 15 13 57 76 157 210 696 696

tot FEBs 168 176 120 144 200 200 240 264 1512tot links 361 417 443 459 425 426 347 336 321

4

FEB Type StatisticsFEB Type StatisticsAu+Au @ 25 AGevAu+Au @ 25 AGev

101077 evt/sec evt/sec40 bit message40 bit messageAllow 20% overhead/headroom


Where to do Opto Conversion ? Where to do Opto Conversion ? Ansatz here:

keep FEB simple FEB has Cu interfaces do electrical-optical conversion in separate unit this unit can act as a 'patch panel' natural to work in bundles of 12 links...FEBaFEBa22

FEBaFEBa22

FEBaFEBa44

FEBaFEBa88FEBaFEBa

88

Cu patchcables

'active'patchpanel

Ribbonfiber

cables


STS QuadrantSTS Quadrant

6 cm6 cm5 cm5 cm

STS-FEBs

Likely not agood placefor patchpanels, facesthe magnetcoil directly

Is this enough space for: - 117 copper links - 10 opto converters - 10 ribbon fiber cablesplus other services like - power distribution - control & monitoring - ,,,,,,,

size of asize of apost cardpost card


STS Interface Converter Board Proposal for STS interfacing

2m copper twinax x1, x2, x4 or x8 copper => optical on Postcard

12.7cm x 8.9cm Conversion to optical signals with

AOCs Does electrical and AOC connector

fit on one Postcard ? Connector density analysis

AOCconnectors

twinaxconnectors

twinaxconnectors

Passive BoardPostcard

miniHTconnector

Slide: U. BrueningSlide: U. Bruening





All STS data can be transported by ~280 ribbon cables Corresponds to 34 cm2 cable cross section

STS Link & Ribbon Cable STS Link & Ribbon Cable DistributionDistribution

Sta0

Sta 1

Sta 2

Sta 3

Sta 4

Sta 5

Sta 6

Sta 7

total

FEBs 168 176 120 144 200 200 240 264 1512

links 361 417 443 459 425 426 347 336 3214

cables/quadrant 8 9 10 10 9 9 8 7cables tot 32 36 40 40 36 36 32 28 280

FEB&Cable StatisticsFEB&Cable StatisticsAu+Au @ 25 AGevAu+Au @ 25 AGev

101077 evt/sec evt/sec48 bit message48 bit message





ROC/ROC/DCBDCB

STS Readout Chain in a STS Readout Chain in a Nutshell....Nutshell....

on/near DetectorCBM Service

Bldg.GSI/FAIR IT Bldg.

~50m~50m ~500m~500m

FLESFLES

12002 chips 1512 FEBs

3300 links 280 cables 410 GB/sec

280 ROCs 1320 links 330 GB/sec

2.5 Gbps 5 Gbps


Physical Location ViewPhysical Location View

FEB ROC/DCS ABBDCB FLES

TNet

Data

Control

Data &Control

Sync

Data

Data

Control

STS,

....

CLOSY

TOF,

...?

ROC

ECS

Cu

optical

on/near Detector CBM Service Bldg. GSI/FAIR IT Bldg.

Data

~50m~50m ~500m~500m

??


Towards a Towards a System ConceptSystem Concept


Loose EndsLoose Ends So far only hit data considered

there are also clock, sync, and control connections

So far we tried to avoid a 'support chip' some data aggregation is needed on detector so far 'distributed solutions' discussed

Alternatives: consider a 'support chip' acting as 'communication hub' look again into the CERN GBT approach


GBT – RevisitedGBT – Revisited

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI

17

GBT to front-end link topology

GBT to Front-end links: 32 (+1) bi-directional serial links (e-links) E-link bandwidth is 80 Mb/s Several e-links can be grouped together to serve a single front-end device achieving

bandwidths that are multiples of 80 Mb/s 16-bits for TTC 8 phase adjustable clocks An E-Link Port Adaptor (EPA) “macro” will be available for integration

in the front-end ASICs

FE ASICsclk

Link controller

Link controller

Data MoverData

Mover

TimingTiming

portport

portport

portport

Controlport

Controlport

Front-Ende-links

GBTManagement

logic

GBTManagement

logic

Slow Control ASICe-link

clk40T<15:0>

e-link

e-port

e-port

ReadoutControl Din

Dout

clk

e-link (6 wires)LVDS: Clk80, Din, Dout

32 ports80Mb/s

Timing

T<15:0>

GBT-SCAGBT-SCA

GBT13

FE ASICsclk

Link controller

Link controller

Data MoverData

Mover

TimingTiming

portport

portport

portport

Controlport

Controlport

Front-Ende-links

GBTManagement

logic

GBTManagement

logic

Slow Control ASICe-link

clk40T<15:0>

e-link

e-port

e-port

ReadoutControl Din

Dout

clk

e-link (6 wires)LVDS: Clk80, Din, Doute-link (6 wires)LVDS: Clk80, Din, Dout

32 ports80Mb/s

Timing

T<15:0>

GBT-SCAGBT-SCA

GBT13

Slide from: TWEPP-07 presentation of Paulo Moreira (see this Slide from: TWEPP-07 presentation of Paulo Moreira (see this link) )

Shown in CBM DAQ

on 24th Jan 2008

http://indico.cern.ch/contributionDisplay.py?contribId=71&sessionId=16&confId=11994

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 18Slide from: TWEPP-09 presentation of Paulo Moreira (see this Slide from: TWEPP-09 presentation of Paulo Moreira (see this link) )



CBM ProfileCBM Profile


Profile of a 'communication hub' Profile of a 'communication hub' II Tuned for very high data bandwidth – downstream side

optical data link usage notnot in RX-TX pairsin RX-TX pairs go for much more TX linksmuch more TX links than RX links TX speed as high as feasible (e.g. 2.5 Gbps w/ 180nm) RX speed doesn't need to be the same handle multiple optical links with one 'hub'

Tune for very high data bandwidth – readout-ASIC side a single readout ASIC can produce 2.5 Gbps go for 'simple to implement' interface plausible choice: LVDS at 250 or better 500 Mbps

The 'hub' must be able to aggregate data from several readout-ASICs onto a single output link.

The 'hub' must handle data, clock, sync, and control


Profile of a 'communication hub' Profile of a 'communication hub' IIII Ribbon cables with 12 fibers are widely used

standard opto converters are: 12TX, 12RX and 4TX-4RX consider 12TX ribbon for data consider 4RX – 4TX for clock, sync and control

For a simple mapping of 'hub's to ribbons it is thus natural to consider 'hub's with either 3, 6 or 12 TX for data and 1RX-1TX for clock, sync and control


Profile of a 'communication hub' Profile of a 'communication hub' IIIIII Now some assumptionssome assumptions on what a feasible 'hub' can

be:

Assume a 'hub' ASIC with 6 fast serializers 6 TX for data 1 RX – 1 TX for clock, sync and control 15 Gbps data bandwitdth 2 'hub's will share a data ribbon 4 'hub's will share a control ribbon

Assume 250 MHz system clock 500 Mbps LVDS seems plausible for readout-ASIC interface support 5-8 times as many LVDS links as TX channels


Readout ASIC InterfaceReadout ASIC Interfacexyterxyter

6 fast LVDS data links500 Mbps

clock, sync, cntlinterface

clock, sync, cntlinterface chainable

in some form

Cu

optical


Readout ASIC Interface – FEB Readout ASIC Interface – FEB ViewViewfebfeb

as many LVDSlinks connected

per chipas needed

one clock, sync, cntlinterface per FEB

A high data rateFEB might have6*8 LVDS links

A low data rateFEB might have1*8 LVDS links

Cu

optical


'Hub' ASIC Interface'Hub' ASIC Interfacehubhub

~48 fast LVDS data links500 Mbps

~6 clock, sync, cntlinterface blocks

6 2.5 Gbpsdata links TX

control link pair

Cu

optical

........

........



Low data rate FEB1*8 LVDS links

1 hub serversseveral FEB

hubhub

Cu

optical

........

........



High data rate FEB6*8 LVDS links

1 hub serversone FEB

hubhub

Cu

optical

........

........


Opto-Converter InterfacesOpto-Converter Interfaces

4 RX – 4 TXfor control linksserves 4 hubs

12 TXfor data linksserves 2 hubs

Cu

optical

optoopto

Data

Clock & Control


All Combined – On Detector All Combined – On Detector PartPartfebfeb

Cu

optical

hubhub optoopto

Data

........

........

Clock & Control


All Combined – Detector & All Combined – Detector & Service Bldg.Service Bldg.

Data

Clock &Control

CBM Service Bldg.

DCB

DCS

on/near Detector

Clock &Sync

febfeb hubhub optoopto

TNetCntlNet

Control

Data


DCSDCS

DCB

DCS

DCB

DCS

Physical Location View - Physical Location View - RevisitedRevisited

FEB ABB

DCB

FLES

Data

Control

Clock &Data &Control

Clock &

Sync

Data

STS,

....

TOF,

...?

ROC

ECS

Cu

optical

on/near Detector CBM Service Bldg. 'Green Cube'

Data

~50m~50m ~500m~500m

Data

Control

'hub' 'opto'Data

Clock & Control

DCB

DCS

CLOSY

DCS

TNetCntlNet

Clock & Sync

Control

Control


DCSDCS

DCB

DCS

DCB

DCS

Data FlowData Flow

FEB ABB

DCB

FLES

Data

Control


Clock &

Sync

Data

STS,

....

TOF,

...?

ROC

ECS

Cu

optical


Data

~50m~50m ~500m~500m

Data

Control

'hub' 'opto'Data

Clock & Control

DCB

DCS

CLOSY

DCS

TNetCntlNet

Clock & Sync

Control

Control


DCSDCS

DCB

DCS

DCB

DCS

Clock & Sync FlowClock & Sync Flow

FEB ABB

DCB

FLES

Data

Control


Clock &

Sync

Data

STS,

....

TOF,

...?

ROC

ECS

Cu

optical


Data

~50m~50m ~500m~500m

Data

Control

'hub' 'opto'Data

Clock & Control

DCB

DCS

CLOSY

DCS

TNetCntlNet

Clock & Sync

Control

Control


DCSDCS

DCB

DCS

DCB

DCS

Control FlowControl Flow

FEB ABB

DCB

FLES

Data

Control


Clock &

Sync

Data

STS,

....

TOF,

...?

ROC

ECS

Cu

optical


Data

~50m~50m ~500m~500m

Data

Control

'hub' 'opto'Data

Clock & Control

DCB

DCS

CLOSY

DCS

TNetCntlNet

Clock & Sync

Control

Control


Where to putWhere to putthe 'hub' ?the 'hub' ?STS viewSTS view

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 39Slide from: S. Igolkin, 3Slide from: S. Igolkin, 3rdrd Meeting of CBM-MPS STS consortium, June 2009 (see this Meeting of CBM-MPS STS consortium, June 2009 (see this link))

https://www.gsi.de/documents/DOC-2010-Jan-12.html


STS Quadrant – A possible STS Quadrant – A possible arrangementarrangement

6 cm6 cm5 cm5 cm

STS-FEBs

STS-'hub' STS-'opto'


STS Mechanical & Connection STS Mechanical & Connection HierarchyHierarchy

Data

Control

CBM Service Bldg.

DCB

DCS

Modulefebfeb hubhub optoopto

Super-Module Station

Clock &Sync

TNetCntlNet

Control


SummarySummary


SummarySummary This is meant to start the discussionThis is meant to start the discussion

no pre-mature conclusions at this point no pre-mature conclusions at this point Decide on general architecture and parameters

back-end links number of links per 'hub' distribution of clock/sync/control/data on links

the GBT scheme with all-on-one bi-direction link pair is one choice asymmetric schemes, using 12TX and 4TX-4RX ribbons is another

front-end links protocol for clock, sync, data and control connections

signaling (LVDS,SLVS), encoding (8/10 or not), framing, .... Decide on required R&D Building Blocks to be tested

serializer; driver & receiver cells; ...


The EndThe End

Thanks for Thanks for your attentionyour attention


BackupsBackups

sts readout chain – revisited readout-asic to roc interconnect walter f.j. müller, gsi, darmstadt...

Documents