Jan.20, 2004
Hawaii, USA
Super B Factory Workshop
Takeo HiguchiInstitute of Particle and Nuclear Studies, KEK;
On behalf of COPPER working group
Readout Electronicsfor the SuperKEKB
Contents
• Introduction• Conceptual design• The COPPER-II• Readout Scheme• COPPER-II Performance• Schedule• Summary
Requirements for Readout Electronics
TriggerSystemTriggerSystem
ReadoutElec.
ReadoutElec.
EventBuilderEvent
Builder
StorageStorage
Detector
DAQ
Trigger rate•> 10 kHz
Data size•~ 300 kB/ev
Readout•~ 50 k channels
Trigger rate•> 10 kHz
Data size•~ 300 kB/ev
Readout•~ 50 k channels
SuperKEKB
Also good for J-PARC, K2K …Also good for J-PARC, K2K …
Conceptual Design
What are keys in designing the readout system?
System Design
• Event buffers for pipelined DAQ
• Data size reduction by processor on data path
ProcessorProcessor
Readout FIFOReadout FIFO
DigitizationDigitization
Data path
Consequent designof readout electronics
from detector
to EB
Design from Cost View
• Flexible system– Common platform + user-defined ADC/TDC.– Implementation of ADC/TDC and CPU as modules.
• Compact design– Less #-of-crates to house platform boards.– Smaller room of electronics hut.
• Commodity usage– Good products at low price in computer market.– Easy to follow/import newest technology.
Design Overview
PMC CPUPMC CPU
local bus PCI bus
Det
ecto
r si
gnal
s mezzanine(add-on) module
Network IFNetwork IF
Trigger Module
Trigger Module
ADC/TDC Module
ADC/TDC Module
ADC/TDC Module
ADC/TDC Module
BridgeBridge
PMCFIFO
Conceptual Design (Front-end I/F)
• User-defined ADC/TDC module
sampling clock
Detectorsignal
0xaa550x02460xf3b7 …
ReadoutFIFOADC/TDCADC/TDC
L1 triggerpipeline
L1 triggerpipeline
trigger signal
Add-on type Module
flexibilityflexibility
compactnesscompactness
Conceptual Design (L1 Trigger Pipeline)
Digitized data in
Data out(to Readout FIFO)
Alw
ays
WR
ITE
busy
Dat
a R
EA
D
Tri
gger
HO
LD
switch
Dat
a R
EA
D
Alw
ays
WR
ITE
Dat
a R
EA
D
switch
EM
PT
Y
L1 trigger Ready
pipelinepipeline
Conceptual Design (FIFO)
Detector
Detector
Detector
Detector
Local bus
BridgeBridgePCI bus
ADC/TDC Module
ADC/TDC Module
ADC/TDC Module
ADC/TDC Module
trigger signal
DMA
32
pipelinepipeline
Conceptual Design (Back-end)
• Data path = PCI bus• PCI mezzanine modules (PMC)
BridgeBridge
CPUCPU
TriggerTrigger
NetworkNetwork
EB
PCI bus
Data suppression
DMA
PCI = Up to 133 MB/sPCI = Up to 133 MB/s
trigger signal
Local bus
online data reductiononline data reduction
flexibilityflexibility
commodity usecommodity use
compactnesscompactness
Conceptual Design (Processor)
• Modular CPU– Modularity = easy to upgrade.– Intel x86 series: most familiar architecture.
• Operating System = Linux– Easy to develop device driver / software.– Low price.
flexibilityflexibility
commodity usecommodity use
PCI Ethernet card PMC Ethernet card
What’s ‘PMC’? - PCI Mezzanine Card Standard
– PMC is 100% compliant with the PCI.– PMC is suitable for high density applications.– Many commercial products are available: Ethernet cards, G
bE cards, memory modules, CPUs, etc.
74×149 mm2
PMC Size CPU
• RadiSys EPC-6315– Equipped with Intel PentiumIII 800 MHz.– RedHat Linux 7.3 runs.– 32-bit 33/66 MHz PCI bus interface.
You can login this CPU like your PC
Even physics-analysis codes run on it!Contains Linux image
Whole Design of the DAQ Platform
PMC CPUPMC CPU
local bus PCI bus
Det
ecto
r si
gnal
s mezzanine(add-on) module
Network IFNetwork IF
Trigger Module
Trigger Module
VME 9U sized board
ADC/TDC Module
ADC/TDC Module
ADC/TDC Module
ADC/TDC Module
BridgeBridge
PMCFIFO
The COPPER-II
This is our readout platform for the Super B factory.
The COPPER-II
PMCProcessor
Trigger
GenericPMC slot
ADC/TDC
ADC/TDC
ADC/TDC
ADC/TDC
VME-9U sized boardVME-9U sized board
•4 ADC/TDC slots•1 MB × 4 FIFO•32-bit local bus•3 PMC slots•32-bit 33MHz PCI bus•Local-PCI bridge•VME I/F
•4 ADC/TDC slots•1 MB × 4 FIFO•32-bit local bus•3 PMC slots•32-bit 33MHz PCI bus•Local-PCI bridge•VME I/F
COmmon Pipelined Platform for Electronics Readout
On-board Ether
ADC/TDC Module
Local bus I/F(for control)
Digitizedsignal output
Trigger busyClock, L1 trigger
Clock = 43MHz,Reduced clock = depends on sub-system
32
Photo of prototype card
76×180 mm2
Readout Scheme
How it works?
Readout Scheme (1)
• L1 Trigger comes…– Trigger module distributes trigger signal to the ADC/TDC modules.– Digitized data are held on the ADC/TDC module.– Data transfer to the COPPER-II starts.
• COPPER-II FIFOs are filled…– The COPPER-II’s FPGA formats data (header and footer).– The COPPER-II’s FPGA counts each event size in FIFOs.– A PCI interrupt is generated when a total data size in the FIFOs excee
d threshold to initiate the DMA.
Readout Scheme (2)
• PCI interrupt is generated…– The CPU checks the size of stored data counted by the COPPER-II.– The CPU starts data transfer from the FIFOs to CPU’s main memory u
sing DMA.
• A user process on the CPU is woke up after DMA…– The user process reads data and performs the data size reduction.
• The data are transferred to the event builder…
COPPER-II Performance
Up to what input rate does COPPER-II work?
Study Setup
ADC proto.ADC proto.
ADC proto.ADC proto.
ADC proto.ADC proto.
ADC proto.ADC proto.Vacant
TriggerTrigger
CPUCPU
Ether
RX
The COPPER-II
Generates virtual ‘ADC’ data.Data size / event is variable.
Self trigger.Trigger rate is variable.
Maximum Trigger Rate
Typical trigger rate
Required trigger rate
@ 416 bytes/ev/ADC-module
The COPPER-II works> 30 kHz input rate
The COPPER-II works> 30 kHz input rate
Belle’s TDC data16 bytes / ch. (drift chamber)
400 bytes/TDC-module corresponds to 100 ch. in one COPPER-II module.
Belle’s TDC data16 bytes / ch. (drift chamber)
400 bytes/TDC-module corresponds to 100 ch. in one COPPER-II module.
Profile of CPU Usage
• User time: ~2%• System time: ~20%• Idle time: ~78% = CPU power that is equivalent to
P3 ~600MHz is still available
• Large idle time fraction indicates the PCI bus works at the full performance.
– 416 bytes / ADC-module / ev × 40 kHz × 4 ADC modules= 67 MB/s.
@ 416 bytes/ev/ADC-module
Performance Degradation by Network Use@ 416 bytes/ev/ADC-module
CPUCPU
CPUCPU
FIFOReadFIFORead
FIFOReadFIFORead
NetworkTransferNetworkTransfer
proc.A
proc.Aproc.B
RX
Ethernet 11MB/s
CPU user time: ~2%CPU system time: ~20%CPU idle time: ~78%
Maximum acceptedtrigger rate: 40 kHz
CPU user time: ~2%CPU system time: ~20%CPU idle time: ~78%
Maximum acceptedtrigger rate: 40 kHz
CPU user time: ~5%CPU system time: ~31%CPU idle time: ~64%
Maximum acceptedtrigger rate: 32 kHz
CPU user time: ~5%CPU system time: ~31%CPU idle time: ~64%
Maximum acceptedtrigger rate: 32 kHz
Still works well.
Schedule
What are we planning?
Schedule in Near Future
• The COPPER-II prototype is being debugged.• We start production of the COPPER-II from the begging of
Feb.2004.
• We are planning to replace some Belle-DAQ parts with COPPER in 2005 summer.
Long Term PlanCOPPER Proto.
Upgrade
Integrated Test
Goal
Per
form
ance
Stu
dy
Per
form
ance
Stu
dy
AS
IC D
evel
opm
ent
AS
IC D
evel
opm
ent
AD
C M
odu
le D
esig
nA
DC
Mod
ule
Des
ign
TD
C M
odu
le D
esig
nT
DC
Mod
ule
Des
ign
Mas
s C
PU
Con
trol
Mas
s C
PU
Con
trol
Use
r I/
F F
ram
ewor
kU
ser
I/F
Fra
mew
ork
Eve
nt
Bu
ildin
g F
arm
Eve
nt
Bu
ildin
g F
arm
L2/
L3
Tri
gger
sL
2/L
3 T
rigg
ers
Pla
tfor
m
Har
dwar
e A
ppli
cati
ons
Sof
twar
e A
ppli
cati
ons
ADC/TDC Proto.We are here.
Summary
• We have developed new readout electronics, COPPER-II, for the Super B factory.
• Design keywords are:Pipeline, Online data reduction,Flexibility, Compactness, and Commodity usage.
• At the typical data size, the COPPER-II works up to 40 kHz trigger rate, which satisfies the requirements.
Backup Slides
Please help me…
Introduction
What is a readout electronics?What is Super B factory requiring for it?
Introduction to Readout Electronics
TriggerSystemTriggerSystem
ReadoutElec.
ReadoutElec.
EventBuilderEvent
Builder
StorageStorage
Detector
Role• Signal digitization• Event buffering• Data suppression• Data transfer to EB
Role• Signal digitization• Event buffering• Data suppression• Data transfer to EB
DAQ
Keywords in the System Design
• Five keywords, again…
pipelinepipeline
online data reductiononline data reduction
flexibilityflexibility
compactnesscompactness
commodity usecommodity use
Add-on-structure module
FIFO chip
Bridge chip
PMC size CPU
PMC size network interface
All Components on a 9U-Size Board
ADC/TDC Module
BridgeBridge
CPUCPU
NetworkNetwork
compactnesscompactness
Level 1 Trigger
Trigger Module
Trigger Module
ADC/TDC Module
trigger
busy
FINESSE TTRX
TRIGGER from the trigger module.
BUSY response by ADC/TDC module.
Data hold.
Data-transfer to FIFO.
ADC/TDC Module Readout FIFO
• Data transfer to readout FIFO
– 32-bit FIFO per ADC/TDC module.– Data size of each event is counted by FPGA
and is stored to FIFO-word-counter FIFO.
ADC/TDC Module
Readout FIFO(1 MB)
32-bit data
FIFO WENA
FIFO WCLK
spied
FIFO-word-counter
FIFO-word-counter
FIFO-word-counter FIFO
To CPU
FPGA
Readout FIFO CPU: DMA
Local-PCIBridge
Local-PCIBridge
CPUCPU
PCI bus
RadiSys EPC-6315PLX-9054
Local bus
Readout FIFO
FIFO-word-counter FIFO
Readout FIFO filled
PCI interrupt
Check event size
Initiate DMA
DMA data transferDMA over
PCI interrupt
COPPER Device Driver
interrupt_handler()interrupt_handler()
do_DMA()do_DMA()
while( data_in_FIFO ){ event_size = read_event_length_FIFO(); DMA_size += event_size;};
start_DMA( DMA_src_addr, DMA_dst_addr, DMA_size );
while( data_in_FIFO ){ event_size = read_event_length_FIFO(); DMA_size += event_size;};
start_DMA( DMA_src_addr, DMA_dst_addr, DMA_size );
interrupt_handler()interrupt_handler()
PCI interruptReadout FIFO ready (every event)
PCI interruptDMA over
DMAbufferDMAbuffer
user_read()user_read()do_DMA()do_DMA() Read new data (if exist)after last do_DMA().
Return to user code.
Return to user code.
Read all available data in FIFO (mostly one event)
CPU Event Builder
EBEthernet
CPUCPU
NetworkNetwork
PCI bus
serial linefor debug
– Are event-header/event-footer correct?
– Are data contents consistent with pre-defined ones?
– Generator module’s event counters = event tag from trigger module?
– Are all event counters from 4 generator modules consistent?
– Do event counters increase by 1 correctly?
– Are event-header/event-footer correct?
– Are data contents consistent with pre-defined ones?
– Generator module’s event counters = event tag from trigger module?
– Are all event counters from 4 generator modules consistent?
– Do event counters increase by 1 correctly?
Maximum Trigger Rate vs. Data Size
Typical data size
Data Transfer Speed on PCI vs. Data Size
Typical data size