ongoing developments for the atlas lar subdetector readout
DESCRIPTION
Ongoing Developments for the ATLAS LAr Subdetector Readout. Possible replacement of VME for the upgrades of ATLAS 12 July 2012 Guy Perrot on behalf of the LAr upgrade collaboration. Outline. Context US Developments (BNL/Stony Brook & U. of Arizona) ATCA Development Blade - PowerPoint PPT PresentationTRANSCRIPT
Ongoing Developments for the ATLAS LAr Subdetector Readout
Possible replacement of VME for the upgrades of ATLAS
12 July 2012Guy Perrot on behalf of the LAr upgrade collaboration
• Context• US Developments (BNL/Stony Brook & U. of Arizona)
– ATCA Development Blade– AMC ROD Injector
• European Developments (Dresden & LAPP)– 10 GbE on Fabric interface – IPM Controller & ATCA Test Blade– ROD Evaluator
• LAr Today’s Path for 2018• Answers to your questions• Summary
Outline
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 2
Current & Future Architectures
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 3
Currentsystem
1600 FEBs192 RODs~120 TBBs
Phase I 2018Digital Trigger
320 scells/LTDB
~120 LTDBs
~ 30 ATCA DPS Blades
~24 Tbps IN~24 Tbps OUT
Up to 1.5 TbpsMonitoring
Phase II 2021FEB & ROD Change
128 cells/FEB~1600 FEBs
~100 ATCA ROD blades ?
150 Tbps IN0.4 Tbps OUT
• BNL ATCA Development Blade– No IPM Controller (was
supposed to be DESY Firmware but lack of time)
– Shortcuts to get power on.– Room for 2 Mezzanines (not
AMC)• Used by Dresden to test 10
GbE on Fabric
US developments (1)
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 4
Base Interface (Gigabit Ethernet)Fabric Interface (10Gig Ethernet)
Zone 3 (to transition module) LVL1 communication
Update ChannelGlobal Clocks
-48v Power Input and IPMI interface
12 fiber inputs SNAP12 format
DDR2SODIMM Memory
Virtex 5 FXT FPGA
Power Conversion
Serial Debug Port
• BNL DPS/ROD sPU AMC– In design phase– DESY/CPPM/CERM MMC (ATMEL
Processor)– Virtex7 FPGA– 48 10Gbps links
• U. of Arizona ROD Injector AMC – 2 x 12 channels @ 6.25 Gbps (Front)– Stratix EP4SGX230KF40– 12 channels @ 5 Gbps (GbE or PCIe) on
Fabric – MMC implemented using an ARM Processor– Future AMCs to use CPPM/CERN MMC– AMC only used in uTCA Shelf
US developments (2)
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 5
• Technische Universitat Dresden: 10 GbE through ATCA Fabric Interface– ATCA shelf: 20port 10GbE Switch, BNL SubROD (Virtex 5 FPGA)– Server: Dual 4-Core Xeon, 4x Dual Port Myricom 10GbE NIC– Link: optical between shelf and server
• Results:– 80 byte UDP packets: ~400Mbps only! with packet losses – 1500 byte packets: ~4.4Gbps with packet losses – 9000 byte jumbo packets: ~10Gbps without packet loss
European developments (TUD)
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 6
• IPM Controller– Communication with Shelf manager through
IPMBus A & B– Hot Swap, Power management etc..
• ATCA blade manager– Communication via Ethernet (front panel or
ATCA Base Interface)– User functions
• Firmware Upgrade • ATCA blade monitoring & configuration• User stuff….
• Software libraries available
• FMC (FPGA Mezzanine Card) compliant– up to 160 customizable links( including 74
differential links) – Low cost
• Features:– ARM cortex M3 processor– Xilinx Spartan 6 for highly configurable user IO– Ethernet / USB / JTAG interfaces
European Developments (LAPP) ATCA Controller Mezzanine
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 7
FPGASpartan 6
µCLM3S9B92
bus
FMCMezzanine Power Supplies
connector
EEPROM
IOUSB
JTAGJTAG JTAG
I2C IPMBusA&B
Ethernet
69mm
76.5mm
µC FPGA IO
Possible VME Replacement for the ATLAS upgrades G.Perrot 8
• ATCA Controller Mezzanine tests– IO connections (JTAG boundary scan
tests)– IPMI management with the Shelf
manager– Ethernet communication through
ATCA Base Interface • ROD Evaluator tests
– Check Blade configuration with the ATCA Controller Mezzanine (Firmware upgrade, Configuration upload etc..)
– Test ATCA compliant power supplies– Check FPGA Design
(communications with DDR3, Flash, configuration with Flash)
European Developments (LAPP)ATCA Test Blade
12/07/2012
• Smaller size: DDR3 VLP Mini-DIMM• Based on 2 µC ARM cortex M4 from ST
Microelectronics• IPMC features
– IPMB_0 with on board buffers, Hardware address detection– Hot Swap management with ATCA Leds and front panel
switch– Management of up to 8 AMC + RTM– On board Event LOG– FRU & SDR via I2C– Access to ATCA board sensors via I2C– Configurable User Signals for Payload management
• JTAG Master– JTAG master via Ethernet (ATCA board debugging,
firmware upgrade)
• Custom interface– Possibility to have a custom interface between the
Mezzanine and the ATCA board for custom functionalities
• Other– USB and UART interfaces (debugging etc..).
European Developments (LAPP) ATCA Controller Mezzanine Version 2
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 982.127 mm
18.30 mm
• ATCA Blade with 4 AMC slots.• MMC mezzanine • Designed to test all the
functionalities of the ATCA controller mezzanine.
• Could be used as “Reference Design” for users.
European Developments (LAPP)ATCA Test Board Version 2
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 10
Mezzanine
DC/DC PowerMgt
AMC 1
AMC 4
PWRMgt
I2CHAEtc..
RJ45switch ETh
BaseInterface
Led, switch
T°
MMC
FPGA
User IORSVD FRU
PWRMgt
• Fully compliant ATCA blade.• To evaluate high speed and high density
optical & electrical transmissions.• To evaluate high power DSP cells from
FPGA• Board management trough Ethernet
(Firmware upgrade, DSP configuration…)
- 48 optical Rx links @8.5Gbit/s- 48 optical Tx links @8.5Gbit/s- 8 Electrical Rx/Tx links @8.5Gbit/s
16 Electrical Rx/Tx LVDS links @ 1.6 Gbit/s between FPGAs
- Connection to Fabric & Base interface
European Developments (LAPP) ROD Evaluator
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 11
FPGA
CPLDFLASH
DDR3
SNAP12 Rx(Carrier)
SNAP12 Tx(Carrier)
ATCA IPMC(FMC carrier)
ATCA DC/DC
DC/DC :3.3V,2.5V,1.8V,1.5V
DC/DC : 0.9V
Phase I Architecture
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 12
Phase I 2018Digital Trigger
320 scells/LTDB
~120 LTDBs
~ 30 ATCA DPS Blades (LDPB)
~24 Tbps IN~24 Tbps OUT
Up to 1.5 TbpsMonitoring
LDPB Data Path Block Diagram
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 13
Data Processing
Memory
MemoryData Processing
DATA 40 MHz RESULTS 40 MHzFE
LTDBL1 CaloFEX
DATA + RESULTS 400 KHz MonitoringData
Collector
DATARESULTS 100 KHz
L0 400 KHzLatency 2.5-10 us
L1 100 KHzLatency 100s of us
LTDB (LAr Trigger Digitizer Board) : 118 ModulesLDPB (LAr Digital Processing Blade) : 29 ½ Blades (4 LTDB / LDPB)FE (Front End) & FEX (Feature Extractor)Monitoring Data Collector
820 Gbps
820 Gbps205 Gbps
24.2 Tbps
49 Gbps (8 Gbps Res. only)1.45 Tbps (240 Gbps)
2 Gbps60 Gbps
24.2 Tbps
MonitoringData
Collector
Back-End Block Diagram (In Work)
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 14
LTDB AMC
Data Processing & Memory
FEX
Switch10
GbE
SwitchGbE
AMCData
Processing & Memory
Switch10
GbE
SwitchGbE
Blade:LDPB
TTC Distribution
TTC
PM PC
MonitoringData
Collector
TTC LDPS
Partition(NEW!)
Shelf Manager
Busy
Scells Raw data
Results to FEXL1 Calo
TTC, Control, Monitoring
Raw data?, result on L0
Raw data?, result on L1
Control, Monitoring
Shelf
IPMC
Blade:Switch
Switch
GbE
TTC
Blade Ctrl.& Monitor.
• What is the expected usage of the backplane?– ATCA IPM control of blades.– Blade & FE LTDB configuration & monitoring via base interface.– Data collection via fabric interface.
• Do you use the fat pipes. If yes which protocol?– For the time being we are thinking 10 GbE.
• Would you agree for a control through Ethernet? – Yes, we plan it that way!
• Do you use synchronisation signals and common clocks? – No common clock for operation of blades.– TTC for clock transmission to FE and Evt signature at FE & BE.
• Do you plan to use redundancy and/or hot swapping– Hot swapping YES, redundancy NO.
• Do you plan full mesh or dual star backplane?– Full mesh, though we are using a dual star architecture, as it can do both.
Answers (1)
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 15
• Do you need a local processor?– For the time being we don’t think so.
• Which shelf manager do you plan to use?– The one that will come with the shelf (we have Pigeon Point).
• Which IPMC do you plan to use?– We are developing an IPMC and are planning to use it.
• Typical expected power consumption?– No idea yet, but probably quite a lot: 200 W/blade => 2.8 KW ???
• Which crate size to you plan to use (i.e. how many slots available)?– ATCA format with 14 slots + shelf manager.
• Do you plan implementation with AMC mezzanine and a mother board?– We are planning to have 4 AMCs on an ATCA carrier blade.
Answers (2)
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 16
• How sure are you that you need ATCA. Could you envisage µTCA?– We went straight for ATCA as we have a very big number of channels.– A study would be needed to understand data throughput on the uTCA backplane compared to
ATCA, nb of slots needed, xTCA switches and external switches. We haven’t done that!
• Do you need zero, one or more ATCA switch blades in your ATCA shelf? Which kind of switch?
– To get the bandwidth for data collection we need 2 switch blades (10GbE) 12 in (fabric) / 12 out (front).
– GbE for the control of blades via base interface.– TTC distribution ?????????
• Which MMC do you want to use on your AMC or in uTCA?– We are planning to use the DESY/CPPM/CERN design.
Answers (3)
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 17
• A basic BE system will be installed in 2013-14 using our ROD Evaluator (2 blades).
• The Digital Trigger system in 2018 (Phase I) will allow us to develop, test and master all the technologies needed for the Phase II developments (replacement of FEBs and RODs)
• It will also allow us to understand what is the best compromise in term of density, power consumption, data collection.
Summary
12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 18