LAr ROD R&D for sLHC

Charlie Armijo, Joel Steinberg, Ken Johns (University of Arizona)Joe Mead, Hucheng Chen, Francesco Lanni (BNL)

Luis Hervas (CERN) Andreas Meyer, Andy Kielburg-Jeka, Arno Straessner (Dresden)

Guy Perrot (LAPP)Gloria Blasetti, Mauro Citterio (INFN, Milano)

Dean Schamberger (SUNY, Stony Brook)

5/26/2009 1H. Chen - Brookhaven National Laboratory

Readout Driver (ROD) Upgrade R&D• Goals

– Prototype a ROD that can receive and process continuously digitized detector signals from proposed front-end architecture

– Investigate and evaluate several critical technologies which could be used in the design of next generation ROD

• R&D Interests– High speed optical link & FPGA SERDES

• Data bandwidth of entire LAr (1524 FEBs) > 150Tbit/s• Industry standard parallel fiber connector MPO/MTP, which has 12 fibers and is about 15mm wide• Xilinx & Altera FPGA embedded SERDES

– Low latency lossless data compression/decompression algorithm• Reduce number of links between FEBs and RODs

– FPGA based Digital Signal Processing• Take the advantage of parallel data processing• Make use of the readily available DSP resources in FPGA

– Explore different system level architecture: AdvancedTCA• Take the advantage of industrial standard and high availability features: redundancy, shelf

management protocols, power management, fabrics• High speed serial bus, PCI Express, RapidIO, 1Gbit/10Gbit Ethernet

– Interface to Digital Level 1 Calorimeter Trigger• Perform level 1 trigger sum digitally• Flexibility: adjustable granularity, sliding window• Latency: optical fiber link, energy/timing calculation, trigger sum• Data distribution (real time data replication) to level 1 system

– Interface to Readout Buffer (ROB)• Explore the possibility to integrate ROB on ROD

5/26/2009 2H. Chen - Brookhaven National Laboratory

ATCA Sub-ROD Development• Following up of the Sub-ROD v1.0 design with many improvements

– AdvanceTCA 8U Module• Fabric interface to implement Gbit, 10Gbit Ethernet, RapidIO or PCI Express protocol• Update channel to facilitate the communication between adjacent modules• Rear transition module interface for off crate communication

– Xilinx Virtex 5 FX series FPGA (XC5VFX70T)• 6.5Gbit/s RocketIO GTX transceiver• PowerPC 440 microprocessor up to 550MHz operation• 550MHz DSP48E slices

– 75Gbit/s Parallel Fiber Optic Transceiver• Reflex Photonics 75Gbit/s SNAP 12 InterBOARD parallel fiber optic transmitter and

receiver• Channel data rate of up to 6.25Gbit/s• Long lead time – expected to be received in July

– Expansion Slot on Board• Samtec connectors with single ended and differential connections to on board FPGA• Daughter board to implement additional functionalities

5/26/2009 3H. Chen - Brookhaven National Laboratory

5/26/2009 4H. Chen - Brookhaven National Laboratory

Sub-ROD Injector Development• Sub-ROD Injector Function

– Data injector for Sub-ROD– Compare/contrast Altera/Xilinx FPGAs

• SERDES• DSP capabilities• NIOS/Microblaze/PowerPC

• Successfully integrated with Sub-ROD v1.0 in Sept. 2008• Recent work

– All transmitter channels successfully tested @ 2.4Gbit/s with 27 PRBS

– All receiver channels passed preliminary testes– Began using external memory for data– Began testing serial links for memory

5/26/2009 5H. Chen - Brookhaven National Laboratory

5/26/2009 6H. Chen - Brookhaven National Laboratory

Status of ATCA Sub-ROD & Sub-ROD Injector

• ATCA Sub-ROD– Fabricated and assembled– Power test is done– Ethernet test is done– Parallel fiber optic link test is going on– DDR2 memory test is going on– Tests of ATCA platform– Tests of cross board communication

• Sub-ROD Injector– Altera Statix GX II FPGA is capable of running 6.375 Gbit/s– Tests at 3.2Gbit/s– Tests at 6 Gbit/s– Tests with 223 and 231 PRBS– Move to AMC form factor and function

• Integration test is planned in Aug./Sept. 2009• Good Evaluation Platform for Collaborators

– ATCA architecture evaluation– 12x6Gbit/s parallel optical transmission evaluation– FPGA DSP/embedded microprocessor evaluation– Flexibility to evaluate different serial protocols– Easy expansion of functionalities with daughter board

5/26/2009 7H. Chen - Brookhaven National Laboratory

Modeling Pile-up• Generate waveforms at high luminosity to test energy and timing

calculations• Program

– Loop over bunch crossings (BC)– For each crossing, generate Poisson random number r with mean =

<min bias>– For each i = 1,..,r generate energy Ei from MC

• Convert Ei to ADC (peak)• Generate ADC waveform Wi that spans 30 BC’s

– Sum Wi for all events in this BC– Sum over Wi all BC– Add pedestal value

• Presently we are tuning the code to ensure sensible results at LHC luminosity

• Expect results by the next meeting85/26/2009 H. Chen - Brookhaven National Laboratory

9

EM3-Barrel Cell Energies

• UL (min bias), UR (j0), LL (j1), LR (ttbar)

5/26/2009 H. Chen - Brookhaven National Laboratory

Waveform• Shape of waveform taken from shaper plus

empirical negative part

105/26/2009 H. Chen - Brookhaven National Laboratory

Some Relevant Ideas

• Multiple Fiber Bundle– http://www.prysmian.com– Rapier – Easy Break-Out Cables

• Up to 432 fibers– Multi Loose Tube Cable

• Up to 720 fibers– Afumex

• Up to 864 fiber

5/26/2009 11H. Chen - Brookhaven National Laboratory

Some Relevant Ideas• Optical Transceiver

– MTP/MPO SNAP-12• http://www.reflexphotonics.com• http://www.emcore.com• http://www.zarlink.com• http://www.avagotech.com• http://www.intexysphotonics.com

– Parallel Active Optical Cables• http://www.finisar.com• http://www.zarlink.com

– LightABLE Parallel Fiber Optical Engine• http://www.reflexphotonics.com

• FPGA SERDES– Xilinx Virtex-6: 40nm, 11.2Gbit/s

• Not available yet– Altera Stratix IV: 40nm, 11.3 Gbit/s

• Overpriced– Xilinx Virtex-5Q: Defense grade FPGA

5/26/2009 12H. Chen - Brookhaven National Laboratory

Some Relevant Ideas• From ATCA to MicroTCA

– ATCA Carriers – Advanced Mezzanine Card

– AMC – PICMG AMC.0 specification– MicroTCA – PICMG MTCA.0 specification

• xTCA for Physics Coordinating Committee– Formed in Mar. 2009 under PICMG– Coordinate Physics issues with PICMG– Growing number of labs using xTCA

products need:• Analog & Digital IO for AMC, ATCA,

MicroTCA• Software design for full xTCA

management, redundancy features• Fast timing and synchronization• Rear Transition Module scheme for

AMC/MicroTCA– “Request” design from industry

• If it is according to any standard, industry sees change to make profit

5/26/2009 13H. Chen - Brookhaven National Laboratory

Development Plan of Next Version of ROD

• ATCA 8U module as mother board– Serve as generic mother board– Sub-ROD daughter board– Sub-ROD Injector daughter board

• AMC module as daughter board– Relatively independent– Can be tested in MicroTCA chassis– Cost effective

• Call for user request of functionalities– Welcome contributions/ideas from

collaborators– Regular phone meeting every 2 or

3 weeks– Discussion of ROD development– Discussion of pile-up simulation

5/26/2009 14H. Chen - Brookhaven National Laboratory