presentation for the nmi january 23rd 2009 john coughlan data acquisition systems for big science dr...

Presentation for the NMIJanuary 23rd 2009 John Coughlan

Data Acquisition Systems for Big Science

Dr John Coughlan

STFC Rutherford Appleton Laboratory


Talk Outline

• STFC Technology, Who we are

• Data Acquisition for Big Science, CERN LHC

• LHC-CMS Data Acquisition boards.

• Next generation systems


STFC

• Science and Technology Facilities Council, STFC

• Largest of 7 UK Scientific Research Councils reporting to DIUS

• 2,000 scientists & engineers at main site Rutherford Appleton Laboratory

STFC, Rutherford Appleton Laboratory, Oxfordshire UK


STFC

• Space Science, Lasers, X-Ray and Neutron sources, Particle Physics +…

• Operate several world-class large-scale research facilities on site, e.g. VULCAN Laser, ISIS Neutron source, Diamond X-Ray Light Source.

• UK Hub supporting activities of University researchers on other international facilities, e.g. ESA, CERN Geneva



Major Site Developments

• Harwell Science and Innovation Campus…

• New Detector Systems centre 2010/11

• Agreement for new ESA research centre



STFC Technology

• STFC Technology Department

• 300 Electronic & Mechanical Engineers with strong Scientific Expertise

• Developing Instrumentation for Large Scale Science Experiments

• Telescopes to Medical Imaging, Micro MEMS to Major engineering structures, Novel sensors to Cryogenic Superconducting Magnets,

Electronics and Microelectronics…

• Europe wide Microelectronics design support. EUROPRACTICE.


STFC Technology Electronics

• PCB Board and System Design

• FPGA Design. (Xilinx and Altera)

• Embedded systems.

• Board and System Test and System Integration

• Microelectronics, ASICs, MAPS sensors

• Current generation of Large Scale Projects for CERN completed, LHC

• New projects starting R&D phase, European Free Electron Laser XFEL


CERN European Laboratory for Particle Physics. Geneva

Large Hadron Collider LHCUnderstanding the Origin of the UniverseStarted Operation (briefly!) in 2008. No Black Holes yet…

Big Science


CMS Experiment at LHC

12,000 tons apparatus~ 50 million electronic readout channelsDAQ rate ~ 100 kHz

2,500 Scientists & Engineers in collaboration


CMS Installation 200 metres Underground

STFC Microelectronics80,000 x Analogue Pipeline ASICs

Extremely Radiation Hard

VME RacksWater Cooled


One Collision

40 Million Collisions every second. Keep images from 100,000 collisions every second. Each image built up from 10 million analogue sensors. Rarest new particles HIGGs ~ few dozen per year?

Need for Massively Parallel (FPGA) based Processing ~ 1 TERA-Bytes / sec

LHC Data Acquisition


LHC-CMS Data Acquisition

Custom

COTS


LHC DAQ Requirements

• Performance driven by science & experiments.• Large channel counts. Large Form Factor PCBs• High bandwidths. Custom data protocols.

• Specific Functions with Flexibility unknown science. FPGAs

• Electronics developed in collaboration with scientists. Large communities.• In parallel with detector development.• Loose specifications. • Fully custom systems where performance and cost justify design

investment

• Long development cycles. Long operating times. 10+10 Years• Larger systems. Designs frozen early. Minimise risk.


LHC-CMS DAQ Board

• Board Features– 9U x 440 mm (VME mechanics)– Optical Inputs, Analogue– 96 ADC Channels 10 bit @ 40 MHz– 10 x XC2V2000– 24 x XC2V40

– 14 Layers– Double sided PCB

• System has– 500 boards (large nr for PP)– 24 Crates– 8 Racks– 50 KW

• Delivered to CERN 2006

Data In ~ 3 GByte/s. Data Out ~ 200 MByte/s


LHC-CMS DAQ Board Development

• Large boards. 9U VME Mechanics. High I/O count

• Analogue & Digital

• Large nr FPGA BGAs. 676 pins 1 mm pitch

• Concerns for BGA assembly

• Design for Test. Boundary Scan

• Design started in 2001.

• Prototypes (25 off) in 2003/4 before production (500 off) in 2005/6.

• FPGA Virtex II. Fix technology early.

• DCMs for channel synchronisation

• DCI useful but power hungry

• Double Data Rate I/O


Board parameters:- 9U x 440 mm VME64x form factor- Optical/Analogue/Digital logic ; 96 ADC @ 40 MHz channels- 14 layers (incl. 6 power). FR4. - Double-sided (secondary side with half of analogue components)- 6,500 components (most passives are 0402). Surface mount- ENIG metal finish, for BGAs. Pb/Sn - 20,000 connections ; 14,000 vias - 100 micron tracking, some 75 micron diff imp, min gap 100 micron, - 37 BGAs (larger FPGA 676 pins on 1mm pitch). All BGAs located on primary side.- Controlled impedance- Boundary Scan JTAG all Digital devices

Analogue components repeated on 2nd side

LHC-CMS DAQ Board Specs


FPGA Assembly on PCBs

3rd Batch Prototypes 2003

ALL 6 failed Boundary Scan on several BGAs.

Shorts under BGAs.

Rework failed too.

17,000 BGAs in system

EU Tender process for PCB & Assembly. Exception Ltd

1 out of 500 production boards failed Tests 2006


LHC CMS CERN Industry Awards

Gold Award : Exception Ltd New Electronics Cover Story


Other LHC-DAQ Boards

•FPGA 130-90 nm generation•1 Gbps serial data. •DDR2•VME Form Factors•Conventional PCB manufacture.

•Step change design required for Next Generation of Projects


Operating in 2013

European X-Ray Free Electron Laser XFELDESY Laboratory, Hamburg

Next Generation Systems


Single Protein Molecules X-ray Imaging

Just before XRay pulse

During the pulse

After pulse

X-ray Diffraction Pattern

30,000 images per second each up to 16 MPixels


X-RAY Pixel Detectors DAQ For XFEL

• 16M + tiled pixel detector at 30K frames/s -> 1 – 6 TByte/s• 128 x FPGA 40nm + 10G links• Off detector DAQ next gen Advanced Telecoms ATCA crates• 2008 - 2013… Scientific requirements ?

1

8

SFP+FPGADetector Pixel Sensors

10G Fibre

30 m

10 GBytes/sec x N cards

Image Builders


Image Builder for XFEL Demonstrator Advanced Mezzanine Card

180 mm

RAMRAM

RAMRAMCONNECTOR

CONNECTOR

CONNECTOR

CONNECTOR

KEEP OUTKEEP OUT

KEEP OUT

KEEP OUT

FPGAFPGAFMCFMC

FPGAFPGA

KEEP OUTKEEP OUT

MGT

MGT

MGT

MGT

MGT

MGT

MGT

MGT

KEEP

OUT

KEEP

OUT

CONNECTOR

CONNECTOR

X-point

X-pointTXTX

TXTXR

XRX

RXRX

FMCFMC

• AMC Form Factor.• Migrating to 8 FPGAs on 8U ATCA?

• FPGA ~ 16 x 3-6 Gbps serial links

• Analogue Cross Point for Image Building. 72x72 @ 3-6 Gbps

• DDR2/3 ~ 2-4 GBytes• B/W 1-2 GBytes/sec In & Out

• VITA57 FMC Mezzanine I/O• 2 x SFP+ opto TRx• 10 Gbps (XAUI or RXAUI PHY)

• mTCA serial backplane


Next Generation Board Issues

• High speed diff pairs 3-6 Gbps. BGA pitch < 1 mm?

• Reduce Fabrication Risk. Advanced PCB design and construction techniques.

• Vias in pad, micro vias, Laser drill. Incremental build up layer PCB.

• FPGA 40nm generation• FPGA to Memory interface. SO-DIMMs WASSO• Memory controllers Hard/Soft IP from FPGA vendors

• 10 Gbps optical interfaces• 3-6 Gbps Serial Backplanes.

• Power. Multiple POL. Analogue.• Decoupling caps. next gen FPGA packages.• Pb Free manufacture?• Tools Signal Integrity analysis, how to measure eye diagrams on

board?


Design Tools

• PCB Design Tool Flow based on CADENCE.

• Fast serial design Signal Integrity analysis, HyperLynx

• FPGA Design Flow Mentor Graphics

• How to integrate PCB and FPGA design flow? Pin outs.

• Need realistic FPGA designs to guide PCB layout, e.g. memory interfaces

• Our expert (Paul Hardy) is here today


Summary

• STFC Technology, Who we are

• Data Acquisition for Big Science, CERN LHC

• LHC Data Acquisition boards.

• Next generation systems, XFEL


http://www.scitech.ac.uk/



Spare Slides

presentation for the nmi january 23rd 2009 john coughlan data acquisition systems for big science dr...

Documents