Presentation for the NMIJanuary 23rd 2009 John Coughlan
Data Acquisition Systems for Big Science
Dr John Coughlan
STFC Rutherford Appleton Laboratory
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Talk Outline
• STFC Technology, Who we are
• Data Acquisition for Big Science, CERN LHC
• LHC-CMS Data Acquisition boards.
• Next generation systems
Presentation for the NMIJanuary 23rd 2009 John Coughlan
STFC
• Science and Technology Facilities Council, STFC
• Largest of 7 UK Scientific Research Councils reporting to DIUS
• 2,000 scientists & engineers at main site Rutherford Appleton Laboratory
STFC, Rutherford Appleton Laboratory, Oxfordshire UK
Presentation for the NMIJanuary 23rd 2009 John Coughlan
STFC
• Space Science, Lasers, X-Ray and Neutron sources, Particle Physics +…
• Operate several world-class large-scale research facilities on site, e.g. VULCAN Laser, ISIS Neutron source, Diamond X-Ray Light Source.
• UK Hub supporting activities of University researchers on other international facilities, e.g. ESA, CERN Geneva
STFC, Rutherford Appleton Laboratory, Oxfordshire UK
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Major Site Developments
• Harwell Science and Innovation Campus…
• New Detector Systems centre 2010/11
• Agreement for new ESA research centre
STFC, Rutherford Appleton Laboratory, Oxfordshire UK
Presentation for the NMIJanuary 23rd 2009 John Coughlan
STFC Technology
• STFC Technology Department
• 300 Electronic & Mechanical Engineers with strong Scientific Expertise
• Developing Instrumentation for Large Scale Science Experiments
• Telescopes to Medical Imaging, Micro MEMS to Major engineering structures, Novel sensors to Cryogenic Superconducting Magnets,
Electronics and Microelectronics…
• Europe wide Microelectronics design support. EUROPRACTICE.
Presentation for the NMIJanuary 23rd 2009 John Coughlan
STFC Technology Electronics
• PCB Board and System Design
• FPGA Design. (Xilinx and Altera)
• Embedded systems.
• Board and System Test and System Integration
• Microelectronics, ASICs, MAPS sensors
• Current generation of Large Scale Projects for CERN completed, LHC
• New projects starting R&D phase, European Free Electron Laser XFEL
Presentation for the NMIJanuary 23rd 2009 John Coughlan
CERN European Laboratory for Particle Physics. Geneva
Large Hadron Collider LHCUnderstanding the Origin of the UniverseStarted Operation (briefly!) in 2008. No Black Holes yet…
Big Science
Presentation for the NMIJanuary 23rd 2009 John Coughlan
CMS Experiment at LHC
12,000 tons apparatus~ 50 million electronic readout channelsDAQ rate ~ 100 kHz
2,500 Scientists & Engineers in collaboration
Presentation for the NMIJanuary 23rd 2009 John Coughlan
CMS Installation 200 metres Underground
STFC Microelectronics80,000 x Analogue Pipeline ASICs
Extremely Radiation Hard
VME RacksWater Cooled
Presentation for the NMIJanuary 23rd 2009 John Coughlan
One Collision
40 Million Collisions every second. Keep images from 100,000 collisions every second. Each image built up from 10 million analogue sensors. Rarest new particles HIGGs ~ few dozen per year?
Need for Massively Parallel (FPGA) based Processing ~ 1 TERA-Bytes / sec
LHC Data Acquisition
Presentation for the NMIJanuary 23rd 2009 John Coughlan
LHC-CMS Data Acquisition
Custom
COTS
Presentation for the NMIJanuary 23rd 2009 John Coughlan
LHC DAQ Requirements
• Performance driven by science & experiments.• Large channel counts. Large Form Factor PCBs• High bandwidths. Custom data protocols.
• Specific Functions with Flexibility unknown science. FPGAs
• Electronics developed in collaboration with scientists. Large communities.• In parallel with detector development.• Loose specifications. • Fully custom systems where performance and cost justify design
investment
• Long development cycles. Long operating times. 10+10 Years• Larger systems. Designs frozen early. Minimise risk.
Presentation for the NMIJanuary 23rd 2009 John Coughlan
LHC-CMS DAQ Board
• Board Features– 9U x 440 mm (VME mechanics)– Optical Inputs, Analogue– 96 ADC Channels 10 bit @ 40 MHz– 10 x XC2V2000– 24 x XC2V40
– 14 Layers– Double sided PCB
• System has– 500 boards (large nr for PP)– 24 Crates– 8 Racks– 50 KW
• Delivered to CERN 2006
Data In ~ 3 GByte/s. Data Out ~ 200 MByte/s
Presentation for the NMIJanuary 23rd 2009 John Coughlan
LHC-CMS DAQ Board Development
• Large boards. 9U VME Mechanics. High I/O count
• Analogue & Digital
• Large nr FPGA BGAs. 676 pins 1 mm pitch
• Concerns for BGA assembly
• Design for Test. Boundary Scan
• Design started in 2001.
• Prototypes (25 off) in 2003/4 before production (500 off) in 2005/6.
• FPGA Virtex II. Fix technology early.
• DCMs for channel synchronisation
• DCI useful but power hungry
• Double Data Rate I/O
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Board parameters:- 9U x 440 mm VME64x form factor- Optical/Analogue/Digital logic ; 96 ADC @ 40 MHz channels- 14 layers (incl. 6 power). FR4. - Double-sided (secondary side with half of analogue components)- 6,500 components (most passives are 0402). Surface mount- ENIG metal finish, for BGAs. Pb/Sn - 20,000 connections ; 14,000 vias - 100 micron tracking, some 75 micron diff imp, min gap 100 micron, - 37 BGAs (larger FPGA 676 pins on 1mm pitch). All BGAs located on primary side.- Controlled impedance- Boundary Scan JTAG all Digital devices
Analogue components repeated on 2nd side
LHC-CMS DAQ Board Specs
Presentation for the NMIJanuary 23rd 2009 John Coughlan
FPGA Assembly on PCBs
3rd Batch Prototypes 2003
ALL 6 failed Boundary Scan on several BGAs.
Shorts under BGAs.
Rework failed too.
17,000 BGAs in system
EU Tender process for PCB & Assembly. Exception Ltd
1 out of 500 production boards failed Tests 2006
Presentation for the NMIJanuary 23rd 2009 John Coughlan
LHC CMS CERN Industry Awards
Gold Award : Exception Ltd New Electronics Cover Story
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Other LHC-DAQ Boards
•FPGA 130-90 nm generation•1 Gbps serial data. •DDR2•VME Form Factors•Conventional PCB manufacture.
•Step change design required for Next Generation of Projects
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Operating in 2013
European X-Ray Free Electron Laser XFELDESY Laboratory, Hamburg
Next Generation Systems
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Single Protein Molecules X-ray Imaging
Just before XRay pulse
During the pulse
After pulse
X-ray Diffraction Pattern
30,000 images per second each up to 16 MPixels
Presentation for the NMIJanuary 23rd 2009 John Coughlan
X-RAY Pixel Detectors DAQ For XFEL
• 16M + tiled pixel detector at 30K frames/s -> 1 – 6 TByte/s• 128 x FPGA 40nm + 10G links• Off detector DAQ next gen Advanced Telecoms ATCA crates• 2008 - 2013… Scientific requirements ?
1
8
SFP+FPGADetector Pixel Sensors
10G Fibre
30 m
10 GBytes/sec x N cards
Image Builders
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Image Builder for XFEL Demonstrator Advanced Mezzanine Card
180 mm
RAMRAM
RAMRAMCONNECTOR
CONNECTOR
CONNECTOR
CONNECTOR
KEEP OUTKEEP OUT
KEEP OUT
KEEP OUT
FPGAFPGAFMCFMC
FPGAFPGA
KEEP OUTKEEP OUT
MGT
MGT
MGT
MGT
MGT
MGT
MGT
MGT
KEEP
OUT
KEEP
OUT
CONNECTOR
CONNECTOR
X-point
X-pointTXTX
TXTXR
XRX
RXRX
FMCFMC
• AMC Form Factor.• Migrating to 8 FPGAs on 8U ATCA?
• FPGA ~ 16 x 3-6 Gbps serial links
• Analogue Cross Point for Image Building. 72x72 @ 3-6 Gbps
• DDR2/3 ~ 2-4 GBytes• B/W 1-2 GBytes/sec In & Out
• VITA57 FMC Mezzanine I/O• 2 x SFP+ opto TRx• 10 Gbps (XAUI or RXAUI PHY)
• mTCA serial backplane
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Next Generation Board Issues
• High speed diff pairs 3-6 Gbps. BGA pitch < 1 mm?
• Reduce Fabrication Risk. Advanced PCB design and construction techniques.
• Vias in pad, micro vias, Laser drill. Incremental build up layer PCB.
• FPGA 40nm generation• FPGA to Memory interface. SO-DIMMs WASSO• Memory controllers Hard/Soft IP from FPGA vendors
• 10 Gbps optical interfaces• 3-6 Gbps Serial Backplanes.
• Power. Multiple POL. Analogue.• Decoupling caps. next gen FPGA packages.• Pb Free manufacture?• Tools Signal Integrity analysis, how to measure eye diagrams on
board?
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Design Tools
• PCB Design Tool Flow based on CADENCE.
• Fast serial design Signal Integrity analysis, HyperLynx
• FPGA Design Flow Mentor Graphics
• How to integrate PCB and FPGA design flow? Pin outs.
• Need realistic FPGA designs to guide PCB layout, e.g. memory interfaces
• Our expert (Paul Hardy) is here today
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Summary
• STFC Technology, Who we are
• Data Acquisition for Big Science, CERN LHC
• LHC Data Acquisition boards.
• Next generation systems, XFEL
Presentation for the NMIJanuary 23rd 2009 John Coughlan
http://www.scitech.ac.uk/
STFC, Rutherford Appleton Laboratory, Oxfordshire UK
Presentation for the NMIJanuary 23rd 2009 John Coughlan
Spare Slides