Initial thoughts on “Pulsar in SRC mode” possibility Ted Liu, Dec. 03, 2003

Some discussions with:SRC side: Petar/Lester/Steve/Satyajit General: Jonathan Lewis and Peter Wilson

Only discuss the technical feasibility:

Is it possible to use Pulsar as SRC IF indeed needed?What is involved at hardware level ?What are the options ?What are the advantages ?What are the concerns ? need experts to point out ! The devil is in the details…

Personal comments: a different view on Trigger/DAQ upgrade Petar is looking at the bright side, so I look at the dark side

• with both SVT and L2 upgrade & UDPS, we MAY be able to run at 40-50KHz (RunIIa-RunIIb spec), at least there is a fair chance at low luminosity

• at higher luminosity, with multiple interactions per BX, there is a fair chance we may no longer be able to run at 40KHz… if we are really unlucky we may still only be able to run at <~20KHz if occupancy doesn’t scale as one would like even after all SVT&L2 upgrade …

Enough safety margin && flexibility is crucial, every micro-second in raw speed may matter be prepared, and invest in raw speed&flexibility early

Some historic comments on Pulsar-SRC

• Last May, after Pulsar design was finalized, Petar & Lester asked me to see if we can make Pulsar compatible for SRC upgrade…• I spent a few weeks to take care of that. In particular, the RF clock & backup local clock (see next slide). But the P3 connection was too late to change in order to reuse the SRCTM (bottom line: SRCTM cannot be reused if using Pulsar).• Since then, there was no more discussion on using Pulsar to upgrade SRC…• We are now looking into this possibility again (requested by Petar & Jonathan). maybe this time we are a bit more serious?

Backplane issue: first few empty slots of VRB P3 backplane can be split (from the rest), or Pulsar-SRC have to live in a different crate nearby…

Control FPGA

DataIO FPGA 1

VME chip

DataIO FPGA 2

Pulsar RF clock & local clock:Use zero-delay glitch-free clock multiplexer:ICS581-01 (found by Mircea)

• local clock• Tevatron RF clockThe two clocksare arranged insuch a way thatwhen the RF clockis lost, the localclock takes overin a few clock ticks.The output clockgoes to PLL clock pin on each FPGA) Designed specifically to meet the unique requirement for SRC

Current SRC interfaces

Current SRC CurrentSRCTM

VRB

Beamstructure

RFclock

TSI(lvds)TS(Taxi,Optical)

GlinkTo FIB

FIBReturnLVDS

RED: already existed in Pulsar designBlack: can be absorbed on a new TM, or 1-2 mezz cards Note: all LVDS interfaces use similar connectors

VRB

Current SRC SRCTM

Pulsar-SRC: A few options to consider

• Option A: Simplest & cleanest

rebuild the SRCTM for Pulsar, and add TSI/beam structure/VRB interfaces on new TM.

Pulsar Control FPGA “sees&controls” everything

• Option B & C: not as clean for VRB, see backup slides…

Option A: build new TM • copy the same FIB interface design …

• add TSI(LVDS)/TS(Taxi optical)Beam structure/VRB…on the other side…

In this case, Control FPGA sees and controls ALL.DataIO FPGAs can be used for house-keeping work

New SRC

PulsarTM isdouble-width….

P3:117 signals

(1) Split the backplane, remove the empty slots

(2) Install P2 style backplane for Pulsar + new TM.(3) Pulsar-SRC and SRC can either live next to each in another VRB

VRB backplane

Pulsar-SRC and SRC can co-exist…can switch back to old SRC easily

NO need to touch the old SRC,Only cables …

ideal for commissioning…

Backplane may notbe an issue:

Advantages: • simple & cleanest from firmware/hardware point of view;• DataIO FPGAs may be used for house keeping tasks;• mezzanine card slots in front can be used for expansion/diagnostics/flexibility

Option A: build new TM (cont.)

New SRC

Option A: build new TM (cont.)

New SRC

diagnostics: one possible example? just a thought last night…• can pump information into a PC and receive PC results via either Gigabit Ethernet (round trip time ~ 30us), or SLINKPCI (round trip ~ a few us): maybe one can try resonance detection? either in firmware or softwareRecall that Pulsar was designed for fast decisions using a PC (firmware/hardware/software infrastructure/expertise all in place)

CPU

Other comments:A few things to consider:

• firmware design issues in each case: option A seems easiest• hardware: option A is cleanest• flexibility: Option A seems more flexible • how to commission ? think backwards! • Option A seems the best from firmware/hardware point of view Hardware needs seem to be simple for all options. However,one may build a Glink Rx mezzanine card as data sink for testingpurpose. Much easier this way to develop & tune the upgrade standalone…Crucial also for Glink Single-Bit-Error testing (down to 10**-14) …Also keep in mind that L2-TSI interface (i.e. L2 raw decisions),all raw L1 trigger bits as well as SVT interface all stillavailable on Pulsar in case someone has good ideas to use them.

Top 10 reasons why Pulsar ispotentially useful for SRC upgrade

• Pulsar motherboard is designed to be compatible with SRC• Can be tailored to meet all SRC interface needs (no kludges) P3 backplane doesn’t seem an issue• Can leave old SRC untouched, can switch back easily, can develop & tune the upgrade standalone• One FPGA “sees&controls” all: firmware design easier/cleaner Low level firmware can be shared, under same CVS…• Fast large SRAMs (two 128K x 36) available on board• Mezzanine card slots can be used for expansion/diagnostics• Can communicate with a PC in semi “real time” (~10 us): e.g. resonance detection: extra safety protection for our silicon • L2 raw decision/L1 raw trigger bits/SVT interface available• Many hardware/firmware expertise/modern firmware tools…• Pulsar-SRC may be also used as a pulser for SVT … (i.e. fake SVX fiber data into Hit Finder)for experts: Do we have good motivation(s) for SRC upgrade ???

• Option B• Option C• the rest of slides are mostly from my SVT talk, keep them here for background information,• also added Pulsar/PC round trip timing measurements (by Upenn) using SLINK-PCI (slide 30-31), with web link to timing measurements with Gigabit Ethernet

More detailed information can be found at:

http://hep.uchicago.edu/~thliu/projects/Pulsar

Back up slides

Pulsar-SRC: A few options to consider

• Option A: Simplest & cleanest rebuild the SRCTM for Pulsar, and add TSI/beam structure/VRB interfaces on new TM. Pulsar Control FPGA “sees&controls” everything

• Option B: Simple & clean use Pulsar TM (AUX) card. Build 1-2 types of simple mezzanine card, with one Glink + LVDS + TAXI. Pulsar DataIO FPGAs handle FIB interfaces, and Control FPGA takes care the rest

•Option C: Simple & clean (avoid the P3 backplane issue completely) Do not use P3 and TM. Build 2 types of mezz cards: one for FIB interfaces (2 Glink per card), the other for the rest. One DataIO handles FIB interface, the other takes care the rest (VRB goes from front panel, or absorb also on mezzanine card…not as clean!)

AUX Card

• DataIO FPGAs take care the FIB interface;• Control FPGA for the rest

glinklvds

BEAMstru

TSILVDS/optical

Option B: Have FIB interface upfront,and the rest on AUX card;With 1-2 mezzanine card(s)

lvdsglink

lvdsglink

lvdsglink

Option B: mezzanine card needs only need 1 or 2 types of mezz card(s) design

1-2 types mezz card: possible absorbed in one PCB design

LVDS

GLINK

LVDS

TAXIoptical

GLINK to FIB

FIB return

TSI/beam structure

TSI (taxi, design alreadyexists on Pulsar mezz)

VRB interface can be absorbed on Pulsar AUX card

glinkglink

Option C: Have all interface upfrontWith two types ofmezzanine cards

glinkglink

lvdslvds

lvdslvds/taxi

If we want tokeep FIB returnLVDS, GlinkMezzanine cardwill be double-width,Pulsar-SRCwill take 3 slotsin this case.

No need for P3.DataIO FPGAsdo most of the work:one for FIB interface,The other for the rest.VRB can be absorbedon mezzanine card

Gigabit EthernetRF clock

SRAMs

Pulsar is designed to be fully self-testable: at board level as well as system level

For every input/output, there can be an output/input,

This allows us to develop&tune an upgrade system standalone. Pulsar design philosophy: (1) In God we trust, everything else we test; (2) One for all, and all for one;

PCI

SVT

SVT

Control/ Merger

TS

L1

Data IOSVT

Mezz card (CMC)connectors

Pulsar design: general purpose board for HEP

9U VME(VME and CDF ctrl signals are visible to all three FPGAs)

P1

P2

3 Altera APEX 20K400_652 (BGA) FPGAs 502 user IO pins each

P3

SLINKsignal lines

Data IO

spare lines

SRAM

SRAM

128K x 36 bits

3 APEX20K400-1XV FPGAs on board = 3 Million system gates/80KB RAM per board2 128K x 36 pipelined SRAMs with No Bus Latency: 1 MB SRAM (~5ns access time)

Pulsar top view

Pulsar bottom view

Control FPGA

DataIO FPGA 1

VME chip

DataIO FPGA 2

VME interface

• VME interface is based on the same chip on all UC boards:

e.g. GB board

• VME interface is available to ALL three FPGAs

Control FPGA

DataIO FPGA 1

VME chip

DataIO FPGA 2

Pulsar clocks• each FPGA has four clock inputs: (1) CDFCLK; (2) user defined clock (40 MHz default, for SLINK interface w P3)(2) Algorithm clock (PLL, tested up to 114MHz); OR (4) Tevatron RF clock (PLL pin on FPGAs)

SVT interfaces1MB SRAM(2 128K x 36): 4 ns access time, pipelined NoBL SRAM

SVT data out

SVT data in

SVT data inSVT data in

Control FPGA

DataIO FPGA 1

DataIO FPGA 2

VME chip

SRAM

SRAM

SVT

2 SLINK

SVT input

SVT output

P2 SVT inter-commLines(5):Master&Slave

As it is, Pulsar can be used as a powerful GhostBuster board

SRAM:CY7C1350

Control FPGA

VME chip

Pulsar P3 interface

P3:117signallines

P3 connectionuses P2 styleconnector,117 signalsmappedto all 5 rows.

Signal Pinmap can bemadecompatiblewith AM board

All signals directly visibleto Control FPGA

43 signal lines each

Control FPGA

DataIO FPGA 1

VME chip

DataIO FPGA 2

Pulsar P2 inter-communication lines

•Pulsar has five SVT style inter-communication lines on P2 visible toall threeFPGAs.

Master andSlave

P2

Level 1 & TS interfaces all Level 1 trigger bits can be directly available to all FPGAs

Control FPGA

DataIO FPGA 1

DataIO FPGA 2

VME chip

SRAM

SRAM

TS

64 bits

L1

TS interface with Control FPGA

AUX card

SLINK/mezzanine interfaces

mezzanine card

mezzanine card

P3

Each mezzanine card has up to 83 user defined signal lines;Four mezzanine card slots up front, 2 in the back of crate. fully compatible with ATLAS DAQ/Trigger

Mezzanine card slotscan be usedfor expandedSRAM,FPGA,inputs/outputs etc.

mezzanine card

mezzanine card

Mezzanine card slots: 5V/3.3V/2.5V power provided by Pulsar

AUX Card

Pulsar

Hotlink Tx

Hotlink Rx

Taxi Tx

Taxi Rx

MOAB (Mother of All Boards) for L2 upgrade

SLINK LSC/LDC (ODIN/HOLA)

ANL SLINK->GBE

                                                                                   

               

• SLINK to Gigabit LSC from ANL, or SLINK LSC/LDC from CERN

NetGear GA621

• Pulsar is compatible with ATLAS DAQ/trigger: via SLINK

Tested with internal clockup to 100 MHz Oct. 21

SLINK LSC

AUX card

PC interface: Gigabit Ethernet / SLINK-PCI or anything one can define on AUX card

SLINK

ANL LSC

Netgear GA621

Performance with Gigabit Ethernet, see:http://www.hep.anl.gov/reb/cdf/risto/risto_talk.pdf

Round-Trip timing:

Round-Trip timing measured & the performance looks good

w/ algo : Blue w/o algos : Red

From CERN

Control FPGA

DataIO FPGA 1

VME chip

DataIO FPGA 2

P2 CDF control signals

•Pulsar sees ALL P2 CDF control signals

•Available to ALL three FPGAs

• can even drive P2 backplaneif necessary

Top 10 reasons for Pulsar to bepotentially useful for SVT upgrade

• Pulsar is a grandson of SVT (young & energetic);• Same P2 inter-communication lines (SVT master&slave);• SVT input is available to all three FPGAs;• fast large SRAMs (two 128K x 36) already available on board;• mezzanine card slots can be used for expansion (extra SRAM/FPGA, extra inputs/outputs);• Back of crate (AUX) card can be used for expansion as well;• P3 interface can be made compatible with AM board;• as it is, Pulsar can be used as a powerful GhostBuster board;• with G-Link mezzanine cards, Pulsar can source/sink SVT system (i.e. SVX fiber data)• communication within Pulsars and to new L2 decision crate can be done with high bandwidth S-LINK free Pulsars for SVT, this is business within family… time to brain storm … see Alex’s talk…

A major fraction of the design effort was dedicated to extensive design optimization and verification by using state-of-the-art CAD tools:

• Leonardo Spectrum for VHDL synthesis;

• Altera Quartus II for place and route, FPGA level simulation

• Mentor Graphics QuickSim for board and multi-board level simulation;

• Interconnect Synthesis tool for trace and cross talk analysis;

• IS_MultiBoard tool for signal integrity checks between motherboard and mezzanine & AUX cards;

we have carefully simulated the board(s) before sending out the prototypes …

Pulsar Design methodology

Example: Multi-board (9 boards) simulation to verify the design (top level schematics)

It took 1.5 GB memory on a 2GHz/2GB modern PC to simulate 9 boards together at the same time

This design methodology allowed us to build flawless prototype boards

The self-test capability of the board design made it possible for us to fully test (all interfaces) the prototype boards within 6 weeks after assembly.

The prototype boards were also tested with on board clock speed up to 114 MHz and no problems were found.

So far, Pulsar has been tested with

~ 1Billion events (Tx->Rx) without

single error…

No single blue wire on all five prototype board (Pulsar + four mezzanine cards)

All firmware (VHDL code) in CVS

Pulsar Design methodology