Introductionto TEL62

(common)firmware

M. SozziPisa - January 30/31, 2014

• Firmware development for the TEL62 must be a collaborative effort: Pisa will drive it, but contribution by at least 1 person per sub-detector is essential (need not be a firmware expert, but somebody willing to devote time to this)

• Mostly centered on TDC-daughtercard operation (but identify common blocks usable for LKr/L0 operation)

• Any sub-detector could be (in principle) L0-triggering or not

• Firmware will likely be (slightly) different for different sub-detectors: strictly confine sub-detector differences in few well-identified blocks.

GeneralitiesMainz 2011

• Use a common development environment• Firmware must be centrally stored, consistent, modular, scalable and understantable to others besides the author• Strict name-space and memory-space constraints• Documentation is not an optional feature to be provided “later”, it must come with the code• Each firmware module must be simulated and should come together with its own test bench code• Live testing and diagnostic features should be incorporated

General principlesMainz 2011

PP0

PP1

PP2

PP3

SL

CCPC(CPU)GLUE

TTCrx

Quad GbE

QDR

DDR

DDR

DDR

DDR

AUX

Daughtercard

(TDCB)

Daughtercard

(TDCB)

Daughtercard

(TDCB)

Daughtercard

(TDCB)

TEL62

Inputbuffer

MON ECS

PP-FPGA V2

Outputbuffer

MON ECS

= OK

= In test

DDRwriter

DDRFramebuffers

PP-SLtester

Test mem 1

ECS

Logger

LogFIFO

ECS

Triginfo RX

SL

Data

FIFOData organizer

DDRarbiterCommon

Monitoring

DDRreader

TDCIB

PP-SLtransmitter

SL

FBmux

DDRmux

ECS

ECS

Datamux

Triggergenerator

Trig

FIFOECS

Triggermux

MON

MON

FIFOarbiterOB

Blackbox

ECS

Data compressor

Hitcounters

Fake trigger generator

PP-SLtransmitter

SL

Triggermux

SubdetectorMonitoring

Choke generatorECS = accessible from CCPC

MON = monitored FIFO

Dataprocessor

All firmware is VHDL files, except test benches for simulation and some obscure verilog simulation files

Pure text: great bonus for version control

Firmware tree

PP-FPGA

SL-FPGA

Sub-detector specific versions later:pp_rich, pp_lav, etc.

Sub-detector specific versions (if needed) later: sl_rich, sl_lav, etc.

TEL62

Common and sub-detector memory spaces

Subdetectorspecific stuff

Example: LAV(by dr. F. Gonnella)

Address spaceAll registers, and most FIFOs/memories have direct ECS access (great bonus for debugging)

Local bus is memory mapped from CCPC (all FPGAs and QDR memory)

Separate sub-detector memory space:COMMON:- Generic registers- Monitoring registers- FIFOs- MemoriesSUB-DETECTOR:- Generic registers- Monitoring registers- FIFOs- Memories

Sticking to the common structure helps readability, maintenance and software integration

Re-usabilitySeveral general-purpose modules available

If you develop a module which can be useful to others you can ask to have it stored in a general library

PP

PP

PP

PP

SL

DD

RD

DR

DD

RD

DR

TDC

TDC

TDC

TDC

Gbit

QD

R

TTCrx

CCPCGLUE(ECS)

Simulation blocks

Pisa

Pisa

Pisa

Pisa

Pisa

Item Notes LHCb? Who Status

Core & monitoring Registers, memories, clocks, spy

YES/NO Pisa Started

Live test blocks Data transfer tests NO Pisa Started

Daughter-card comm For TDCB NO Pisa Started

Data correction Offsets, data manipolation NO

Data formatter/writer Time framing, counting NO Pisa

DDR control Burst writing/reading, arbitration NO Pisa Started

Data extraction Timestamp translation, framing NO Pisa

Trigger handling Trigger type handling, arbitration YES/NO Pisa

Trigger primitive gen RICH version (time multiplicity) Can be different for other SD

NO Perugia Started

Inter-PP comm Trigger primitive exchange NO

PP-FPGA FIRMWARE BLOCKS

Item Notes LHCb? Who Status

Core & monitoring Registers, memories, clocks, spy

YES/NO Pisa Started

Live test blocks Data transfer tests NO Pisa

Data merger Merge 4 PP data YES

Data formatter/writer Multi-event packets YES

Gigabit control Ethernet (as in TELL1 + receiver)

YES/NO

QDR control Burst writing/reading, arbitration YES Pisa

Trig primitive merger Merge 4 PP trigger data NO

Trig primitive handling

Merge other board trigger data,format, send to Gbit

NO

TTCrx handling Timestamps, trigger type, resets YES/NO Pisa

AUX-board handling Inter-board communication NO

SL-FPGA FIRMWARE BLOCKS

Item Notes LHCb? Who Status

TDC simulation block NO Pisa Started

DDR simulation block NO Pisa

QDR simulation block NO Pisa

TTCrx simulation block YES/NO Pisa

CCPC simulation block YES/NO Pisa

Gbit simulation block YES/NO

AUX card Design and building --

JTAG test vectors -- Roma 2

OTHER TASKS

Some NA62 TDAQ concepts

Refer to NA62 TDR for more information

Three root concepts characterize the NA62 TDAQ

1.Full-digital integrated Trigger and DAQ(no separate sources for trigger and main data)

2.Unique communication channels(all commands through TTC, all replies through data path)

3.Strong state checking (mandatory reply to commands)

Timing (NA62)sob (pulse): start of burst (from TTC system)eob (pulse): end of burst (from TTC system)done (pulse): cleanup finished after eob (from every module)in_burst, in_endburst (levels)

When in RUN mode:

in_burst=0in_endburst=0

in_burst=1in_endburst=0

in_burst=0in_endburst=1

sob

eob

done

(L0) triggers (NA62)Valid sequence (see NA62 TDR):

SOB signalSOB triggerAny other triggerEOB signalEOB trigger

Any other trigger sequence is invalid.

Note that “EOB” is derived from SPS machine signal EE with a (common) arbitrary delay to allow for out-of-beam triggers, calibration, etc.

Both “signals” and “triggers” are actually transmitted through TTC- Signal have direct effect on hardware- Triggers have a “L1” (TTC naming for LHC) pulse determining the 25ns timestamp, followed by a mandatory “Broadcast message” (6-bit trigger type for NA62)

Choke/error (NA62)Experiment-wide levels from each sub-detector to central L0 Trigger Processor (only 1 choke + 1 error line for a whole sub-detector)

CHOKE: anomalous condition with buffers filling up, system still working fine, no data loss

ERROR: anomalous condition, possibly unrecoverable, which might have caused data loss

All systems are notified of start and finish of such conditions and must acknowledge this

Monitored and recorded

Some project-wide signalsClocks: mostly 160 MHzMultiple clock domains sometimes a necessity:- ECS works at 20 MHz- TDCBs communicate at 40 MHz- GbE works at 120 MHz

Reset (pulse): everything, at initializationreset_error (pulse): clear error conditions (normally not to be done at SOB)reset_monitor (pulse): clear counters, etc.freeze (level): stop everything for debugging (level)snapshot (pulse): store some monitoring values in registerssob: start of burst (pulse)eob: end of burst (pulse)done: cleanup finished after eob (pulse)in_burst, in_endburst

Module tests

Providing a module self-test feature is recommended:

-Poor man’s way: a script setting registers properly and triggering a sequence of actions resulting in some known data to be displayed

-Nicer way: use the internal self-test firmware framework (TEST mode): inject test stimuli stored in ROM on a “test start” pulse and set “test result” lines

Error reportingMost modules are allowed a (single) error lineError condition must be sticky (not automatically reset with new burst)Error conditions can raise global ERROR lineIf more than one error condition must be distinguished, connection to the logger module must be used (recommended for debugging)Severe errors must be unrecoverable

Also consider “crazy” illegal input to your module

Protocol for handling of each kind of error will consolidate with time

Modules can also provide input (maskable) for CHOKE generation

LoggingThe logger module (both in each PP and in SL) is a small (circular or linear) memory recording desired module conditions (usually errors) with automatic time-stamping and storage capability of a few module-specific 32-bit words

In practice: a LOGGER_INTERFACE module exists to simplify operation (see examples). If you have multiple sources (i.e. state machines) for your (single) log lane some tricks are required (see examples).

End of Burst data

In NA62 End of Burst data (monitoring, status, diagnostics, statistics, etc.) is stored together with event data, making offline data quality selection much easier and convenient.

This is obtained because EOB data is the same as event data, with a different format as allowed for a different trigger.

Your modules might want to provide some EOB data: describe the format and interface to the appropriate module(TRIGINFO_RX in PP and SL_DATA_SOURCE in SL)

EOB data is (mostly) stored in ASCII format (!) for ease of inspection in raw data files.