David AbbottDavid Abbott

CODA3 - DAQ and Electronics CODA3 - DAQ and Electronics Development for the 12 GeV UpgradeDevelopment for the 12 GeV Upgrade

OutlineOutline

What is CODAWhat is CODA DAQ Issues / Aging TechnologiesDAQ Issues / Aging Technologies Requirements for 12 GeVRequirements for 12 GeV CODA3 DevelopmentsCODA3 Developments

SoftwareSoftware HardwareHardware

SummarySummary

The CODA missionThe CODA mission

CODA is a software toolkit with specialized CODA is a software toolkit with specialized hardware supporthardware support

Modular software components use the Modular software components use the networknetwork for for inter-process communication and event transport.inter-process communication and event transport.

Use Use open standardsopen standards and minimize the use of and minimize the use of commercial software while maximizing use of commercial software while maximizing use of commercial hardware.commercial hardware.

DAQ systems for each experimental Hall can be DAQ systems for each experimental Hall can be ““built-upbuilt-up” from common components to fit their ” from common components to fit their needs.needs.

What is CODAWhat is CODACODA is a software toolkit from which data acquisition systems with varying degrees of complexity can be built.

A typical system might look

Single Board Computer

FASTBUSVMECAMAC

ROC rcServer

EB

ROC

Network: Ethernet FDDI ATM

UNIX/LINUX

DISK/TAPEET

User Proc.

RunControlGUI

ER

cdev

dp_tcl

mSQL Database

msqld cmlogServer

cmlogDatabase

Data Acquisition StatusData Acquisition Status The Data Acquisition Group is charged with the task of The Data Acquisition Group is charged with the task of

providing a functional, unified acquisition system providing a functional, unified acquisition system appropriate for appropriate for ALLALL experimental programs at JLAB. experimental programs at JLAB.

Recent experiments have begun to test the limits of the Recent experiments have begun to test the limits of the current CODA system.current CODA system.

Development in both hardware and software technologies Development in both hardware and software technologies will be necessary to provide continued support of the future will be necessary to provide continued support of the future 6GeV program.6GeV program.

Current DAQ projects reflect the philosophy that we can Current DAQ projects reflect the philosophy that we can progress to support the physics of the 12 GeV program progress to support the physics of the 12 GeV program through an “through an “upgradeupgrade” of the existing proven system.” of the existing proven system.

Front-End Issues…Front-End Issues…

The CODA ROC is a customizable software object that The CODA ROC is a customizable software object that provides a standard for the User to access front-end provides a standard for the User to access front-end hardware - whatever that may be. It must evolve.hardware - whatever that may be. It must evolve.

Obsolete FASTBUS digitizing electronics needs a Obsolete FASTBUS digitizing electronics needs a replacement alternative (in VME). There are far fewer replacement alternative (in VME). There are far fewer appropriate commercial solutions today.appropriate commercial solutions today.

Real-Time readout on a per event basis limits the maximum Real-Time readout on a per event basis limits the maximum accepted L1 trigger rate (~10 KHz).accepted L1 trigger rate (~10 KHz).

32 crate limit on the trigger distribution system is nearly 32 crate limit on the trigger distribution system is nearly reached in Hall B.reached in Hall B.

DAQ - Trigger - Electronics - All must be designed to work DAQ - Trigger - Electronics - All must be designed to work together, efficiently, “customized” !! together, efficiently, “customized” !!

Other DAQ LimitationsOther DAQ Limitations

Event transport limitations in the Event transport limitations in the current CODA architecture are being current CODA architecture are being seen for moderately complex seen for moderately complex systems.systems.

Aging software technologies and Aging software technologies and reliance on third party packages are reliance on third party packages are making code portability and upkeep making code portability and upkeep difficult.difficult.

Monitoring and control of large Monitoring and control of large numbers of distributed objects are numbers of distributed objects are not handled in a consistent way (too not handled in a consistent way (too many protocols).many protocols).

Slow controls only minimally Slow controls only minimally supportedsupported

~30 ---> 1

EB

ROC ROC ROC

50-70 MB/sec

3 - 9 KHz Trigger

Hall B “bottlenecks”

A General PlanA General Plan Replace aging technologiesReplace aging technologies

Run ControlRun Control Tcl/Tk based DAQ ComponentsTcl/Tk based DAQ Components mSQLmSQL FASTBUS/CAMACFASTBUS/CAMAC

Hall D (GlueX) requirements drive development directionHall D (GlueX) requirements drive development direction Maintain Cross platform supportMaintain Cross platform support

Linux - Solaris - vxWorks - OS X - 64 bit…Linux - Solaris - vxWorks - OS X - 64 bit… Support and use new commercial advances where Support and use new commercial advances where

possible.possible. More custom hardware design will be required and an More custom hardware design will be required and an

electronics support group to maintain.electronics support group to maintain.

CODA3 - Requirements/GoalsCODA3 - Requirements/Goals Pipelined Electronics (FADC, TDC)Pipelined Electronics (FADC, TDC)

Dead-timeless systemDead-timeless system Replacement for obsolete electronicsReplacement for obsolete electronics Eliminate large numbers of delay cablesEliminate large numbers of delay cables

Integrated L1/L2 Trigger and Trigger Distribution SystemIntegrated L1/L2 Trigger and Trigger Distribution System Support up to 200 KHz L1 TriggerSupport up to 200 KHz L1 Trigger Use FADC for L1 trigger inputUse FADC for L1 trigger input Support 100+ crates Support 100+ crates

Parallel/Staged Event BuildingParallel/Staged Event Building Handle ~100 of input data streamsHandle ~100 of input data streams Scalable (>1 GByte/s) aggregate data throughputScalable (>1 GByte/s) aggregate data throughput

L3 Online FarmL3 Online Farm Online (up to x10) reduction in data to permanent storageOnline (up to x10) reduction in data to permanent storage

Integrated Experiment ControlIntegrated Experiment Control DAQ RunControl + “Slow” control/monitoringDAQ RunControl + “Slow” control/monitoring Distributed, scalable, and “intelligent” Distributed, scalable, and “intelligent”

Anatomy of a CODA3 DAQ System

Example - Hall D DAQExample - Hall D DAQ

ExistingHalls

Front-End GoalsFront-End Goals

L1 Trigger rate - 200 KHzL1 Trigger rate - 200 KHz Block up Events (200 event block -> 2kHz)Block up Events (200 event block -> 2kHz) Move some ROL runtime code to modules (FPGAs)Move some ROL runtime code to modules (FPGAs) ADCs provide L1 trigger data ( hence we need a ADCs provide L1 trigger data ( hence we need a

distributed high speed clock - 250 MHzdistributed high speed clock - 250 MHz

New Trigger SupervisorNew Trigger Supervisor Perhaps 100+ cratesPerhaps 100+ crates Support pipeline, event blockingSupport pipeline, event blocking Manage flow control into DAQ system backendManage flow control into DAQ system backend

Front-End SystemsFront-End Systems

VME CPUVME CPU -(MV6100) PowerPC, GigEvxWorks or LinuxCODA ROCReadout ~160+ MB/s

Trigger InterfaceTrigger Interface - (V3)

Pipeline TriggerEvent BlockingClock distributionEvent ID Bank Info

F1 TDCF1 TDCFlash ADCFlash ADC

R&D to support fully pipelined crates capable of 200 KHz trigger rates

Level 1 TriggerLevel 1 Trigger Distributed high speed clockDistributed high speed clock

• derived from Accel clock 499 MHzderived from Accel clock 499 MHz• stepped down at crate - inter crate jitter <50ps stepped down at crate - inter crate jitter <50ps

Subset of ROCs collect summed ADC dataSubset of ROCs collect summed ADC data and send it to L1 Trigger - in syncand send it to L1 Trigger - in sync 12 bit sums/crate x 250MHz --> 3 Gbit/s links12 bit sums/crate x 250MHz --> 3 Gbit/s links Trigger decision goes to Trigger SupervisorTrigger decision goes to Trigger Supervisor for pipeline distribution to all crates for pipeline distribution to all crates

ROC ROC ROC ROC ROC ROC ROC ROC ROCROC ROC ROC

L1 Trigger

Trigger Supervisor

200 KHz

Staged/Parallel Event BuildingStaged/Parallel Event Building• Divide total throughput into N streams (1GB/sec -> N*xMB/sec).Divide total throughput into N streams (1GB/sec -> N*xMB/sec).• Two stages - Data Concentration -> Event Two stages - Data Concentration -> Event BuildingBuilding..• Each EMU is a software component running on a separate host.Each EMU is a software component running on a separate host.

2N hosts are interconnected though one BIG switch

Event Management UnitEvent Management Unit

• EMU built around the ETEMU built around the ET system for customizablesystem for customizable processing/distribution ofprocessing/distribution of event streams.event streams.

Examples:Examples:• Data Concentrator for ROCsData Concentrator for ROCs• Sub-Event builderSub-Event builder• Farm distribution pointFarm distribution point• Event RecorderEvent Recorder• User Processes can attach toUser Processes can attach to any EMU in the systemany EMU in the system

Level 3 FarmLevel 3 Farm

Can be used for analysis Can be used for analysis or filteringor filtering

Support 100s of nodesSupport 100s of nodes Nodes can come and go Nodes can come and go

during event takingduring event taking Filtered events will be Filtered events will be

time ordered on the time ordered on the back-end. back-end.

1 GB/s

100 MB/s

Current DAQ Projects - OverviewCurrent DAQ Projects - Overview

Components:Components: Run Control Coda ROC Coda EB/ER (EMU)

Software Tools:Software Tools: cMsg ET evio

Hardware:Hardware: FADC/F1TDC Trigger Interface (VME/PCI) Trigger/Clock Distribution

R&D:R&D: Embedded Linux Experiment Control Staged/Parallel Event Building VXS - High speed serial 200KHz Trigger/readout Low jitter clock distribution

Some Current ProjectsSome Current Projects

cMsgcMsg - homegrown publish-subscribe messaging - homegrown publish-subscribe messaging which will become the basis for DAQ online which will become the basis for DAQ online communicationcommunication

JAVA Agent-based Run ControlJAVA Agent-based Run Control EMUEMU - C/C++ based, modular, threaded - C/C++ based, modular, threaded Event BlockingEvent Blocking

Hardware - New TI to support existing Trigger Sup.Hardware - New TI to support existing Trigger Sup. Software - new ROC, parallel readout lists, EVIOSoftware - new ROC, parallel readout lists, EVIO

Embedded LinuxEmbedded Linux - establish a stable maintainable - establish a stable maintainable Linux distribution for SBCs with access to VME and Linux distribution for SBCs with access to VME and PCI hardware.PCI hardware.

Software ToolsSoftware Tools

cMsgcMsg - Homegrown publish-subscribe - Homegrown publish-subscribe messaging - the basis for all DAQ online messaging - the basis for all DAQ online communication.communication.

ET ET - Event Transport - High speed, shared - Event Transport - High speed, shared memory based event distribution system. memory based event distribution system. Also supports network transport.Also supports network transport.

EVIOEVIO - Library/Tools for reading, writing, - Library/Tools for reading, writing, building, analyzing and displaying events in building, analyzing and displaying events in the CODA framework. the CODA framework.

CODA is a software toolkit from which data acquisition systems with varying degrees of complexity can be built.

A typical system might look

Single Board Computer

FASTBUSVMECAMAC

ROC rcServer

EB

ROC

Network: Ethernet FDDI ATM

UNIX/LINUX

DISK/TAPEET

User Proc.

RunControlGUI

ER

cdev

dp_tcl

mSQL Database

msqld cmlogServer

cmlogDatabase

cMsg - CODA Messaging SystemcMsg - CODA Messaging System

cMsg

Publish-subscribe messaging system JAVA,C++,C implementation and APIs Replaces older separate packages with one maintainable system

Experiment ControlExperiment Control

JVM

Host 1

A

ROC

AHV

RC

CODA Java-Based (v 1.5) “Intelligent” agents (AFECS)CODA Java-Based (v 1.5) “Intelligent” agents (AFECS) JADE extensions provide a runtime “distributed” JVM.JADE extensions provide a runtime “distributed” JVM. Agents provide a customizable intelligence and communication with Agents provide a customizable intelligence and communication with

external processes.external processes.

Host 1 Host 2

Host 3

A

A

ROC

HV

RC

CODA ROC

CODA EMU

EPICS IOC 1

EPICS CAG

Trigger soft

Trigger hard

Online ANA

A

A

A

A

A

A

A

S

S

S

S

S

NR NA NC

Physical ComponentsNormative Agents

Supervisor agent

Grand supervisor

Front-End

ACC

AFECS Platform

IPC

GUI/user

IPC

WEB

IPC

Hierarchy of Control

IPC

Run Control GUIRun Control GUI

Experiment Monitoring and Experiment Monitoring and ControlControl

JLAB Pipeline TDCJLAB Pipeline TDC

010002000300040005000600070008000

0800016000240003200040000480005600064000

Normal Resolution

Delay (ns)

Count

slope = 118.94 ps/cnt

High Resolutionslope = 59.47 ps/cntTDC ASIC (8 channels)

Hits

Trigger

Time -->

JLAB Flash ADCJLAB Flash ADCfADC250 @ 250 MSPS

Pulse on CH 1: 30 MHz, tr & tf = 5 ns, Width = 8 ns

0

200

400

600

800

1000

0 10 20 30 40 50

Sample # (4 ns/sample)

ADC Value

FPGAs

Hits

TriggerTime -->

8µs

VME64X - VXS InterconnectVME64X - VXS Interconnect

J0 - 200 total pins 45 differential pairs 6 GHz Bandwidth

18 VME Payload Slots

2 Switching slots

Electronics DevelopmentElectronics Development

VXS - L1 Trigger VXS - L1 Trigger

VME CPUVME CPU -(V7865) Intel, GigELinux CODA ROCVME Readout of Event Data

Switch Sum and Trigger Switch Sum and Trigger Distribution Modules (VXS)Distribution Modules (VXS)

Collect Sums/HitsPass Data to Master L1Clock distributionTrigger Distribution

Flash ADCFlash ADC

Use VXS High speed serial backplane (P0) to collect Energy sum and hit data from FADCs

Flash ADCFlash ADC

P0P0

ATCA as a StandardATCA as a Standard

Becoming a widely adoptedBecoming a widely adoptedCommercial standardCommercial standard

8U Blade Form-factor 8U Blade Form-factor

High-Power/High AvailabilityHigh-Power/High Availability

Few applications in Physics Few applications in Physics yet and not particularly cheapyet and not particularly cheap

Possible use for Master Possible use for Master Trigger processor and TriggerTrigger processor and TriggerSupervisorSupervisor

SummarySummary

CODA version 3 is now being molded - and CODA version 3 is now being molded - and can be adapted to future requirements!! can be adapted to future requirements!!

Our plan is to phase in new tools to provide Our plan is to phase in new tools to provide a smooth transition from CODA2 --> a smooth transition from CODA2 --> CODA 3 (CODA 2.6 Release)CODA 3 (CODA 2.6 Release)

New custom electronics will be available for New custom electronics will be available for use soon.use soon.