7. cbm collaboration meetingxdaq evaluation - j.adamczewski1

7. CBM collaboration meeting XDAQ evaluation - J.Adamczewski 1


Outline

• Requirements of DAQ framework for CBM

• The CMS XDAQ framework

• XDAQ on the CBM IB-Cluster

• Summary and further evaluation


DAQ for CBM

• Triggerless data transport until filter farm

• B-net: 1000 x 1000 nodes, high speed connections

• Linux may run on all DAQ nodes (even FPGAs)

• Test cluster with InfiniBand:

small „demonstrator“ set-up within next year


Data Acquisition framework requirements

• Configuration of multiple nodes

(connections and algorithms, database?)

• Controls and monitoring

• Data transport (format, protocol, hardware drivers?)

• Message logging

• Error handling and failure recovery (ARMOR?)

• Modular architecture, extensability! (sub-detector tests)

• User interface?


Standard DAQ framework for LHC CMS experiment

(Orsini, Gutleber) http://xdaqwiki.cern.ch

• C++ libraries on Linux, modular packages

• Each node: context, xdaq process with

embedded xdaq applications

• Configuration: XML

• Data transport: I2O protocol (Intelligent IO)

• Communication: http, cgi; SOAP messages

The CMS XDAQ framework


XDAQ features continued... http://xdaqwiki.cern.ch

• State machines (sync/async FSM)

• Message logger, error handler

• Monitoring tool

• Hardware access library (HAL)

• Front End Driver (FED kit, for CMS!)

• Job Control (task handler for node control)

• others:

• exceptions

• threads

• infospace

• data (de)serializers , ...

The CMS XDAQ framework


XDAQ: messaging and peer transport

0..*

0..*

0..1

0..1

pt::JxtaMessenger

+send:void

service:std::string

exception

UnknownProtocolOrService DuplicateEndpoint AddressMismatch DuplicateListener InvalidFrame Exception NoListener SendFailure PeerTransportNotFound InvalidAddress ReceiveFailure NoMessenger MaxMessageSizeExceeded

pt::JxtaListener

+processIncomingMessage:void

service:std::string

pt::BinaryMessenger

+~BinaryMessenger+send:void

localAddress:pt::Address::Reference destinationAddress:pt::Address::Reference

pt::Messenger

+~Messenger

service:std::string

pt::HTAccessSecurityPolicy

+checkAccess:bool+checkAuth:bool+isAccessLimited:bool+isAuthLimited:bool

policyName:std::string name:std::string type:std::string

pt::PeerTransport

+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+isServiceSupported:bool

type:TransportType protocol:std::string supportedServices:std::vector<std::string>

pt::PeerTransportAgent

+addPeerTransport:void+removePeerTransport:void+getPeerTransport:pt::PeerTransport*+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+getMessenger:pt::Messenger::Reference+removeListener:void+getListener:pt::Listener*+getListeners:std::vector<pt::Listener*>+removeAllListeners:void+addListener:void

peerTransports:std::vector<pt::PeerTransport*>

pt::SecurityPolicyFactory

-instance_:SecurityPolicyFactory*-policies_:std::map<std::string,pt::SecurityPolicy*,std::less<std::string>>

+getInstance:SecurityPolicyFactory*+destroyInstance:void+getSecurityPolicy:pt::SecurityPolicy*+addSecurityPolicyImplementation:void+removeSecurityPolicyImplementation:void+getSecurityPolicyNames:std::vector<std::string>-SecurityPolicyFactory-~SecurityPolicyFactory

pt::SecurityPolicy

+~SecurityPolicy

policyName:std::string

pt::PeerTransportSender

+getMessenger:pt::Messenger::Reference

pt::PeerTransportAgentImpl

-instance_:PeerTransportAgentImpl*-senders_:std::vector<pt::PeerTransport*>-receivers_:std::vector<pt::PeerTransport*>-services_:std::map<std::string,pt::Listener*,std::less<std::string>>

+~PeerTransportAgentImpl+instance:PeerTransportAgent*+addPeerTransport:void+removePeerTransport:void+getPeerTransport:pt::PeerTransport*+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+getMessenger:pt::Messenger::Reference+addListener:void+removeListener:void+getListener:pt::Listener*+getListeners:std::vector<pt::Listener*>+removeAllListeners:void

peerTransports:std::vector<pt::PeerTransport*>

pt::Address

+~Address+toString:std::string+equals:bool

protocol:std::string service:std::string serviceParameters:std::string

pt::BinaryListener


pt::SOAPListener

+processIncomingMessage:xoap::MessageReference

service:std::string

pt::Listener

service:std::string

pt::SOAPMessenger

+~SOAPMessenger+send:xoap::MessageReference

service:std::string localAddress:pt::Address::Reference destinationAddress:pt::Address::Reference

pt::PeerTransportReceiver

-listeners_:std::map<std::string,pt::Listener*,std::less<std::string>>

+~PeerTransportReceiver+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+isExistingListener:bool+getListener:pt::Listener*+config:void

peer transport

xoap::SOAPBodyElementxoap::SOAPFault

+SOAPFault+getFaultCode:std::string+getFaultString:std::string+setFaultCode:void+setFaultString:void

xoap::StandardObjectPolicy

+dispose:void

exception

HTTPException Exception

xoap::Endpoint

#id_:std::string

+Endpoint+toString:std::string

xoap::SOAPConnection

#httpURLConnection_:HTTPURLConnection

+SOAPConnection+call:xoap::SOAPMessage

xoap::URLEndpoint

+URLEndpoint

URL:std::string

xoap::SOAPAllocator

+allocate:void *+dispose:void+testMemoryLeakage:void

xoap::MimeHeaders

-headers_:std::multimap<std::string,std::string,std::less<std::string>>

+MimeHeaders+getHeader:std::vector<std::string>+setHeader:void+addHeader:void+removeHeader:void+removeAllHeaders:void

allHeaders:std::multimap<std::string,std::string,std::less<std::string>>

xoap::XStr

-fUnicodeForm:XMLCh*

+XStr+XStr+~XStr+unicodeForm:const XMLCh*+operatorconstXMLCh*:

xoap::HTTPURLConnection

#socket_:int

+HTTPURLConnection+~HTTPURLConnection+receiveFrom:std::string+sendTo:void+close:void+connect:void#receive:int#send:void#open:void#extractMIMEBoundary:std::string#writeHttpPostMIMEHeader:void#writeHttpPostHeader:void

xoap::SimpleReferenceCount

-counter:unsigned long *

+SimpleReferenceCount+init:void+dispose:void+increment:void+decrement_and_is_zero:bool

xoap::SOAPName

#prefix_:std::string#name_:std::string#uri_:std::string

+SOAPName+operator==:bool

qualifiedName:std::string localName:std::string prefix:std::string URI:std::string

T:typenameSimpleReferenceCount:typename CounterPolicy =StandardObjectPolicy:typename ObjectPolicy =

CounterPolicyObjectPolicy

xoap::CountingPtr

-object_pointed_to:T*

+CountingPtr+CountingPtr+CountingPtr+~CountingPtr+operator=:CountingPtr<T,CP,OP>&+operator=:CountingPtr<T,CP,OP>&+operator->:T*+operator*:T&+isNull:bool-init:void-attach:void-detach:void

LISTENER:classxoap::Method

+obj_:LISTENER*+func_:xoap :: MessageReference+name_:std::string+namespaceURI_:std::string

+~Method+type:std::string+invoke:xoap::MessageReference+name:std::string+name:void+namespaceURI:void+namespaceURI:std::string

toolbox::lang::Methodxoap::MethodSignature

+invoke:xoap::MessageReference+name:std::string

i2o

0..*

0..1

1..*

0..1

0..1

0..1

0..1

xdaq::WebApplicationxdata::ActionListener

PeerTransportTCP

-pts_:tcp::PeerTransportSender*-ptr_:tcp::PeerTransportReceiver*-autosize_:xdata::Boolean-maxPacketSize_:xdata::UnsignedLong-poolName_:xdata::String

+XDAQ_INSTANTIATOR:+PeerTransportTCP+~PeerTransportTCP+Default:void+apply:void+failurePage:void-actionPerformed:void-reset:void

tcp::Transmitter

+isConnected_:bool+retry_:unsigned long

+Transmitter+~Transmitter+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool

pt::Addresstcp::Address

#url_:toolbox::net::URL#service_:std::string

+Address+~Address+toString:std::string+equals:bool

protocol:std::string service:std::string serviceParameters:std::string host:std::string port:std::string URL:std::string path:std::string socketAddress:sockaddr_in

i2o::Messengertcp::I2OMessenger

-pt_:tcp::PeerTransportSender*-channel_:tcp::Channel*-destination_:pt::Address::Reference-local_:pt::Address::Reference

+I2OMessenger+~I2OMessenger+send:void


tcp::Channel

#sockaddress_:sockaddr_in#sockaddressSize_:socklen_t#socket_:int

+Channel+~Channel+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool

exception

CannotConnect Exception

pt::PeerTransportSenderTask

tcp::PeerTransportSender

-outputQueue_:toolbox::squeue<tcp::PostDescriptor>-activated_:bool-done_:bool

+PeerTransportSender+~PeerTransportSender+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+isServiceSupported:bool+getMessenger:pt::Messenger::Reference+svc:int+post:void+start:void+stop:void

type:pt::TransportType protocol:std::string supportedServices:std::vector<std::string>

toolbox::lang::Classtcp::ReceiverLoop

-sockets_:std::vector<int>-address_:pt::Address::Reference-fdset_:fd_set-allset_:fd_set-timeout_:timeval-done_:bool-stop_:bool-maxfd_:int-current_:int-listenfd_:int-accepted_:int-nochannels_:int-nready_:int-autoSize_:bool-maxPacketSize_:unsigned long-listener_:i2o::Listener*-pool_:toolbox::mem::Pool*-manager_:toolbox::mem::MemoryPoolFactory*-process_:toolbox::task::ActionSignature*-logger_:Logger

+ReceiverLoop+~ReceiverLoop+receive:int+send:void+disconnect:void+connect:void+close:void+process:bool+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+activate:void+cancel:void-accept:int-isConnected:bool-isActive:bool-onRequest:void-safeReceive:void

_enum ReceiverLoop h 50

address:pt::Address::Reference pool:toolbox::mem::Pool* maxPacketSize:unsigned long autoSize:bool

structtcp::PostDescriptor

+ref:toolbox::mem::Reference*+channel:tcp::Channel*+handler:toolbox::exception::HandlerSignature*+context:void *

pt::PeerTransportReceivertcp::PeerTransportReceiver

#autoSize_:bool#maxPacketSize_:unsigned long#loop_:std::vector<tcp::ReceiverLoop*>#pool_:toolbox::mem::Pool*#logger_:Logger

+PeerTransportReceiver+~PeerTransportReceiver+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+isServiceSupported:bool+config:void+start:void+stop:void

type:pt::TransportType protocol:std::string supportedServices:std::vector<std::string> pool:toolbox::mem::Pool* maxPacketSize:unsigned long autoSize:bool

tcp0..1

1..*

0..1

exception

Exception

i2o::Messengerfifo::I2OMessenger

-pt_:fifo::PeerTransport*

+I2OMessenger+send:void


pt::PeerTransportSenderpt::PeerTransportReceiver

toolbox::lang::Classfifo::PeerTransport

#sync_:BSem*#mutex_:BSem*#fifo_:toolbox::rlist<fifo::PostDescriptor>#pending_:int#i2oListener_:i2o::Listener*#messenger_:pt::Messenger::Reference#localAddress_:pt::Address::Reference#process_:toolbox::task::ActionSignature*#logger_:Logger

+PeerTransport+~PeerTransport+post:void+process:bool+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+isServiceSupported:bool+getMessenger:pt::Messenger::Reference+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+config:void

_enum PeerTransport h 119


pt::Addressfifo::Address

#url_:toolbox::net::URL#service_:std::string


protocol:std::string service:std::string serviceParameters:std::string host:std::string port:std::string URL:std::string path:std::string

xdaq::ApplicationPeerTransportFifo

-pt_:fifo::PeerTransport*

+XDAQ_INSTANTIATOR:+PeerTransportFifo+~PeerTransportFifo

structfifo::PostDescriptor

+ref:toolbox::mem::Reference*+handler:toolbox::exception::HandlerSignature*+context:void *

fifo0..1

0..1

0..1

0..*

pt::SOAPMessengerhttp::SOAPLoopbackMessenger

-local_:pt::Address::Reference-destination_:pt::Address::Reference

+SOAPLoopbackMessenger+~SOAPLoopbackMessenger+send:xoap::MessageReference


http::Utils

+receiveHeaderFrom:char *+receiveBodyFrom:char *+receiveFrom:char *+sendTo:void+replyTo:void+replyBodyTo:void+replyHeaderTo:void#extractMIMEBoundary:std::string#writeHttpPostMIMEHeader:void#writeHttpPostHeader:void#writeHttpReplyMIMEHeader:void#writeHttpReplyHeader:void

toolbox::lang::Classhttp::ReceiverLoop

-sockets_:std::vector<int>-clientIP_:std::map<int,std::string,std::less<int>>-clientHost_:std::map<int,std::string,std::less<int>>-address_:pt::Address::Reference-fdset_:fd_set-allset_:fd_set-maxfd_:int-current_:int-listenfd_:int-accepted_:int-nochannels_:int-nready_:int-listener_:pt::SOAPListener*-cgiListener_:xgi::Listener*-process_:toolbox::task::ActionSignature*-httpRootDir_:std::string-logger_:Logger-policy_:pt::HTAccessSecurityPolicy*-is_:xdata::InfoSpace*-aliasName_:std::string-aliasPath_:std::string

+ReceiverLoop+~ReceiverLoop+receive:int+send:void+disconnect:void+connect:void+close:void+process:bool+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+activate:void-accept:int-isConnected:bool-isActive:bool-onRequest:void-reply:void-authenticateUser:bool-verifyAccess:bool-isBrowserSupported:bool

_enum ReceiverLoop h 40

address:pt::Address::Reference

exception

CannotConnect Exception

http::ClientChannel

+ClientChannel+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool

pt::Addresshttp::Address

#url_:toolbox::net::URL*#service_:std::string


protocol:std::string service:std::string serviceParameters:std::string host:std::string port:std::string URL:std::string path:std::string socketAddress:sockaddr_in

pt::PeerTransportReceiverhttp::PeerTransportReceiver

#loop_:std::vector<http::ReceiverLoop*>#logger_:Logger#is_:xdata::InfoSpace*

+PeerTransportReceiver+~PeerTransportReceiver+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+addServiceListener:void+removeServiceListener:void+removeAllServiceListeners:void+isServiceSupported:bool+config:void


pt::PeerTransportSenderhttp::PeerTransportSender

#sync_:BSem*#mutex_:BSem*#logger_:Logger

+PeerTransportSender+~PeerTransportSender+createAddress:pt::Address::Reference+createAddress:pt::Address::Reference+isServiceSupported:bool+getMessenger:pt::Messenger::Reference


http::Channel

#sockaddress_:sockaddr_in#sockaddressSize_:socklen_t#socket_:int#mutex_:BSem

+Channel+~Channel+connect:void+disconnect:void+receive:int+send:void+close:void+isConnected:bool+lock:void+unlock:void

xdaq::ApplicationPeerTransportHTTP

-pts_:http::PeerTransportSender*-ptr_:http::PeerTransportReceiver*-aliasName_:xdata::String-aliasPath_:xdata::String

+XDAQ_INSTANTIATOR:+PeerTransportHTTP+~PeerTransportHTTP

pt::SOAPMessengerhttp::SOAPMessenger

-channel_:http::ClientChannel*-local_:pt::Address::Reference-destination_:pt::Address::Reference

+SOAPMessenger+~SOAPMessenger+send:xoap::MessageReference


http

0..1

0..1

i2o::Listener

+getService:std::string+processIncomingMessage:void

service:std::string

LISTENER:classstruct

i2o::Method

+obj_:LISTENER*+func_:void+fid_:unsigned long

+type:std::string+invoke:void+fid:unsigned long+fid:void

toolbox::lang::Methodi2o::MethodSignature

+invoke:void+fid:unsigned long

i2o::Messenger

+getService:std::string

service:std::string

exception

Exception

i2o::utils::Dispatcher

#logger_:Logger#registry_:xdaq::ApplicationRegistry*#context_:xdaq::ApplicationContext*#addressMap_:i2o::utils::AddressMap*

+Dispatcher+~Dispatcher+processIncomingMessage:void

pt::BinaryListener


pt::BinaryMessenger

+~BinaryMessenger+send:void+getLocalAddress:pt::Address::Reference+getDestinationAddress:pt::Address::Reference localAddress:pt::Address::Reference destinationAddress:pt::Address::Reference

pt::Listener

+getService:std::string

service:std::string

pt::Messenger

+~Messenger+getService:std::string

service:std::string

i2o::utils::AddressMap

-instance_:AddressMap*-tidToApplication_:std::map<unsignedlong,xdaq::ApplicationDescriptor*,std::less<unsignedlong>>-applicationToTid_:std::map<xdaq::ApplicationDescriptor*,unsignedlong,std::less<xdaq::ApplicationDescriptor*>>

+getApplicationDescriptor:xdaq::ApplicationDescriptor*+getTid:unsigned long+setApplicationDescriptor:void+removeTid:void+clear:void+getInstance:AddressMap*+destroyInstance:void

instance:AddressMap*

SOAP


XDAQ: I2O messaging and peer transport


XDAQ: web server

hyperdaq web interface


XDAQ executive with applications

Common XML configuration file:

• Each node knows all

applications on all nodes!

• Unique addressing by

context (url) and application id


XDAQ: SOAP messaging

xrelay: send SOAP control messages

• example commands:

Configure, Enable, Halt, Reset,..

• any new commands may be defined

• web interface not suited as

real control system UI!

• XDAQ applications may exchange

SOAP messages with other UI

(Labview?)


XDAQ on the CBM InfiniBand cluster

Installation problems:

• XDAQ distribution not 64 bit save!

Code adjustments required

• kernel modules for CERN linux excluded!

• drivers for CMS hardware (FED) excluded

• coretools done, worksuite partially,…

Basic evaluation possible!


XDAQ tests on CBM InfiniBand cluster

Tested features on IB cluster:

http://wiki.gsi.de/cgi-bin/view/Daq4FAIR/DaqClusterSoftwareXdaqStatus

• hyperdaq webserver (main test controls ui)

• xml configuration setup

• control variable access and export (infospace)

• SOAP messaging (test controller application)

• state machines (sync./async. with web/SOAP ui)

• monitoring application (very raw ui!)

• multithreading (workloop framework)

• exceptions, data wrappers, message logging, toolbox,…

• i2o data transport (tcp roundtrip example)


I2O roundtrip on CBM InfiniBand cluster

Standard XDAQ roundtrip benchmark:

• sender node posts I2O frame to receiver node

• receiver callback sends frame back to sender

• sender callback sends frame back again, etc…

• perfmeter class records bandwidth, latency,…

• web display of results on sender node (java!)

Code independent of transport implementation!

XML configuration file does set up (tcp ethernet, tcp IB, fifo)


XDAQ executive

MyRoundTrip

HyperDAQ


sender receiver

XDAQ executive

MyRoundTrip

token() token()

EnableAction()

default()

I2O callbackI2O callback

web application

state machine

tcp over IB


XDAQ pseudo roundtrip with udapl rdma

Developed udapl roundtrip in xdaq application:

• Use of C++ wrapper for udapl (S. Linev)

• XDAQ roundtrip example, but without I2O

• no package return in receiver callback,

but IB-RDMA write-read from sender

• measurements with XDAQ perfmeter tool

XDAQ compatible with UDAPL library (multithreading!)

Overhead compared with plain UDAPL!


XDAQ executive

MyDAPLRoundTrip

HyperDAQ

XDAQ pseudo roundtrip with udapl rdma

sender receiver

XDAQ executive

MyDAPLRoundTrip

Benchmark()

ConfigureAction()

EnableAction()

default()

state machine

workloop

(thread)

web application

ConfigureAction()

state machine

allocated memuDAPL/RDMA



955 MB/splain RDMA

813 MB/s

XDAQ with RDMA

234 MB/s

XDAQ, I2O, TCP over IB

penalty of XDAQ framework?

measurement? procedure?

minimum latency: 55 µs 30 µs4 µs


Summary and Outlook

• CBM DAQ system needs software framework

• XDAQ offers many required features

• Tests on CBM IB cluster:

• SOAP controls, web server, monitor, i2o over tcp

• XDAQ works together with uDAPL library (IB)

• To do:

• I2O implementation of uDAPL – performance?

• Configuration and control system?

• XDAQ scalability? Job Control?

• Implementations for DAQ demonstrator hardware?

7. cbm collaboration meetingxdaq evaluation - j.adamczewski1

Documents