EPICS Internals

Nick Rees

Outline• Introduction• Libraries and modules• Data Structures• Process types• Examples of processing

INTRODUCTION

Background• When I accepted this challenge I thought there

was an EPICS internals talk I could copy.– It turns out there wasn’t.

• I decided the aim of the talk was to provide EPICS developers with a vague idea of what to expect to see when they look at a running EPICS system.– It turns out this wasn’t that easy– It also was very difficult to do this in 40 minutes.– By focussing on this I missed out more than I covered.

• The result is something which even I find a bit difficult to understand (and I wrote the talk), so I don’t know about the rest of you.– For the non-programmers out there, good luck.

What is EPICS core?

What is EPICS core?• 109 directories• 1567 files, 239241 lines of all types

– 281 files, 93451 lines of C– 203 files, 49807 lines of C++– 329 files, 34985 lines of header files– 28 files, 14964 lines of html

• Over 20 years of development by a competent world-wide team.

• Not an enormous body of code.• A pretty reliable body of code that does what it is

designed to do.• A reasonably useful, flexible and portable tool

What isn’t EPICS core?• Consistently designed• Designed• Buzzword compliant, or even object-oriented• Well understood by lots of people.• Stylistically consistent.• Pretty, elegant or pure

Another view• The Application Developers guide states:

– “EPICS consists of a set of core software and a set of optional components. The core software, i.e. the components of EPICS without which EPICS would not function, are:

• Channel Access - Client and Server software

• IOC Database

• Scanners

• Monitors

• Database Definition Tools

• Source/Release”

• This talk focuses on the first 4 points, with most emphasis on the IOC Database.

LIBRARIES AND MODULES

Helper libraries• If you write EPICS base software you must know,

understand and use the basic helper libraries it provides. For example:– ellLib – doubly linked list library identical to VxWorks

listLib– gpHash – efficient hashes– freeList – malloc replacement giving efficient handling

of fixed size blocks.– epicsRingBytes – ring buffers.– epicsThread – thead creation and control– epicsMutex – OS independent mutexes– etc...

Essential EPICS code• dbStaticLib.c

– Static database access routines• Used during EPICS development or before iocInit.

• dbAccess.c– Run-time database access routines.

– dbNameToAddr, dbPutField, dbPut, dbGetField, dbGet, dbProcess etc.

• dbConvert.c– Type conversions

• camessage.c– Channel Access server function table implementations.

• dbEvent.c– Implements database event handling

• etc.....

What EPICS libraries are there?

• Common– Cap5– ca– cas– Com– gdd

• Host and IOC:– asHost– asIoc– dbStaticHost– dbStaticIoc

• IOC only– dbIoc– dbtoolsIoc– miscIoc– recIoc– registryIoc– rsrvIoc– softDevIoc– testDevIoc

EPICS library dependencies

Conclusion• If you want to know more, consult the source.

DATA STRUCTURES

dbBase.h• Basic database data structures are in dbBase.h• These define the structures that hold the

database.• The are defined in the dbd and db files, and

created by dbStaticLib.• At run-time, they are accessed by dbAccess

routines.

dbBase.h record tree• dbBase

└ recordTypeList└ dbRecordType

└ recordList└ dbRecordNode└ name└ precord└ dset└ lset etc.└ ...

└ deviceList└ ...

└ dbFldDes[]└ rset└ ...

dbBase.h tree

dbBase definitiontypedef struct dbBase {

ELLLIST menuList;ELLLIST recordTypeList;ELLLIST drvList;ELLLIST registrarList;ELLLIST functionList;ELLLIST variableList;ELLLIST bptList;void *pathPvt;struct dbPvd *ppvd;struct gphPvt *pgpHash;short ignoreMissingMenus;short loadCdefs;

}dbBase;

dbRecordType definitiontypedef struct dbRecordType {

ELLNODE node;ELLLIST attributeList; /*LIST of attributes*/ELLLIST recList; /*LIST of sorted dbRecordNodes*/ELLLIST devList; /*LIST of device support*/ELLLIST cdefList; /*LIST of Cdef text items*/char *name;short no_fields; /* number of fields defined */short no_prompt; /* number of fields to configure*/short no_links; /* number of links */short no_aliases; /* number of aliases in recList */short *link_ind; /* addr of array of ind in papFldDes*/char **papsortFldName;/* ptr to array of ptr to fld names*/short *sortFldInd; /* addr of array of ind in papFldDes*/dbFldDes *pvalFldDes; /*pointer dbFldDes for VAL field*/short indvalFlddes; /*ind in papFldDes*/dbFldDes **papFldDes; /* ptr to array of ptr to fldDes*//*The following are only available on run time system*/struct rset *prset;int rec_size; /*record size in bytes */

}dbRecordType;

dbRecordNode definitiontypedef struct dbRecordNode {

ELLNODE node;

void *precord;

char *recordname;

ELLLIST infoList; /*LIST of info nodes*/

int flags;

}dbRecordNode;

Record (dbCommon) definitiontypedef struct dbCommon {

char name[61]; /* Record Name */char desc[41]; /* Descriptor */char asg[29]; /* Access Security Group */epicsEnum16 scan; /* Scan Mechanism */epicsEnum16 pini; /* Process at iocInit */epicsInt16 phas;/* Scan Phase */epicsInt16 evnt; /* Event Number */epicsInt16 tse; /* Time Stamp Event */DBLINK tsel; /* Time Stamp Link */epicsEnum16 dtyp; /* Device Type */epicsInt16 disv; /* Disable Value */epicsInt16 disa; /* Disable */DBLINK sdis; /* Scanning Disable */epicsMutexId mlok; /* Monitor lock */ELLLIST mlis; /* Monitor List */epicsUInt8 disp; /* Disable putField */epicsUInt8 proc; /* Force Processing */epicsEnum16 stat; /* Alarm Status */epicsEnum16 sevr; /* Alarm Severity */epicsEnum16 nsta; /* New Alarm Status */epicsEnum16 nsev; /* New Alarm Severity */

epicsEnum16 acks; /* Alarm Ack Severity */epicsEnum16 ackt; /* Alarm Ack Transient */epicsEnum16 diss; /* Disable Alarm Sevrty */epicsUInt8 lcnt; /* Lock Count */epicsUInt8 pact; /* Record active */epicsUInt8 putf; /* dbPutField process */epicsUInt8 rpro; /* Reprocess */struct asgMember *asp; /* Access Security Pvt */struct putNotify *ppn; /* addr of PUTNOTIFY */struct putNotifyRecord *ppnr; struct scan_element *spvt; /* Scan Private */struct rset *rset; /* Address of RSET */struct dset *dset;/* DSET address */void *dpvt; /* Device Private */struct dbRecordType *rdes; /* dbRecordType */struct lockRecord *lset; /* Lock Set */epicsEnum16 prio; /* Scheduling Priority */epicsUInt8 tpro; /* Trace Processing */char bkpt; /* Break Point */epicsUInt8 udf; /* Undefined */epicsTimeStamp time; /* Time */DBLINK flnk; /* Forward Process Link */

} dbCommon;

dbAddr definition• Virtually all runtime access to the database is via a dbAddr

handle.

typedef struct dbAddr { struct dbCommon *precord; /* address of record */ void *pfield; /* address of field */ struct dbFldDes *pfldDes; /* address of fldDes */ long no_elements; /* number of elements */ short field_type; /* type of field */ short field_size; /* size of the field */ short special; /* special processing */ short dbr_field_type; /* request type */ /* DBR_STRING,...,DBR_ENUM,DBR_NOACCESS */} dbAddr;

PROCESS TYPES

EPICS and threads• A typical C/C++ programmer’s first reaction is

that EPICS has lots of threads• However, they all have there uses, and once you

understand them they are not so confusing.• The need arises from the need to have non-

blocking processing and strict priorities.• It is actually simpler and more efficient than a lot

of select() calls.

EPICS ThreadsName PrioritycbHigh 71

timerQueue 70

scanOnce 70scan0.1 66

scan0.2 65

cbMedium 64

scan0.5 64scan1 63

scan2 62

scan5 61scan10 60

cbLow 59

CAC-event 51dbCaLink 50

CAS-client 20

CAS-event 19

CAS-TCP 18CAS-beacon 17

CAS-UDP 16

errlog 10taskwd 10

• IOC management:– timerQueue– taskwd– errlog

• Database processing:– cb*– scan*

• Channel Access:– CAS-*– CAC-event– dbCaLink

• Plus various driver threads...

IOC management threads• timerQueue

– Implements all EPICS base delays

• taskwd– Monitors tasks for any suspensions

• errlog– Handles asynchronous processing of log messages.– Forwards log messages to the console or to files at low

priority.

Database processing threads• cb*

– callback tasks – used for event and I/O Interrupt scanned records.

– Calls any specified function asynchronously– 3 priorities (low, medium high)

• scan*– periodic scan tasks– Calls record process() routines at regular intervals.– Higher scan rates have higher priorities.

• + all driver threads– Every asyn port has its own thread

Channel Access threads• CAS-*

– Channel access server tasks• CAS-UDP listens for channel lookups

• CAS-TCP listens for TCP connections

• CAS-beacon looks for CA beacons

• One CAS-client and CAS-event task per client

• dbCaLink– Channel Access client that processes CA requests on

behalf of the database.

• CAC-event– Handles the CA client callback events (I think)

EXAMPLES OF PROCESSING

Example sequence diagram

IOC function callsIOC tasksClient

Task name C filename

C function

CA server call processing• Every CA client message has a function code defined

in caProto.h of the form CA_PROTO_XXXX– E.g.: CA_PROTO_WRITE

• This will convert into a function in one of the CA jump tables (tcpJumpTable or udpJumpTable). Functions are off the form xxxx_action(mp,pPayload,client)– Parameters are pointers to

• CA header block

• Data

• client structure.

– Performs:• Net to host byte swapping

• Access security checks

– E.g.: write_action(mp,pPayload,client)

CA server call processing• xxxx_action often calls a “CA db_access” routine

with underscores, which handles an ugly “old to new” type enumeration conversion– E.g.: db_put_field(pAddr, oldDataType,pPayload,count)

• This calls a dbAccess routine which is part of the database code.– E.g.: dbPutField(pAddr,newDataType,pDate,count)

• This locks the database and implements the appropriate action.

caget from a record

Channel life cycle

caput to an ai record

dbPutFieldlong epicsShareAPI dbPutField(DBADDR *paddr, short dbrType, const void *pbuffer, long nRequest)

{

long status = 0;

dbFldDes *pfldDes = paddr->pfldDes;

dbCommon *precord = paddr->precord;

short dbfType = paddr->field_type;

/* Various sanity tests deleted here... */

dbScanLock(precord);

status = dbPut(paddr, dbrType, pbuffer, nRequest);

if (status == 0) {

if (paddr->pfield == (void *)&precord->proc ||

(pfldDes->process_passive && precord->scan == 0 && dbrType < DBR_PUT_ACKT)) {

if (precord->pact) {

precord->rpro = TRUE;

} else {

/* indicate that dbPutField called dbProcess */

precord->putf = TRUE;

status = dbProcess(precord);

}

}

}

dbScanUnlock(precord);

return status;

}

dbPutdbPut(DBADDR *paddr, short dbrType, const void *pbuffer, long nRequest){ long special = paddr->special; short field_type = paddr->field_type; long no_elements = paddr->no_elements;

if (special) dbPutSpecial(paddr, 0);

if ( IS AN ARRAY ) prset->get_array_info(paddr, &dummy, &offset);

dbPutConvertRoutine[dbrType][field_type](paddr, pbuffer, nRequest, no_elements, offset);

if ( IS AN ARRAY ) prset->put_array_info(paddr, nRequest);

if (special) dbPutSpecial(paddr,1);

if ( not ( VALUE FIELD and PP )) db_post_events(precord, paddr->pfield, DBE_VALUE | DBE_LOG); return;}

camonitor from an ai record

caput callback from an ai record

What’s the stack trace like?Call Locationclone() libc.so.6

start_thread() libpthread.so.0

start_routine(arg) osdThread.c:320

camsgtask(pParm) camsgtask.c:123

camessage(client) camessage.c:2442

write_action(mp,pPayload,client) camessage.c:810

db_put_field(paddr,src_type,psrc,no_elements) db_access.c:1216

dbPutField(paddr,dbrType,pbuffer,nRequest) dbAccess.c:1257

dbProcess(precord) dbAccess.c:630

process(pao) librecIoc.so

writeValue(pao) aoRecord.c:537

write_ao(pao) devAoSoft.c:72

CAS-client thread

Socket recv call (caput starts here)

Process CA messagescaput CA action

caput to an ai recordCall Locationclone() libc.so.6

start_thread() libpthread.so.0

start_routine(arg) osdThread.c:320

camsgtask(pParm) camsgtask.c:123

camessage(client) camessage.c:2442

write_action(mp,pPayload,client) camessage.c:810

db_put_field(paddr,src_type,psrc,no_elements) db_access.c:1216

dbPutField(paddr,dbrType,pbuffer,nRequest) dbAccess.c:1257

dbProcess(precord) dbAccess.c:630

process(pao) librecIoc.so

writeValue(pao) aoRecord.c:537

write_ao(pao) devAoSoft.c:72

CAS-client thread

Socket recv call (caput starts here)

Process CA messagescaput CA action

caput callback to an ai recordCall Locationclone() libc.so.6

start_thread() libpthread.so.0

start_routine(arg) osdThread.c:282

camsgtask(pParm) camsgtask.c:123

camessage(client) camessage.c:2508

write_notify_action(mp,pPayload,client) camessage.c:1823

dbPutNotify(ppn) dbNotify.c:342

putNotifyCommon(ppn,precord) dbNotify.c:236

dbProcess(precord) dbAccess.c:643

process(precord) aiRecord.c:177

recGblFwdLink(precord) recGbl.c:267

dbNotifyCompletion(precord) dbNotify.c:422

callbackRequest(pcallback) callback.c:154

CAS-client task

camonitor from an ai record

Call Locationclone() libc.so.6start_thread() libpthread.so.0start_routine(arg) osdThread.c:282periodicTask(arg) dbScan.c:553scanList(psl) dbScan.c:658dbProcess(precord) dbAccess.c:643process(prec) calcRecord.c:126recGblFwdLink(precord) recGbl.c:265dbScanFwdLink(plink) dbAccess.c:519dbScanPassive(pfrom,pto) dbAccess.c:490dbProcess(precord) dbAccess.c:643process(precord) aiRecord.c:175monitor(prec) aiRecord.c:403db_post_events(pRecord,pField,caEventMask) dbEvent.c:767db_post_single_event_private(event) dbEvent.c:715

scan1 task

Camonitor event

Call Locationclone() libc.so.6start_thread() libpthread.so.0start_routine(arg) osdThread.c:282event_task(pParm) dbEvent.c:941event_read(ev_que) dbEvent.c:865read_reply(pArg,paddr,eventsRemaining,pfl) camessage.c:627cas_send_bs_msg(pclient,lock_needed) caserverio.c:63send() libc.so.6

CAS-event task

caget from an ai record

Call Locationclone() libc.so.6start_thread() libpthread.so.0start_routine(arg) osdThread.c:282camsgtask(pParm) camsgtask.c:123camessage(client) camessage.c:2508read_notify_action(mp,pPayload,client) camessage.c:757read_reply(pArg,paddr,eventsRemaining,pfl) camessage.c:557db_get_field(paddr,buffer_type,pbuffer,no_elements,pfl) db_access.c:543dbGetField(paddr,dbrType,pbuffer,options,nRequest,pflin)dbAccess.c:989dbGet(paddr,dbrType,pbuffer,options,nRequest,pflin) dbAccess.c:1009getOptions(paddr,poriginal,options,pflin) dbAccess.c:393dbFastGetConvertRoutine[field_type][dbrType] (localAddr.pfield, pbuffer, &localAddr)

dbAccess.c:1069

CAS-client task

The end• What did I miss?

– IOC initialisation– Database definition– Static database access– dbStatic– Test facilities– Access security– IOC shell– Registry– OSI– libCom– Build rules– Record, device and driver support

• Basically everything.....

Resources• EPICS Application Developers Guide• Channel Access Protocol Specification

– http://epics.cosylab.com/cosyjava/JCA-Common/Documentation/CAproto.html

• Training slides on EPICS web site• Eclipse is your friend...• “module load ddd” gives you the ddd graphical

debugger.