Diane Fairley11/11/2004

Slc-aware Ioc

SLC Executive

Message Service

SLC Executive Slc-aware IOC Initialization

The SLC Executive Thread

Startup Process

Shutdown Process

Some General Thread Issues

Utilities

Overall Architecture

Proxy

slcIoc

Alpha

slcExec

msgSend msgRecv

msgHdlr

Slc Database

cstrAsync

bpmHdlr

mgntHdlr

mgntAsync

Epics database

Epics Land

dbSenddbRecv

dbHdlr

Msg service

DB service

Db Ctl Channel access

iocsh

slcIoc Initialization The saIocMain.cpp file

Epics initialization via st.cmd file Read environment variables

slcIoc name (like the microname: XL01, XL02, … ) Proxy IP address (ip address of PX01 for now)

Load epics databases Currently a Restart/Stop subroutine record

start cmlog client

iocInit()

slcStart()

iocsh(NULL)

slcStart()

Initializes slc globals

Starts the slcExec thread

Executes the Restart function

SLC Globals

discovered_name Name string, from environment variable

proxy_ip_address from environment variable The ip address string in xxx.xxx.xxx.xxx format

Db downloadEvent semaphore

Name Type Represents

pName char * Name of thread

tid epicsThreadId Thread identifier

active epicsBoolean True if thread is active / looping; set by thread

stop epicsBoolean True if thread should stop; set by slcExec

mqId epicsMessageQueueId Message queue identifier

Cmlog_client_h long cmlog client handle

• slcThreads[ thread enum]– Array of thread information and global resources

saIocJobs[jobcode] = thread enum converts micro job code to an slcIoc thread

enumeration

slcSockets[ socket enum] Array of socket descriptors

dbExists

The SLC Executive Thread

Manages the slc startup, creating all other slc threads in order

Supervises threads and globals

Manages a clean shutdown of slc threads

Responds to slcRestart (IPL a micro)

Responds to slcStop (RESET a micro)

SlcExec thread initialization slcCmlogOpen Create a message queue Set slcExec global active=True Go into loop waiting for messages

slcRestart Commanded via IPL on VMS control system

TST_SLC_NOTIFY message “BOOT” Received by msgRecv, forwarded to msgHdlr

Commanded from ioc shell >slcRestart

Commanded from channel access via subroutine record Sets CSTR:<micro>:1:REST.A=0, EDM display can set this

Commanded from slcStart() initialization

SLC Restart

slcIoc

Epics

slcExec

Restart SubRecord

Epics database

Msg service

DB service

Db CtlChannel access

iocsh

Initialization

slcRestart

Send TEST_START message to slcExec From msgHdlr, or from slcRestart iocsh command, or from subRecord

slcExec process: Phase I If threads are running, call slcStop

Wait for all threads to stop

Create downloadEvent semaphore for threads to wait on

Create all threads except msgRecv, dbRecv threads

Wait for these threads to set their global active=True

Restart: Phase I

slcIoc

slcExec

msgSend

msgHdlr

cstrAsync

bpmHdlr

mgntHdlr

mgntAsync

Epics database

Epics Land

dbSend

dbHdlr

Msg service

DB service

Db CtlChannel access

Slc database

DownloadEvent

iocsh

slcRestart

Restart Phase II Create the msgRecv and dbRecv threads

Wait for the Recv threads to set their global active=True

Send a db_download_req message to dbSend message queue.

Goes back to its loop waiting for messages and checking on thread active flags

Restart: Phase II

slcIoc

slcExec

msgSend

msgHdlr

Slc Database

cstrAsync

bpmHdlr

mgntHdlr

mgntAsync

Epics database

Epics Land

dbSend

dbHdlr

Msg service

DB service

Db CtlChannel access

msgRecv dbRecv

Db download req

iocsh

slcRestart

Restart Phase III Database Service downloads the database

Database Service sets the dbExists flag to true

Database Service signals the downloadEvent semaphore to allow threads to proceed.

Ready to go

Proxy

slcIoc

Alpha

slcExec

msgSend msgRecv

msgHdlr

Slc Database

cstrAsync

bpmHdlr

mgntHdlr

mgntAsync

Epics database

Epics Land

dbSenddbRecv

dbHdlr

Msg service

DB service

Db CtlChannel access

iocsh

General Thread Initialization

slcCmlogOpen

Create resources Some will create a message queue Recv threads create a socket Async threads may create a timer Create any other thread specific resources

Set active=True

Perform dblists (optional) Wait on database event

Go into loop waiting for messages or wake-up

slcStop Commanded via RESET on VMS control

system TST_SLC_NOTIFY message “RSET” Received by msgRecv

Commanded from the ioc shell >slcStop

Commanded from channel access via subroutine record Sets CSTR:<micro>:1:REST.A = 1 EDM display can set this

slcStop Send TEST_STOP message to slcExec

From msgHdlr, or from iocsh command, or from subRecord

slcExec process Set all the thread global stop=True; Send TEST_STOP message to all threads with message

queues Wait for all thread global “active” flags to be set false Wait for all threads to stop Delete the global db downloadEvent Set dbExists to false; Set all thread global “stop” flags back to False; Could re-initialize all globals, do final global resource

cleanup…. go back to loop waiting for messages

General Thread Shutdown Release resources

msg queues, sockets, heap memory, timers, semaphores

slcCmlogClose()

set active=False

Set tid=NO_THREAD

Return

General slcIoc thread assumptions There are three basic thread types

‘socket’ threads that wait at sockets ‘message queue’ threads that wait at queues Async threads that wake up periodically

Non-message queue threads must poll the stop flag! No ‘kill a thread’ function in epics thread library Allows for orderly clean-up in all threads at all times ‘waits’ should have a timeout Possibly, message queue threads may wait indefinitely at

the queue because they can receive a TEST_STOP message.

Sockets should be non-blocking or interruptable

Some Utilities Thread Utilities

slcThreadNameEnum getThreadEnum(epicsThreadId tid); slcThreadNameEnum getThreadEnumByName(const char* name); char* getMicroname(void); eg: ‘XL02’ char* getProxyIPString(void); eg: ‘142.67.94.25’ some new wait(…) utilities

To be used by threads when waiting (with a timeout) on a flag or socket or ….

slcCmlog Utilities int slcCmlogOpen(void); int slcCmlogClose(void); int slcCmlogLogMsg (…,.,..,…., ...);

Message queue utilities epicsMessageQueueId createMsgQ( slcThreadNameEnum thrd ); vmsstat_t sendToMsgQ ( slcThreadNameEnum thrd, void* msg, int size );

slcIoc Message Service Threads overview

Message structure

Some Assumptions

Message Flow

Utilities

Message Service

slcIoc

slcExec

msgSend msgRecv

msgHdlr

Slc Database

bpmHdlr

mgntHdlr

Epics database

Epics Land

Msg service

Channel access

Slc Message Service The msgRecv thread

“owns” the msg service socket; creates and destroys it Receives all incoming messages at the msg service socket Sends the incoming message to the proper slc Hdlr thread

Uses the slcIocJobs[job code] global

The msgSend thread Sends all outgoing message at the message service socket “uses” the message service socket When the connection fails, waits for the msgRecv thread to

re-connect. Checks for a valid socket in slcSockets[sdMsg]

The msgHdlr thread Handles messages with the TEST function code Handles the new TST_SLC_NOTIFY messages

Message StructureName Data type Represents

fwd_hdr_ts

ip_port_u

len

user

cmd

crc

ip_port_tu (int4u)

int4u

user_field_ts(int2u)

int1u

int1u

Lower half of the ip address, and the port number

Message bytecount minus this fwdheader

User defined; chunk count for large buffers;

Fwd_server command, e.g.

8 bit crc over header; currently set to 0x55

msgheader_ts

source[4]

dest[4]

timestamp[2]

func;

datalen;

char

char

int4u

int2u

int2u

Name of Alpha job that sent the message *

Name of destination micro *

VMS format timestamp

Function code (job code + function code)

Word count of the data in Reqdata

reqdata[] Array of int2u function-specific message data; packed; max size is NETVAXMSGLEN – sizeof(msgheader_ts)/2

NOTE: this structure is shared with VMS and micros

Message Service Assumptions Sockets are non-blocking or interruptable

They must be able to see the global thread stop flag

Large messages will need heap memory allocations Message queues do not (efficiently) allow variable size msgs Use a memory pool to avoid memory leaks, fragmentation

Messages from the VMS control system are little-endian, VMS format, and packed. The message service threads are responsible for converting

the header portions of the message

The other ‘job’ threads are responsible for converting the function specific portion of the message (reqdata)

Message Service message flow The msgRecv thread

Receives the message into a local inMsg variable using a getBuffer utility.

interprets the fwdheader Allocate memory to store a multi-packet message.

converts/unpacks the msgheader into native format

replaces timestamp in msgheader

Translates the incoming job code to an slcIoc thread id

sends the entire message on to the correct queue. Message queue send actually copies the message Large messages: reqdata portion would include a pointer

Message Service message flow The job thread

Reads the message into an inMsg variable. Ignores the fwdheader portion

Reads the message data length from the msgheader

converts/unpack the incoming function specific data Releases large message memory when it can, if necessary

uses the data – unidentified requirements here.

converts/packs the function specific reply into an outMsg variable.

Allocates memory for large response message

fills in the new msgheader. (leaves it in native format)

The outMsg reply is sent to the msgSend queue.

Message service message flow

The msgSend thread Reads the message into a msg variable

converts/packs the msgheader

fills in the fwdheader

sends reply to VMS control system Creates and sends multiple buffers for large messages De-allocates memory used by large messages

Converting and packing/unpacking Forward header and Msgheader

The forward header follows networking standard in VMS and micros – uses hton and ntoh

Jobs don’t need to see the forward header Conversion and unpacking handled in by Message Service

Function Specific Data (requests and replies) Typically a structure is defined for the data and shared by

the VMS, micros and slcIocs See nfs_ref/inc_c for xxx_reqreply_struc.*

To unpack, must memcpy data field-by-field from incoming msg.reqdata[] to a variable of this structure, then convert.

To pack, do the reverse, convert then memcpy field-by-field from structure to outgoing msg.reqdata[]

Utilities will be written for each shared structure.

Conversion Utilities void sswap(const unsigned short* s); void lswap(const unsigned long* l);

void swap_int2u(void* pOut2u, const void* pIn2u); void swap_int4u(void* pOut4u, const void* pIn4u);

void cvt_vms_to_ieee(void* pOutR4, const void* pInR4); void cvt_ieee_to_vms(void* pOutR4, const void* pInR4);

int getVMSTime(void* pDest); int epicsTimeToVMSTime(void* pDest, const

epicsTimeStamp* eTime);

long int diffVMSTimes_10usec(void* pRef, void* pLater); long int diffVMSTimes_10msec(void* pRef, void* pLater);

TCP Utilities SOCKET getConnection(unsigned short port, unsigned short socket);

Gets a connection with the proxy Registers with the proxy Reports error conditions

vmsstat_t getBuffer(SOCKET sd, fwd_hdr_ts *pFwd, void *pBuf, int bytesReqd, int *pBytesRcvd); Reads incoming message Allocates memory and re-creates full message from multi-packet Reports error conditions