1. The rsyslog v8 engine A developer's view Rainer Gerhards

2. What is New? Output part totally revamped Simplified script execution Even faster execution speed Less complex code Even higher scalability Global variable support Required changes to output module interfaceRainer Gerhards * http://blog.gerhards.net 3. The v7 rule engine Queue worker rsyslog coreQueue workerqueueSingle-thread compartmentAction instanceQueue workerFilter processing Message formatting Actual output action, like sending msg Kept simple & single threaded Works well with fast actions Has problems with slow ones, e.g. via HTTP (like Elasticsearch) Rainer Gerhards * http://blog.gerhards.net 4. V7 Action data structures Global data, once per module Solely for setting default parameters pData: per action-instance Automatically created by framework Holds all data necessary for doing action Config parameters Connection handles Other stateRainer Gerhards * http://blog.gerhards.net 5. Real-World Sample: ommysql typedef struct _instanceData { MYSQL *f_hmysql; char f_dbsrv[MAXHOSTNAMELEN+1]; unsigned int f_dbsrvPort; char f_dbname[_DB_MAXDBLEN+1]; char f_dbuid[_DB_MAXUNAMELEN+1]; char f_dbpwd[_DB_MAXPWDLEN+1]; unsigned uLastMySQLErrno; uchar *f_configfile; uchar *f_configsection; uchar *tplName; } instanceData;Green: per-connection parameters Rest is config part. Rainer Gerhards * http://blog.gerhards.net 6. The v8 rule engine Queue worker queueQueue workerAction wrkr inst.Queue workerrsyslog coreAction wrkr inst.Action wrkr inst.Now multiple instances per action! Queue worker pool automatically scales outbound connection count by spawning more worker instances Works well with ElastiSearch etc. Inherantly serial outputs (e.g. local files!) must serialize themselves Rainer Gerhards * http://blog.gerhards.net 7. V8 Action data structures Global data, once per module Solely for setting default parameters pData: per action-instance Automatically created by framework Primarily config parameters Maybe some global state like overall counters, mutexes, ... pWrkrData: per action-worker-instance Connection handles Other live state Rainer Gerhards * http://blog.gerhards.net 8. Real-World Sample: ommysql typedef struct _instanceData { char dbsrv[MAXHOSTNAMELEN+1]; unsigned int dbsrvPort; char dbname[_DB_MAXDBLEN+1]; char dbuid[_DB_MAXUNAMELEN+1]; char dbpwd[_DB_MAXPWDLEN+1]; uchar *configfile; uchar *configsection; uchar *tplName; } instanceData; typedef struct wrkrInstanceData { instanceData pData; /* points back to action instance */ MYSQL *hmysql; unsigned uLastMySQLErrno; } wrkrInstanceData_t;Config and per-connection parameters now split, multiple instances of per-connection data. Rainer Gerhards * http://blog.gerhards.net 9. V8 data structure sample One action module With 2 actions One being executed on two, one on three threadsAction instance 2 Rainer Gerhards * http://blog.gerhards.netwrkr 1 wrkr 2 wrkr 1 wrkr 2 wrkr 3External Connectionsmodule global dataAction instance 1 10. What if the destination cannot handle multiple workers? A typical example is file output Framework will still call into multiple workers Worker must use pData mutex to protect itself this is not done automatically! Basically like in v7, but plugin needs to take care Action instance 1 Action instance 2wrkr 1 wrkr 2real code Action-instance mutexeswrkr 1 wrkr 2real codewrkr 3 Rainer Gerhards * http://blog.gerhards.net 11. Real-World Sample: omfile BEGINcommitTransaction instanceData *__restrict__ const pData = pWrkrData->pData; unsigned i; CODESTARTcommitTransaction pthread_mutex_lock(&pData->mutWrite); for(i = 0 ; i < nParams ; ++i) writeFile(pData, pParams, i); if(pData->bFlushOnTXEnd && pData->pStrm != NULL) { /* if we have an async writer, it controls the flush via * a timeout. */ if(!pData->bUseAsyncWriter) CHKiRet(strm.Flush(pData->pStrm)); } finalize_it: pthread_mutex_unlock(&pData->mutWrite); ENDcommitTransactionRainer Gerhards * http://blog.gerhards.net 12. Anything else? A couple of more things have changed,e.g. New callbacks for wrkr Instance handling New transaction processing callback New data structures for message data Changes are hidden mostly in macro layer But still there is work to do Usually quite easy Presentation on how to do this will follow, in the meantime use http://blog.gerhards.net/2013/11/samples-for-v8-module-conversion.htmlRainer Gerhards * http://blog.gerhards.net 13. Even more? Yup... We now support external plugins, which need not be written in C! Use any language you like, e.g. Python, Perl or Java... More info at http://www.slideshare.net/rainergerhards1/writing-rsyslog-p Note: action worker instances facilitate writing great performance external plugins Rainer Gerhards * http://blog.gerhards.net