!chaos control system

102
C. Bisegni Control system based on a Highly Abstracted and Open Structure school 2012 http://chaos.infn.it 1 venerdì 14 dicembre 12

Upload: claudio-bisegni

Post on 18-Dec-2014

293 views

Category:

Technology


0 download

DESCRIPTION

The !CHAOS project is aiming at the development of a new concept of control system and data acquisition framework providing, with a high level of abstraction, all the services needed for controlling and managing a large scientific, or non-scientific, infrastructure.

TRANSCRIPT

Page 1: !Chaos Control System

C.  Bisegni

Control  system  based  on  aHighlyAbstracted  andOpenStructure

school  2012

http://chaos.infn.it

1venerdì 14 dicembre 12

Page 2: !Chaos Control System

the  scene...

imagine  to  have  this...

2venerdì 14 dicembre 12

Page 3: !Chaos Control System

the  scene...

with  many  distributed

controls  rooms

3venerdì 14 dicembre 12

Page 4: !Chaos Control System

the  scene...

with  many  distributeddata  center

4venerdì 14 dicembre 12

Page 5: !Chaos Control System

the  scene...

with  only  ONEControl  System  with  

complex  automated  task

5venerdì 14 dicembre 12

Page 6: !Chaos Control System

the  scene...

somewhere  in  the  future  perhaps  will  have  this

6venerdì 14 dicembre 12

Page 7: !Chaos Control System

the  scene...

with  only  ONE  Control  System  with  even  more  complex  automated  task

7venerdì 14 dicembre 12

Page 8: !Chaos Control System

the  scene...

securely,  there  will  be  a  lot  of  

chaos

8venerdì 14 dicembre 12

Page 9: !Chaos Control System

the  scene...

but  only  ONE!CHOASinstance

9venerdì 14 dicembre 12

Page 10: !Chaos Control System

10

Objec>ve

What  is  needed  to  control  the  CHAOTIC  

complexity  of  big  apparatus?

10venerdì 14 dicembre 12

Page 11: !Chaos Control System

11

Objec>veWhat  is  needed  to  control  the  CHAOTIC  

complexity  of  big  apparatus?

•  a  design  that  permits  to  scale  to  achieve  high  speed  in  the  data  

acquisi@on  and  elabora@on  and  for  fault  tolerant;

•  possibility  to  aBach  and  detach  an  en@re  subsystem;

•  possibility  to  “configure”  a  specified  instrument  without  halt  the  full  system;

•  possibility  to  hot  insert  an  instrument  or  a  subsystem;

11venerdì 14 dicembre 12

Page 12: !Chaos Control System

12

Objec>veWhat  is  needed  to  control  the  CHAOTIC  

complexity  of  big  apparatus?

•  give  the  flexibility  to  implements  control  algorithm  on  abstracted  

instrument  class;

•  work  on  data  abstrac@on,  without  fixed  schema;

•  give  a  data  system  that  permit  to  monitor  the  instrument  channels  and  query  the  history    instruments  data;

•  the  data  can  be  managed  into  domains

•  every  domain  can  be  implemented  with  different  technology

12venerdì 14 dicembre 12

Page 13: !Chaos Control System

13

Objec>ve

13venerdì 14 dicembre 12

Page 14: !Chaos Control System

14

What  is!CHAOS  (Control  system  based  on  Highly  Abstracted  and  Open  Structure),  is  an  experiment  of  CSN  5(technological  research  experiments)  at  the  LNF  and  Roma-­‐TV  sec@on.

The  project  has  been  appreciated  by  educa@onal  world  and  the  University  of  Tor  Vergata  and  Cagliari  have  joined  the  development  team.

UniversitàTorVergata LNF - Roma2

Universitàdi Cagliari

14venerdì 14 dicembre 12

Page 15: !Chaos Control System

15

Objec>ve

!CHAOS   has   been   designed   to   widely   scale   and   for   manage   large  

apparatus  with  different  needs  in  term  of  storage  technology  and  speed  

for  data  acquisi@on

15venerdì 14 dicembre 12

Page 16: !Chaos Control System

16

Objec>ve

•  new  topology  in  the  CS  that  permit  to  integrate  the  triggered  DAQ;

•  define  a  new  way  to  access  real@me  data;

•  data  acquisi@on,  storage  and  indexing  fully  scalable;

•  high  scalability  in  every  cri@cal  point.

16venerdì 14 dicembre 12

Page 17: !Chaos Control System

17

Objec>ve

•  99%  of  adapta@on  @me  of  instrument  to  !CHOAS  done  by  framework

•  simplify  the  crea@on  of  the  user  interfaces  

•  simplify  the  crea@on  of  the  control  algorithms  basing  it  on  an  abstrac@on  

of  the  I/O  channels  or  full  dataset  not  on  physical  device

•  All  service  for  data  management  and  monitoring  already  done  and  fully  

customizable

A  CPP  Framework  and  services  that  permit  to  have:

17venerdì 14 dicembre 12

Page 18: !Chaos Control System

18

!CHAOS  Simplify  Vision

18venerdì 14 dicembre 12

Page 19: !Chaos Control System

Implementation

Impl

emen

tatio

nImplem

entation

Control'sAlgorithm

User InterfaceInstrument's Driver

Implementation

Data Cloud

Live Data

History Data

network

DirectoryServerCloud

CHAOS  Boundary

User  specialized  Boundary

Simplify  Vision

Abstract RPC Driver

AbstractEvent Drivernetwork

AbstractDirect I/O

Implementation

Data Cloud

Live Data

History Data

network

Implementation

Data Cloud

Live Data

History Data

19venerdì 14 dicembre 12

Page 20: !Chaos Control System

20

!CHAOS  Real  Model

20venerdì 14 dicembre 12

Page 21: !Chaos Control System

Live / History domain

Live / History domain

Data  Boundary

Common Toolkit

User Interface Toolkit

InformaIon  Boundary

ChaosDirectory

Nodes

Common Toolkit

Control Unit Toolkit

Co

mm

on

To

olk

it

Exe

cuti

on

Un

it T

oo

lkit

RPC - Event - Direct I/O

RPC - Event - Direct I/O

RP

C -

Eve

nt

- D

ire

ct I

/O

HT

TP

- A

PI

Node Information

Node Activity

Security

CQL Proxy Nodes

Indexer Nodes

Management Nodes

AbstracIon  Boundary

Driver for sensor and instruments

User Interface for monitoring and controls

ControlsAlgorithm

User  CustomizaIon  Boundary

21venerdì 14 dicembre 12

Page 22: !Chaos Control System

22

Toolkit  and  Services

!CHAOS  is  a  set  of  toolkit  and  services  that  will  permit  the  realizaAon  of  an  control  system

22venerdì 14 dicembre 12

Page 23: !Chaos Control System

23

!CHAOS  Toolkit  and  Services

Common  Toolkit

CU  Toolkit

UI  Toolkit

EU  Toolkit

DirectoryServices

CQL  Proxy

Indexer

Managment

Live  and  History  Data

23venerdì 14 dicembre 12

Page 24: !Chaos Control System

24

!CHAOS  Toolkit

implements   the   fundamental   soZware's   layer   to  RPC,   I/O,   Event   and   other’s   u@lity   code   and   for  logic

Common  Toolkit

abstracts   the   !CHAOS   resources   to   the   device  drivers  developers.

abstracts  the  CHAOS  resources  to  the  developers  that   need   to   develop   control   and   compu@ng  algorithms

abstracts  the  CHAOS  resources  to  the  developers  that  need  to  develop  control  and  monitoring  user  interface

CU  Toolkit

UI  Toolkit

EU  Toolkit

24venerdì 14 dicembre 12

Page 25: !Chaos Control System

!CHAOS  Services

DirectoryServices

gives   services   via   RPC   (or   with   other   higher  transport   layers)   for   get   device   structured  informa@on,  system  topology  etc...

CQL  Proxy

Indexer

Management

give  the  access   to  insert  and  query  opera@on  on  live  and  history  data  to  other  CHOAS  node  using  CHOAS  Query  Language  (CQL)

performs  the  indexing  of  the  new  inserted  data

performs   the   management   of   the   datas   chunk  saved  on  the  storage  (erase,  move,  etc.)

25venerdì 14 dicembre 12

Page 26: !Chaos Control System

26

!CHOAS  Scalability  and  Fault-­‐Tolerant

Some  node  needs  to  be  scaled  for  performance  reason  or  for  fault-­‐tolerant:

•  Directory  Nodes•  Live  and  History  nodes

26venerdì 14 dicembre 12

Page 27: !Chaos Control System

27

!CHOAS  Scalability  and  Fault-­‐Tolerant

Server P1

Server P2

Server PN

Device 1Driver

Framework

Server P1

Server P1

Server P2

Server PN

Client ReaderDriver

Framework

Server P2

X

XX

Clients  that  need  to  talk  to  scaled  services,  has  more  than  one  ip  address  for  

every  service

Every  ip  address  has  a  priority,  the  client,  as  first  address,  uses  the  one  with  

highest  prioritywhen  a  server  goes  down,  another  ip  will  be  used

27venerdì 14 dicembre 12

Page 28: !Chaos Control System

28

!CHOAS  Scalability  and  Fault-­‐Tolerant

the  re-­‐balance  occurs  when  a  server  with  a  highest  priority,  previously  goes  

down  or  stopped  for  maintenance,  becomes  up  and  running

Server P1

Server P2

Server PN

Device 1Driver

Framework

Server P1

Server P1

Server P2

Server PN

Client ReaderDriver

Framework

Server P2

use at TS1

use atTS1

ok

the  new  server  will  be  used  starIng  from  a  pre-­‐determinated  Ime  stamp  (TS1)all  the  operaIons  and  the  calculaIon  of  TS1  are  coordinated  by  one  of  the  

directory  nodes

28venerdì 14 dicembre 12

Page 29: !Chaos Control System

11/dic/2011 29

!CHAOS  Abstrac>on  of  the  Data

29venerdì 14 dicembre 12

Page 30: !Chaos Control System

30

!CHAOS  use  a  structured  serializa@on  for  data  descrip@on  and  serializa@on

•  Data  is  managed  by  CDataWrapper  cpp  object  that  wrap  the  serializa@on;  

•  BSON  (hBp://bsonspec.org/)  has  been  chosen,  very  similar  to  JSON  but  it  is  binary;

•  serializa@on  is  wrapped  into  a  CPP  class  so  it  can  be  changed  in  future;

Abstrac>on

30venerdì 14 dicembre 12

Page 31: !Chaos Control System

31

 it  is  used  in  (for  now):

•Hardware  dataset  descrip@on's  and  acquired  data  packaging  (“channel  desc”  =  value)

•RPC  message

•Chaos  Query  Language•  Directory  Server  communicaIon

Abstrac>on

31venerdì 14 dicembre 12

Page 32: !Chaos Control System

32

•  Hardware  aBributes  and  I/O  channels  are  described  into  the  DATASET•  Each  aBribute  is  defined  by

•  Name•  Descrip@on•  Type  (type  ex:  INT32,  INT64,  DOUBLE,  STRING,  BINARY,  STRUCT)•  Cardinality  (single  or  array)•  Flow  (Input,  Output,  Bidirec@onal)•  Range•  default  value•  other  will  came

Abstrac>on  for  Hardware  Dataset

32venerdì 14 dicembre 12

Page 33: !Chaos Control System

11/dic/2011 33

VOLT_CHANNEL_1

RESET

DATASETname:VOLT_CHANNEL_1type:VOLT32flow:outputrange:1-­‐20card:  1

name:VOLT_CHANNEL_2type:VOLT32flow:outputrange:1-­‐20card:  1

name:  RESETtype:  BYTEflow:  inputcard:  1

VOLT_CHANNEL_2

Hardware  Dataset

33venerdì 14 dicembre 12

Page 34: !Chaos Control System

11/dic/2011 34

Hardware  DATASET

The  JSON  visualiza>on  of  BSON  DS  representa>on

{“device_id”: DEVICE_ID

“device_ds”:  { "name" : "VOLT_CHANNEL_1", "desc" : "Output volt...", "attr_dir" : 1, "type" : 4, "cardinality" : 1 ! }

....

}

VOLT_CHANNEL_1

RESET

VOLT_CHANNEL_2

34venerdì 14 dicembre 12

Page 35: !Chaos Control System

11/dic/2011 35

!CHAOS  Communica>on  Systems

35venerdì 14 dicembre 12

Page 36: !Chaos Control System

36

!CHOAS  has  three  different  communicaIon  system:

RPC,  used  for  command  (messages  that  need  to  be  securely  received  and  

recorded  and  for  message  that  need  a  response  [  asynchronous  ])

Events,  used  for  inform  to  a  wide  range  of  nodes,  that  something  has  

happened  (device  heartbeat,  alert  on  device  channel,  received  command,  

etc..)

Direct  I/O,  used  for  streams  to  and  from  the  live,    historical  data  or  inter  

node  data  exchange

Communica>on  Systems  

36venerdì 14 dicembre 12

Page 37: !Chaos Control System

37

Every  !CHAOS  node  has  at  last  one  

client  and  server  instance  of  any  of  

the  three  system

Communica>on  Systems  

37venerdì 14 dicembre 12

Page 38: !Chaos Control System

38

•RPC  layer  is  implemented  as  a  plug-­‐ins  System,  it's  is  abstracted  to  internal  CHAOS  framework  so  we  con  change  it

•  used  to  send  important  messages  that  need  to  be  sure  of  it's  delivering  status

•  can  be  used  to  receive  asynchronous  responses•  the  set  command  for  input  channel  on  control  unit  is  done  with  RPC

RPC  System  

38venerdì 14 dicembre 12

Page 39: !Chaos Control System

39

RPC  Message  Flow•  request  message  is  constructed  with  BSON,  serialized  and  sent  to  RPC  end  point•  if  the  requester  want  a  response  it  put  answer-­‐code  and  sender-­‐address  in  BSON  pack,and  automa@cally  RPC  System  wait  for  the  answer•  the  request  and  response  involves  the  forwarding  of  the  ack  message,  from  the  receiver  to  the  sender  in  synchronous  mode

RPC  System  

Node A Node B

Messagge forwarding

ack [sync]Node A Node B

Response forwarding

ack [sync]

Message Answer  [opIonal]

39venerdì 14 dicembre 12

Page 40: !Chaos Control System

40

!CHAOS  rpc  system

•  Two  kind  of  data  forwarding:•Message,  the  informaBon  are  sent  to  remote  host

•Request,  the  informaBon  are  sent  to  remote  host,  and  an  answer  is  

awaited

•In  either  types  of  data  forwarding,  a  result  of  submissions  is  sent  to  the  

client  when  a  server  has  received  the  data

40venerdì 14 dicembre 12

Page 41: !Chaos Control System

41

RPC  ac>on  message

•In  either  types  of  data  forwarding,  the  data  informaBon  are  stored  into  a  

k/v  message:

•NODE_DST,  internal  MessageBroker  address[string]

•ACT_ALIAS_DST,  alias  for  the  call  acBon  on  the  node[string]•ACT_MSG,  k/v  object  used  as  param  for  the  acBon[BSON]

“NODE_DST”:  “3b95d9b5”“ACT_ALIAS_DST”:  “alias”“ACT_MSG”:  message

BSON Document for the message

41venerdì 14 dicembre 12

Page 42: !Chaos Control System

42

RPC  ac>on  request

the  request,  add  some  other  informaBon  to  the  message

BSON Document for the request

“NODE_DST”:  “3b95d9b5”“ACT_ALIAS_DST”:  “alias”“ACT_MSG”:  messagem

essa

gela

yer

requ

est

laye

r

“ANSW_ID”:  1“ANSW_NODE”:  “44342364”“ANSW_ACT”:  “alias”“ANSW_IP”:  “ip:port”

42venerdì 14 dicembre 12

Page 43: !Chaos Control System

43

!CHAOS  submessage

•  a  message  can  bring  a  sub  message

•  the  sub  message  is  dispatched  aLer  the  main  message  has  been  

successfully  executed

43venerdì 14 dicembre 12

Page 44: !Chaos Control System

44

!CHAOS  submission  result•the  message  or  request  submission  is  a  synchronous  operaBon

•aLer  received  the  server  send  back  to  the  client  the  result,  wrapped  into  a  BSON  object  with  these  key:

•SUBMISSION_ERROR_CODE,  integer  number  that  represents  the  

error  code[0  mean  no  error]

•SUBMISSION_ERROR_MESSAGE,  string  that  represents  what  is  

happened

•SUBMISSION_ERROR_DOMAIN,  string  that  represents  the  locaBon  

where  the  error  is  occurred

“SUBMISSION_ERROR_CODE”:  1“SUBMISSION_ERROR_MESSAGE”:  “error  string”“SUBMISSION_ERROR_DOMAIN”:  “class::method”su

bmis

sion

resu

lt

44venerdì 14 dicembre 12

Page 45: !Chaos Control System

45

!CHAOS  answer  to  a  request

•the  answer  to  a  request,  is  composed  by:

•  the  error  fields  shown  in  previous  slides,  in  case  of  error  there  will  be  all  tree

•  the  BSON  object  that  represents  the  answer,  in  case  of  success

“SUBMISSION_ERROR_CODE”:  1

“ACT_MSG”:  Result  BSON  Docansw

erre

sult

45venerdì 14 dicembre 12

Page 46: !Chaos Control System

09/dic/2011 46

•  Event  is  implemented  as  a  plug-­‐ins  System  it's  is  abstracted  to  internal  CHAOS  framework  so  we  con  change  it

•  there  are  four  type  of  event:•  Alert  (highest  priority)•  Instrument

•  Command

•  Custom  (user  specific  event  type)

•  all  informaAon  are  packed  into  1024  byte  and  transported  by  protocols  that  support  mulAcast  (UDP,  PGM[hVp://en.wikipedia.org/wiki/PragmaAc_General_MulAcast])

Event  System  

46venerdì 14 dicembre 12

Page 47: !Chaos Control System

09/dic/2011 47

Some  numbers  about  event  data

•  Every  event  type  has  2^16  sub  type•  Every  sub  type  can  have  up  to  2^16  priority  level•  Every  subtype  can  have  up  to  255  byte  of  informaAon

Event  System  

47venerdì 14 dicembre 12

Page 48: !Chaos Control System

09/dic/2011 48

•  Direct  I/O  is  implemented  as  a  plug-­‐ins  system

•  used  to  make  insert  or  select  into  the  live  and  the  history  data

•  can  be  used  as  stream  for  receive  result  data  of  the  query  or  read  the  stored  raw  data  file  of  the  devices

•  fastest  way  to  transfer  data  between  node

Direct  I/O  System  

48venerdì 14 dicembre 12

Page 49: !Chaos Control System

49

Communica>on  Systems  

NetworkBrokerRPC Event Direct I/O

Client Server Client Server Client Server

Network

CommonToolkit,  implementa  a  so_ware  switch  that  permit,  to  all  other  class,  to  access  all  three  protocoll.

Node NodeNode

Actio

n

Actio

n

RPCChannel

Answer Actio

n

EventChannel

Handler

49venerdì 14 dicembre 12

Page 50: !Chaos Control System

50

NetworkBroker  For  RPC  and  Event

DeclarAction

EventAction

CPP Class

NetworkBroker

RegisterAction

addActionDescritionInstance

virtual void handleEvent(const event::EventDescriptor * const event) = 0;

RegisterEventAction

50venerdì 14 dicembre 12

Page 51: !Chaos Control System

51

NetworkBroker  For  RPC  and  Event

the  dispatcherfind  des>na>on  queue

Domain  Queue  is  filled

ac>on

NetworkBrokerSERVER

Client

Communication Implementation

data

data

data

data

data

51venerdì 14 dicembre 12

Page 52: !Chaos Control System

52

NetworkBroker  For  RPC  and  Event

MessageChannel

NetworkBroker

Get Message Channel API

Get Event Channel APIEventChannel

DeviceMessageChannel

MDSMessageChannel

AlertEVTChannel

DeviceEVTChannel

CmdEVTChannel

CustomEVTChannel

52venerdì 14 dicembre 12

Page 53: !Chaos Control System

53

NetworkBroker  For  RPC  and  Event

MessageChannel

EventChannel

security check for

who can send what(TODO)

GetMessageChannel

GetEventChannel

NetworkBroker

53venerdì 14 dicembre 12

Page 54: !Chaos Control System

11/dic/2011

!CHAOS  ControlUnit  toolkit

54venerdì 14 dicembre 12

Page 55: !Chaos Control System

•CUToolkit  help  the  Control  Unit  development

•A  ControlUnit  is  and  “Instrument”  driver  that  aBach  an  hardware  to  !CHAOS

•Developer  need  to  extend  only  one  class  to  create  a  CHAOS  driver,  the  “AbstractControlUnit”

•AbstractControlUnit  expose  all  the  APIs  needed  for  interac@ng  with  !CHOAS

CU  Toolkit

55venerdì 14 dicembre 12

Page 56: !Chaos Control System

•defineAc@onAndDataset•init•run•stop•deinit

AbstractControlUnit

Write  a  Control  Unit

AbstractControlUnit  is  an  abstract  cpp  class,  that  force  developer  to  implement  some  method

Common Toolkit

Control Unit Toolkit

developer  need  to  extends  this  class  to  create  a  !CHOAS  device  driver

56venerdì 14 dicembre 12

Page 57: !Chaos Control System

Write  a  Control  UnitdefineAcIonAndDataset,  is  the  method  where  the  developer  needs,  using  api,  describe  the  aBached  device.

void setDefaultScheduleDelay(int32_t);

void addDeviceId(string);

void addAttributeToDataSet(const char*const deviceID, const char*const attributeName, const char*const attributeDescription, DataType::DataType attributeType, DataType::DataSetAttributeIOAttribute attributeDirection);

Add  a  new  device  iden@fica@on  to  the  control  unit

Add  a  global  default  scheduling  rate  for  run  method

Add  a  new  aBribute  to  the  device  dataset

57venerdì 14 dicembre 12

Page 58: !Chaos Control System

Write  a  Control  Unit

template<typename T>void addInputXXXXAttributeToDataSet(const char*const deviceID, const char*const attributeName, const char*const attributeDescription, T* objectPointer, typename TDSObjectHandler<T, XXXX>::TDSHandler objectHandler)

Add  a  new  input  aBribute  of  the  type  XXXX  to  the  device  dataset  giving  ha  handler  for  manage  it

typedef void (T::*TDSHandler)(const std::string & , const XXXX&);

The  handler  is  every  method  of  an  class  that  respect  this  prototype:

58venerdì 14 dicembre 12

Page 59: !Chaos Control System

Write  a  Control  Unit

A  device  driver  can  also  define  custom  ac@ontemplate<typename T>AbstActionDescShrPtr addActionDescritionInstance(T* actonObjectPointer, typename ActionDescriptor<T>::ActionPointerDef actionHandler, const char*const actionAliasName, const char*const actionDescription);

typedef CDataWrapper* (T::*ActionPointerDef)(CDataWrapper*, bool&);

An  ac@on  is  every  method  of  a  class  that  respect  this  prototype:

59venerdì 14 dicembre 12

Page 60: !Chaos Control System

Write  a  Control  Unit

init,  is  the  method  where  the  developer  needs  to  ini@alize  his  hardware.

The  local  dataset  is  filled,  from  controlling  client,  with  the  current  default  value,  for  every  channel  or  custom  property.  The  developer  can  read  these  value  and  setup  some  useful  things.

60venerdì 14 dicembre 12

Page 61: !Chaos Control System

Write  a  Control  Unit

Send  a  data  buffer  to  the  output  buffer  for  a  determinated  device  idvoid pushDataSetForKey(const char *deviceID, CDataWrapper*);

CDataWrapper *getNewDataWrapperForKey(const char*); Return  a  new  CDataWrapper  prepared  to  be  push  on  data  output  queue

All  the  data  acquired  in  the  driver,  that  are  associated  to  the  output  channels,  need  to  be  insert  into  a  CDataWrappedObject  to  be  published

The  data  is  filled  with  the  default  chaos  key  for:  @mestamp,  deviceID,  triggerCode

run,  is  the  method  where  the  developer  needs,  using  api,  check  the  state  of  the  device  and  acquire  the  value  of  output  channel,  or  make  control  logic

61venerdì 14 dicembre 12

Page 62: !Chaos Control System

Write  a  Control  Unit

stop,  inform  the  developer  that  has  been  stopped  the  device

deinit,  inform  the  developer  that  has  been  requested  the  deini@aliza@on  of  the  device.  So  all  resource  of  the  hardware  need  to  be  deallocated.

62venerdì 14 dicembre 12

Page 63: !Chaos Control System

Write  a  Control  Unit

•  A  CU  is  a  C++  class.  

•  CUToolkit  help  to  create  a  deamon,  and  take  care  to:

•  talk  with  MDS  for  registra@on;

•  make  periodical  check  for  network(s@ll  in  development)  availability;

•  take  care  to  configure  all  driver  with  all  the  ip  for  the  server  that  will  manage  the  acquired  data;

•  other  secret  things...

summarizing

63venerdì 14 dicembre 12

Page 64: !Chaos Control System

Write  a  Control  UnitA  typical  CUToolkit  main  for  a  CU,  is  the  code  below:

int main (int argc, char* argv[] ) {

// Initialize the CUToolkitChaosCUToolkit::getInstance()->init(argc, argv);

// Add a new control unit identified by the class WorkerCUChaosCUToolkit::getInstance()->addControlUnit(new WorkerCU());

// Start the all the server and client for RPC and execute all// work to startup and register the instance of WorkerCU class// and wait an shutdown command via RPC or SIGINT signalChaosCUToolkit::getInstance()->start();

return 0;

}

the  method ChaosCUToolkit::getInstance()->addControlUnit(...) can be called for other instance of WorkerCU Class or other control unit

64venerdì 14 dicembre 12

Page 65: !Chaos Control System

11/dic/2011 65

!CHAOS  UI  Toollkit

65venerdì 14 dicembre 12

Page 66: !Chaos Control System

UIToolkit  helps  the  user  interface  development

Gives   API   that   help   to   develop:   monitoring   processes,   device  controller  etc...

Gives  an   high   performance   queue   using   "locking   free"   design.     The  queue   can   manage   quan@za@ons   ,   over   @me,   for   the   dataset   data  acquisi@on

UI  Toolkit

UIToolkit

Live Acquisition Thread Pool

Absr

actIO

Drive

r

Absr

actIO

Drive

r

DeviceControl DeviceControl

InterThread Cache Logic

Network

66venerdì 14 dicembre 12

Page 67: !Chaos Control System

At  the  moment  the  whole  package  of  UI   is   in  study  but  some  things  has  been  done  to  help  developer  to  control  hardware.

The  DeviceController  class  can  help  developer  to  connect  to  a  specific  device  using  the  ID

UI  Toolkit

67venerdì 14 dicembre 12

Page 68: !Chaos Control System

UI  Toolkit

DeviceController

MDSChannel

DeviceChannel

EventHandler

Directory  Server

DeviceID

Live Data DriverMemcached  Instances(for  now)

68venerdì 14 dicembre 12

Page 69: !Chaos Control System

UI  Toolkit

A  “c”   interface  is  provide  to  access  UIToolkit   into  sofware  that  don’t  permit  to  use  C++  as  developer  language.

This  permit   to   realize   an  user   interface   or   controller   algorithm   into  another  soZware  as  plugin,  as  for  example  LabView.

The   “C”   proxy   has  a  minimal   set  of   interface   to   simplify   the   device  controller  alloca@on,  dealloca@on  and  usage.

69venerdì 14 dicembre 12

Page 70: !Chaos Control System

11/dic/2011 70

!CHAOS  EU  Toolkit

70venerdì 14 dicembre 12

Page 71: !Chaos Control System

helps  the  development  of  distributed  algorithms  (Execu@on  Unit)  that  can  be  used  to  compute  or  automate  the  devices  control

The  EU  can  have,   in   input   the  device  channels  or   all   the  Dataset  descrip@on,  so   it  work  on  a  "Class"  of  device  not  directly  on  one  device.

The   EU   can   be   also   used   to   perform   calcula@on   instead   control  algoritm;

on   EU   can   be   preset   in   many   instance   distributed   on   different  fiscal  server

EU  Toolkit

71venerdì 14 dicembre 12

Page 72: !Chaos Control System

EU  Toolkit

Input Parameter Output Parameter

ExecutionQueue

Result Queue

Specialized Execution Unit

Abstract Execution Unit

Input Param Pack Output Result Pack

scheme  of  flow  of  input  or  output  of  an  EU

72venerdì 14 dicembre 12

Page 73: !Chaos Control System

EU  Toolkitexcep@on  could  be  concatenated  crea@ng  a  sequence  of  any  combina@on  of:

•  CompuIng  algorithm

•  Control  algorithm

Input Parameter Output Parameter

ExecutionQueue

Result Queue

Specialized Execution Unit

Abstract Execution Unit

Input Param Pack Output Result Pack

Input Parameter Output Parameter

ExecutionQueue

Result Queue

Specialized Execution Unit

Abstract Execution Unit

Input Param Pack

73venerdì 14 dicembre 12

Page 74: !Chaos Control System

14/dic/2012 74

Execu>on  Unit  Example  1Thinks  about  an  EU  that  make  some  logic  on  CU  data  and  control  another  CU

The  EU  read  the  needed  data  from  the  Output  aBribute  of  the  HW  from  the    CU_1  data

The  EU  write  the  result  of  computa@on  to  a  CU_2  for  set  the  Input  aBribute  of  the  HW

CU_1 CU_2

74venerdì 14 dicembre 12

Page 75: !Chaos Control System

14/dic/2012 75

Execu>on  Unit  Example  1this  is  the  real  data  flow

live  dataCU_1  DATA  BLOCK

The  Input  ABribute  on  CU_2is  set  with  RPC  command

Output  CU_1  aBributedata  is  read  from  live

Output  aBribute  data  is  pushed

on  live  memory  blockfrom  CU_1

some  logic  is  done

CU_1 CU_2

Ouput  aBribute  data  isread  from  device

The  ABribute  is  set  on  HWby  CU_2

75venerdì 14 dicembre 12

Page 76: !Chaos Control System

14/dic/2012 76

Execu>on  Unit  Example  2

Now  think  to  an  execu@on  unit  that  send  output  to  another  execu@on  unit  and  so  on...

the  idea  is  to  create  a  simil-­‐language  for  describe  a  “program”  that  can  be  executed  by  a  “sequence”  of  EU;

76venerdì 14 dicembre 12

Page 77: !Chaos Control System

11/dic/2011 77

!CHAOS  Data  Management

77venerdì 14 dicembre 12

Page 78: !Chaos Control System

78

Data  Management

!CHOAS  has  two  different  management  system  for  the  data  apparatus  

controlled  environment  data

Live  DataHistory  Data

78venerdì 14 dicembre 12

Page 79: !Chaos Control System

11/dic/2011 79

!CHAOS  Live  Datalive

cachedev_1

push data

ui_1

ui_2

eu_1

79venerdì 14 dicembre 12

Page 80: !Chaos Control System

80

Live  Data

The  system  of  the  Live  Data,  is  ideated  for  the  sharing  of  the  real  @me  

state  of  the  instruments  (CU)  to  other  nodes

80venerdì 14 dicembre 12

Page 81: !Chaos Control System

81

Live  Data

1.  every  node  can  read  data  from  the  central  cache  at  any  Rme[polling]

2.  the  nodes  can  register  to  central  cache  for  receive  update[push]

livecachedev_1

push data

ui_1

ui_2

eu_1

Every  instrument  pushes  the  data  of  his  I/O  channels  into  the  shared  

cached  through  the  controller  CU

81venerdì 14 dicembre 12

Page 82: !Chaos Control System

82

Live  Data

livecachedev_1

push data

ui_1

ui_2

eu_1

livecachedev_2

push data

scalability  &  reliability

All  data  of  all  instruments  are  

distributed  across  many  cache  server  

82venerdì 14 dicembre 12

Page 83: !Chaos Control System

83

Live  Data

livecachedev_1

push data

ui_1

ui_2

eu_1

livecachedev_2

push data

X

scalability  &  reliability

with  the  algorithm  previously  shown,  

when  one  server  goes  down,  all  client  

that  push's,  or  read,  data  on  that  

server,  use  another  server,  with  highest  priority

83venerdì 14 dicembre 12

Page 84: !Chaos Control System

84

Live  Data

The  usage  of  the  two  modes  permits  to  manage,  more  efficiently,  the  

bandwidth  of  the  flows  of  the  real  @me  data  (read  and  write)

not  all  nodes  need  real@me  data  at  the  same  speed:

•  the  mode  (1)  is  for  monitoring  and  control

•  the  mode  (2)  is  for  control  algorithm  that  can’t  lose  packages

84venerdì 14 dicembre 12

Page 85: !Chaos Control System

11/dic/2011 85

!CHAOS  History  Data

85venerdì 14 dicembre 12

Page 86: !Chaos Control System

86

Live  &  History  Data

Storage Driver

Index Driver

Com

mon Layer

Abstract History Logic

Proxy CQL Interpreter

Cache D

river

Cache Logic

ChaosQL  Proxy

the   live  data  management   is  merged   into  history  data  management   into  

the  ChaosQL  node  with  the  Cache  Driver

86venerdì 14 dicembre 12

Page 87: !Chaos Control System

87

History  Data

The   System   for   History   Data   has   been   designed   to   efficiently  

store  the  serialized  data  pack,  manage  different  type  of  index  on  

demand,  and  managing  data  file

87venerdì 14 dicembre 12

Page 88: !Chaos Control System

History  Data

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

CQ

L In

terp

rete

r

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

Man

agem

ent I

ndex

Log

ic

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

Stor

age

Man

agem

ent L

ogic

It  is  realized  by  three  different  services  and  two  driver

ChaosQL  Proxy Indexer  Node Management  Node

Storage Driver Index Driver

Storage  Driver  abstrac@on Index  Driver  abstrac@on

88venerdì 14 dicembre 12

Page 89: !Chaos Control System

89

Chaos  Query  Language

 ChaosQL  implement  the  following  operaBon  on  data

•  push  history/live  data  per  device•  create  index  on  device::aXribute::{rule}•  delete  index  on  device::aXribute•  retrieve  data  with  logic  operaBon  on  aXribute  and  index

89venerdì 14 dicembre 12

Page 90: !Chaos Control System

90

History  Data

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

CQ

L In

terp

rete

r

ChaosQL  Proxy

Manages    the  insert  and  select  opera@ons    

by  parsing  the  Chaos  Query  Language

Insert   -­‐  as  fast  as  possible  stores   the  data  into  the  storage  system

select   -­‐   decodes   the   query,   asks   at   the  indexer   the   posi@on   in   the   storage   of   the  

data  that  match  the  criteria

90venerdì 14 dicembre 12

Page 91: !Chaos Control System

91

History  Data

All   nodes,   using   the   ChaosQL   Driver   (that  

uses  direct   I/O)  can  query   the  history  and  

live  data.

the   results   are   received   in   asynchronous  

mode

Storage Driver

Index Driver

Comm

on Layer

Abstract History Logic

Proxy CQL Interpreter

UI or CU or EU

UI-CU-EU Toolkit

Chaos Common

Query Re

sult

CQL I/O Driver

Direct I/O Driver

91venerdì 14 dicembre 12

Page 92: !Chaos Control System

11/dic/2011

Index  Database

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

Push  Data CQL

Prox

y

Indexer  Cluster

Store  Data

Launch  Index

get  data  pack

Store  Index

Management  Cluster

92venerdì 14 dicembre 12

Page 93: !Chaos Control System

11/dic/2011

Index  Database

Storage

CommonTK

CUTK

CQL Proxy

CQL Proxy

CommonTK

EUTK

UITK

CommonTK

CQL

Prox

y

Query  Live  Data

Query  History  Data

Indexer  Cluster

send  search  on  idx

execute  search  on  idx  DB

get  pack  at  locaIon

Receive  History  Data

Management  Cluster

93venerdì 14 dicembre 12

Page 94: !Chaos Control System

94

History  Data

the  query  can  be  spliXed   in  more  than  one  Bme  

period  from  the  receiving  proxy.  

The   spliXed   query   are   sent   to   other   proxy   that  

will  process  the  sub  query

Storage Driver

Index Driver

Comm

on Layer

Abstract History Logic

Proxy CQL Interpreter

UI or CU or EU

UI-CU-EU Toolkit

Query

Resu

lt

Storage Driver

Index Driver

Comm

on Layer

Abstract History Logic

Proxy CQL Interpreter

Storage Driver

Index Driver

Comm

on Layer

Abstract History Logic

Proxy CQL Interpreter

Result

CQLPackOriginal

CQLPack1

CQLPack2

CQLPack3

Sub Query

Result

QueryChannel

ChaosCommon

CQL I/O Driver

Direct I/O Driver

94venerdì 14 dicembre 12

Page 95: !Chaos Control System

95

History  Data

Executes  the  necessary  opera@ons  to  index  

the  data  saved  on  the  storage  system

two  type  of  index:

•  TMI  -­‐  Time  Machine  index

•  DSI  -­‐  Dataset  Index

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

Man

agem

ent I

ndex

Log

ic

Indexer  Node

95venerdì 14 dicembre 12

Page 96: !Chaos Control System

96

History  Data

 TMI  -­‐  Time  Machine  index

•This  index  permits  to  catalog  every  data  pack  for  a  device  for  the  

@mestamp.  

•  all  the  data  pack  are  recorded  and  associated  to  their  @mestamp

•  this  index  is  managed  by  system  

96venerdì 14 dicembre 12

Page 97: !Chaos Control System

97

History  Data

 DSI  -­‐  Dataset  Index

•This   index   permits   to   catalog   aBributes   of   the   dataset   for   a  

determinated  instrument

•  there  will  be  possibility  of  customize  the  index  algorithm

•  this  index  is  created  on  demand  and  on  specific  @me  periods

97venerdì 14 dicembre 12

Page 98: !Chaos Control System

98

History  Data

Executes  management   opera@ons   on   files  

that  store  the  data  packs

some  data  pack  could  be  subject  to  ageing

the   ageing   state   will   determine   the   erase  

or  long  term  storage  of  a  data  pack

Storage Driver

Index Driver

Common LayerAbst

ract

His

tory

Log

ic

Stor

age

Man

agem

ent L

ogic

Management  Node

98venerdì 14 dicembre 12

Page 99: !Chaos Control System

23/02/12INFN-­‐Vitrociset  mee@ng

Directory server

CU Toolkit

CU

modules (init)

CU

modules (config)

CU

modules (set)

CU

modules (de-init)

Control UnitC

U m

odules (stop)

infrastructure

Control Unit

CU

modules (init)

CU

modules (config)

CU

modules (set)

CU

modules (de-init)

CU

modules (stop)

CU-LV interface

distributedHistory

Data service

distributedLive Dataservice

I/O m

odules

I/O Unit Exec Unit

I/O m

odules

I/O m

odules

EU m

odules

EU m

odules

EU m

odules

Common Toolkit

C User Interface LV User Interface MatLab UI

User Interface Toolkit 20%

Common Toolkit

Project  Status

99venerdì 14 dicembre 12

Page 100: !Chaos Control System

23/02/12INFN-­‐Vitrociset  mee@ng

infrastructure

developments’  details

LV User Interface: prototype in preparation (shlib for LV)

Live Data: single-instance (no-distributed) prototype ready

RPC client/server: prototype ready (msgpack and zmq impl)

History Data service: performance test (mongodb) ongoing (UniTV)

Directory services: DS simulator ready; DS under developmentscalability & availability: test facility in preparation

UI Toolkit: prototype ready

Common Toolkit: prototype ready

CU Toolkit: prototype ready

CU modules: prototype ready (C++)

CU-LV Interface prototypes& modules: almost ready

QT User Interface: prototype ready (Qt)

Event client/server: prototype ready (boost asio impl)Direct I/O c/s: todo

C wrapper UI Toolkit: prototype ready

100venerdì 14 dicembre 12

Page 101: !Chaos Control System

101

...closing

in   the   next   year   all   development   force   will   be   focused   to   realize   the  

system  for  the  history  data    and  for  the  directory  service,  a  prototype  for  

each  system  will  be  available  for  2013  2Q

101venerdì 14 dicembre 12

Page 102: !Chaos Control System

102

thanks  for  the  >me

CHOAS  GroupClaudio  Bisegni,  Luciano  Catani,  

Giampiero  Di  Pirro,  Luca  FoggeBa,  

Mazziteli  Giovanni,  MaBeo  Mara,  Antonello  Paolew,  Stecchi  Alessandro,  

Federico  Zani

thesis'sFlaminio  Antonucci  (tor  Vergata  

Ingengeria),  Andrea  Capozzi  (Tor  Vergata  Ingengeria),  Francesco  Iesu  (Cagliari  

Infroma@ca),

102venerdì 14 dicembre 12