diamon/laser design proposal
DESCRIPTION
Diamon/Laser Design proposal. Marek Misiowiec BE/CO/AP May 20 1 0. Core Overview. Multi-tier, distributed, fail-safe system processing diagnostic data collected repeatedly from a variety of sources . Raw data of different shapes is transformed into unified formats and further - PowerPoint PPT PresentationTRANSCRIPT
Diamon/Laser Design proposal
Marek Misiowiec BE/CO/APMay 2010
Core OverviewMulti-tier, distributed, fail-safe system processing diagnostic data collected repeatedly from a variety of sources. Raw data ofdifferent shapes is transformed into unified formats and furtheranalyzed on a middle layer according to well-defined rules.
Actions may be taken, in form of alarms or monitoring informationdelivered to clients. Processed data is made available as JAPCparameters or for dedicated GUIs. It is stored in long term archive
too. Downward communication with the source devices is possible.2
devs
apps DAQ
raw data
unifieddata
SRV
clients
client view
Core Overview Diamon/Laser core responsibilities:
• data acquisition for various sources – push & pull modes• communication through layers, publishing• data transformation, processing, analysis, conditioning• failover, database separation, scaling
complementary functionalities:• rules engine• logging, archiving, configuration updates
non-core:• definitions & rules, data provider’s workflow• GUIs, dedicated client applications
3
Importance of a changeNowadays: upgrades mess
• stale sources, legacy constraints, dusty components meeting only part of the requirements
• partial data, simplistic thresholds, no conditioning monitoring based on alarms - Diamon on Laser
• misuse of the system, overhead• mediocre scaling• badly-behaving sources/clients cannot be detached/isolated
legacy solutions hiding in Laser world• SonicMQ, OC4J, EJBs, Hibernate, Sleepycat
...and no standard CO components in place
4
Importance of a change
We would like to: widely use CO solutions and help to establish others worth it better drive interactions with the bottom and upper tiers consolidate logics in system core be able to draw more conclusions on collected data sustain upgrades, updates, dependency changes
5
Transition
6
JMSNotifier
oc4j-diamon
AlarmArchive
Agent Messages
Checks surveillance
Alarms, filters non registered
Alarms
Result Messages
Sends update/Requests config
CLIC Agents
Central Agents
CMWJMXWIE TWIDO SNMP
PING
jdaemon
Configuration
Agentsto be
configured
configures
commands
commands
Router
CCC Consoles
LASER
ALARMS
YAMI protocol
JMS
Transition
7
JMSNotifier
oc4j-diamon
AlarmArchive
Agent Messages
Checks surveillance
Alarms, filters non registered
Alarms
Result Messages
Sends update/Requests config
CLIC Agents
Central Agents
CMWJMXWIE TWIDO SNMP
PING
jdaemon
Configuration
Agentsto be
configured
configures
commands
commands
Router
CCC Consoles
LASER
ALARMS
YAMI protocol
JMS
OC4J, EJBs, Hibernate,
BerkeleyDB
OC4J, EJBs, Hibernate,
BerkeleyDB
secondSonicMQ
firstSonicMQ
In-house DB
In-house DB
SNMP, CMW, variety of data
formats
directly to devices
not enough data
YAMI specific protocol
Transition
8
JMS
spring AlarmArchive
Agent Messages
Result Messages
Sends update/Requests config
CLIC Agents
Central Agents
CMWJMXWIE TWIDO SNMP
PING
jdaemon
Configuration
Agentsto be
configured
configures
commands
commands
CCC Consoles
YAMI protocol
JMS
LASERPROXY
LASERCMW
Transition
9
spring
AlarmArchive
Result MessagesSends update/
Requests config
CLIC Agents
Central Agents
CMWJMXWIE TWIDO SNMP
PING
jdaemon
Configuration
configures
commands
CCC Consoles
YAMI protocol
JMS
LASERPROXY
LASERCMW
Transition
10
spring
AlarmArchive
Result Messages
Data Acquisition Layer
CLIC AGENTS
CMWJMXWIE TWIDO SNMP
PINGConfiguration
commands
CCC Consoles
JMS
LASERPROXY
LASERCMW
Rules, thresholds, distribution, analysis, publishing...
JAPC
other applications
Technical outcomeThen: unified view of complete data sound construction on CO materials
• OC4J -> Spring, Hibernate -> JDBC Templates, EJBs -> POJOsSonicMQ -> ActiveMQ, multitude of data formats -> JAPCin-house database -> CCDB + Logging
• wider choice of middleware: CMW, YAMI, JMS focusing logics in one place
• rules, thresholds, alarm generation• conditioning: machine mode, working hours, state of subsystems
driving the whole process • detachable sources, clients• less restart/reboot downtime• convergence with similar efforts
11
information exchange
12
japc-ext-yami japc-ext-cmw japc-ext-snmp jms-to-japc proxy japc-ext-?
Core Services
rules thresholds
alarm generator
externalsconditions
publisher w-flow
consoles
client middleware: JMSclientinter
actions
other apps
JAPCprovider
ConfigDB
archiverArchive
DB
Core
DAQ [1] DAQ [2] DAQ [n]
tackleproblematic
sources
Clicagents CMW
srcsLaserWie, Twi
?other
sources
source middleware protocols
Available technologies
Building blocks: CO standard components eg. JAPC CO less standard, but demanded eg. japc-ext-yami Laser dev code base, ready to reuse eg. caching, distribution TIM being refurbished and struggling with similar future other CO projects: INCA
13
Lessons learned from Laser eradication of legacy concepts & technologies pays off to scale well, we need to distribute processing power
• a supervisor over multiple workers• dividing is difficult
separation of concerns single processing unit as a set of loosely coupled actions
• patterns: chain of responsibilities, observable• upward flow: source input -> processing -> publishing
cater for database outages through local caching JMS+RMI as a middleware for all parties involved
• more advanced features of JMS channel help ...but no JAPC concepts were introduced at all
14
Device/property model to clarify the current incoherent view strong incentives:
• speak one language• let everyone fit (Logging)• common tools, well-established in CO world
migration of legacy sources• expose properties• eradicate laser-source as it is
interaction with devices through JAPC calls domain entities:
• Alarm, Alarms, Diagnostic,...
15
JAPC extensions to unify view on a variety of devices/applications
available now:• CMW (japc-ext-cmwrda) and JMS (remote) as most popular
we need some more:• YAMI through japc-ext-yami is already available,
used in new Clic agent• SNMP through japc-ext-snmp under development
16
Bottom tierExperience gained in: current Diamon and Laser
• eg. CMW Alarm Monitor as a huge data acquisition process LHC Concentrators , similar problems INCA data acqusition layer
Progress:
17
source type current planned/done
Clic Agents JMS, YAMI japc-ext-yami
CMW Alarm Monitor Laser gateway JAPC deviceLaser sources C/Java laser-source Laser-to-JAPC proxyWiener, Twido SNMP central agents japc-ext-snmpPing central agent JAPC device
Already developed JAPC extensions current version of YAMI meets our requirements Diamon Clic agent
• fully YAMI-based, JAPC values CMW Alarm Monitor refurbished
• 6000 subscriptions to GM and FESA devices• will be incorporated into new core
proxy between Laser production alarms and JAPC values• for the legacy sources
18
Open questionsStill to think over: best components and collaborations to choose level of distribution and scaling shared memory, distributed cache storage: short-term, long-term, Logging DB rules handling: homemade, proprietary ...and many others
19