c2mon - a highly scalable monitoring platform for big data scenarios @cern by matthias brager
TRANSCRIPT
![Page 1: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/1.jpg)
![Page 2: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/2.jpg)
C2MON A highly scalable monitoring platform for Big Data scenarios at CERN
Matthias Bräger, CERN [email protected] 20th – 21st of May 2016 J on the Beach, Malaga, Spain
![Page 3: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/3.jpg)
About me
▪ Responsible for Technical Infrastructure Monitoring (TIM) service at CERN ▪ Head of the CERN Control and Monitoring Platform (C2MON): http://cern.ch/c2mon ▪ Working at CERN since Dec. 2007
Before CERN: ▪ 2 years at LOGICA space department for ESOC, Darmstadt, Germany ▪ 4 years Java developer at IBM, Mainz, Germany
3
![Page 4: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/4.jpg)
European Organization for Nuclear Research
▪ Founded in 1954 (60 years ago!) ▪ 21 Member States
▪ ~ 3’360 Staff, fellows, students... ▪ ~ 10’000 Scientists from
113 different countries
▪ Budget: 1 billion CHF/year
http://cern.ch
4
![Page 5: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/5.jpg)
From Physics to Industry
5 http://cern.ch/knowledgetransfer
![Page 6: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/6.jpg)
ATLAS
CMS LHCb
Alice LHC
The worlds biggest machine
Generated 30 Petabytes in 2012 > 100 PB in total!
6
![Page 7: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/7.jpg)
LHC - Large Hadron Collider
27km ring of superconducting magnets Started operation in 2010 with 3.5 + 3.5 TeV, 4 + 4 TeV in 2012 2013 – 2015 in Long Shutdown 1 (machine upgrade) Restarted in April 2015 with 6.5 + 6.5 TeV max
7
![Page 8: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/8.jpg)
Some ATLAS facts ▪ 25m diameter, 46m length, 7’000 tons ▪ 100 million channels ▪ 40MHz collision rate (~ 1 PB/s)
8
![Page 9: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/9.jpg)
9
![Page 10: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/10.jpg)
Physics data(>100 PB)
Metadata of physics data
Sensor Data of technical installations
Log data Configuration data
Documents
Media data
Others
10
![Page 11: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/11.jpg)
Is Hadoop used for storing the ~30 PB/year of physics data ?
No ;-( Experimental data are mainly stored on tape
CERN uses Hadoop e.g. for storing the metadata of the experimental data
11
![Page 12: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/12.jpg)
Physics Data Handling ▪ Run 1: 30 PB per year
demanding 100’000 processors with peaks of 6 GB/s writing to tape spread across 80 tape drives
▪ Run 2: > 50 PB per year
with peaks of 10 GB/s writing to tape
CERN’s Computer Center (1st floor)
12
![Page 13: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/13.jpg)
Physics Data Handling 2013 already more than 100 PB stored in total! ▪ > 88 PB on 55’000 tapes ▪ > 13 PB on disk (total disk space 45 PB) ▪ > 140 PB free tape storage waiting for
Run 2
CERN’s tape robot
13
![Page 14: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/14.jpg)
Why tape storage? ▪ Cost of tape storage is a lot less than
disk storage ▪ No electricity consumption when tapes
are not being accessed ▪ Tape storage size = Data + Copy
Hadoop storage size = Data + 2 Copies
▪ No requirement to have all recorded physics data available within seconds
CERN’s tape robot
14
![Page 15: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/15.jpg)
Physics data(>100 PB)
Metadata of physics data
Sensor Data of technical installations
Log data Configuration data
Documents
Media data
Others
15
![Page 16: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/16.jpg)
A lot of systems to control
16
85’000 Devices > 2 Million I/O Endpoints
Much more when including subsystems!
Electricity
Magnets
Cryogenics
Controls Computers
Vacuum
Ventilation
Cooling
Safety
![Page 17: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/17.jpg)
Main systems controlled from one central point: The CERN Control Centre
![Page 18: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/18.jpg)
A complex controls infrastructure...
• Each dot is a process • Each line is a network connections
![Page 19: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/19.jpg)
• Many different types of data sources and protocols
• Complex data structure • Different data rates
The configuration hell
19
How to subscribe to my data?
![Page 20: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/20.jpg)
? C2MON
What if…
20
History
… there is a platform that: • handles low level data subscriptions • monitors the different data sources • reconfigures acquisiton processes at runtime
• standardises messages and data storage • reduces data streams to relevant information • always keeps the latest values available
• provides custom data streams • provides access to history … and is modular and open source?
myApp
MySQL HSQL Oracle Elasticsearch
![Page 21: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/21.jpg)
C2MON – A great platform for many use cases
21
C2MON History
Use C2MON… • to feed your analytics framework • to structure persist your data in ES for
offline analytics • as backbone for your SCADA system • as configurable data proxy • to write innovative new Java and Web
applications • …
MySQL HSQL Oracle Elasticsearch
myApp
![Page 22: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/22.jpg)
Use C2MON to realise IoT scenarios
22
C2MON System X
System Z
System Y
![Page 23: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/23.jpg)
C2MON - CERN Control and Monitoring Platform
▪ Modular and scalable at all layers ▪ Optimized for high availability & big data volume
▪ Server based on In-Memory cache solution Two big monitoring services (TIM & DIAMON) running in production with C2MON at CERN Central LHC alarm system (“LASER”) in migration phase
Ready for outside collaborations!
C2MONServer
ClientAPImyApp
DAQAPImyDAQ
Acquisi?onFilteringValida?on
Business
ClientApps
myMod
http://cern.ch/C2MON
![Page 24: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/24.jpg)
Architecture
Client API
C2MON Server
History / Backup
In-Memory
- configuration - rule logic - latest sensor values - assuring high availability
DAQs
No downtime, if DB is not available.
24
Sensor == Tag
![Page 25: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/25.jpg)
In-Memory approach: Scale with data and processing needs
Increase Data in Memory
Reduce database dependency
DB
Application
DB
Application
In-Memory
Distributed In-Memory
DB
Application
In-Memory
Application
In-Memory
25
![Page 26: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/26.jpg)
In-Memory Data Grid solutions
Forrester Wave™: In-Memory Data Grids, Q3 2015
Popular Open Source solutions:
![Page 27: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/27.jpg)
The Tag family
Tag
ControlTag
DataTag
RuleTag
• Internally used for Process and Equipment surveillance
• Used for data acquisition *
• (#123 + #234) > 2 [ERROR], true [OK]
• id • name • value • Quality • timestamp
* Support of primitive arrays and arbitrary Objects
![Page 28: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/28.jpg)
TagService tagService = C2monServiceGateway.getTagService();
tagService.subscribeByName(”my.tags.foo1", tag -> {
// String name = tag.getName(); // Object value = tag.getValue(); // boolean valid = tag.isValid();});
28
Example: Subscription to one Tag
C2MON
myApp
![Page 29: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/29.jpg)
TagService tagService = C2monServiceGateway.getTagService();
tagService.subscribeByName(”my.tags.*", tag -> {
// String name = tag.getName(); // Object value = tag.getValue(); // boolean valid = tag.isValid();});
29
Example: Subscription to Tag stream
C2MON
myApp
my.tags.foo1 my.tags.foo2 … my.tags.xyz900.joe
Wildcard support
![Page 30: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/30.jpg)
Acquisition layer
DAQ Process
Equipment (or Service)
SubEquipment
Machine running DAQ process
DAQ Process takes care of: • Equipment/Service monitoring • Data acquisition for configured Tags • Raw data validation & filtering • Sending data to server tier
![Page 31: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/31.jpg)
Raw data validation & filtering on DAQ layer
Dynamic Filtering ▪ Dynamic Time dead-band filtering for Protecting
against data bursts Static Filtering ▪ Static time dead-band filtering ▪ Value redundancy ▪ Value dead-band filtering Data Validation ▪ Value in defined range? ▪ Correct value data type? ▪ Source timestamp in the future? ▪ Outdated information?
C2MONServer
DAQAPImyDAQ
Acquisi?onFilteringValida?on
BusinessmyMod
31
Configurable by Tag
![Page 32: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/32.jpg)
Basic configuration structure Process
Equiment [0..*]
SubEquipment [0..*]
DataTag(s)
Metadata [0..*] Alarm [0..*]
Metadata [0..*]
DataTag(s)
Metadata [0..*] Alarm [0..*]
Metadata [0..*]
Commands [0..*]
![Page 33: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/33.jpg)
ConfigurationService confService = C2monServiceGateway.getConfigurationService();
confService.createProcess("P_JONTHEBEACH");
confService.createEquipment( "P_JONTHEBEACH”, "E_REST”, "cern.c2mon.daq.rest.RESTMessageHandler”);
33
Example: New DAQ configuration
Inject your own DAQ flavour
C2MON Server
P_JONTHEBEACH
E_REST
![Page 34: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/34.jpg)
HashMap<String, String> tagAddress = new HashMap<>(); tagAddress.put("mode", "POST"); tagAddress.put("postFrequency", "60");
confService.createDataTag( "E_REST", DataTag.create( "Temperature_Malaga", Integer.class, new DataTagAddress(tagAddress)).build() );
34
Example: Create DataTag
C2MON Server
P_JONTHEBEACH
E_REST
![Page 35: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/35.jpg)
HashMap<String, Object> metas = new HashMap<>(); metas.put(”GPS", "N 36° 43' 16.585'', W 4° 25' 17.035''"); metas.put("Country", "Spain");
confService.updateTag( DataTag.update("Temperature_Malaga”).metadata(metas).build() );
35
Example: Add metadata to Tag
C2MON
myApp
![Page 36: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/36.jpg)
36
C2MONServer
DAQAPI
JSON/XML
JSON
ClientAPI
BootstrapWEBConsole
ClientAPIAlarmCon
sole
Database
Open Source in all layers
Easy prototyping due to Spring Boot
![Page 37: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/37.jpg)
as timeseries data storage
ß C2MON à
![Page 38: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/38.jpg)
C2MON example Technical Infrastructure Monitoring (TIM)
▪ Operational since 2005
▪ Used to monitor and control infrastructure at CERN ▪ 24/7 service
▪ ~ 100 different main users at CERN
▪ Since Jan. 2012 based on
new server architecture with C2MON
CERN Control Center at LHC startup
38
![Page 39: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/39.jpg)
TIM – Main features
▪ Unifies sensor data from a multitude of sources and protocols
▪ Provides simple dashboarding and access to historical values
▪ Central configuration management
▪ Filters raw data streams
Workflow based sensor and alarm declaration
39
![Page 40: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/40.jpg)
Cooling Safety Systems Electricity Access Network and Hardware Controls
Cryogenics
TIM Server based on C2MON
Client Tier
Data Analysis Video Viewer TIM Viewer Access Management
Alarm Console
Data Acquisition & Filtering
> 1200 commands > 1300 rules
> 91k data sensors > 41k alarms
Web Apps
40
![Page 41: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/41.jpg)
TIM Server based on C2MON
Client Tier
Data Analysis Video Viewer TIM Viewer Access Management
Alarm Console
Data Acquisition & Filtering
ca. 400 million raw values per day
Filtering ca.2 million updates
> 1200 commands > 1300 rules
> 91k data sensors > 41k alarms
Web Apps
41
![Page 42: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/42.jpg)
C2MON in Action
DAQ
C2MONServer TerracoPa
…
C2MONServer
DAQ
…
AccessDashboardDataAnalysis
TerracoPaStandby
Acquisition & Filtering > 15 Equipment types > 400 Million updates/day (TIM)
Business Logic Layer > 91k data sensors > 300k alarms > 20K commands > 48.000 business rules > 28 Million updates/day (DIAMON)
Client
OPC PLC
AlarmWebInterface
42
![Page 43: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/43.jpg)
What’s coming next?
▪ Offering C2MON to the Open Source community
http://cern.ch/c2mon ▪ Collaboration with Universities and Industry ▪ Providing new simple acquisition API for Zero Configuration ▪ Introduction of Streaming Analytics module
43
![Page 44: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/44.jpg)
Technologies used for C2MON C2MON server: No J2EE server and only open source! ▪ Java 7, Spring 4.2 ▪ persistence framework: MyBATIS (server), Hibernate (client) Dependency management through Maven Supported Databases: Oracle, HSQL, MySQL, Elasticsearch Middleware: JMS ActiveMQ ▪ no direct communication between message publisher and consumer ▪ secured broker access ▪ broker clustering ▪ persistent queues ▪ topics for broadcast/reply messages Message transport format: XML and JSON Remote caching solution for C2MON server cluster: Terracotta/Ehcache, (support for other caches foreseen) ▪ horizontally scalable ▪ proven technology ▪ open source ▪ support contract possible
![Page 45: C2MON - A highly scalable monitoring platform for Big Data scenarios @CERN by Matthias Brager](https://reader031.vdocuments.us/reader031/viewer/2022030308/58ede1df1a28abc1508b4659/html5/thumbnails/45.jpg)
Questions? Thank you for your attention!