deep dive in zabbix pre-processing · 9 text preprocessing retrieve a value by using regex:...
TRANSCRIPT
![Page 1: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/1.jpg)
DEEP DIVE INZABBIX PRE-PROCESSING
![Page 2: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/2.jpg)
2
WHY PREPROCESSING?
The retrieved data is freeform and is not fit for calculations, aggregations and/or optimal data storage:
![Page 3: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/3.jpg)
3
WHY PREPROCESSING?
HOW CAN I SOLVE THIS?
USE PREPROCESSING!
● Text preprocessing
● Structured data
● Arithmetic
● Deltas
● Numeral systems
● Javascript
● Validation
● Prometheus Exporter
![Page 4: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/4.jpg)
DEEP DIVE INZABBIX PRE-PROCESSING
THE EVOLUTION OF PREPROCESSING
![Page 5: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/5.jpg)
5
THE LEGACY WAY
Limited preprocessing support in versions <3.4:
● Custom multiplier● Numeral system transformations● Delta calculation (speed per second/simple change)
What if you needed more?
Preprocess the data by using your own scripts!
![Page 6: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/6.jpg)
6
VERSIONS 3.4 AND 4.2
ZABBIX VERSION 3.4
Introducing - Preprocessing!
New ways to transform data:
• Regex• Trim• XML XPath• JSON Path
ZABBIX VERSION 4.2
● Extended preprocessing● Validation and throttling ● Custom error handling● Preprocessing support by
Zabbix Proxy● Prometheus exporter
support
What has changed?
![Page 7: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/7.jpg)
7
VERSION 4.4
• Preprocessing of XML data via Xpath
• JSONPath aggregation and search
• Extended Custom error handling
• Introducing CSV to JSON preprocessing
• WMI, JMX and ODBC data collection returns JSON arrays – ready to preprocess via JSONPath!
What has changed?
![Page 8: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/8.jpg)
DEEP DIVE INZABBIX PRE-PROCESSING
THE MANY WAYS OF PREPROCESSING
![Page 9: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/9.jpg)
9
TEXT PREPROCESSING
Retrieve a value by using Regex:
Temperature: 36C PCRE 36
Trim the retrieved value and store it as a number:
36C Right Trim 36
![Page 10: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/10.jpg)
10
STRUCTURED DATA
Retrieve value from JSON or XML data:
![Page 11: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/11.jpg)
11
ARITHMETIC
Custom Multipliers to transform numeric data:
● Difference between current and previous value
● Change per second
DELTA CALCULATIONS
![Page 12: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/12.jpg)
12
INTRODUCED IN VERSION 4.2
● Custom scripts● Validation● Throttling● Prometheus
![Page 13: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/13.jpg)
13
JAVASCRIPT
Implemented with Duktape JavaScript engine!
With JavaScript you can perform:
● Data transformation● Data aggregation● Data filtering● Logical expressions● etc.
...All done internally by Zabbix
![Page 14: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/14.jpg)
14
JAVASCRIPT
Convert diskstats to JSON:
![Page 15: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/15.jpg)
15
JAVASCRIPT
Preprocessed data - ready for LLD!
Initial data
![Page 16: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/16.jpg)
16
JAVASCRIPT
Result:
The items have been discovered and created by our LLD!
![Page 17: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/17.jpg)
17
VALIDATION
Validate data against validation logic:
● In range● Matches regular expression● Does not match regular expression● Check for errors in JSON/XML or by using regex
CUSTOM ON FAILDefine the behavior in cases when the preprocessing fails:
● Discard value● Set value to● Set error to
![Page 18: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/18.jpg)
18
VALIDATION
Discard the value outside of defined range:
Match a regular expression and set the value to Unknown if no match is obtained:
![Page 19: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/19.jpg)
19
VALIDATION
Check for an application-level error message located at JSONPath/XPath:
{
"Software":"Zabbix",
"Version":"4.2.0",
"OS":"CentOS 7",
"ErrorMessage":"Service Down"
}
Error message!
![Page 20: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/20.jpg)
20
THROTTLING
Enables high frequency monitoring with minimal performance impact:
● Discard repeating values● Discard repeating values with a heartbeat
Useful when we are receiving a lot of duplicate data at a very high frequency!
Application service monitoring:
Time: 00:00 00:02 00:04 00:06 00:08 00:10 00:12
Data: Up Down Down Up Up Up Up
![Page 21: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/21.jpg)
21
BEFORE THROTTLING
History
0
10000110
Zabbix Server
0
10000110
![Page 22: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/22.jpg)
22
WITH THROTTLING
History
0
10000110
Zabbix Server
0
10---1-0
![Page 23: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/23.jpg)
23
THROTTLING WITH A HEARTBEAT
History
0
10000110
Zabbix Server
0
10--0 (Heartbeat)1-0
![Page 24: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/24.jpg)
24
PROMETHEUS
● Query Prometheus endpoint with HTTP checks● Use preprocessing to obtain metrics● Use within LLD to discover components monitored by
Prometheus
…And many more
![Page 25: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/25.jpg)
25
PROMETHEUS PATTERN
● Create master HTTP item● Create dependent items with “Prometheus pattern”
Retrieve a metric:
Retrieve a label value:
![Page 26: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/26.jpg)
26
PROMETHEUS LLD
Create a dependent item LLD with a “Prometheus to JSON”preprocessing step.
Discover all CPU’s:
![Page 27: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/27.jpg)
27
PROMETHEUS LLD
Retrieved JSON:
![Page 28: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/28.jpg)
28
PROMETHEUS LLD
Retrieved JSON:
Macros:{#CPUNUM}, {#MODE} JSONPath:$.labels.cpu, $.labels.mode
Macro:{#HELP} JSONPath:$.help
![Page 29: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/29.jpg)
29
PROMETHEUS LLD
Item prototype with “Prometheus pattern” preprocessing step:
node_cpu_seconds_total{cpu="{#CPUNUM}",mode="{#MODE}"}
![Page 30: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/30.jpg)
30
PROMETHEUS LLD
Use {#HELP} to populate the Description field:Macro:{#HELP} JSONPath:$.help
![Page 31: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/31.jpg)
31
PROMETHEUS LLD
![Page 32: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/32.jpg)
32
PROMETHEUS LLD
![Page 33: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/33.jpg)
33
PREPROCESSING BEFORE VERSION 4.2
All of the preprocessing is being done by the server!
HistoryZabbix ServerZabbix Proxy
Preprocessed dataRaw data Compressed raw data
![Page 34: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/34.jpg)
34
PREPROCESSING WITH VERSION >= 4.2
Preprocessing is performed on the proxy!
HistoryZabbix ServerZabbix Proxy
Preprocessed dataRaw data Compressed preprocessed data
![Page 35: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/35.jpg)
DEEP DIVE INZABBIX PRE-PROCESSING
PREPROCESSING UNDER THE HOOD
![Page 36: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/36.jpg)
36
PREPROCESSING WORKFLOW
Add result to queue
Send value to worker
Enqueue item value
Execute preprocessing steps
Value stored
Data source
Preprocessing manager
Preprocessing worker
![Page 37: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/37.jpg)
37
PREPROCESSING MANAGER
● Added in version 3.4● Enqueues the items in the preprocessing queue● Assigns the preprocessing tasks to preprocessing workers● Flushes the preprocessed values from the queue
● StartPreprocessors defines the value of pre-forked preprocessing workers
● Number of workers determined by:○ Count of preprocessable items○ Count of preprocessing steps○ etc.
PREPROCESSING WORKERS
![Page 38: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/38.jpg)
38
LET’S RECAP
• Use master items with LLD!
• Discover your metrics with preprocessing!
• Data validation!
• Custom behavior with advanced preprocessing rules!
• Discard unnecessary data with throttling!
• Improve performance by preprocessing on the proxy!
• Transform your data!
• Enable data aggregation and calculation with preprocessing!
Automate!
Customize! Transform!
Improve!
![Page 39: DEEP DIVE IN ZABBIX PRE-PROCESSING · 9 TEXT PREPROCESSING Retrieve a value by using Regex: Temperature: 36C PCRE 36 Trim the retrieved value and store it as a number: 36C Right Trim](https://reader036.vdocuments.us/reader036/viewer/2022081611/5f0218be7e708231d4028da4/html5/thumbnails/39.jpg)
THANK YOU!