Download - Rihards Olups - Zabbix at Nokia - Case Study
![Page 1: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/1.jpg)
Zabbix at Nokia
September 9-10, Zabbix ConferenceRiga, Latvia
![Page 2: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/2.jpg)
INTRODUCTION
About the Speaker
Using Zabbix since 2001
Pleasure to work with the Zabbix team for 5+ years
A couple of books on Zabbix
![Page 3: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/3.jpg)
INTRODUCTION
Who We Were
3310
![Page 4: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/4.jpg)
INTRODUCTION
![Page 5: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/5.jpg)
INTRODUCTION
Who We Are
Telecommunication and infrastructure
Hardware and software
![Page 6: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/6.jpg)
INTRODUCTION
What We Use
Zabbix at (a single division of) Nokia
Not just Zabbix – a lot of different solutions
Not an endorsment
![Page 7: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/7.jpg)
INTRODUCTION
What We Have
Zabbix 2.4 instance, production
Zabbix 2.4 instance, testing
Zabbix 2.4 instance, development
Zabbix 2.2, production
...these are the "new" systems
![Page 8: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/8.jpg)
INTRODUCTION
What We Still Have
Zabbix 1.8, production
... x 2
Also, agents
![Page 9: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/9.jpg)
INTRODUCTION
THAT OLD !>!!??1111one
"We released version n 3 months ago, WHY HAVEN'T THEY UPGRADED YET ?"
"But the new version has nnnnn..."
"____FILL_IN____"
![Page 10: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/10.jpg)
INTRODUCTION
The Reasons
Reason 1: the rule of .4
Reason 2: if it works...
Reason 3: it's complicated
![Page 11: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/11.jpg)
INTRODUCTION
There's More
A few more Zabbix instances
Various versions
Planning to deploy 3.0 – cutting edge
![Page 12: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/12.jpg)
INTRODUCTION
Backend
Oracle for 1.8
MySQL/MariaDB for anything after that
Main reasons:● Licencing● Reliability (sharing the database)
![Page 13: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/13.jpg)
INTRODUCTION
Important Building Blocks
Scripts to collect the data
API-using tools
JMX (Java is popular – why?)
![Page 14: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/14.jpg)
INTRODUCTION
The Three (Main) Topics
The experience of upgrading Zabbix
Other trouble
Suggested practice and solutions
![Page 15: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/15.jpg)
UPGRADING PROBLEMS SOLUTIONS
Upgrading 1.8 to 2.4
Why do companies sit on old releases?
Upgrading is an effort.
![Page 16: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/16.jpg)
UPGRADING PROBLEMS SOLUTIONS
Something Breaks
Investigation starts● There are items● Items stopped getting data● Person who wrote this has moved on
![Page 17: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/17.jpg)
UPGRADING PROBLEMS SOLUTIONS
When Something Breaks
Have to figure out:● What does it do● How to fix it● Before that – where the hell is it
![Page 18: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/18.jpg)
UPGRADING PROBLEMS SOLUTIONS
One Example of an API-using Script
"user.authenticate" method removed
'auth' not allowed in user.login
item "description" changed to "name"
"exists" methods removed
"& |" changed to "and or"
zabbix_sender changed "failed 0" to "failed: 0;"
![Page 19: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/19.jpg)
UPGRADING PROBLEMS SOLUTIONS
Solved All That?
Still fails against one of the "new" systems
Remember, one of them was still 2.2
...so "& |" needed instead of "and or"
![Page 20: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/20.jpg)
UPGRADING PROBLEMS SOLUTIONS
Three
Three cases:● Old API, sender● New API, sender + "& |"● New API, sender + "and or"
![Page 21: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/21.jpg)
UPGRADING PROBLEMS SOLUTIONS
Click Me
Links in alert emails – 3 versions again:● 1.8 – patched version to show graph● 2.2 – history.php?itemid=● 2.4 – history.php?itemids[ ]=
![Page 22: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/22.jpg)
UPGRADING PROBLEMS SOLUTIONS
Java GW Trouble
Java GW 2.0 – patched (endpoints, ports)
2.0 never times out, no time to forward-port
Solution – a remote command to restart...
![Page 23: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/23.jpg)
UPGRADING PROBLEMS SOLUTIONS
Zabbix Server Trouble
Didn't use a feature before
Started using it, server crashes
Have an action on another Zabbix server to restart this one -> ● WORKSFORME
(that server was upgraded quickly, though)
![Page 24: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/24.jpg)
UPGRADING PROBLEMS SOLUTIONS
More Inter-version Template Fun
Have an updated template in 2.4
Import it in 2.2. Fail.● Change "and or" back to "& |"● ...remember to use the HTML entity● Change some more● Did they use spaces? Where they updated?● ...dependencies
![Page 25: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/25.jpg)
UPGRADING PROBLEMS SOLUTIONS
Migrating Users
Get their media, too – API method user.get
...doesn't return media in 1.8
DB it is
![Page 26: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/26.jpg)
UPGRADING PROBLEMS SOLUTIONS
The Great Things When Upgrading
It's the little things● Usability● Links● Maintainability
![Page 27: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/27.jpg)
UPGRADING PROBLEMS SOLUTIONS
What We Notice
Newlines in trigger expressions
Links, links, links● Link in simple graphs lost in 3.0...
Death to the dropdowns
Split users/groups in the administration section
![Page 28: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/28.jpg)
UPGRADING PROBLEMS SOLUTIONS
The Non-shiny
Change design? The overworked peasant never notices. Used to "enterprise" design.
Runtime loglevel changing? A lifesaver.
Less bugs. Really.
Human-friendly errors. Or any at all.
![Page 29: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/29.jpg)
UPGRADING PROBLEMS SOLUTIONS
Upgrading to 3.0
...
...
eh ?
![Page 30: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/30.jpg)
UPGRADING PROBLEMS SOLUTIONS
Templates and Upgrading to 3.0
Import a template that uses SNMP LLD
Get burnt
ZBX-10758● XML import does not convert SNMP LLD rule
![Page 31: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/31.jpg)
UPGRADING PROBLEMS SOLUTIONS
It's Not Perfect
The cynical doctor – "can you get to the door?"
...more like "got pimples? well, live with it"
![Page 32: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/32.jpg)
UPGRADING PROBLEMS SOLUTIONS
Discovering Trouble
Network discovery script, returns tags
Rewritten, some tags reused
Misconfigured discovery ->● all messed up (not getting away with it)● audit not helpful
![Page 33: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/33.jpg)
UPGRADING PROBLEMS SOLUTIONS
Action Fun
No built-in way to test actions
Create a trapper item+trigger
Limit an action to that single trigger
![Page 34: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/34.jpg)
UPGRADING PROBLEMS SOLUTIONS
...Is Actionable
Successfully test the action
Delete test item & trigger
Email saying it's all good now
![Page 35: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/35.jpg)
UPGRADING PROBLEMS SOLUTIONS
Action Trouble Is Here
What's wrong?
Action got silently disabled
https://github.com/whosgonna/Zabbix-Tiny.pm/blob/master/ examples/example_check_action_by_id.pl
![Page 36: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/36.jpg)
UPGRADING PROBLEMS SOLUTIONS
New Item Not Working
...no, just the oldest value[s] missing
https://support.zabbix.com/browse/ZBX-9236
![Page 37: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/37.jpg)
UPGRADING PROBLEMS SOLUTIONS
Graph Says It's All Good
Something alerts
Check the graph – straight line at 1, all good
...it's a trapper item
![Page 38: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/38.jpg)
UPGRADING PROBLEMS SOLUTIONS
My Favourite Things
Monitoring -> Triggers
...made nearly useless by new triggers blinking
ZBX-7559 (single 'ok' event considered as a change)
![Page 39: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/39.jpg)
UPGRADING PROBLEMS SOLUTIONS
That Comma
Suddenly items start failing
Value 82,82 not suported for float
iostat uses user locale – who starts the daemon
![Page 40: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/40.jpg)
UPGRADING PROBLEMS SOLUTIONS
And You, curl
The same with curl● $ curl -s -w '%{time_total} - %{speed_download}\n'
www.zabbix.com -o /dev/null
● 0,701 – 55300,000
Initscripts or external checks/userparams
![Page 41: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/41.jpg)
UPGRADING PROBLEMS SOLUTIONS
The Silence of Triggers
Scripts break, items get misconfigured
We get no alerts. Review unsupported items.
3.2 will be one hell of a time... but better
![Page 42: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/42.jpg)
UPGRADING PROBLEMS SOLUTIONS
nodata() on Unsupported
<q1x> HALLEHLUJAH
<volter> Oh yes!
<Silvery> finally
![Page 43: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/43.jpg)
UPGRADING PROBLEMS SOLUTIONS
When Scripts Roam Free
Lockfiles not being removed
...just monitor for thatUserParameter=vfs.files.older_than[*],find "$1" -ctime +$2 | wc -l
![Page 44: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/44.jpg)
UPGRADING PROBLEMS SOLUTIONS
A (Quite) Rocky Horror Picure Show
Can't migrate template from newer -> older
Redo manually, forget the LLD filter
Get hundreds of thousands of items● On a busy DB server
![Page 45: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/45.jpg)
UPGRADING PROBLEMS SOLUTIONS
Services + Ports
Multiple JVMs or any other service
Often just a single port/item
Template proliferation
![Page 46: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/46.jpg)
UPGRADING PROBLEMS SOLUTIONS
Multiple JVMs
Several JVMs on same host – a problem● Patched Java GW● Separate hosts● Works out better maintenance-wise
![Page 47: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/47.jpg)
UPGRADING PROBLEMS SOLUTIONS
Scripting the API
Creating items, triggers
Generating graphs and screens
General maintenance (users, host groups)
![Page 48: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/48.jpg)
UPGRADING PROBLEMS SOLUTIONS
API Scripts
Weird load on Zabbix, triggers deleted/created
Function get_trigger_id unconditionally deletes the trigger
Check your scripts, including logout
![Page 49: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/49.jpg)
UPGRADING PROBLEMS SOLUTIONS
API Issues
New users are obliterated by validation/error messages
Missing functionality – but gets better
API mostly works in recent versions
![Page 50: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/50.jpg)
UPGRADING PROBLEMS SOLUTIONS
Audit Log
Many operations not recorded
Significant issue with many admin users
![Page 51: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/51.jpg)
UPGRADING PROBLEMS SOLUTIONS
Syncing the Templates
Needed the templates to be the same across 5 Zabbix servers
Manual syncing
Looking into the API – after decomissioning 1.8
![Page 52: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/52.jpg)
UPGRADING PROBLEMS SOLUTIONS
The Daily WTF
These problems weren't massive
But the small problems eat your time
Death by a thousand typos
![Page 53: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/53.jpg)
UPGRADING PROBLEMS SOLUTIONS
ONCALL
Fueled by Zabbix notifications
A surprise of "Latest 20 issues" in the dashboard
Can't reorder elements in 1.8● ...do collapsed elements still load data?
![Page 54: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/54.jpg)
UPGRADING PROBLEMS SOLUTIONS
Supports Other Decisions
Something's down
Software acting weird
Capacity planning
![Page 55: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/55.jpg)
UPGRADING PROBLEMS SOLUTIONS
Don't Use /
Data transfer on / can indicate problems● Misconfiguraton● Somebody outputing to /
Added special monitoring on Solaris DB servers
![Page 56: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/56.jpg)
UPGRADING PROBLEMS SOLUTIONS
SSSSolaris
Discover partitions along their mountpoints● 'join' can hang, 'awk' versions...#!/bin/bashwhile read partitionlongid partitionid mountpoint fstype; do partitionlist="$partitionlist,"'{"{#PARTITIONID}":"'$partitionid'","{#PARTITIONLONGID}":"'$partitionlongid'","{#FSTYPE}":"'$fstype'","{#MOUNTPOINT}":"'$mountpoint'"}'done < <(/usr/xpg4/bin/awk '(NR == FNR){i[$1] = $2; n[$1] = $3; next}{print $1, $2, i[$1], n[$1]}' <(mount -p | /usr/xpg4/bin/awk '{sub(".*/","",$1); sub("s0$","",$1); print $1, $3, $4}') <(paste -d" " <(iostat -xn | awk '{print $NF}') <(iostat -x | awk '{print $1}') | tail +3))
echo '{"data":['${partitionlist#,}']}'
![Page 57: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/57.jpg)
UPGRADING PROBLEMS SOLUTIONS
Have Guardrails
Write basic guidelines
http://zabbix.org/wiki/Docs/template_guidelines
It's easy to forget basics like usermacros
![Page 58: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/58.jpg)
UPGRADING PROBLEMS SOLUTIONS
Paint Them Your Style
All triggers must have comments
Don't make triggers fire upon "it got worse"
Avoid cronjobs to feed data● http://zabbix.org/wiki/Escaping_timeouts_with_atd
![Page 59: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/59.jpg)
UPGRADING PROBLEMS SOLUTIONS
Trigger This
Most triggers are very simple
Some are so complex nobody understands them
This presentation resulted in a few triggers getting fixed
![Page 60: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/60.jpg)
UPGRADING PROBLEMS SOLUTIONS
Be Reasonable
Don't monitor what you don't need● ...or monitor it infrequently● ...and have triggers on it
It can reveal problems in an unexpected way
![Page 61: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/61.jpg)
UPGRADING PROBLEMS SOLUTIONS
Shots in the Dark
CPU load alone can expose a lot
I/O load / iowait
It can also be used in unintented ways
![Page 62: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/62.jpg)
UPGRADING PROBLEMS SOLUTIONS
For Your Own Safety
A file must be owned by root, permissions 100
World-readable on some systems
Userparameter for owner/permissions
![Page 63: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/63.jpg)
UPGRADING PROBLEMS SOLUTIONS
SSL/TLS Certs
You want to renew them on time
There are a lot of certs
Monitor them all
#!/bin/bashdate -d "$(echo | openssl s_client -connect "$1":"$2" 2>/dev/null | \openssl x509 -noout -enddate | sed 's/^notAfter=//')" "+%s"
![Page 64: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/64.jpg)
UPGRADING PROBLEMS SOLUTIONS
How We Alert on Expiry
An alert goes out● 60 days in advance● 30 days in advance● 15 days in... advance
![Page 65: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/65.jpg)
UPGRADING PROBLEMS SOLUTIONS
Hooking Into the DB
Maintaining maintenance
Acknowledging things
Acknowledging all on host missing in the frontend
![Page 66: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/66.jpg)
UPGRADING PROBLEMS SOLUTIONS
Try Not to Assume
Never assume others see what you see
Zabbix links are no good
Copy Zabbix graphs
![Page 67: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/67.jpg)
UPGRADING PROBLEMS SOLUTIONS
Simple Minded
MySQL monitoring that's simple, robust, lightweight...
Dump show variables; and vfs.file.regexp
UserParameter=mysql.table.discovery[*],for table in $(mysql $1 -Ne "show tables;"); do tablelist="$tablelist,"'{"{#TABLE}":"'$table'"}'; done; echo '{"data":['${tablelist#,}']}'
+ default MySQL table size userparameter
![Page 68: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/68.jpg)
UPGRADING PROBLEMS SOLUTIONS
No Sharing
Do not share monitoring DB with production● ...or testing, or QA, or whatever
We woke up IT, network guys, before we figured out that one system was hammering the shared DB
![Page 69: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/69.jpg)
UPGRADING PROBLEMS SOLUTIONS
If You Share Your Monitoring DB...
...monitoring will get broken when you already have other trouble
Which is exactly when you want a reliable monitoring
![Page 70: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/70.jpg)
UPGRADING PROBLEMS SOLUTIONS
Glad You Solved It. Now Maintain It
Don't modify the product too much
Maintenance costs are higher that implementation
![Page 71: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/71.jpg)
UPGRADING PROBLEMS SOLUTIONS
KISS
Simple is good
Documented is better
"Perfect is the enemy of good"
![Page 72: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/72.jpg)
UPGRADING PROBLEMS SOLUTIONS
Nirvana Fallacy
When you never even begin an important task because you feel reaching perfection is too hard
![Page 73: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/73.jpg)
UPGRADING PROBLEMS SOLUTIONS
A Pretty Fly
"If you never miss a plane, you're spending too much time at the airport"
Economist George Stigler
![Page 74: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/74.jpg)
UPGRADING PROBLEMS SOLUTIONS
Explain and Document
False "job security" – you never get promoted
Two Zabbix script repos● Some people work on one, some on another
![Page 75: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/75.jpg)
UPGRADING PROBLEMS SOLUTIONS
Break Stuff
How can a webpage testing script fail?● Connection fails● Unexpected content● Never times out
![Page 76: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/76.jpg)
UPGRADING PROBLEMS SOLUTIONS
It Takes Time
Solving a problem is often easy
Solving it properly is hard● Handle all the edge cases, write comments● Accept named parameters● Have some debug output● ...
![Page 77: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/77.jpg)
UPGRADING PROBLEMS SOLUTIONS
if you pay attention, nothing is simple
to explain anything, reduce it to simple
![Page 78: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/78.jpg)
UPGRADING PROBLEMS SOLUTIONS
Wild West
Wild West people rush in, save the day, get the fame
Poor peasants try to maintain that
Don't allow that to happen
![Page 79: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/79.jpg)
UPGRADING PROBLEMS SOLUTIONS
Maintenance Is the King
Low maintenance cost makes Java popular
Neat but complicated will be thrown out in favour of simple and robust (easy to maintain)● Documentation can help a bit
![Page 80: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/80.jpg)
USABILITY COMMUNITY CONCLUSION
Usability
Surprise topic
Based on comments people make about Zabbix
![Page 81: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/81.jpg)
USABILITY COMMUNITY CONCLUSION
It's Very Decent
<DXManiac> I tell people here at the office almost daily "Yes, the UI is cluttered and things are complicated, but I wouldn't know how to cram so many features in such a small place in any better way" :)
![Page 82: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/82.jpg)
USABILITY COMMUNITY CONCLUSION
You Better You Bet
![Page 83: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/83.jpg)
USABILITY COMMUNITY CONCLUSION
Zabbix Has These Issues, But...
<fracklen> But on the other hand - zabbix does have the feel of being well mature
<fracklen> and a great community
![Page 84: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/84.jpg)
USABILITY COMMUNITY CONCLUSION
No Surprises
Zabbix positions itself as an enterprise solution
Enterprises want predictability
Knowing about changes in detail and in advance can spare painful surprises
![Page 85: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/85.jpg)
USABILITY COMMUNITY CONCLUSION
What's Important?
Built-in data gathering support less important than:● Input data fits the Zabbix data model● Core system is functional, stable, easy to
debug● Healthy, transparent development
![Page 86: Rihards Olups - Zabbix at Nokia - Case Study](https://reader031.vdocuments.us/reader031/viewer/2022021416/586e8c3a1a28aba0038b83ad/html5/thumbnails/86.jpg)
Thank You
Zabbix team for building it
Participants of this great conference
Zabbix community
The conference team