Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
Programming
Summary
Write your own monitoringsoftware with RRDtool
Tobias Oetiker
ISG.EE - ETH Zürich
OSCON 2005Portland, Oregon, August 3, 2005
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
Programming
Summary
Netflow Data presented withFluxscope
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
MotivationThe world today
About data collection
A graph says more than1’000 words
About RRDtool
Programming
Summary
Content
MotivationThe world todayAbout data collectionA graph says more than1’000 words
About RRDtool
Programming
Summary
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
MotivationThe world today
About data collection
A graph says more than1’000 words
About RRDtool
Programming
Summary
Numbers everywhere
I 15 users onlineI 1.34 PB free tape spaceI 31’223 ifInOctetsI 100 degrees in the server
roomI 87 Mb/s transfer rate
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
MotivationThe world today
About data collection
A graph says more than1’000 words
About RRDtool
Programming
Summary
All these questions
I Are we running out ofdiskspace?
I Is there a pattern in the highserver load?
I Is there really globalwarming?
I Can we do something aboutit?
I Collect that data!
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
MotivationThe world today
About data collection
A graph says more than1’000 words
About RRDtool
Programming
Summary
How to collect data
I collect everything, analyzelater (aka never)
I data with known propertiesmakes life simpler
I from research to production
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
MotivationThe world today
About data collection
A graph says more than1’000 words
About RRDtool
Programming
Summary
Requirements for a solution
I handle counters and gaugesI be nice to the serverI setup for specific taskI data-pre processingI and forget about it . . .
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
MotivationThe world today
About data collection
A graph says more than1’000 words
About RRDtool
Programming
Summary
Condor use at UW
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
MotivationThe world today
About data collection
A graph says more than1’000 words
About RRDtool
Programming
Summary
Making data accessible
I graphicalrepresentation
I eye candy for thepeople
I analysis at a glanceI make my boss look
good to his boss
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
MotivationThe world today
About data collection
A graph says more than1’000 words
About RRDtool
Programming
Summary
Data post-processing
I highlight interestingproperties
I combine data fromseveral sources
I add extra Information
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtoolHistory
Features
Graphing features
Programming
Summary
Content
Motivation
About RRDtoolHistoryFeaturesGraphing features
Programming
Summary
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtoolHistory
Features
Graphing features
Programming
Summary
1995: MRTG was only a start
I logfiles in text formatI gnuplot for graphsI 1996 - MRTG is used
for amazing tasksI more performanceI more flexibility
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtoolHistory
Features
Graphing features
Programming
Summary
The rrd TOOL
I a building blockI basis for a better
MRTG (2nd system)I basis for hundreds of
other toolsI Google says: “industry
standard”
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtoolHistory
Features
Graphing features
Programming
Summary
The Round Robin Database
I lossy storage, fixed file sizeI current data is the most
interestingI consolidation functions for
feature extractionI artificial data-sourcesI holt-winters aberrant behavior
detectionI constant step sizeI fixed disk space
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtoolHistory
Features
Graphing features
Programming
Summary
Feeding Data
I combat jitter by takingacquisition time intoaccount.
I preserve data-volumeI on-the-fly data
validationI deal with unknown
dataI time is a one way
street
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtoolHistory
Features
Graphing features
Programming
Summary
Basic Graphing
I auto scalingI auto labelingI sensible defaultsI quick resultsI anti-aliased outputI multiple formats
(png/eps/pdf/svg)
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtoolHistory
Features
Graphing features
Programming
Summary
Advanced Graphing
I change colors, fonts,sizes
I data from severaldatabases
I data processing withRPN math
I alpha transparency(rrggbbaa)
I support of localebased characterencoding
I truetype fonts
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Content
Motivation
About RRDtool
ProgrammingLanguage bindingsBasic operationsAdvanced Graphing
Summary
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
On the command line
Just another Unix command.coderrdtool cmd arg
examplerrdtool info demo.rrd
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Via STDIN
Save startup time by feeding several commands.codeecho cmd arg | rrdtool -
exampleecho info demo.rrd | rrdtool -
I several commandsI fast operationI simple interface
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
perl module
I love coding in perl . . .codeuse RRDs;my $return = RRDs::cmd arg;
exampleuse RRDs;use Data::Dumpermy $ret = RRDs::info "demo.rrd";print Dumper $ret;
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Other Bindings
I RRDcgiI Perl Pipes - RRDpI PythonI TclI and more externally
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Creating Round Robin Databases
I what data do I want to look at?I how long do I want to keep it?I what are the interesting properties of my data?I what to keep in one rrd file?
Commandrrdtool create filename [–start start time] [–step step][DS:ds-name:DST:heartbeat:min:max][RRA:CF:xff:steps:rows]
DST: COUNTER, GAUGE, DERIVE, ABSOLUTE,New in 1.2: COMPUTECF: AVERAGE, MIN, MAX, LAST,New in 1.2: Aberrant Behavior Detection.
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Create Example
examplerrdtool create example.rrd --step=60\
DS:in:COUNTER:600:0:1000 \RRA:AVERAGE:0.5:1:100 \RRA:AVERAGE:0.5:10:100 \RRA:MAX:0.5:10:100
I resolution 60 secondsI COUNTER data sourceI updates at least every 600 secondsI accept rates between 0 and 1000I three archive with 100 entriesI store unknown unless half are known.
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Adding Data into a Round RobinDatabase
Commandrrdtool update filename [–template ds:ds:. . . ]time:value:value. . . [. . . ]
I data value and acquisition timeI timestamps have to increase with every updateI instead of time use N for nowI templates are for redundancy only
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Update Example
examplerrdtool update example.rrd \
--template in N:39344
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Data re-bin-ing
300s 300s
1
3
2280koctets
400s
Data Aquisition
700 octets/sec
300s 300s
Data Storage
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Data Consolidation
AVERAGE RRA with 3 PDP
AVERAGE RRA with 2 PDP
RRA with 1 PDP
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
The first graph
Commandrrdtool graph graph.png DEF:var=rrdfile:DS-name:CFLINE1:var#rrggbb:label
1. prepare data (DEF)2. draw graph (LINE1)
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Data processing with RPN(remember the HP days)What if you have collected octetsbut want to present bits?Calculated time series and variablesCDEF:var=RPN expression
Examples
a,8,* a ∗ 8 (bit to byte conversion)a,b,+ a + ba,b,c,IF if (a!=0) then b else ca,1800,TREND half-hour sliding window averagea,b,c,LIMIT if (a>b AND a<c) then a else ’UNKN’
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Data processing with RPN
Calculated single value variableVDEF:var=RPN expression
var is associated with a time and a data value.Examples
a,TOTAL rate multiplied with intervala,AVERAGE average value of aa,95,PERCENT find 95-percentile
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
MRTG like Graph
Examplerrdtool graph graph.png \
DEF:in=data.rrd:input:AVERAGE \DEF:out=data.rrd:output:AVERAGE \AREA:in#ff0000:incoming \LINE2:out#00ff00:outgoing
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Graph with marked maximum
Examplerrdtool graph graph.png \
DEF:a=data.rrd:a:AVERAGE \VDEF:max=a,MAXIMUM \LINE2:a#00ff00:outgoing \LINE1:max#ff0000:maximum\\g \VRULE:max#ff0000 \GPRINT:max:"at %.2lf"
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Graph with marked maximum
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Graph with 95 percentile
Examplerrdtool graph.png \
DEF:in=data.rrd:input \DEF:out=data.rrd:output \CDEF:base=in,out,MAX \VDEF:n95=base,95,PERCENT \AREA:in#0000b0:incoming \AREA:out#00ff0080:outgoingLINE1:base#000000:maximum \LINE1:n95#ff0000:"95%-tile "\GRPINT:n95:"at %.2lf %s"
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Graph with 95 percentile
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
ProgrammingLanguage bindings
Basic operations
Advanced Graphing
Summary
Mailserver Statistics
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
Programming
Summary
Content
Motivation
About RRDtool
Programming
Summary
Write your ownmonitoring
software withRRDtool
Tobias Oetiker
Motivation
About RRDtool
Programming
Summary
Summary
I carefully design yourRRD structure
I use RPN math to beatyour data into form
I write your ownmonitors, RRDtooldoes the boring stuff
I look good
More information on . . .http://www.rrdtool.org