ubiquitous system analysis performance co pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · ubiquitous...
TRANSCRIPT
![Page 1: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/1.jpg)
Ubiquitous System Analysis
Performance Co Pilot
Abegail JakopLukas BerkRed HatOct. 23, 2014
![Page 2: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/2.jpg)
FSOSS 2014 | Oct 23, 20142
Introduction
● PCP Overview● Introduction ● Components
● Recent Developments● PAPI pmda● pmwebd● Deeper metrics
● Questions?
![Page 3: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/3.jpg)
FSOSS 2014 | Oct 23, 20143
Analyzing Performance
How is this typically/historically done?● rsyslog/syslog-ng/journald● top/iostat/vmstat/ps● Mixture of scripting languages (bash/perl/python)● Specific tools vary per platform● Proper analysis requires more context
![Page 4: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/4.jpg)
FSOSS 2014 | Oct 23, 20144
Introducing
PERFORMANCECO-PILOT
![Page 5: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/5.jpg)
FSOSS 2014 | Oct 23, 20145
Performance Co-Pilot
Points of interest● Unix-like component design● Complements existing system functionality● Cross platform● Consistent unit measurement● Extremely extensible● Open Source!
![Page 6: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/6.jpg)
FSOSS 2014 | Oct 23, 20146
Performance Co-Pilot
Two Underlying Components
1) Performance Metric Domain Agents
2) Performance Metric Collection Daemon
![Page 7: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/7.jpg)
FSOSS 2014 | Oct 23, 20147
Performance Co-Pilot
Two Underlying Components
1) Agents
2) Performance Metric Collection Daemon
![Page 8: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/8.jpg)
FSOSS 2014 | Oct 23, 20148
Performance Co-Pilot
Two Underlying Components
1) Agents
2) PMCD
![Page 9: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/9.jpg)
FSOSS 2014 | Oct 23, 20149
Performance Co-Pilot
Kernel
Network
Webserver
ApplicationSpecific
PMCD
Agents
![Page 10: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/10.jpg)
FSOSS 2014 | Oct 23, 201410
Performance Co-Pilot
Number of metrics exposed by agents?● A lot! (~1500 from a default fedora install)● Huge variation in what they're measuring● How do you reliably and predictably name them?
![Page 11: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/11.jpg)
FSOSS 2014 | Oct 23, 201411
Performance Co-Pilot
root
hw
...
network routerkernel
... ... ... ... ...
all percpu udp tcp recv
total_utilrcvpacksyscall
● Performance Metric Name Space
network .tcp .rcvpack
![Page 12: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/12.jpg)
FSOSS 2014 | Oct 23, 201412
Performance Co-Pilot
Kernel
Network
Webserver
ApplicationSpecific
PMCD
pmval
pmstat
pmfind
pminfo
Host A Host B
![Page 13: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/13.jpg)
FSOSS 2014 | Oct 23, 201413
Performance Co-Pilot
Where to start?
pminfo – display information about metrics$ pminfo -t
![Page 14: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/14.jpg)
FSOSS 2014 | Oct 23, 201414
Performance Co-Pilot
Where to start?
pminfo – display information about metrics$ pminfo -t$ pminfo -t papi
![Page 15: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/15.jpg)
FSOSS 2014 | Oct 23, 201415
Performance Co-Pilot
Where to start?
pminfo – display information about metrics$ pminfo -t
papi.system.REF_CYC [Reference cycles]
papi.system.L3_TCA [L3 cache accesses]
papi.system.L2_TCA [L2 cache accesses]
papi.system.L3_TCH [L3 cache hits]
papi.system.L2_TCH [L2 cache hits]
$ pminfo -t papi
![Page 16: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/16.jpg)
FSOSS 2014 | Oct 23, 201416
Performance Co-Pilot
Where to start?
pmval – current value of a metric$ sudo pmval papi.system.TOT_CYC
metric: papi.system.TOT_CYC
host: toium
semantics: cumulative counter (converting to rate)
units: none (converting to / sec)
samples: all
7.869E+04
9.186E+04
9.240E+04
![Page 17: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/17.jpg)
FSOSS 2014 | Oct 23, 201417
Performance Co-Pilot
Kernel
Network
Webserver
ApplicationSpecific
PMCD
Host A Host B
pmval
pmstat
pmfind
pminfo
![Page 18: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/18.jpg)
FSOSS 2014 | Oct 23, 201418
Performance Co-Pilot
Kernel
Network
Webserver
ApplicationSpecific
PMCD
pminfo
pmlogger
Host A Host B
![Page 19: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/19.jpg)
FSOSS 2014 | Oct 23, 201419
Performance Co-Pilot
Kernel
Network
Webserver
ApplicationSpecific
PMCD
pminfo
pmlogger
Host A Host B
Archives
![Page 20: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/20.jpg)
FSOSS 2014 | Oct 23, 201420
Performance Co-Pilot
pmlogger creates logs for future analysis● Enables us to use tools on older data, retrospectively● Default around 5mb a day, rotates and compresses● Metrics organized, no need to stick them into elastic
search
![Page 21: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/21.jpg)
FSOSS 2014 | Oct 23, 201421
Performance Co-Pilot
● Recent and future developments● PAPI pmda● pmwebd● Enabling deeper system introspection
![Page 22: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/22.jpg)
FSOSS 2014 | Oct 23, 201422
“Only two hard parts of computer science, cache invalidation, naming things, and off-by-one errors”
- Unknown
Performance Co-Pilot – Recent Developments
![Page 23: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/23.jpg)
FSOSS 2014 | Oct 23, 201423
“Only two hard parts of computer science, cache invalidation, naming things, and off-by-one errors”
- Unknown
Performance Co-Pilot – Recent Developments
![Page 24: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/24.jpg)
FSOSS 2014 | Oct 23, 201424
Performance Co-Pilot – Recent Developments
PAPI – Performance API● Cross platform● Uses dedicated hardware counters for perf metrics
● Cache hits/misses, total instructions/cycles● By writing a pmda (agent) for PAPI, we can expose these metrics
Webserver
ApplicationSpecific
PMCD
![Page 25: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/25.jpg)
FSOSS 2014 | Oct 23, 201425
Performance Co-Pilot – Recent Developments
PAPI – Performance API● Cross platform● Uses dedicated hardware counters for perf metrics
● Cache hits/misses, total instructions/cycles● By writing a pmda (agent) for PAPI, we can expose these metrics
Webserver
ApplicationSpecific
PMCD
PAPI
![Page 26: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/26.jpg)
FSOSS 2014 | Oct 23, 201426
Performance Co-Pilot – Recent Developments
We could just view the raw values$ sudo pmval papi.system.TOT_CYC
metric: papi.system.TOT_CYC
host: toium
semantics: cumulative counter (converting to rate)
units: none (converting to / sec)
samples: all
7.869E+04
9.186E+04
9.240E+04
![Page 27: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/27.jpg)
FSOSS 2014 | Oct 23, 201427
We could just view the raw values● Ratios and relative percentages are more insightful● Perfect for the pmie tool!
Performance Co-Pilot – Recent Developments
PMCD
pmie
pmtools
![Page 28: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/28.jpg)
FSOSS 2014 | Oct 23, 201428
Performance Co-Pilot – Recent Developments
Performance Metrics Inference Engine● Allow you to form metrics-based expressions for
evaluation● Ratios, counts, aggregates, conditionals● Raise alarms, logging entries, shell commands● Run on live data or logs● Run rules across data from multiple hosts
![Page 29: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/29.jpg)
FSOSS 2014 | Oct 23, 201429
Example pmie expression:
Performance Co-Pilot – Recent Developments
(papi.system.L3_TCM / papi.system.TOT_INS)
![Page 30: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/30.jpg)
FSOSS 2014 | Oct 23, 201430
Example pmie expression:
Performance Co-Pilot – Recent Developments
((papi.system.L3_TCM / papi.system.TOT_INS) * 100) (papi.system.L3_TCM / papi.system.TOT_INS)
![Page 31: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/31.jpg)
FSOSS 2014 | Oct 23, 201431
Example pmie expression:
Performance Co-Pilot – Recent Developments
((papi.system.L3_TCM / papi.system.TOT_INS) * 100) (papi.system.L3_TCM / papi.system.TOT_INS) > 2
![Page 32: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/32.jpg)
FSOSS 2014 | Oct 23, 201432
Example pmie expression:
Performance Co-Pilot – Recent Developments
some_inst
((papi.system.L3_TCM / papi.system.TOT_INS) * 100) (papi.system.L3_TCM / papi.system.TOT_INS) > 2
![Page 33: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/33.jpg)
FSOSS 2014 | Oct 23, 201433
Example pmie expression:
Performance Co-Pilot – Recent Developments
some_inst
((papi.system.L3_TCM / papi.system.TOT_INS) * 100)
-> syslog “Percentage of Level 3 Cache misses > 2%”
(papi.system.L3_TCM / papi.system.TOT_INS) > 2
![Page 34: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/34.jpg)
FSOSS 2014 | Oct 23, 201434
Performance Co-Pilot – Recent Developments
pmwebd● We already ship a gui tool (pmchart)
● Several feature full graphing tools available● PCP's architecture and design makes integration easy
PMCD
pmtools
![Page 35: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/35.jpg)
FSOSS 2014 | Oct 23, 201435
Performance Co-Pilot – Recent Developments
pmwebd● We already ship a gui tool (pmchart)
● Several feature full graphing tools available● PCP's architecture and design makes integration easy
PMCD
pmwebd
pmtools
![Page 36: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/36.jpg)
FSOSS 2014 | Oct 23, 201436
Performance Co-Pilot – Recent Developments
pmwebd● We already ship a gui tool (pmchart)
● Several feature full graphing tools available● PCP's architecture and design makes integration easy
PMCD
pmwebd
pmtools
Grafana/Graphite
![Page 37: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/37.jpg)
FSOSS 2014 | Oct 23, 201437
Performance Co-Pilot – Recent Developments
![Page 38: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/38.jpg)
FSOSS 2014 | Oct 23, 201438
Performance Co-Pilot – Recent Developments
![Page 39: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/39.jpg)
FSOSS 2014 | Oct 23, 201439
Performance Co-Pilot – Current Developments
PCP offers wide variety of metrics● What if we want 'under the hood' metrics?● Need a system-wide, tool with live data to help...
![Page 40: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/40.jpg)
FSOSS 2014 | Oct 23, 201440
Introducing:
![Page 41: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/41.jpg)
FSOSS 2014 | Oct 23, 201441
What is SystemTap
Tool for examining live system events
● Communicated through scripts● Links the strengths of tracers, profilers, and
debuggers.
// SystemTap scriptprobe tcp.sendmsg { gather_info; print(info) }
Linux Kernel Module
Summary Reportsent packet of size ... to ...sent packet of size ... to ...... the all-seeing
Linux Kernel
![Page 42: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/42.jpg)
FSOSS 2014 | Oct 23, 201442
Usage
Two major components of scripts:
1) Probe Points
2) Handlers
![Page 43: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/43.jpg)
FSOSS 2014 | Oct 23, 201443
Example Probe
Simple Hello world
probe begin { println ("Hello, World!")}
Or tracking when a new bash process is started
probe process("bash").function("main") { println("A bash process has started")}
![Page 44: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/44.jpg)
FSOSS 2014 | Oct 23, 201444
Example Probe
Or something a little more complicated:● Listing functions in the order that a process calls them
$ cat bash_functioncalls.stpprobe process("bash").function("*").call { printf ("bash called function %s\n", ppfunc())}
$ stap bash_functioncall.stpbash called function _startbash called function __libc_csu_initbash called function _initbash called function frame_dummybash called function register_tm_clonesbash called function mainbash called function xtrace_init...
![Page 45: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/45.jpg)
FSOSS 2014 | Oct 23, 201445
Getting Started
● Where do you start? Figure out what you can probe.
● If you don't know what probe points types there are:
$ stap --dump-probe-types
java(number).class(string).method(string)kernel.function(number)module(string).statement(string)process(string).function(string).calleesprocfs(string).readtimer.usec(number)...
![Page 46: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/46.jpg)
FSOSS 2014 | Oct 23, 201446
Language
What can you include in handlers?● Ordinary features you'd find in a language:● Globals, locals, string, integers, loops, conditionals,
functions, arrays, error handling and more
Additional, handy features:● Associative arrays, foreach loop, aggregates, macros,
regex matching
![Page 47: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/47.jpg)
FSOSS 2014 | Oct 23, 201447
Probe Points
How does one start writing a script?● Listing mode is a great starting point● Lists possible probe points
$ stap -l 'process("stap").function("symbol_*")'
process("stap").function("[email protected]:1092")process("stap").function("[email protected]:424")
![Page 48: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/48.jpg)
FSOSS 2014 | Oct 23, 201448
Context Variables
Probes can access context variables
The context variables start with “$”
$ stap -L 'kprocess.create'
kprocess.create task:long new_pid:long new_tid:long $return:struct task_struct* $clone_flags:long unsigned int ...
![Page 49: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/49.jpg)
FSOSS 2014 | Oct 23, 201449
Tapsets
A library for systemtap scripts● Their purpose is provide a level of abstraction● Users don't have to know the exact details
For example:
For a list of all the aliased probes
kprocess.create = kernel.function("copy_process").return
stap --dump-probe-alias
![Page 50: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/50.jpg)
FSOSS 2014 | Oct 23, 201450
Tapsets
There are also helper functions
For a list of available helper functions
$ cat kprocess_list.stpprobe kprocess.create { printf ("Process %s was started\n", pid2execname(new_pid))}
$ stap kprocess_list.stpProcess bash was startedProcess bash was startedProcess soffice.bin was startedProcess soffice.bin was startProcess udisksd was startedProcess firefox was started
stap –-dump-functions
![Page 51: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/51.jpg)
FSOSS 2014 | Oct 23, 201451
Tapsets
And helper variables
$ cat helper_vars.stpprobe syscall.* { printf ("syscall: %s, parameters: %s\n", name, $$parms$$)}
$ stap helper_vars.stpsyscall: read, parameters: fd=4 buf=140736613309360 count=8196syscall: fcntl, parameters: fd=4 cmd=4 arg=32770syscall: kill, parameters: pid=4200 sig=10syscall: fcntl, parameters: fd=4 cmd=4 arg=34818syscall: read, parameters: fd=4 buf=140736613309360 count=8196syscall: fcntl, parameters: fd=4 cmd=4 arg=32770syscall: pselect6, parameters: n=5 inp=140736613308976 outp=0 exp=0 tsp=0 sig=140736613308864...
![Page 52: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/52.jpg)
FSOSS 2014 | Oct 23, 201452
Example
![Page 53: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/53.jpg)
FSOSS 2014 | Oct 23, 201453
terminator.c
1:#include <stdlib.h> 2:#include <stdio.h> 3: 4:int sleeper () { 5: static int num = 0; 6: sleep(1); 7: return num; 8:} 9:10:int main () {11: int num = 0;12: while (num < 10) {13: num = sleeper ();14: printf("a second has passed\n");15: }16: printf("10 seconds have passed\n");17: return 0;18:}
![Page 54: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/54.jpg)
FSOSS 2014 | Oct 23, 201454
10th line == 10th second
$ ./terminatora second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passed
terminator.c
![Page 55: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/55.jpg)
FSOSS 2014 | Oct 23, 201455
terminator.c
Figure out what's going on● Where to probe?● Does function X ever call function Y? And with what
parameters?● What can be done if something's not quite right?
![Page 56: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/56.jpg)
FSOSS 2014 | Oct 23, 201456
terminator.c
Where to probe?
Does main() ever call sleeper()? ● Check what functions main() calls.
$ stap -L 'process("terminator").function("*")'
$ stap -L 'process("terminator").function("main").callee("*")' process("terminator").function("[email protected]:10").callee("[email protected]:4") $num:int const
![Page 57: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/57.jpg)
FSOSS 2014 | Oct 23, 201457
terminator.c
That was just a starting point!● What determines when the loop ends?● What about the return value from sleeper?
$ cat sleeper_return_check.stp
global old_num=-1
probe process("./terminator").function("sleeper").return { if ($num <= old_num) error("num is not increasing!") old_num = $num}
![Page 58: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/58.jpg)
FSOSS 2014 | Oct 23, 201458
terminator.c
Running the script
Well, that explains things.● So how can we fix this?
$ stap sleeper_return_check.stp -c ./terminator a second has passeda second has passedERROR: num is not increasing!WARNING: Number of errors: 1, skipped probes: 0WARNING: /home/ajakop/work/codebase/install/bin/staprun exited with status: 1Pass 5: run failed. [man error::pass5]
![Page 59: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/59.jpg)
FSOSS 2014 | Oct 23, 201459
terminator.c
1:#include <stdlib.h>2:#include <stdio.h>3:4:int sleeper () { 5: static int num = 0;6: sleep(1);7: return num;8:}9:10:int main () {11: int num = 0;12: while (num < 10) {13: num = sleeper ();14: printf("a second has passed\n");15: }16: printf("10 seconds have passed\n");17: return 0;18:}
num++; ?
![Page 60: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/60.jpg)
FSOSS 2014 | Oct 23, 201460
terminator.c
Can't do that if the program can't be stopped.● Alternative: write a script to do it!
$ cat fix_terminator.stp global actual_num=-1
probe process("./terminator").function("sleeper").return { if ($num <= actual_num) { actual_num++ $return = actual_num } else actual_num = $num}
![Page 61: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/61.jpg)
FSOSS 2014 | Oct 23, 201461
terminator.c
The result:
Yay! It worked!
$ stap -g fix_terminator.stp -c ./terminatora second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passeda second has passed10 seconds have passed
![Page 62: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/62.jpg)
FSOSS 2014 | Oct 23, 201462
Performance Co-Pilot – Current Developments
Systemtap fits the bill for what we need● Malleable output● Able to specify various probe points● Exposes low level information, safely
PAPI
ApplicationSpecific
PMCD
ApplicationSpecific
![Page 63: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/63.jpg)
FSOSS 2014 | Oct 23, 201463
Performance Co-Pilot – Current Developments
Systemtap fits the bill for what we need● Malleable output● Able to specify various probe points● Exposes low level information, safely
PAPI
ApplicationSpecific
PMCD
ApplicationSpecificsystemtap
![Page 64: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/64.jpg)
FSOSS 2014 | Oct 23, 201464
Performance Co-Pilot – Current Developments
Example● Can we determine network latency on a network device?
PAPI
ApplicationSpecific
PMCD
ApplicationSpecific
systemtap
stap.ko
![Page 65: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/65.jpg)
FSOSS 2014 | Oct 23, 201465
Performance Co-Pilot – Current Developments
# stap ./net_xmit.stp eth0 dev1 dev2
# pminfo -df stap_json
stap_json.json.net_xmit_data.xmit_latency Data Type: 64-bit int InDom: 130.0 0x20800000 Semantics: counter Units: none inst [0 or "dev1"] value 0 inst [1 or "dev2"] value 0 inst [2 or "eth0"] value 319
stap_json.json.net_xmit_data.xmit_count Data Type: 64-bit int InDom: 130.0 0x20800000 Semantics: counter Units: none inst [0 or "dev1"] value 0 inst [1 or "dev2"] value 0 inst [2 or "eth0"] value 2304551
![Page 66: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/66.jpg)
FSOSS 2014 | Oct 23, 201466
Questions?
Performance Co-Pilot
![Page 67: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/67.jpg)
FSOSS 2014 | Oct 23, 201467
Get Involved!
IRC: irc.freenode.net
#pcp
#systemtap
Web:
http://pcp.io
http://sourceware.org/systemtap
Email:
![Page 68: Ubiquitous System Analysis Performance Co Pilot - pcp.iopcp.io/papers/fsoss-2014.pdf · Ubiquitous System Analysis Performance Co Pilot Abegail Jakop Lukas Berk ... Performance Metric](https://reader031.vdocuments.us/reader031/viewer/2022022509/5ad57f837f8b9a075a8cf81e/html5/thumbnails/68.jpg)
FSOSS 2014 | Oct 23, 201468
Get Involved!
IRC: irc.freenode.net
#pcp
#systemtap
Web:
http://pcp.io
http://sourceware.org/systemtap
Email: