© 2019 Percona. 1

Peter Zaitsev, CEO Percona

Using eBPF for Linux Performance AnalysesIn 25 Minutes…

February 3rd, 2019

FOSDEM, Monitoring and ObservabilityBrussels, Belgium

© 2019 Percona. 2

Question

Who is familiar with eBPF ?

© 2019 Percona. 3

About Myself

Performance Geek turned CEO, who kept

his passion

© 2019 Percona. 4

About the Presentation

eBPF Overview

Practical examples of eBPF based tools

© 2019 Percona. 5

eBPF History and Linux Support

© 2019 Percona. 6

eBPF - Extended Berkeley Packet Filter

Berkeley Packet Filter - Originated in 1992 as efficient virtual machine for Packet Filtering

Extended Berkeley Packet Filter – Extended Version found in Linux

General Event Processing Framework

JIT Compiler for high efficiency

© 2019 Percona. 7

eBPF in Linux

Has been in Linux Kernel since 2014

Actively being improved

Integrated in “perf” tooling

system

© 2019 Percona. 8

Improvements in recent Kernels

© 2019 Percona. 9

eBPF Programs

Linux Kernel can load programs in custom byte code

Programs verified before load to prevent misuse

LLVM Clang can compile to eBPF byte code

This compilation is kernel-dependent

Few will need to write eBPF programs Directly

© 2019 Percona. 10

eBPF Code Example

© 2019 Percona. 11

eBPF User Space vs Kernel

Source: http://www.brendangregg.com/ebpf.html

© 2019 Percona. 12

eBPF in Linux Summary

© 2019 Percona. 13

eBPF features in different kernel versions

© 2019 Percona. 14

Tracing Landscape per Brendan Gregg

© 2019 Percona. 15

Things to note

•perf-tools-unstable

•Iovisor/bcc

•Iovisor/ply

•…

Not all the tools

available in the single package

© 2019 Percona. 16

Iovisor bcc installation

Detailed Instructions: https://github.com/iovisor/bcc/blob/master/INSTALL.md

Note: Tools installed to /usr/share/bcc/tools not added to PATH automatically

© 2019 Percona. 17

Tools in Action

© 2019 Percona. 18

Profile: Better “poor man’s profiler”

© 2019 Percona. 19

Biolatency: Block Device Latency

© 2019 Percona. 20

Biosnoop: Block Device IO Tracing

© 2019 Percona. 21

Ext4dist: Filesystem Latency per Operation

© 2019 Percona. 22

Ext4slower: Trace Slow (or all) IO

© 2019 Percona. 23

Cachestat: File System Cache Performance

© 2019 Percona. 24

Runqlat: CPU RunQueue Latency

© 2019 Percona. 25

Execsnoop: Trace Program Starts

© 2019 Percona. 26

Opensnoop: Trace Opened Files

© 2019 Percona. 27

Tcpconnect: Trace TCP Connections

© 2019 Percona. 28

Tcpretrans: TCP Retransmits Details

© 2019 Percona. 29

Gethostlatency: DNS Lookup Latency

© 2019 Percona. 30

All tools available with BCC:

© 2019 Percona. 31

Ply: Simple Scripting for eBPF

Is not available as packages yet!

git clone https://github.com/iovisor/plycd ply./autogen.sh./configuremake

More Details: https://wkz.github.io/ply/

© 2019 Percona. 32

Ply Example

© 2019 Percona. 33

Ply Language#!/usr/bin/env ply

kprobe:SyS_execve {

@exec[tid()] = mem(arg(0), "128s");

i = 0;

unroll(16) {

argi = mem(arg(1) + i * sizeof("p"), "p");

if (!argi)

return;

@argv[tid(), i] = mem(argi, "128s");

i = i + 1;

}

}

© 2019 Percona. 34

Lets Get Some History !

© 2019 Percona. 35

eBPF and Trending

Command Line Frontends are great for interactive troubleshooting of current problems

Not very helpful if you want to analyze the past

How do we get the data into Monitoring System ?

© 2019 Percona. 36

Meet Cloudflare’s eBPF Exporter

Get results of your eBPF Probes to Prometheus

Can plot Prometheus data using Grafana and other Tools

Use for Alerting with Prometheus Alert Manager

More Info: https://blog.cloudflare.com/introducing-ebpf_exporter/

© 2019 Percona. 37

In Search of the Response Time Histograms

Autodesk Research: https://www.autodeskresearch.com/publications/samestats

© 2019 Percona. 38

eBPF Exporter Architecture

Uses BCC Frontend (Library)

Can be used to take BCC Programs output and expose them in Prometheus Format

Mind Performance Overhead of eBPF Probes

© 2019 Percona. 39

eBPF Overhead

eBPF Programs can be run million+ times per second per core

More Details: https://github.com/cloudflare/ebpf_exporter/tree/master/benchmark

© 2019 Percona. 40

Installing eBPF Exporter

• Install Exporter (Binaries Available)

• https://github.com/cloudflare/ebpf_exporter/releases

• Provide Instrumentation Configuration

• https://github.com/cloudflare/ebpf_exporter/tree/master/examples

© 2019 Percona. 41

Percona Monitoring and Management (PMM)

100% Free and Open Source

Purpose Built for Open Source Databases

Based on Grafana and Prometheus

Easy to Install and Use

Where should we visualize it ?

© 2019 Percona. 42

Adding eBPF Exporter to PMM

pmm-admin add external:service ebpf --service-port=9435

© 2019 Percona. 43

Device IO Response Histogram

© 2019 Percona. 44

PMM System Overview Dashboard

© 2019 Percona. 45

Adding Latency Histograms

© 2019 Percona. 46

Can see outliers

© 2019 Percona. 47

eBPF Future

© 2019 Percona. 48

Bpftrace – Dtrace Replacement

See More: http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html

© 2019 Percona. 49

Further Reading List

https://github.com/zoidbergwill/awesome-ebpf

https://slideplayer.com/slide/12710510/

http://www.brendangregg.com/ebpf.html

http://vger.kernel.org/netconf2018_files/BrendanGregg_netconf2018.pdf

http://www.brendangregg.com/Slides/Velocity2017_BPF_superpowers.pdf

https://lwn.net/Articles/740157/

© 2019 Percona. 50

Thank You!