dtrace overview

COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.

Jérôme GauthierMay 2011

DTRACEOverview

AGENDA

1. Context

2. DTrace Framework

3. System inspection and analysis


2

4. DTrace Toolkit

5. Demo

Context

DTrace Framework

System inspection and analysis


3


DTrace Toolkit

Demo

CONTEXT

We need tools to debug and observe different situations …

Analyze the systemperformance

Understand the system utilization or

saturation


4

Debug anapplication

saturation

Debug a system/kernel crash

CONTEXT

Processcontrol

… in several areas

Processstatistics

Processdebugging

pgreppkill

pstoppargs pflags mdbtruss


5

Kerneldebugging

Kernelstatistics

Systemstatistics

iostatvmstat

cpustat kstat kmdblockstat

Examples of some Solaris commands

CONTEXT

• Some existing utilities are process based

• Some are only inspecting certain parts of the system

• Disks

• Virtual memory

• Network

• Kernel


6

• Kernel

Need a tool to simplify the analysis of all parts of a system

This tool must not make the system any slower

DTrace Framework

Context



7


DTrace Toolkit

Demo

DTRACE FRAMEWORK

• DTrace : A facility for dynamic instrumentation of production systems

� Another way to debug and observe the entire system and understand the big picture

• Open source (CDDL = Common Development and Distribution License)


8

• Supported platforms : Unix-like systems :

License)

DTRACE FRAMEWORK

Operating System Programs

Kernel level

User level

DTRACE


9

• Examine how the system works

• Track down performance problems across many layers of software

• Locate the cause of aberrant behavior

Does not replace or retire other system utilities

• A framework for real-time analysis and observation

• Designed for live production systems

� A totally safe way to inspect live data on production systems

� No need to stop or restart applications

DTRACE FRAMEWORK


10

� Live measurement and interpretation

• Over 30 000 data monitoring points spread in all system

• Zero effect when disabled, safe and no system slowdown when enabled

• Everything about DTrace is virtualized per consumer

� There is no limit on the number of concurrent DTrace consumers

• Possibility to create custom programs to dynamically instrument the system

DTRACE FRAMEWORK


11

• Example of problems that can be detected and analyzed:

� High CPU utilization

� Acute memory pressure

� Abnormal I/O activity

� Excessive network traffic


Context

DTrace Framework


12


DTrace Toolkit

Demo

• Probes are programmable sensors placed all over the system

• A probe fires when the event happens

� Then, certain actions are executed only if the predicate expression is true

SYSTEM INSPECTION AND ANALYSISPROBES


13

• General form of a probe clause :probe description

/predicate/

{

actions

}


probe description

/predicate/

{

actions

}• A predicate allow action to only be taken when certain conditions are met

� Example : Look only for a process which has the pid=1203

Match a process which has the name firefox-bin


14

• An action can be “ record a stack trace, a timestamp, or the argument to a function ”

• When some probes fire, DTrace gathers the data and reports it back to you

� If no action specified, DTrace will just take note of each time the probe fires


Type of actions Example Explanation

Data recordingtrace() Records the result of trace to the buffer

printf() Traces a D expression

• Actions are used to record data to a DTrace buffer

• Different types of actions:


15

stack() Records a kernel stack trace

Destructivestop() Stops the process which has executed the probe

raise() Signal a process at a precise point during execution

panic() Force a crash dump

Specialexit() Stop tracing and exits

copyin() Creates a buffer and returns its address

strlen() Returns the length of a string in bytes

• Each probe is uniquely identified by a 4-tuple :


< provider, module, function, name >

ModuleThe name of the module in which the probe is located� Either the name of a kernel module or the name of a user library

probe description

/predicate/

{

actions

}

Probe description


16

FunctionIf this probe correspond to a specific program location, it’s the name of the program function in which the probe is located.

NameThe final component of the tuple is the string name of the probe that give you some idea of the probe’s semantic meaning.

• Some probes does not have a module and function (e.g. : BEGIN, END)

� We can identify a probe only by its name

• DTrace probes come from a set of kernel modules called providers

� Which correspond to a particular kind of instrumentation

SYSTEM INSPECTION AND ANALYSISPROVIDERS

< provider, module, function, name >


17

• We can list the probes that a provider can transmit to the DTrace Framework

• A provider pass the control to DTrace when you decide to enable one of its probes

• Example of some providers :

SYSTEM INSPECTION AND ANALYSISPROVIDERS

SYSCALL

• Holds the entire communication from userland to kernel space• Every system call on the

PROC

• Handle process creation and termination• Sending and handling signals

SCHED

• CPU scheduling: sleeping, running threads• CPU time: which threads are run by which CPU and


18

• Every system call on the system

signals

IO

• I/O system• Disk input and output requests• I/O by device, process, size, filename

MIB

• Counters for management information bases• IP, IPv6, ICMP, IPSec

are run by which CPU and for how long

PROFILE

• Time based probing at specific interval of times• To sample anything in the system every unit time

• A D program consist of one or more probe clauses

• D Language is like C language with some constructs similar with awk :

• Support ANSI C operators

Support Strings

SYSTEM INSPECTION AND ANALYSISD LANGUAGE

/* D Program */

probe1 description

/predicate/

{

actions

}

probe2 description

/predicate/


19

• Support Strings

• D expressions are based on built-in variables : pid, execname, timestamps, curthread

• No control-flow constructs : loops, if statements

• Floating-point arithmetic is not permitted (only Integers)

/predicate/

{

actions

}

probe3 description

/predicate/

{

actions

}

• Used to aggregate data and look for trends

• Simple to generate report about :

Total system calls by a process or an application

SYSTEM INSPECTION AND ANALYSISD LANGUAGE : AGGREGATIONS

@name[keys] = aggfunction(args)


20

• Total system calls by a process or an application

• Total number of read or write by process

• Aggregating functions: • count()

• sum()

• avg()

• min()

• max()

• quantize()

SYSTEM INSPECTION AND ANALYSISD LANGUAGE : SECURITY

• D programs are compiled into a safe intermediate form that is used for execution when a probe fires

D program source files

DTrace consumer: dtrace()

DTrace providers• Programming mistakes : DTrace will

kernel

userland

DTrace safe execution environment


21

� It is impossible to construct a bad script that would cause damage to the system

• DTrace handle run-time errors and report them (dividing by zero, dereferencing invalid memory, …)

• Programming mistakes : DTrace will report errors and disable instrumentation

Context

DTrace Framework



22

DTrace Toolkit

Demo


DTRACE TOOLKIT

• Collection of over 100 useful documented scripts built on top of DTrace Framework

� Developed by the OpenSolaris DTrace community

• The toolkit contains :

� the scripts


23

� the man pages

� the example documentation

� the notes files

� the tutorials

• The script are sorted by categories

DTRACE TOOLKITCATEGORIES

DTrace ToolkitApplications

Extra, User, System


24

DTrace Framework

Applications

CPU

Disk

Kernel

Network

Memory

Processes

System

Extra, User, System

Context

DTrace Framework



25

Demo


DTrace Toolkit

DEMO

Command line scripts

tcpstat

Chime scripts

system calls

� System calls counter by process,


26

� TCP bytes received and send

httpdstat

� Real-time Apache Web Server stats

� System calls counter by process, function, module, …

php calltime

� Measure PHP elapsed times for functions

Questions?


27

[email protected]

REFERENCES & ADDITIONAL INFORMATION

• PowerPoint presentations

• Joyent Performance Visualization – Brendan Gregg

• DTrace & DTrace Toolkit – Stefan Parvu

• Links

• DTrace community site:


28

• DTrace community site:http://hub.opensolaris.org/bin/view/Community+Group+dtrace/WebHome

• Brendan Gregg site: http://www.brendangregg.com/dtrace.html

• DTrace Blogs: http://dtrace.org/blogs/

• DTrace online guide: http://wikis.sun.com/display/DTrace/Documentation

• DTrace wiki: http://www.solarisinternals.com/wiki/index.php/DTraceToolkit

• DTrace training: http://dtracehol.com


29

dtrace overview

Technology

dtrace framework dtrace

system utilities

probe fires

system slowdown

entire system

probe clause

probe descriptionthe

probe thatname