SYSTEMTAP FOR LINUX PLATFORMS
Khaled Ahmed
Shareef Bassiouny
Mohamed Sohail
2014 EMC Proven Professional Knowledge Sharing 2
Table of Contents
Introduction ................................................................................................................................ 3
Document Conventions .............................................................................................................. 5
Conceptual Overview ................................................................................................................. 5
SystemTap Scripts ..................................................................................................................... 6
Tapsets ...................................................................................................................................... 6
Commands and Privileges ......................................................................................................... 7
Important Files and Directories .................................................................................................. 8
Installing SystemTap .................................................................................................................. 8
SystemTap first run .................................................................................................................... 9
SystemTap Flight recorder mode ............................................................................................... 9
How SystemTap works .............................................................................................................11
SystemTap Functions ...............................................................................................................17
SystemTap Variables ................................................................................................................21
Target Variables ........................................................................................................................23
SystemTap Language Syntax ...................................................................................................26
SystemTap Associative arrays ..................................................................................................29
User-Space Probing ..................................................................................................................35
Useful SystemTap scripts .........................................................................................................39
Conclusion ................................................................................................................................52
Appendix ...................................................................................................................................52
Disclaimer: The views, processes, or methodologies published in this article are those of the authors. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.
2014 EMC Proven Professional Knowledge Sharing 3
Introduction
Today’s computer systems have become more complicated and sophisticated in many aspects.
The huge number and capabilities of modern software and integration of these softwares can
create unusual errors and bugs that can’t be diagnosed correctly. Help is needed to dig deep
into the issue to know the root cause of the problem. One of the new known technologies in this
field is SystemTap.
SystemTap is a tracing programming language that enables system administrators to trace and
profile a large scale of Linux computing environments including mission-critical production Linux
servers and even embedded devices without affecting the reliability, stability, or performance of
these environments. SystemTap (stap, for short) is a set of programmatic instructions that the
administrator develops to be compiled and executed on the target system similar to any
scripting language like Perl or Python with the difference that these two programming languages
are interpreted but SystemTap is compiled to a binary form known as a “Linux Kernel Module”.
SystemTap provides a command line interface and a scripting language to examine the
activities of a running Linux system, particularly the kernel, in fine detail. SystemTap scripts are
written in the SystemTap scripting language, are then compiled to C-code kernel modules, and
inserted into the kernel. The scripts can be designed to extract, filter, and summarize data, thus
allowing the diagnosis of complex performance or functional problems. SystemTap provides
information similar to the output of tools like netstat, ps, top, and iostat. However, more
filtering and analysis options can be used for the collected information."
SystemTap can extract all sort of information from a live Linux system in order to diagnose a
complex performance issues or functional problems in a very safe way through “probes”. These
probes can be considered as software sensors that the administrator plugs on the Kernel parts
that needs to be diagnosed including Linux system calls, low level kernel functions, and
subsystems functions.
SystemTap was originally developed by RedHat Corporation and was included in 2005 in
RedHat Enterprise Linux 4 update 2 as a technology preview. Starting 2011, SystemTap was
running a wide variety of Linux systems including Fedora, Ubuntu, Debian, SuSE, CentOS.
Later, support of diagnosing userland applications (normal user programs) was added alongside
diagnosing the Java Virtual Machine and CPython VM. SystemTap was originally influenced by
the DTrace framework developed by Sun Microsystem for their Solaris system and later many
2014 EMC Proven Professional Knowledge Sharing 4
BSD UNIX distributions including Apple’s Mac OS X. Now, companies like RedHat, Oracle, IBM,
and Hitachi are contributing to development of this promising technology.
Actually, SystemTap is designed to provide the infrastructure for diagnosing / tracing a running
Linux kernel and provide a detailed report of its status. This can enable system administrators
and developers to write efficient scripts to identify where performance issues and / or bugs are
caused. Prior to SystemTap, the procedure of profiling a running Linux kernel was a long
process consisting of adding very long instruments to the kernel then recompile the whole
kernel to generate a new kernel image with the custom monitoring instruction and reboot the
Linux system to this kernel. SystemTap eliminates all of this, allowing users to gather the same
information by simply running user-written SystemTap scripts.
This Knowledge Sharing article:
Introduces users to SystemTap, familiarizes them with its architecture, and provides
setup instructions.
Presents how SystemTap can be very useful for diagnostics, developing any of the EMC
platforms running Linux, especially when generic tracing and troubleshooting tools seem
inefficient to provide good insight to a certain problem.
Provides examples of what SystemTap can do with sample general-purpose scripts
Offers a complete solution based on SystemTap diagnostic methods applied on EMC
NetWorker® installed on a RedHat Linux system.
Shares tips and tricks with Linux-based product, Avamar®.
2014 EMC Proven Professional Knowledge Sharing 5
Document Conventions
This document uses the following conventions:
Source code listings are in blue:
probe begin{ printf(“Hello World!”); } Code execution output is in red: [khaled@fedora20 systemtap]$ stap helloworld.stp Hello World! Parts of code or code execution that needs attention are highlighted in yellow: [khaled@fedora20 systemtap]$ ls -l forktracker.stp -rwxr-xr-x. 1 khaled users 584 Dec 18 23:50 forktracker.stp Commands are written in italic font: stap -L 'kernel.function("vfs_read")'
Conceptual Overview
Each time you run a SystemTap script, a SystemTap session is started. A number of passes are
done on the script before it is allowed to run, at which point the script is compiled into a kernel
module and loaded. In case the script has already been executed before and no changes
regarding any components have occurred (for example, regarding compiler version, kernel
version, library path, script contents), SystemTap does not compile the script again, but uses
the *.c and *.ko data stored in the SystemTap cache (~/.systemtap). The module is unloaded
when the tap has finished running. For an example, see the test run in Section “Installation and
Setup” and the respective explanation.
2014 EMC Proven Professional Knowledge Sharing 6
SystemTap Scripts
SystemTap usage is based on SystemTap scripts (*.stp). They tell SystemTap which type of
information to collect, and what to do once that information is collected. The scripts are written
in the SystemTap scripting language that is similar to AWK and C.
The essential idea behind a SystemTap script is to name events, and to give them handlers.
When SystemTap runs the script, it monitors for certain events. When an event occurs, the
Linux kernel runs the handler as a sub-routine, then resumes. Thus, events serve as the
triggers for handlers to run. Handlers can record specified data and print it in a certain manner.
The SystemTap language only uses a few data types (integers, strings, and associative arrays
of these), and full control structures (blocks, conditionals, loops, functions). It has a lightweight
punctuation (semicolons are optional) and does not need detailed declarations (types are
inferred and checked automatically).
Tapsets
Tapsets are a library of pre-written probes and functions that can be used in SystemTap scripts.
When a user runs a SystemTap script, SystemTap checks the script's probe events and
handlers against the tapset library. SystemTap then loads the corresponding probes and
functions before translating the script to C. Like SystemTap scripts themselves, tapsets use the
filename extension *.stp.
However, unlike SystemTap scripts, tapsets are not meant for direct execution—they constitute
the library from which other scripts can pull definitions. Thus, the tapset library is an abstraction
layer designed to make it easier for users to define events and functions. Tapsets provide useful
aliases for functions that users may want to specify as an event (knowing the proper alias is
easier than remembering specific kernel functions that might vary between kernel versions).
2014 EMC Proven Professional Knowledge Sharing 7
Commands and Privileges
The main commands associated with SystemTap are stap and staprun. To execute them, you
either need root privileges or must be a member of the stapdev or stapusr group.
stap
SystemTap front-end. Runs a SystemTap script (either from file, or from standard input). It
translates the script into C code, compiles it, and loads the resulting kernel module into a
running Linux kernel. Then, the requested system trace or probe functions are performed.
staprun
SystemTap back-end. Loads and unloads kernel modules produced by the SystemTap front-
end.
For a list of options for each command, use --help. For details, refer to the stap and the staprun
man pages.
To avoid giving root access to users just for running SystemTap, you can use the following
SystemTap groups. They are not available by default on SUSE Linux Enterprise, but you can
create the groups and modify the access rights accordingly.
stapdev
Members of this group can run SystemTap scripts with stap, or run SystemTap instrumentation
modules with staprun. As running stap involves compiling scripts into kernel modules and
loading them into the kernel, members of this group still have effective root access.
stapusr
Members of this group are only allowed to run SystemTap instrumentation modules with
staprun. In addition, they can only run those modules from
/lib/modules/kernel_version/systemtap/. This directory must be owned by root and must only be
writable for the root user.
2014 EMC Proven Professional Knowledge Sharing 8
Important Files and Directories The following list provides an overview of the SystemTap main files and directories.
/lib/modules/kernel_version/systemtap/
Holds the SystemTap instrumentation modules.
/usr/share/systemtap/tapset/
Holds the standard library of tapsets.
/usr/share/doc/packages/systemtap/examples
Holds a number of example SystemTap scripts for various purposes. Only available if the
systemtap-docs package is installed.
~/.systemtap/cache
Data directory for cached SystemTap files.
/tmp/stap*
Temporary directory for SystemTap files, including translated C code and kernel object.
Installing SystemTap
To install SystemTap, you will need the following RPMs to be installed on the system:
• systemtap
• systemtap-runtime
If you are using the yum Linux package manager, these two RPMs can be installed by
executing the command “yum install systemtap systemtap-runtime”. Note that before you can
use SystemTap, you will still need to install the required kernel information RPMs.
To install the required Kernel information RPMs, SystemTap requires that you install the kernel
information packages or what some people call, “Kernel Debugging Information”.
This information allows SystemTap to generate the code for the instrumentation (Probing). The
following are the package names for the kernel information symbols.
kernel-debuginfo
kernel-debuginfo-common
2014 EMC Proven Professional Knowledge Sharing 9
To install these packages using “yum”, execute “yum install kernel-debuginfo kernel-debuginfo-
common”
SystemTap first run
Perform these steps to validate the SystemTap installation.
- Boot into the kernel where you downloaded the debugging information for it.
- Run this command to get a list of all the Kernel functions that we can probe
stap -l 'kernel.function("*")'
- Run this command, “stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'” to capture all the files read operations on the system.
To generate instrumentation for another computer, do these steps:
1. Set up a machine with the required kernel information packages installed.
2. Create the SystemTap script; compile it on this development machine.
3. Only install the systemtap-runtime package on the target machine.
4. Copy the generated kernel module file from the development machine to the target machine.
5. Use the “staprun” command to load the probe kernel module into the running kernel and
capture the output.
SystemTap Flight recorder mode
Flight recorder mode enables the system administrator to run SystemTap for a long period of
time on the target system and have control over when to capture the output. The flight recorder
mode is enabled by invoking the “-F” parameter to the stap command when running the
SystemTap script.
Flight recorder mode has 2 modes
1. In-memory
2. Disk file
2014 EMC Proven Professional Knowledge Sharing 10
When you run the flight recorder in-memory mode, the SystemTap script output will be sent to
the Kernel memory buffer. This is a place in the kernel memory where we can send information
to be used by the administrator or other kernel components later. To achieve this, execute these
steps:
Create the SystemTap script i.e. nettop.stp
Run the script with the -F parameter “stap -F nettop.stp”
You will see output similar to the below
[khaled@fedora20 systemtap]$ stap -F nettop.stp Disconnecting from systemtap module. To reconnect, type "staprun -A stap_496c96571c016d03d66f26f8d577296d__711"
When the probe action to be captured is triggered, you will need to reattach to the kernel
memory buffer to see the output by executing this command.
staprun -A stap_496c96571c016d03d66f26f8d577296d__711
Executing the stap run command against the given SystemTap session ID enables you to see
the script output on the console.
To run SystemTap Flight Recorder in File mode, perform these actions:
1. Create the SystemTap script file .stp
2. Execute “stap -F -o /tmp/cputest.log 1,2 cputest.stp”
3. Wait until the probe event triggers, then kill the stap process “kill -4 <PID>” where PID is
the process id in the system.
4. Analyze the generated log file.
2014 EMC Proven Professional Knowledge Sharing 11
How SystemTap works
SystemTap is used to extract, filter, and summarize useful data from a functional Linux system
that will make diagnosing the Linux system much more easier.
The idea behind SystemTap is to work on an event and assign them a handler. This handler is
invoked when the event is triggered. When the SystemTap script is executed, SystemTap
monitors for the events to occur and then executes the handler code written by the script writer
to print the information desired when this event occurs.
Event types are:
Kernel or User-Space programs Functions enter/exit
Timer events
Session events
When SystemTap script is executed with the command “stap <script.stp>”, the SystemTap
starts a new session for executing this script and returning the output. This session sequence is
as follows:
1. The stap interpreter is checking the script for any programming problems that may cause
problems if they are passed to the Kernel, such as Null pointer referencing and division
by zero. If any of these programming defects exist, the code execution will terminate with
code parsing error.
2. The interpreter is checking if any Tapsets are included in the script, Tapsets are the
SystemTap library like the standard library in any programming language which means a
code that can be included and reused in many scripts. If any of the Tapsets is included,
the interpreter will make sure to load this code appropriately.
3. The interpreter then will translate the SystemTap instructions into a C code and will invoke
the C compiler against it to create a Linux Kernel Loadable Module.
4. The generated kernel module includes all the probing C code that will be loaded to the
kernel to extract the required information.
5. Once we terminate the SystemTap script execution, the kernel module will be
automatically unloaded from the kernel.
2014 EMC Proven Professional Knowledge Sharing 12
The SystemTap script is the sole part of the SystemTap session. The SystemTap script is
composed of event declaration that instructs which event(s) needs to be monitored and also the
handler code for every event. Note that you can assign more than one event to the same
handler so in case any of these events occur, the handler will be fired to collect the desired
information.
This is how you can define the event/handler code in a typical SystemTap script.
probe EventName {
Handler Code
}
The above code will execute the code “Handler Code” if the “EventName” is triggered.
There are 2 types of Events:
Synchronous
Asynchronous
Synchronous events are simply events that correspond to existing Kernel functions.
Examples of synchronous events are
kernel.function(“copy_from_user”) – this call will probe every invocation of the function
“copy_from_user”
syscall.fcntl # this will probe every invocation of the system call function “fcntl”
vfs.file_operation – you can substitute “file_operation” with all the Virtual File System operations
like read, write, open, llseek etc.
Please note that “syscall” and “vfs” are tabsets, which means that SystemTap programmers are
defining these components to be used by us.
Note that you can gather the information before and after the invocation of the event function
that is called from any process / thread on the system like this:
2014 EMC Proven Professional Knowledge Sharing 13
kernel.function(“copy_from_user”) – this will be triggered when the function
copy_from_user is invoked by any thread in the system.
kernel.function(“copy_from_user”).return – appending the “.return” instruction will make
the event trigger fires when we leave the function.
This is very useful when needing to know what this specific function has performed.
The same principle applies on the syscall and vfs components:
syscall.fcntl, syscall.fcntl.return
vfs.read, vfs.read.return
You can use the wildcard asterisk symbol “*” to probe for a set of functions simultaneously.
kernel.function(“*@kernel/signal.c”) {}
And
kernel.function(“*@kernel/signal.c”).return {}
The above two probe definitions will be fired upon the invocation and return of all the functions
defined in the file “kernel/signal.c” in the Linux kernel source tree. This will allow us to track all
signal-related function invocations in the target kernel.
SystemTap also provides a mechanism to probe events that are statically marked as
tracepoints, like so:
kernel.trace(“tracepoint”)
Statically marked tracepoint events are simply kernel functions that developers choose to mark
as a tracepoint because it is frequently invoked by the Kernel relevant subsystem(s)
An example of this is when we need to see what happens every time the EXT4 filesystem
allocates a new inode
kernel.trace(“ext4_allocate_inode”)
2014 EMC Proven Professional Knowledge Sharing 14
SystemTap can also probe kernel module functions, like so:
probe module(“module_name”).function(“function_name”) {}
And
probe module(“module_name”).function(“function_name”).return {}
i.e. probe module("fuse").function("fuse_copy_page”) # this probe will be triggered every time a
fuse-managed filesystem (i.e. NTFS or DBFS) will copy a memory page.
SystemTap will only be able to probe open source modules that are part of the kernel source
and have their debugging symbols installed. If you are trying to probe proprietary kernel
modules such as ATI or NVidia graphics cards that are closed source at the time of writing this
article, SystemTap will not help unless the corresponding debugging symbols are installed. This
is not likely to happen because of the source code nature of these modules.
Asynchronous events
Asynchronous events are those that are not relevant to any kernel function or tracepoint. These
events are the timers, counters, and start / end instructions of the SystemTap script execution.
Examples of Asynchronous events are:
begin
The handler code behind the “begin” event will be triggered on the startup of the SystemTap
script. It is intended to be used when it is desired to display a message when the stap script
starts to run.
end
The “end” will be triggered at the end of the SystemTap session.
timer
The “timer” event is used to execute handler code on a timely basis based on the assigned
timer interval in the probe instruction.
probe timer.s(10){
printf (“hello world!\n”)
}
2014 EMC Proven Professional Knowledge Sharing 15
In the above example, we defined the timer to be triggered every 10 seconds as the “.s”
instructs every time this timer event occurs it will print “hello world!”. All the options for the timer
event are shown below:
timer.ms(milliseconds)
timer.us(microseconds)
timer.ns(nanoseconds)
timer.hz(hertz)
timer.jiffies(jiffies)
Let’s now move to the explanation of the Handler section.
Consider the following example:
probe begin{
printf(“Hello World!\n”)
exit()
}
This example defines the probe event “begin” so the code enclosed between the two curly
braces that follows will be executed at the start of the SystemTap session. Here, the code block
only prints the phrase “Hello World!” followed by a new line and then terminates the script
execution by calling the exit() function.
The printf function is the same as its equivalent in the C programming language.
Consider the following example to demonstrate the “printf” usage
probe syscall.open.return {
filename = user_string($filename)
if (execname() != "yakuake"){
printf("%s(%d) open (%s)\n", execname(),pid(),filename)
}
}
2014 EMC Proven Professional Knowledge Sharing 16
The above script is tracking all the calls to the Linux system call “open” and prints the process
name, process ID, and the opened file for every process on the system except for the process
named “yakuake” which is not desired to capture the same output for it. A sample output of the
execution of this script is shown below:
[khaled@doma stap_scripts]$ stap -v open_syscall.stp
Pass 1: parsed user script and 155 library script(s) using
379788virt/192528res/3128shr/192764data kb, in 2610usr/210sys/2858real ms.
Pass 2: analyzed script: 6 probe(s), 7 function(s), 5 embed(s), 0 global(s) using
578352virt/239928res/4416shr/238700data kb, in 1700usr/380sys/2110real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/4d/stap_4d6c65ccacf4a018ed647c8041d75871_11031.c
Pass 4: using cached
/home/khaled/.systemtap/cache/4d/stap_4d6c65ccacf4a018ed647c8041d75871_11031.ko
Pass 5: starting run.
kded4(14293) open (/etc/mtab)
systemd-udevd(177) open (/sys/fs/cgroup/systemd/system/systemd-
udevd.service/cgroup.procs)
crond(360) open (/etc/passwd)
crond(7376) open (/etc/pam.d/crond)
crond(7376) open (/usr/lib64/security/pam_access.so)
unix_chkpwd(7377) open (/lib64/libnss_files.so.2)
unix_chkpwd(7377) open (/etc/passwd)
unix_chkpwd(7377) open (/etc/shadow)
dbus-daemon(356) open (/proc/7376/cmdline)
systemd-logind(344) open (/etc/systemd/sleep.conf)
systemd-logind(344) open (/sys/power/state)
systemd-logind(344) open (/proc/14293/cgroup)
systemd-logind(344) open (/proc/7376/cgroup)
systemd-logind(344) open (/proc/7376/sessionid)
systemd-logind(344) open (/etc/passwd)
systemd-logind(344) created directory /run/systemd/sessions
systemd-logind(344) open (/run/systemd/sessions/324.ref)
2014 EMC Proven Professional Knowledge Sharing 17
systemd-logind(344) open (/run/systemd/sessions/324.ref)
systemd-logind(344) created directory /sys/fs/cgroup/systemd/user/1000.user/324.session
systemd-logind(344) open (/sys/fs/cgroup/systemd/user/1000.user/324.session/tasks)
systemd-logind(344) open (/sys/fs/cgroup/cpu/tasks)
systemd-logind(344) created directory /run/systemd/sessions
systemd-logind(344) open (/run/systemd/sessions/.324m38cfG)
pulseaudio(14468) open (/run/systemd/users/1000)
systemd-logind(344) created directory /run/systemd/users
SystemTap Functions
SystemTap has many built-in functions like printf() and exit() that are ready to be used by
programmers. Below are some of the commonly used SystemTap functions:
execname() Returns the process name of the process currently active in the kernel.
pid() Returns the process ID of the current active process.
tid() Returns the current thread ID if the process is multi-threaded.
uid() Returns the User ID that owns a process or thread.
cpu() Returns the current CPU number.
pp() Short for “probe point”, it returns the current probe point name if there are multiple probe
points in the script.
ppfunc() Returns the current function, called in the probe point.
target() Used when we need to dynamically pass information to the SystemTap script from the
command line, i.e. passing process ID to a SystemTap script to make it portable by not
hardcoding the PID inside the script which does not make sense because the PID number is a
frequently changing value. The following example illustrates use of the target() function.
probe syscall.*{
procid = target()
if (pid() == procid)
printf("%s(%d) %s\n", execname(),pid(),name)
}
2014 EMC Proven Professional Knowledge Sharing 18
Invoking this script with the “-x <pid>” parameter will return all the system calls executed by the
passed PID number. Check the output below:
[khaled@fedora20 systemtap]$ stap –v targeter.stp -x 345
Pass 1: parsed user script and 97 library script(s) using
212012virt/27444res/3120shr/24780data kb, in 100usr/10sys/108real ms.
Pass 2: analyzed script: 394 probe(s), 20 function(s), 26 embed(s), 0 global(s) using
500584virt/163192res/4400shr/159232data kb, in 1720usr/200sys/1923real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/62/stap_62e6796e8290d19288a40bebff0142d9_164703.c
Pass 4: using cached
/home/khaled/.systemtap/cache/62/stap_62e6796e8290d19288a40bebff0142d9_164703.ko
Pass 5: starting run.
crond(345) stat
crond(345) select
crond(345) open
crond(345) fstat
crond(345) mmap2
crond(345) read
crond(345) close
crond(345) munmap
crond(345) open
crond(345) fstat
crond(345) mmap2
crond(345) read
crond(345) close
crond(345) munmap
crond(345) rt_sigprocmask
crond(345) rt_sigaction
crond(345) rt_sigprocmask
crond(345) nanosleep
The target() function can only capture either PIDs or operating system commands from the
SystemTap script command line invocation. PID number is captured with the “-x” parameter
while OS commands are passed using the “-c” parameter.
2014 EMC Proven Professional Knowledge Sharing 19
thread_indent() This function is very useful in formatting and indenting the output of the
SystemTap script. The example below demonstrates its use:
probe kernel.function("*@net/socket.c").call {
printf ("%s -> %s\n", thread_indent(1), ppfunc())
}
probe kernel.function("*@net/socket.c").return {
printf ("%s <- %s\n", thread_indent(-1), ppfunc())
}
This script reports the calls to all of the Kernel socket functions in a multi-level hierarchical view.
The following is an example of this script execution.
[khaled@doma stap_scripts]$ stap -v socket_trace.stp
Pass 1: parsed user script and 155 library script(s) using
379788virt/192520res/3128shr/192764data kb, in 2620usr/210sys/2857real ms.
Pass 2: analyzed script: 178 probe(s), 8 function(s), 3 embed(s), 2 global(s) using
591272virt/250900res/4344shr/251620data kb, in 1290usr/360sys/1676real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/96/stap_962ee921a7558049a549dcaec3c4841c_36115.c
Pass 4: using cached
/home/khaled/.systemtap/cache/96/stap_962ee921a7558049a549dcaec3c4841c_36115.ko
Pass 5: starting run.
0 sshd(8942): -> sock_poll
4 sshd(8942): <- sock_poll
0 sshd(8942): -> sock_aio_write
5 sshd(8942): -> alloc_sock_iocb
9 sshd(8942): <- alloc_sock_iocb
140 sshd(8942): <- sock_aio_write
0 sshd(8942): -> sock_poll
4 sshd(8942): <- sock_poll
0 sshd(8942): -> sock_aio_write
4 sshd(8942): -> alloc_sock_iocb
2014 EMC Proven Professional Knowledge Sharing 20
7 sshd(8942): <- alloc_sock_iocb
164 sshd(8942): <- sock_aio_write
0 sshd(8942): -> sock_poll
4 sshd(8942): <- sock_poll
0 sshd(8942): -> sock_aio_write
4 sshd(8942): -> alloc_sock_iocb
7 sshd(8942): <- alloc_sock_iocb
162 sshd(8942): <- sock_aio_write
0 sshd(8942): -> sock_poll
Please note that every “.call” or entering to a function is represented by “->” symbol that was
defined in the first probe handler and every “.return” or function exiting is represented by “<-” as
defined in the second probe handler.
These were some of the standard SystemTap library functions but you can still write your own
functions to eliminate repeating the code for the tasks that will be executed frequently inside the
SystemTap script.
This syntax is to define and auxiliary function:
function func1 (arg1, arg2, argN) {
Function Code
}
Still, you can define the return value and the parameters type for this function the same as in the
C language as shown below:
function func1:string (arg1:int, arg2:long, argN:string) {
Function Code
}
Notice that we declared the function as returning string and the parameters to be of types int,
long, string.
2014 EMC Proven Professional Knowledge Sharing 21
SystemTap Variables
Like all programming languages SystemTap is using variables to store script execution data in
memory to do the processing on them and use these processed data as desired.
SystemTap variables are dynamically typed, meaning that the programmer doesn’t need to
explicitly assign the appropriate data type for each variable, unlike the C programming language
that needs to explicitly assign each variable with an appropriate data type, i.e. in the C
programming language:
#include <stdio.h>
void main(){
char* greet = "Hello World!";
printf("%s\n",greet);
}
The above example simply prints the text “Hello World!” to the shell and exits. Notice that we
explicitly defined the variable “greet” as type “char*”. Otherwise, the program won’t compile.
In SystemTap syntax
probe begin {
greet = "Hello World!"
printf("%s\n",greet)
exit()
}
SystemTap will implicitly detect the correct data type for every declared variable as illustrated by
the above examples.
Variables are only visible in the scope of the owner event, This means that if a variable “var1” is
defined in “eventA”, this variable will only be accessible to the “eventA” handler code but not
from any other event.
2014 EMC Proven Professional Knowledge Sharing 22
Consider the following example:
probe begin{
x = 1
println(x)
exit()
}
probe end{
x += 4
println(x)
}
This script simply should print the value of the variable “x” when the script starts and when it
exits. Below is the output:
[khaled@fedora20 ~]$ stap vars.stp
1
4
The output shows that the value of variable “x” was 1 and became 4 although it should be 5 at
the end of the script because we added 4 to the original value of “x”, The cause of this situation
is that “x” is defined twice within two separate events.
To allow the variable “x” to be accessible through all events, simply declare it as global at the
beginning of the script, like so:
global x
probe begin{
x = 1
println(x)
exit()
}
2014 EMC Proven Professional Knowledge Sharing 23
probe end{
x += 4
println(x)
}
Now the execution of this code shows the output as follows:
[khaled@fedora20 ~]$ stap vars.stp
1
5
Because “x” is defined as global, It is now accessible from both events.
Target Variables
Target variables are those that correspond to an actual location of code like a kernel function or
a system call.
We can probe a target variable to examine its value at any certain time or condition, Consider
the following example that shows each invocation to the system call “open” and also the file
name being accessed by the “open” system call.
probe syscall.open.return{
file = user_string($filename)
printf("%s(%d) is opening: %s\n",execname(),pid(),file)
}
The output is as follows:
[khaled@fedora20 ~]$ stap -v open_trace.stp
Pass 1: parsed user script and 100 library script(s) using
215592virt/30740res/3120shr/28340data kb, in 120usr/50sys/222real ms.
Pass 2: analyzed script: 4 probe(s), 7 function(s), 5 embed(s), 0 global(s) using
404764virt/67356res/4400shr/63392data kb, in 880usr/600sys/2279real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/e3/stap_e352b975473f530ebfec4fa432d8de64_10345.c
Pass 4: using cached
/home/khaled/.systemtap/cache/e3/stap_e352b975473f530ebfec4fa432d8de64_10345.ko
2014 EMC Proven Professional Knowledge Sharing 24
Pass 5: starting run.
vminfo(434) is opening: /var/run/utmp
systemd-udevd(296) is opening: /sys/fs/cgroup/systemd/system.slice/systemd-
udevd.service/cgroup.procs
ls(639) is opening: /etc/ld.so.cache
ls(639) is opening: /lib64/libselinux.so.1
ls(639) is opening: /lib64/libcap.so.2
ls(639) is opening: /lib64/libacl.so.1
ls(639) is opening: /lib64/libc.so.6
ls(639) is opening: /lib64/libpcre.so.1
ls(639) is opening: /lib64/libpthread.so.0
ls(639) is opening: /lib64/libdl.so.2
ls(639) is opening: /lib64/libattr.so.1
ls(639) is opening: /usr/lib/locale/locale-archive
ls(639) is opening: /usr/share/locale/locale.alias
ls(639) is opening: /usr/share/locale/en_US.UTF-8/LC_TIME/coreutils.mo
ls(639) is opening: /usr/share/locale/en_US.utf8/LC_TIME/coreutils.mo
ls(639) is opening: /usr/share/locale/en_US/LC_TIME/coreutils.mo
ls(639) is opening: /usr/share/locale/en.UTF-8/LC_TIME/coreutils.mo
ls(639) is opening: /usr/share/locale/en.utf8/LC_TIME/coreutils.mo
ls(639) is opening: /usr/share/locale/en/LC_TIME/coreutils.mo
ls(639) is opening: /usr/lib64/gconv/gconv-modules.cache
ls(639) is opening: /sys/fs/selinux/mls
ls(639) is opening: /etc/nsswitch.conf
ls(639) is opening: /etc/ld.so.cache
ls(639) is opening: /lib64/libnss_files.so.2
ls(639) is opening: /etc/passwd
ls(639) is opening: /etc/group
ls(639) is opening: /usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo
ls(639) is opening: /usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo
ls(639) is opening: /usr/share/locale/en_US/LC_MESSAGES/coreutils.mo
ls(639) is opening: /usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo
ls(639) is opening: /usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo
ls(639) is opening: /usr/share/locale/en/LC_MESSAGES/coreutils.mo
2014 EMC Proven Professional Knowledge Sharing 25
ls(639) is opening: /etc/localtime
vminfo(434) is opening: /var/run/utmp
To get a list of all the target variables for a kernel function or system call execute the command:
stap -L ‘syscall.open’
Output:
[khaled@fedora20 ~]$ stap -L 'syscall.open'
syscall.open name:string filename:string flags:long mode:long argstr:string $filename:long int
$flags:long int $mode:long int
Please note that we used the SystemTap built-in function “user_string” to convert the data type
of the “$filename” variable from “long” to “string” so we can print the actual human readable file
name to the console.
Also, SystemTap is capable of querying structure records and get the information of any desired
“struct” member, please check the following example:
probe kernel.function("vfs_read"){
printf("current files_stats->max_files: %d\n", @var("files_stat@fs/file_table.c")->max_files)
exit()
}
This probe gets the max files kernel variable, knowing that this variable is tunable by the system
administrator by using the utility “sysctl”. See the output below:
[khaled@fedora20 ~]$ stap -v struct_probe.stp
Pass 1: parsed user script and 100 library script(s) using
215596virt/30736res/3120shr/28344data kb, in 100usr/20sys/117real ms.
Pass 2: analyzed script: 1 probe(s), 2 function(s), 0 embed(s), 0 global(s) using
400432virt/163160res/102220shr/61540data kb, in 720usr/110sys/830real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/30/stap_30b6b460586de308b8ee87cda75040ad_1966.c
Pass 4: using cached
2014 EMC Proven Professional Knowledge Sharing 26
/home/khaled/.systemtap/cache/30/stap_30b6b460586de308b8ee87cda75040ad_1966.ko
Pass 5: starting run.
current files_stats->max_files: 99807
Pass 5: run completed in 0usr/40sys/349real ms.
[khaled@fedora20 ~]$ sysctl fs.file-max
fs.file-max = 99807
The code above is using the “@var” statement to instruct SystemTap to locate the variable
“files_stat” which is of type “struct” inside the Linux kernel‘s file “fs/file_table.c”, get the member
named “max_files” from this kernel data structure, and then print the value to the terminal.
SystemTap Language Syntax
1. Conditional Statements
Sometimes the programmer needs to limit the script execution flow to certain decisions
This can be achieved with the IF/ELSE clause. The format of the IF/ELSE clause is:
if (condition) {
Statement 1
}
else{
Statement 2
}
Statement 1 will be executed if the “condition” is met. Otherwise, statement 2 will be
executed.
Please check this example code that illustrates the usage of the IF/ELSE clause.
global readcount, writecount
probe vfs.read,vfs.write{
if (probefunc() == "vfs_read"){
readcount++
2014 EMC Proven Professional Knowledge Sharing 27
}else{
writecount++
}
}
probe timer.s(5){
printf("VFS stats: read(%d), write(%d)\n", readcount,writecount)
}
The output is:
[khaled@fedora20 ~]$ stap -v vfs_counts.stp
Pass 1: parsed user script and 100 library script(s) using
215600virt/30736res/3116shr/28348data kb, in 110usr/10sys/120real ms.
Pass 2: analyzed script: 3 probe(s), 6 function(s), 6 embed(s), 2 global(s) using
496648virt/158520res/8048shr/150116data kb, in 1620usr/360sys/1978real ms.
Pass 3: translated to C into
"/tmp/staplp5zC2/stap_ded65077fe34cbf9f4d921789eea9d9f_4054_src.c" using
489024virt/156980res/7488shr/150116data kb, in 130usr/80sys/210real ms.
Pass 4: compiled C into "stap_ded65077fe34cbf9f4d921789eea9d9f_4054.ko" in
3030usr/460sys/3285real ms.
Pass 5: starting run.
VFS stats: read(41), write(0)
VFS stats: read(75), write(2)
VFS stats: read(109), write(4)
VFS stats: read(143), write(6)
VFS stats: read(182), write(24)
VFS stats: read(217), write(26)
VFS stats: read(252), write(28)
2. The WHILE loop
2014 EMC Proven Professional Knowledge Sharing 28
The WHILE loop syntax is:
while (condition){
statement
}
As long as the condition is a value greater than zero, the statement will be executed.
3. The FOR loop
The FOR loop syntax is:
for (initialization ; conditional ; value_change){
statement
}
i.e.
for (x = 0 ; x <= 9 ; x++){
printf(“%d\n”,x)
}
The loop above simply loops on the x variable that has the initialization value of zero until it
reaches 9. Every time it loops, it prints its value.
4. Same as the C programming language, these are the conditional operators:
The logical equal is “==” pronounced “is equal to”
Greater than or equal is “>=”
Less than or equal is “<=”
Not equal is “!=”
Less than “<”
Greater than “>”
2014 EMC Proven Professional Knowledge Sharing 29
SystemTap Associative arrays
Associative arrays are the type of variables that hold a collection of values in the form of
Keys that corresponds to values,
Arrays are defined as:
Array_name[Index_value]
i.e.
employee[“name”] = “John”
employee[“id”] = 90
employee[“department”] = “IT”
procinfo[pid()] = gettimeofday_s
Accessing Array elements
To access the array elements, simply call the array member element as below:
printf(“EmpID: %d corresponds to Emp name: %s\n”, employee[“id”],employee[“name”])
Processing multiple members in an associative array
The “foreach” statement is used to perform processing on multiple array elements. The
example below demonstrates the “foreach” keyword usage.
global reads
probe vfs.read{
reads[execname()]++
}
probe timer.s(3){
foreach(count in reads){
printf("%s : %d\n",count,reads[count])
}
printf("===========\n")
}
2014 EMC Proven Professional Knowledge Sharing 30
Note that the variable named “count” in the “foreach” statement is declared within the
foreach statement and it will only be accessible within the scope of the foreach
statement code.
Incrementing Array elements
To increment array elements, use the “++” operator as is used with the C language:
global reads
probe vfs.read{
reads[execname()]++
}
probe timer.s(3){
foreach(count in reads){
printf("%s : %d\n",count,reads[count])
}
printf("===========\n")
}
Output:
[khaled@fedora20 systemtap]$ stap -v vfsreads.stp
Pass 1: parsed user script and 100 library script(s) using
215596virt/30740res/3120shr/28344data kb, in 110usr/10sys/122real ms.
Pass 2: analyzed script: 2 probe(s), 1 function(s), 5 embed(s), 1 global(s) using
496632virt/158464res/8060shr/150100data kb, in 1580usr/410sys/1978real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/f2/stap_f246d60ea643aa5ff07d567a6eda42f7_2571.c
Pass 4: using cached
/home/khaled/.systemtap/cache/f2/stap_f246d60ea643aa5ff07d567a6eda42f7_2571.ko
Pass 5: starting run.
stapio : 16
2014 EMC Proven Professional Knowledge Sharing 31
vminfo : 5
systemd-udevd : 4
===========
stapio : 16
sshd : 1
vminfo : 5
===========
stapio : 16
sshd : 1
===========
Pass 5: run completed in 10usr/50sys/16393real ms.
Testing for member existence
To check for a particular member in an associative array, the syntax in the example below can
be used.
global reads
probe vfs.read{
reads[execname()]++;
}
probe timer.s(3){
printf("============\n");
foreach(count in reads+){
printf("%s : %d\n",count, reads[count]);
if (["stapio"] in reads){
printf("stapio detected.. Existing\n");
exit();
}
}
}
2014 EMC Proven Professional Knowledge Sharing 32
As the above example shows, we checked for the existence of the member “stapio” in the
“reads” array with this statement
if (["stapio"] in reads)
This statement will evaluate to true if the element named “stapio” is a member in the “reads”
array; otherwise, it will be false.
Deleting array elements
Deleting array elements is done by using the “delete” keyword which has the same effect as in
the C programming language.
global reads
probe vfs.read{
reads[execname()]++
}
probe timer.s(3){
foreach(count in reads){
printf("%s : %d\n",count,reads[count])
}
printf("===========\n")
delete reads
}
Computing statistical aggregates
Aggregate instances are used to collect statistics on numerical values when it is important to
accumulate new data quickly and in large volume. These instances store only aggregated
stream statistics. Aggregates make sense only for global variables. They are stored individually
or as elements of an associative array.
- The aggregation operator "<<<"
2014 EMC Proven Professional Knowledge Sharing 33
The aggregation operator is "<<<", The left operand specifies a scalar or array-index l-value,
which must be declared global. The right operand is a numeric expression. The meaning is
intuitive: add the given number as a sample to the set of numbers to compute their statistics.
The specific list of statistics to gather is given separately by the extraction functions. The
following is an example:
reads[execname()] <<< count
For each instance of a distinct extraction function operating on a given identifier, the translator
computes a set of statistics. With each execution of an extraction function, the aggregation is
computed for that moment across all processors. The first argument of each function is the
same style of l-value as used on the left side of the aggregation operation.
The following functions provide methods to extract information about aggregate.
@count(s)
This statement returns the number of samples accumulated in aggregate s.
@sum(s)
This statement returns the total sum of all samples in aggregate s.
@min(s)
This statement returns the minimum of all samples in aggregate s.
@max(s)
This statement returns the maximum of all samples in aggregate s.
@avg(s)
This statement returns the average value of all samples in aggregate s.
2014 EMC Proven Professional Knowledge Sharing 34
Example:
global reads;
probe vfs.read{
reads[execname(), pid()] <<< 1;
}
probe timer.s(1){
foreach([x, y] in reads){
printf("%s (%d) : %d\n", x, y, @count(reads[x,y]));
}
printf("==============\n");
}
Output:
[khaled@fedora20 systemtap]$ stap -v aggregates1.stp
Pass 1: parsed user script and 101 library script(s) using
215736virt/30996res/3120shr/28484data kb, in 110usr/10sys/118real ms.
Pass 2: analyzed script: 2 probe(s), 2 function(s), 5 embed(s), 1 global(s) using
496736virt/158560res/8056shr/150204data kb, in 1590usr/390sys/2008real ms.
Pass 3: translated to C into
"/tmp/stapzj0Hxk/stap_6c120c380184bf84fea63efa119daf5a_2676_src.c" using
489096virt/156412res/6888shr/150204data kb, in 10usr/80sys/88real ms.
Pass 4: compiled C into "stap_6c120c380184bf84fea63efa119daf5a_2676.ko" in
1470usr/370sys/1907real ms.
Pass 5: starting run.
stapio (10184) : 7
==============
sshd (572) : 1
stapio (10184) : 13
==============
vminfo (438) : 7
sshd (572) : 2
systemd-udevd (292) : 3
2014 EMC Proven Professional Knowledge Sharing 35
systemd-udevd (10185) : 1
stapio (10184) : 19
==============
vminfo (438) : 7
sshd (572) : 3
systemd-udevd (292) : 3
systemd-udevd (10185) : 1
stapio (10184) : 25
==============
User-Space Probing
Up to now, we discussed the instrumentation of the Linux kernel using SystemTap. Now we are
going to explain how to probe the user-space processes or what is usually known as “User
Programs”
User-space probing requires the running kernel to have the “utrace” kernel extension, The
following command checks the existence of the utrace kernel extension:
grep CONFIG_UTRACE /boot/config-`uname -r`
Please note that the above command will dynamically get the kernel version number and attach
it to the pattern “config-“. Thus, if the running kernel version is 2.6.32-1, the file to check will be:
grep CONFIG_UTRACE /boot/config-2.6.32-1
If the utrace kernel extension is compiled with the kernel, the output should be:
CONFIG_UTRACE=y
As with kernel and systemcall probing, the “process” keyword is used to create an event for a
certain process. This process can be the full path to the process executable file or only the
executable name if this executable is in the PATH of the user executing the SystemTap script.
Check the syntax below.
process(“PATH”).function(“function”)
As in kernel functions, the process is accepting wild cards for the process functions and also
accepting the “.return” suffix to probe the exiting point of a function.
2014 EMC Proven Professional Knowledge Sharing 36
Consider the following example that demonstrates the user-space process probing code.
probe process("/lib64/security/pam_unix.so").function("*"){
printf("Function: %s\n",ppfunc());
}
Output:
[khaled@fedora20 systemtap]$ stap -v pam.stp
Pass 1: parsed user script and 101 library script(s) using
215744virt/31000res/3124shr/28492data kb, in 110usr/50sys/206real ms.
Pass 2: analyzed script: 270 probe(s), 1 function(s), 0 embed(s), 0 global(s) using
218016virt/33720res/3840shr/30512data kb, in 10usr/10sys/28real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/14/stap_1462efec64d08d99f11ca5a1542fe68e_60553.c
Pass 4: using cached
/home/khaled/.systemtap/cache/14/stap_1462efec64d08d99f11ca5a1542fe68e_60553.ko
Pass 5: starting run.
Function: pam_sm_acct_mgmt
Function: _set_ctrl
Function: search_key
Function: getline
Function: __strsep_3c
Function: __strsep_3c
Function: __strsep_3c
Function: __strsep_3c
Function: get_account_info
Function: is_pwd_shadowed
Function: _unix_run_verify_binary
Function: memset
Function: memcpy
This example simply lists all the PAM (Pluggable Authentication Modules) functions called when
the system is authenticating a user by SSH (Secure Shell) or SU (Super User) utilities.
2014 EMC Proven Professional Knowledge Sharing 37
The following example demonstrates the power of SystemTap in capturing parameters passed
to functions in the user-space,
IMPORTANT NOTICE: The example code shown below is used only to demonstrate
SystemTap capabilities and should not be used on any system without proper awareness of the
individuals using this system,
global pass, username, host
probe process("/usr/bin/ssh").function("readpassphrase").return{
pass = user_string($buf);
}
probe process("/usr/bin/ssh").function("ssh_userauth2").return{
host = user_string($host);
username = user_string($server_user);
printf("%s@%s (%s)\n",username,host,pass);
}
This example is used to probe the process “/usr/bin/ssh” for the login information of the remote
servers to which this ssh client is authenticating. The output should be as follows:
[khaled@fedora20 systemtap]$ stap -v ssh.stp
Pass 1: parsed user script and 101 library script(s) using
215736virt/31004res/3124shr/28484data kb, in 100usr/30sys/126real ms.
Pass 2: analyzed script: 4 probe(s), 5 function(s), 2 embed(s), 9 global(s) using
377360virt/38696res/4972shr/34196data kb, in 80usr/400sys/1059real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/73/stap_73dbec385d2791b3202e97603c280b58_6470.c
Pass 4: using cached
/home/khaled/.systemtap/cache/73/stap_73dbec385d2791b3202e97603c280b58_6470.ko
Pass 5: starting run.
Now, open another terminal and connect to any server using SSH
[khaled@fedora20 ~]$ ssh khaled@aqua
2014 EMC Proven Professional Knowledge Sharing 38
Password:
Last login: Wed Jan 8 01:30:07 2014
[khaled@aqua ~]$
After successfully authenticating to the remote server, you will notice that SystemTap
successfully captured the login information and printed it at the end of the SystemTap script
execution.
[khaled@fedora20 systemtap]$ stap -v ssh.stp
Pass 1: parsed user script and 101 library script(s) using
215736virt/31004res/3124shr/28484data kb, in 100usr/30sys/126real ms.
Pass 2: analyzed script: 4 probe(s), 5 function(s), 2 embed(s), 9 global(s) using
377360virt/38696res/4972shr/34196data kb, in 80usr/400sys/1059real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/73/stap_73dbec385d2791b3202e97603c280b58_6470.c
Pass 4: using cached
/home/khaled/.systemtap/cache/73/stap_73dbec385d2791b3202e97603c280b58_6470.ko
Pass 5: starting run.
khaled@aqua (p@ASsW0rD)
Please keep in mind that the above example is not a security break this output won’t appear
unless the debug-info package for the SSH binary is installed on the system which requires that
the root privilege be installed.
2014 EMC Proven Professional Knowledge Sharing 39
Useful SystemTap scripts
1. tcpdumplike.stp
probe begin, timer.s(1) {
printf("-----------------------------------------------------------------\n")
printf(" Source IP Dest IP SPort DPort U A P R S F \n")
printf("-----------------------------------------------------------------\n")
}
probe tcp.receive {
printf(" %15s %15s %5d %5d %d %d %d %d %d %d\n",saddr, daddr, sport, dport,
urg, ack, psh, rst, syn, fin)
}
Output:
[khaled@fedora20 ~]$ stap -v tcpdumplike.stp
Pass 1: parsed user script and 101 library script(s) using
215740virt/31000res/3120shr/28488data kb, in 110usr/10sys/121real ms.
Pass 2: analyzed script: 4 probe(s), 28 function(s), 4 embed(s), 0 global(s) using
490344virt/152960res/4680shr/148612data kb, in 1460usr/300sys/1760real ms.
Pass 3: translated to C into
"/tmp/stapUhUO6T/stap_1d61bd824dbb56f190b5b219a27b9115_17035_src.c" using
487504virt/154916res/6860shr/148612data kb, in 10usr/80sys/90real ms.
Pass 4: compiled C into "stap_1d61bd824dbb56f190b5b219a27b9115_17035.ko" in
1620usr/330sys/1941real ms.
Pass 5: starting run.
-----------------------------------------------------------------
Source IP Dest IP SPort DPort U A P R S F
-----------------------------------------------------------------
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 1 0
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0
2014 EMC Proven Professional Knowledge Sharing 40
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
-----------------------------------------------------------------
Source IP Dest IP SPort DPort U A P R S F
-----------------------------------------------------------------
192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0
192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0
This script generates information about the incoming/outgoing TCP connection to/from the
system and lists the source, destination IP addresses alongside the source, destination ports,
and also the TCP packet flag of each captured packet.
Note: The above probe event “tcp.receive” is defined in the standard SystemTap Tapset library.
2. iotime.stp
global start
global time_io
function timestamp:long() { return gettimeofday_us() - start }
function proc:string() { return sprintf("%d (%s)", pid(), execname()) }
probe begin { start = gettimeofday_us() }
2014 EMC Proven Professional Knowledge Sharing 41
global filehandles, fileread, filewrite
probe syscall.open.return {
filename = user_string($filename)
if ($return != -1) {
filehandles[pid(), $return] = filename
} else {
printf("%d %s access %s fail\n", timestamp(), proc(), filename)
}
}
probe syscall.read.return {
p = pid()
fd = $fd
bytes = $return
time = gettimeofday_us() - @entry(gettimeofday_us())
if (bytes > 0)
fileread[p, fd] += bytes
time_io[p, fd] <<< time
}
probe syscall.write.return {
p = pid()
fd = $fd
bytes = $return
time = gettimeofday_us() - @entry(gettimeofday_us())
if (bytes > 0)
filewrite[p, fd] += bytes
time_io[p, fd] <<< time
}
probe syscall.close {
if ([pid(), $fd] in filehandles) {
2014 EMC Proven Professional Knowledge Sharing 42
printf("%d %s access %s read: %d write: %d\n",
timestamp(), proc(), filehandles[pid(), $fd],
fileread[pid(), $fd], filewrite[pid(), $fd])
if (@count(time_io[pid(), $fd]))
printf("%d %s iotime %s time: %d\n", timestamp(), proc(),
filehandles[pid(), $fd], @sum(time_io[pid(), $fd]))
}
delete fileread[pid(), $fd]
delete filewrite[pid(), $fd]
delete filehandles[pid(), $fd]
delete time_io[pid(),$fd]
}
Output:
[khaled@fedora20 systemtap]$ stap iotime.stp
779922 438 (vminfo) access /var/run/utmp read: 1536 write: 0
779931 438 (vminfo) iotime /var/run/utmp time: 62
2995593 292 (systemd-udevd) access /sys/fs/cgroup/systemd/system.slice/systemd-
udevd.service/cgroup.procs read: 4 write: 0
2995602 292 (systemd-udevd) iotime /sys/fs/cgroup/systemd/system.slice/systemd-
udevd.service/cgroup.procs time: 29
5782516 438 (vminfo) access /var/run/utmp read: 1536 write: 0
5782523 438 (vminfo) iotime /var/run/utmp time: 43
10785159 438 (vminfo) access /var/run/utmp read: 1536 write: 0
10785168 438 (vminfo) iotime /var/run/utmp time: 56
15787737 438 (vminfo) access /var/run/utmp read: 1536 write: 0
15787746 438 (vminfo) iotime /var/run/utmp time: 286
This script gathers the time spent executing the “open”,”read”,”write”,”close” system calls for
each process.
Tip: We can limit the process selection to the Avamar processes so we can gather all the disk
read / write operations timings for Avamar.
2014 EMC Proven Professional Knowledge Sharing 43
3. inodewatch.stp
probe vfs.write, vfs.read
{
if (ino == $1)
printf ("%s(%d) %s %u\n",execname(), pid(), probefunc(),ino);
}
This script monitors all the read / write operations performed on a specific file recognized by its
inode number, please follow the sequence below to test the execution of this script:
Create a file using the “touch” command create a file named “test1.txt”
[khaled@fedora20 ~]$ touch test1.txt
List the file inode number by executing the command “ls -li test1.txt”
[khaled@fedora20 ~]$ ls -li test1.txt
3518 -rw-r--r--. 1 khaled users 0 Jan 9 13:34 test1.txt
Notice that the inode number for the “test1.txt” file is “3518”
Start the script against inode “3518” and in another terminal window, create some read / write
actions against the “test1.txt” file:
[khaled@fedora20 ~]$ echo "This is a test file" > test1.txt
[khaled@fedora20 ~]$ cat test1.txt
This is a test file
Now, check the script execution for the output:
[khaled@fedora20 ~]$ stap -v inodewatch.stp 3518
Pass 1: parsed user script and 101 library script(s) using
215740virt/30996res/3116shr/28488data kb, in 100usr/20sys/122real ms.
Pass 2: analyzed script: 2 probe(s), 14 function(s), 6 embed(s), 0 global(s) using
496728virt/158588res/8040shr/150196data kb, in 1680usr/380sys/2066real ms.
Pass 3: translated to C into
2014 EMC Proven Professional Knowledge Sharing 44
"/tmp/stapISnw0t/stap_19c82375a02625d38102e7614aedd7ae_7294_src.c" using
489104virt/157036res/7468shr/150196data kb, in 130usr/70sys/211real ms.
Pass 4: compiled C into "stap_19c82375a02625d38102e7614aedd7ae_7294.ko" in
3160usr/510sys/3534real ms.
Pass 5: starting run.
bash(573) vfs_write 3518
cat(8105) vfs_read 3518
cat(8105) vfs_read 3518
Notice the write action from “bash” pid 573 (which was the “echo” against the file) then read by
“cat” pid 8105.
4. inodewatch2.stp
global ATTR_MODE = 1
probe kernel.function("setattr_copy"){
inode_nr = $inode->i_ino
if ( inode_nr == $1 && $attr->ia_valid & ATTR_MODE)
printf ("Process: %s(%d) has changed inode %u to mode %o by userid %d\n",execname(),
pid(), inode_nr, $attr->ia_mode, uid())
}
The inodewatch2.stp script watches a specific file for file system permission changing.
Please follow this sequence to test its functionality:
1. Execute the inodewatch2.stp against inode number “3518”
2. On another terminal window, change the file permissions mode for the “test1.txt”
[khaled@fedora20 ~]$ chmod 600 test1.txt
[khaled@fedora20 ~]$ chmod 700 test1.txt
3. Watch the script execution output for results:
[khaled@fedora20 systemtap]$ stap -v ./inodewatch2.stp 3518
Pass 1: parsed user script and 101 library script(s) using
215744virt/31000res/3120shr/28492data kb, in 110usr/20sys/120real ms.
2014 EMC Proven Professional Knowledge Sharing 45
Pass 2: analyzed script: 1 probe(s), 6 function(s), 2 embed(s), 1 global(s) using
404628virt/67084res/4376shr/63256data kb, in 750usr/180sys/946real ms.
Pass 3: using cached
/home/khaled/.systemtap/cache/f1/stap_f1e2c7f2cd8d62479e2a2225018e2a81_4720.c
Pass 4: using cached
/home/khaled/.systemtap/cache/f1/stap_f1e2c7f2cd8d62479e2a2225018e2a81_4720.ko
Pass 5: starting run.
Process: chmod(9532) has changed inode 3518 to mode 100600 by userid 1000
Process: chmod(9534) has changed inode 3518 to mode 100700 by userid 1000
The functionality of the above inode scripts would be ideal for diagnosing EMC backup and
recovery solutions such as Avamar and Data Domain® because they provide a kernel level
monitor on the actions performed on the files stored on the file systems of these respective
products.
5.Networker.stp
Program Code:
#!/usr/bin/stap -v # This program monitors the EMC networker # processes and get the reads, writes, # connections and sent / recieved signals information. # Author: Khaled Ahmed ([email protected]) global pname = "nsrmmd"; global pname_rpc_handler = "nsrexecd"; global readcount,writecount; global totalreadcount,totalwritecount; global bool_tcp_con_recv = 0; #global bool_tcp_con_send = 0; probe vfs.read{ if (execname() == pname){ readcount++; } } probe vfs.write{ if (execname() == pname){ writecount++; } } probe kernel.function("tcp_accept").return?, kernel.function("inet_csk_accept").return? {
2014 EMC Proven Professional Knowledge Sharing 46
if (execname() == pname_rpc_handler ){ sock = $return; if (sock != 0){ bool_tcp_con_recv = 1; printf("\nIncoming TCP connection to the Networker process\n"); printf("%s %6s %16s\n", "PID", "PORT", "COMING FROM"); printf("%d %6d %16s\n\n", pid(), inet_get_local_port(sock), inet_get_ip_source(sock)); } } } probe signal.send{ if (execname() == pname || execname() == pname_rpc_handler){ printf("\nSIGNAL SENT\n"); printf("%-8s %-16s %-5s %-16s %6s %-16s\n", "SPID", "SNAME", "RPID", "RNAME", "SIGNUM", "SIGNAME"); printf("%-8d %-16s %-5d %-16s %-6d %-16s\n\n", pid(), execname(), sig_pid, pid_name, sig, sig_name); }else{ if (pid_name == pname || pid_name == pname_rpc_handler){ printf("\nSIGNAL RECIEVED\n"); printf("%-8s %-16s %-5s %-16s %6s %-16s\n", "SPID", "SNAME", "RPID", "RNAME", "SIGNUM", "SIGNAME"); printf("%-8d %-16s %-5d %-16s %-6d %-16s\n\n", pid(), execname(), sig_pid, pid_name, sig, sig_name); } } } probe timer.s(3){ printf("(%s): reads = %d, write = %d\n",pname, readcount, writecount); totalreadcount += readcount; totalwritecount += writecount; readcount = 0; writecount = 0; } probe timer.s(10){ printf("Total Counts: READS(%d) WRITES(%d)\n", totalreadcount, totalwritecount); }
Sample Output:
[khaled@centos6 ~]$ stap -v networker.stp Pass 1: parsed user script and 96 library script(s) using 200576virt/26776res/3152shr/24072data kb, in 210usr/60sys/416real ms. Pass 2: analyzed script: 7 probe(s), 18 function(s), 30 embed(s), 7 global(s) using 439148virt/130892res/6780shr/122216data kb, in 2570usr/4320sys/20351real ms. Pass 3: using cached /home/khaled/.systemtap/cache/e1/stap_e1c1c67f347eda5734bdd60cd7a9cb0f_25284.c Pass 4: using cached /home/khaled/.systemtap/cache/e1/stap_e1c1c67f347eda5734bdd60cd7a9cb0f_25284.ko Pass 5: starting run. (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(0) WRITES(0) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(0) WRITES(0) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0
2014 EMC Proven Professional Knowledge Sharing 47
(nsrmmd): reads = 0, write = 0 Total Counts: READS(0) WRITES(0) (nsrmmd): reads = 0, write = 0 SIGNAL RECIEVED SPID SNAME RPID RNAME SIGNUM SIGNAME 20629 kill 20448 nsrexecd 15 SIGTERM
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20474 nsrmmd 15 SIGTERM
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20473 nsrmmd 15 SIGTERM SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20472 nsrmmd 15 SIGTERM SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20471 nsrmmd 15 SIGTERM
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20458 nsrsnmd 15 SIGTERM
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20448 nsrexecd 15 SIGTERM
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20448 nsrexecd 15 SIGTERM
Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 (nsrmmd): reads = 4, write = 3 SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20472 nsrmmd 20458 nsrsnmd 17 SIGCHLD
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20473 nsrmmd 20458 nsrsnmd 17 SIGCHLD SIGNAL RECIEVED SPID SNAME RPID RNAME SIGNUM SIGNAME 20458 nsrsnmd 20448 nsrexecd 17 SIGCHLD
2014 EMC Proven Professional Knowledge Sharing 48
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20448 nsrexecd 15 SIGTERM Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20448 nsrexecd 15 SIGTERM
Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20474 nsrmmd 15 SIGTERM
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20471 nsrmmd 15 SIGTERM
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20448 nsrexecd 15 SIGTERM
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20471 nsrmmd 1 init 17 SIGCHLD
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20474 nsrmmd 1 init 17 SIGCHLD
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20448 nsrexecd 15 SIGTERM
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 1 init 17 SIGCHLD (nsrmmd): reads = 0, write = 5 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process
2014 EMC Proven Professional Knowledge Sharing 49
PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20682 nsrexecd 20679 nsrexecd 15 SIGTERM SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20679 nsrexecd 20677 networker 17 SIGCHLD
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:0a4c:c52b Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:0a4c:c52b (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process
2014 EMC Proven Professional Knowledge Sharing 50
PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process
2014 EMC Proven Professional Knowledge Sharing 51
PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001 Total Counts: READS(4) WRITES(8) Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001
Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001 Pass 5: run completed in 20usr/380sys/106690real ms.
2014 EMC Proven Professional Knowledge Sharing 52
Conclusion
At the end of this article, I can confidently say that there is a limitless amount of information that
could be retrieved from either the operating system or the user applications that can assist in
gaining a deep understanding of how the OS is interacting with its components and also with the
user space applications that run on top of it. I couldn’t provide intensive examples on the
Avamar or NetWorker user space processes as the debugging information is not available for
either product because of the nature of their source code,
If the engineering team wishes to play around with these products and probe the product
functions on the runtime using SystemTap, it will be important to modify the build “Makefile” of
those products and add the option “-g” to the “GCC” C compiler in order to generate the
required debugging information to be used with SystemTap. Otherwise, SystemTap will be
unable to diagnose either the OS kernel or a user-space applications if the debugging
information is not installed on the system.
Appendix
1- SystemTap Language Reference https://sourceware.org/systemtap/langref 2- SystemTap Beginners Guide https://sourceware.org/systemtap/SystemTap_Beginners_Guide 3- http://doc.opensuse.org/products/draft/SLES/SLEStuning_sd_draft/cha.tuning.systemtap.html
2014 EMC Proven Professional Knowledge Sharing 53
EMC believes the information in this publication is accurate as of its publication date. The
information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION
MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO
THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an
applicable software license.