1

Abstracting the Content of System

Call Traces

Waseem FadelAbdelwahab Hamou-Lhadj

Department of Electrical and Computer EngineeringConcordia UniversityMontréal, QC, Canada{w_fadel, abdelw}@ece.concordia.ca

2

Objective of the Trace Abstraction and Analysis Track (Reminder)

Develop trace abstraction techniques and tools to facilitate the understanding and

analysis of the content of large event-based system call traces

3

Benefits Help users understand the behavioural aspects of a

system Allow automated comparison of multiple traces

based on the system behaviour and not a mere set of events

Monitor the system performance to detect service degradation

Ensure that subsequent revisions of a software system have not introduced programming errors

Compare traces of redundant servers, performing the same work in order to detect any malfunction possibly caused by a security breach

4

Proposed Approach

Linux KernelDocumentation

Sample LTTng Traces

Expert Knowledge

Trace Generated from

Linux Kernel(LTTng events)

Trace Abstraction Algorithm

Based on Pattern Matching and

Filtering of Noise

High-Level Trace

Pattern Library

5

Pattern Library

We built a pattern library that contains several patterns that represent key Linux kernel operations File, socket and process management operations

The patterns are modeled as state machines States represent system modes (user_mode, sys_call

mode, etc.) Events consist of LTTng events

6

Patterns we have so far

We completed the pattern library to include the following operations: File Management (Open, Read, Write, Seek,

Access, File Manipulation with fcntl, Stat, Close) Socket Management for both TCP and UDP

(Create, Connect, Bind, Listen, Accept, Send, Receive, Close)

Process Management (Execution with exec and execve, Exit, Fork, Clone, Get Resource Limit, Get Time of The Day, New UName, Unlink, Read Link)

7

Filtering of Trace Noise

We define noise in an LTTng trace as any event associated with memory management, page faults, and interrupts Are dependent on a specific kernel version Can occur anywhere in the trace and in any order Are treated similarly to the way utilities have been treated

in related work

Associated events are treated as a set i.e. order of occurrence of detailed events is ignored

8

Validation of Patterns

The patterns have been validated by Pierre-Marc Fournier and Mathieu Desnoyers from École Polytechnique de Montréal

Regular meetings with them have also helped in the process of understanding the markers, the system calls, and the main modes of execution

Both users agreed with the way we defined noise found in traces

9

The Linux Kernel Trace Abstraction Tool

The tool takes as input a trace generated from LTTng tracer

It applies our approach to that trace Outputs the trace in its abstracted format It has been developed in Java as an Eclipse

plug-in

10

The Linux Kernel Trace Abstraction Tool (cont.) The tool was designed to accept patterns defined as external XML files

<?xml version="1.0" encoding="ISO-8859-1"?><pattern name="Sample Pattern" type="HighLevelSampleConstrcut"

noise="false"> <event name="syscall_entry" syscall_name="sys_sample"

order="1" prev_state="IGNORE"> <current_state>SYSCALL_SAMPLE</current_state> </event>

<event name="sample" order="2" prev_state="SYSCALL_SAMPLE"> <current_state>SYSCALL_SAMPLED</current_state> </event>

<event name="syscall_exit" order="LAST" prev_state="SYSCALL_SAMPLED"> <current_state>USER_MODE_SAMPLED</current_state> </event>

</pattern>

11

The Linux Kernel Trace Abstraction Tool (cont.)

12

The Linux Kernel Trace Abstraction Tool (cont.)

The tool consists of three parts: The top part, through which the developer can

provide the tool with the required information The middle part, displays the trace resulting

from the abstraction process The bottom part, displays the information

related to the original and abstracted traces The following slides show the architecture

and the class diagram

13

Architecture

Partitioned both horizontally and vertically Easy to extend (to add new system calls

patterns or even patterns for different calls) Easy to maintain (to modify existing patterns

or the design itself)

14

Horizontal Partitioning

Horizontal partitioning is performed by defining the main domains of the system

15

Vertical Partitioning

16

Vertical Partitioning (cont.)

Divide the system into different layers Define the interfaces between layers Presentation layer can be developed without

affecting lower layers (Multiple GUIs can be provided for the same data)

Components from different layers can be designed, implemented and maintained independently

17Class Diagram

18

Class Diagram (cont.)

Adding new patterns and high-level constructs can easily be done by sub-classing the appropriate classes and interfaces

Multiple implementations representing different trace formats can be applied using the same interfaces

High level constructs are easy to further abstract

19

Case Studies

We applied our approach to large traces generated while running different processes

One process was the java virtual machine which was running a distributed file server and a client

Another process was the eclipse framework Third process was gedit Fourth process was GIMP image editor The final process was firefox

20

Case Studies (cont.)

ProcessInitial size

Size after Abstraction

Compression Ratio

Eclipse122698532590273%

GIMP84757522977873%

Firefox64671025728260%

Gedit1861679048251%

JVM47271303393%

21

Case Studies (cont.)

SEQ(1) Socket Create: family = 2, type = 2, protocol = 0, sock = 0xd563d340, ret = 8SEQ(1) Socket Connect: fd = 8, uservaddr = 0x80569c8, addrlen = 28, ret = 0SEQ(2) File Stat: ip = 0xb7f78430, syscall_id = 221 [sys_fcntl64+0x0/0xb0]SEQ(1) Get Time of Day: ip = 0xb7f78430, syscall_id = 78 [sys_gettimeofday+0x0/0x80]SEQ(1) Unknown Event: Event name: pollfd, Params: fd = 8SEQ(1) Unknown Event: Syscall name: sys_poll, Params: ip = 0xb7f78430, syscall_id = 168

[sys_poll+0x0/0xc0]SEQ(1) Socket Send: call = 9, a0 = 8SEQ(4) Unknown Event: Event name: pollfd, Params: fd = 8SEQ(1) Process Schedule: prev_pid = 0, next_pid = 23566, prev_state = 0

…

SEQ(1) Unknown Event: Syscall name: sys_futex, Params: ip = 0xb7f78430, syscall_id = 240 [sys_futex+0x0/0x130]

SEQ(1) Unknown Event: Syscall name: sys_clock_gettime, Params: ip = 0xb7f78430, syscall_id = 265 [sys_clock_gettime+0x0/0xa0]

SEQ(1) Get Time of Day: ip = 0xb7f78430, syscall_id = 78 [sys_gettimeofday+0x0/0x80]SEQ(1) File Access: ip = 0xb7f78430, syscall_id = 33 [sys_access+0x0/0x30]SEQ(1) File Write: fd = 19SEQ(1) Socket Receive: call = 10, a0 = 19SEQ(1) Process Exit: Process Exit: pid = 23580, cpu_id = 1, state = 1, Send Signal: pid = 23610, cpu_id

= 1, state = 1, Sched Try Wakeup: pid = 23972, cpu_id = 1, state = 1

A snapshot of the traces resulting from the abstraction of the JVM process

22

Remaining Challenges

Continuous improvement of the pattern library Defining additional patterns Dealing with new LTTng events

Using higher level constructs to further abstract the resulting traces

Improving the algorithm in terms of performance

Embedding the tool with the LTTv

23

Conclusion

We introduced techniques to abstract execution traces resulting from the Linux kernel

Our approach is based on building a pattern library that consists of patterns of the most common operations in Linux

We also defined noise patterns that result from memory management operations and page faults

We introduced an algorithm to abstract the system call traces by using the pattern library

We applied our techniques to traces generated from several processes

24

Pattern Library

25

File Management: Open & Close

SYSCALL_OPEN

SYSCALL_FILE_OPENED

USER_MODE_FILE_OPENED

syscall_entry(sys_open)

open

syscall_exit

Within a Process

SYSCALL_CLOSE

USER_MODE_FILE_CLSOED

syscall_entry(sys_close)

close

UESR_MODE_FILE_OPENED

SYSCALL_CLOSED

syscall_exit

Open File Close File

26

File Management: Read & Write

SYSCALL_READ

SYSCALL_DATA_READ

syscall_entry(sys_read)

read syscall_exit

UESR_MODE_FILE_OPENED

SYSCALL_WRITE

SYSCALL_DATA_WRITTEN

syscall_entry(sys_write)

syscall_exit

UESR_MODE_FILE_OPENED

write

Write to File Read from File

27

Socket Management (1)

SYSCALL_SOCKETCALL

SYSCALL_SOCKET_CREATION

USER_MODE_SOCKET_CREATED

syscall_entry(sys_socketcall)

socket_call (call = 1)

socket_create

Within a Process

SYSCALL_SOCKET_CREATED

syscall_exit

SYSCALL_SOCKETCALL

SYSCALL_SOCKET_BINDING

USER_MODE_SOCKET_BOUND

syscall_entry(sys_socketcall)

socket_call (call = 2)

socket_bind

Within a Process

SYSCALL_SOCKET_BOUND

syscall_exit

SYSCALL_SOCKETCALL

SYSCALL_SOCKET_LISTENING

USER_MODE_SOCKET_LISTENING

syscall_entry(sys_socketcall)

socket_call (call = 4)

socket_listen

Within a Process

SYSCALL_SOCKET_STARTED_LISTENING

syscall_exit

Create Bind Listen

28

Socket Management (2)

SYSCALL_SOCKETCALL

SYSCALL_SOCKET_CONNECTING

USER_MODE_SOCKET_CONNECTED

syscall_entry(sys_socketcall)

socket_call (call = 3)

socket_connect

Within a Process

SYSCALL_SOCKET_CONNECT

syscall_exit

SYSCALL_SOCKETCALL

SYSCALL_SOCKET_SEND

USER_MODE_SOCKET_SEND

syscall_entry(sys_socketcall)

socket_call (call=9 || call=11)

Within a Process

SYSCALL_DEV_XMIT

syscall_exit [napi_completed=true]

SYS_CALL_NAPI_SCHEDULE

napi_schedule

SYSCALL_NAPI_POLL

napi_poll

SYSCALL_DEV_RECEIVE

dev_receive

entry/napi_comleted=true

SYSCALL_NAPI_COMPLETE

napi_complete

SYSCALL_TIMER_SET

timer_set

timer_set

dev_xmit

Connect Receive Send

SYSCALL_SOCKETCALL

SYSCALL_SOCKET_RECEIVE

USER_MODE_SOCKET_RECEIVE

syscall_entry(sys_socketcall)

socket_call (call=10 || call=12)

dev_xmit

Within a Process

SYSCALL_DEV_XMIT

syscall_exit

SYS_CALL_NAPI_SCHEDULE

napi_schedule

SYSCALL_NAPI_POLL

napi_poll

SYSCALL_DEV_RECEIVE

dev_receive

SYSCALL_NAPI_COMPLETE

napi_complete

SYSCALL_TIMER_SET

timer_set

29

Socket Management (3)

entry/isBound=true

BindConnect

entry/isListening=true

Listen

Accept

Send

Receive

socket_call (call = 9)

socket_call (call = 9)

socket_call (call = 10)

socket_call (call = 10)

Close

SYSCALL_SOCKETCALL

sys_socketcall

socket_call (call = 1)

sys_socketcall

sys_socketcall

sys_close

socket_call (call = 9)

socket_call (call = 10)

sys_close

sys_close

sys_close

sys_close sys_close

Create

Within a Process

sys_socketcall

socket_call (call = 2)

socket_call (call = 3)

SYSCALL_SOCKETCALL [SOCKET_CREATED]

sys_close

socket_call (call = 4) [isBound==true] socket_call (call = 5) [isListening==true]

TCP Connection

30

Process Management (1)

Cloning/Forking

Within a Process

ptregs_clone

File Management

TCP SocketsWithin a Child Process

sys_waitpid

SYSCALL_WAIT

SYSCALL_WAITING

process_wait

UDP Sockets

Execution with execve

ptregs_execve

Exit Process

sys_exit_group

ptregs_clone

sched_schedule

SYSCALL_SCHED_SCHEDULE

USER_MDOE_SCHEDULED

syscall_exit

ptregs_clone

31

Process Management (2)

SYSCALL_EXEC

USER_MODE_EXEC

syscall_exit

Within a Process

exec

SYSCALL_EXECVE

USER_MODE_EXECVE

syscall_exit

Within a Process

syscall_entry(ptregs_execve)

SYSCALL_SCHED_TRY_WAKEUP

sched_try_wakeup

SYSCALL_SCHED_SCHEDULE

sched_schedule

SYSCALL_EXIT_GROUP

Within a Process

syscall_entry(sys_exit_group)

SYSCALL_PROCESS_EXIT

SYSCALL_SEND_SIGNAL

process_exit

send_signal

sched_try_wakeup

SYSCALL_SCHED_TRY_WAKEUP

Execution with exec with execve Exit Cloning

SYSCALL_CLONE

process_fork

Within a Process

syscall_entry(ptregs_clone)

SYSCALL_MIGRATE_TASK

sched_migrate_task

SYSCALL_PROCESS_FORK

SYSCALL_WAKEUP_NEW_TASK

sched_wakeup_new_task

syscall_exit

USER_MODE_FORK

sched_schedule (child process)

SYSCALL_SCHED_SCHEDULE

USER_MODE_SHCEDULED

syscall_exit

32

Other Patterns

File Control Read Link Unlink Get Resource Limit New UName

33

Thank You!

Questions?