dtrace talk at oracle open world

1

Thursday, September 23, 2010

<Insert Picture Here>

Oracle Performance Measurement and Tuning with Solaris DTraceJeff Savit Principal Sales Consultant

Angelo RajaduraiPrincipal Software Engineer

Thursday, September 23, 2010

3

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Thursday, September 23, 2010

4

<Insert Picture Here>

Agenda

• Basic Concepts of Performance Tuning• Performance Tuning Features in Solaris• DTrace - Dynamically Observing the Entire

Application Stack• Dynamic Tracing Oracle WebLogic Server with

DTrace• Further Reading

Thursday, September 23, 2010

5

Basic Concepts for Performance Management

Define Goals

Measure Performance

Analyze Model &Deduce

Take Action

PerformanceModel

Thursday, September 23, 2010

Start with the Best PlatformSolaris - Designed for Performance & Scale

6

Dynamic, multithreaded, preemptive kernel and process modelTightly integrated file system and virtual memory64-bit kernel, supporting both 32-bit and 64-bit applicationsResource managementOptimized, SMP-hot network stackCPU and memory scalability from low to high. Today, up to:

256 CPUs on T5440, 512 on M9000; 4TB RAM on M9000Optimized for modern multicore, multithread, NUMA systemsPremier x86 support optimized for Intel Nehalem and AMD OpteronSolaris Containers (Zones): negligible-overhead, built in virtualization

Thursday, September 23, 2010

• High system time is not a problem• You may be asking the system to do stuff

• Slow I/O is not a problem• You may be streaming bulk I/O

• 100% utilization is not a problem• You are using all the box you paid for :-)• Modern CPUs, NICs, disks, can do more than 1 thing at a

time without adding delay• Fine even if saturated unless some work unable to get

enough cycles to meet SLA

• None of these are problems unless they relate to business metrics

Define the Problem

7

Thursday, September 23, 2010

8

Native Apps

ruby

C C++

Database Oracle 11g

Operating System Solaris/OEL/Windows

Container WebLogic Server

Frameworks

Monitor: Right Location & Tools

PresentationJSP/Javascript/Ajax/JavaFX

python

php

perl

shell

Apps run in VM

Thursday, September 23, 2010

Solaris Tool chest

9

Proc tools

cputrack - per-processor hw counters pargs – process arguments pflags – process flags pcred – process credentialspldd – process's library dependencies plockstat – process lock statistics psig – process signal disposition pstack – process stack dumppmap – process memory map pfiles – open files and names prstat – process statistics ptree – process treeptime – process microstate times pwdx – process working directorypgrep – grep for processes pkill – kill processes list pstop – stop processes prun – start processes prctl – view/set process resources pwait – wait for processpreap – reap a zombie process

Tracing & Debugging

abitrace - trace ABI interfaces mdb – debug process/kernel

truss – trace function and system calls

System Statistics

acctcom – process accounting busstat – Bus hardware counters

cpustat – CPU hardware counters iostat – IO & NFS statistics

kstat – display kernel statistics mpstat – processor statistics

netstat – network statistics nfsstat – nfs server stats

sar – kitchen sink utility vmstat – virtual memory stats

Toolkits

DTraceToolkit – DTrace Tools K9Toolkit – perl perf tools

nicstat – network stats

Thursday, September 23, 2010

10

perfbarsee the big picture from across the room

Don't be shy about starting with a simple graphical tool!

http://blogs.sun.com/partnertech/resource/tools/perfbar.{sparc,i386}

See:LDOM Booth for use of perfbar

Thursday, September 23, 2010

11

FenxiReading a 1000 lines of text does not show you the big picture

Open source tool written at Sun for aggregating and displaying vmstat, mpstat, iostat data

https://fenxi.dev.java.net/

Thursday, September 23, 2010

12

So why do we need another tool?

Thursday, September 23, 2010

13

Native Apps

ruby

C C++

Database Oracle 11g

Operating System Solaris/OEL/Windows

Container WebLogic Server

Frameworks

Observing the Stack is hard

PresentationJSP/Javascript/Ajax/JavaFX

python

php

perl

shell

Apps run in VM

Thursday, September 23, 2010

The Blame Game

14

Devel

oper:

“I’d as

k the

datab

ase ad

min” Database Admin:

“May be the middleware guys

will know”

Sysadmin

“I don’t know! Ask the

Developer” Middlew

are Ex

pert:

“Lets

ask th

e Sysa

dmin”

Thursday, September 23, 2010

Static instrumentation is time consuming

15

Instrument Validate

Run in production

Collect infoProcess

Add Code Compile

Custom Instruments

Stop AppRestart w/ flags

Prebuilt Instruments

Thursday, September 23, 2010

Debugging Transient Problems In production. Impossible!

16

Please let me observe a live system.

Thursday, September 23, 2010

• turn instrumentation ON and OFF dynamically

• probe any arbitrary location• not just pre-defined probe points

• collect arbitrary data • have low probing overhead• have no overhead for disabled probes

If Only We Can ...

17

Thursday, September 23, 2010

Dynamic Tracing

18

Native

cc+ph

perl pyth

oru

b

shelDatabase

Operating System Solaris/Linux/OS X/

Container Glassfish/

Frameworks

Application Logic

PresentationJSP/Javascript/Ajax

12::SELECT id, name, symbol, conversion_rate FROM currencies WHEREstatus = 'Active' and deleted = 012::SELECT category, name, value FROM config12::SELECT id FROM outbound_email WHERE type = 'system'

-> foo()-> bar()<- bar()

<- foo()

Thursday, September 23, 2010

19

/ /predicate when

{

}

actions do what

D - Scripts

: : :provider module probe name whereprobe

Thursday, September 23, 2010

DTrace examples

20

nthreads{ printf("new thread created by %s(%d)\n", execname,pid);}

pid$1::malloc:entry{

@=quantize(arg0);}

malloc size observer

Thread monitor

pid$1:::entry{

@[probefunc]=count();}

hot methodfinder

Thursday, September 23, 2010

More DTrace examples

21

profile-101/execname==”java”/{

@[tid]=count();}

pid$1::sleep:entry{

@[ustack()]=count();}

Who went to sleep?

How many Java threads are active?

io:::done{

@=quantize(args[0]->b_bcount);}

Size distribution of IO

Thursday, September 23, 2010

DTrace and Java

• DTrace can observe the following in Java• When VM starts and ends• When thread starts and ends• When class gets loaded and unloaded• When object allocated and freed• When GC starts and ends• JNI calls• When a method is called and method returns

22

Thursday, September 23, 2010

DTrace and Java - Example

23

hotspot$1:::class-loaded{ printf("%s loaded\n", copyinstr(arg0,arg1));}

hotspot$1:::method-entry{

@[copyinstr(arg1,arg2), copyinstr(arg3,arg4)]=count();}

JVM needs to be started with -XX:+ExtendedDTraceProbes

Class Loading

Java Methods

Thursday, September 23, 2010

DTrace and Java - More Example

24

GC

#!/usr/sbin/dtrace -qshotspot$1:::gc-begin{ printf("GC started at %Y\n",walltimestamp); printf("%20s | %-10s\n", "pool", "time(ms)"); printf(" ------------------------------------\n"); self->ts = timestamp;}

hotspot$1:::mem-pool-gc-begin{ self->pool_ts[copyinstr(arg2,arg3)] = timestamp;}

hotspot$1:::mem-pool-gc-end{ self->pool = copyinstr(arg2,arg3); printf("%20s | %-10d\n", self->pool, (timestamp - self->pool_ts[self->pool])/1000);}

hotspot$1:::gc-end/self->ts/{ printf(" ------------------------------------\n"); printf("%20s | %-10d\n\n","Total GC time", (timestamp - self->ts)/1000);}

Thursday, September 23, 2010

Other Providers

25

DTrace providers

javaperlruby

javascript mysqlpostgres

cpciptcpudp

plockstatnfsv3

nfsv4

kerberos

sh

iSCSI

USDT

FC

DTracecore provider

DTracecore

IOsyscall

profilesysinfo

vminfo

PID

mib

fbtproc

lockstat

python

Thursday, September 23, 2010

DTrace Eco-Systems

26

• DTraceToolkit• Collection of useful D-scripts

• Chime• DTrace Data Visualization

• D-Light• Sun Studio based tool to observe application using DTrace

• DTrace analytics• DTrace based front end for Sun ZFS storage

Thursday, September 23, 2010

DTrace & Oracle WebLogic Server

27

Thursday, September 23, 2010

Instrumenting JVM vs Byte Code

28

JVM

Garbage Collector

Class Loader

JIT Compiler

Execution Engine

Byte Code

gc-begingc-end

class-loadedclass-unloaded

method-compile-beginmethod-compile-end

method-entrymethod-return

Thursday, September 23, 2010

Instrumenting JVM vs Byte Code

29

JVM

Garbage Collector

Class Loader

JIT Compiler

Execution Engine

Byte Code @OnMethod{ clazz=”myclass”, method=”mymeth”}

BTrace Supports other annotations@OnTimer, @OnError, @OnEvent & @OnLowMemory

Thursday, September 23, 2010

30

{

}

action do what

BTrace Code Structure

@OnMethod ( clazz = “java.lang.thread”,method = “start” where

import com.sun.btrace.annotations.*;import statuc com.sun.btrace.BTraceUtil.*;@BTracepublic class BTraceSample {

)

public static void func()

Thursday, September 23, 2010

Sample B-script

31

import com.sun.btrace.annotations.*;import static com.sun.btrace.BTraceUtils.*;

@BTrace public class ThreadStart { @OnMethod( clazz="java.lang.Thread", method="start" ) public static void onnewThread(@Self Thread t) { println(strcat("starting ", name(t))); }}

New Java Thread

Tracer

Thursday, September 23, 2010

More B-scripts

32

@BTrace public class AllMethods { private static Aggregation count = newAggregation(AggregationFunction.COUNT);

@OnMethod( clazz="/javax\\.swing\\..*/", method="/.*/" ) public static void m(@ProbeClassName String probeClass, @ProbeMethodName String probeMethod) { AggregationKey key = newAggregationKey(strcat(probeClass,strcat(".",probeMethod))); addToAggregation(count,key,1); }

@OnEvent public static void onEvent() { truncateAggregation(count,10); printAggregation("Count", count); }}

Count all javax.swing methods

Thursday, September 23, 2010

DTrace Vs BTrace

33

DTrace for JVM BTrace

Dynamic tracing entire system Only within a Java Process

Instrument JVM Instrument Java Byte Code

D-script Java like instrumentation Lang

Link Java to entire stack Just within Java

High overhead for method tracing

Low overhead for method tracing

Thursday, September 23, 2010

Combining DTrace & BTrace

34

DTrace BTraceBest

ofBoth

Worlds

DFr

ame

Thursday, September 23, 2010

Dynamic Tracing Framework for Oracle WebLogic Server

35

Thursday, September 23, 2010

The DFrame config file• The Framework is controlled using the config file• Two types of probes

• mapped• config file format

class:method:entry/exit:logical name• probe format

weblogic*:::event(logical name, class.method)• unmapped

• config file formatclass:method

• probe formatweblogic*:::method-entry(class, method)weblogic*:::method-return(class,method)

36

Thursday, September 23, 2010

How to use the Framework• Only one command to remember

embedProbes• Usage

embedProbes <pid>embedProbes -f configFile <pid>embedProbes -d [debug on]embedProbes -B [save B-script]

• The command reads config file, creates the b-script and embedded dprobes into weblogic

37

Thursday, September 23, 2010

38

Demonstration

Thursday, September 23, 2010

Reducing the Observer Effect

39

• Use low overhead tools✓dframe overhead for most probes very low (<3%)

• Ability to dynamically enable and disable probes✓dframe can be turned on and off dynamically

• Measure the probe effect before running in production✓ex: for medrec application on a Sun T5140

for processing 2016 transactions,this is 487,240 events the overhead us just %3.8

Thursday, September 23, 2010

Learn more

• White paper on DFrame– http://bit.ly/DFrame

• Need pre-release access?– email dframe-interest_us@oracle.com

40

Thursday, September 23, 2010

DTrace Resources

41

• DTrace Community Page• http://hub.opensolaris.org/bin/view/Community+Group+dtrace/

• DTrace Documentation• http://wikis.sun.com/display/DTrace/Documentation

• DTraceToolkit• http://hub.opensolaris.org/bin/view/Community+Group+dtrace/dtracetoolkit

• DTrace Chime• http://hub.opensolaris.org/bin/view/Project+dtrace-chime/

• DTrace Hands on Lab• http://dtracehol.com/ - AMI ami-22fd164b (dtrace-hol)

Thursday, September 23, 2010

42

Thursday, September 23, 2010