java black box profiling

Java black box profiling

Alexey Ragozinalexey.ragozin@gmail.com

The Problem

• There are application• It doesn’t meat its SLA• Users/client/stakeholders are unhappy

You are one to fix it!

What to do?

• Do not panic!• Write out all moving parts in system• Elaborate acceptance criteria• Make sure you understand KPI and SLA• Is problem in Java application?

Yes - You are ready to start profiling

Moving parts

ServerBrowser

Application

Database

Caching

JavaScript

HTTP

Network Network

SQL performance

CPU Memory / Swapping

Disk IO Virtualization

Understanding targets

• Business transaction ≠ Page• Business transaction ≠ SQL transaction• Page ≠ HTTP request• HTTP request ≠ SQL transaction

Make sure you know how your KPI are translating into milliseconds

Just be before you start profiling

Take a deep breath Make sure problem is in Java part Make sure you understand end goal

These points are trivial so it is very easy to forget them under stress.Do not make that mistake!

Few more rule before start

Every system is a black box Do you think that know what is inside? You don’t!

There 3 kinds of data produced by profiling Lies Darn lies Statistics

Data incorrectly interpreted

Data incorrectly collected

Profiling

Reproducing performance issue

Stationary process• Uniform load reasonable simulating real one• Goal is optimizing average performance indicators

Transition process• Goal is optimize prolonged process• Load profile is changing over time systematically

Profiling single operation• Goal is to optimize particular operation

Profiling in JVM

• Stack trace sampling• Byte code instrumentation• Thread MBeans• GC logs• Heap dumps / Heap histogramMore diagnostic sources• JMX• Performance counters• Flight recorder / mission control

Stack trace sampling

• Work best with stationary process• Uniformly impacts system under test• Do not require “calibrating”• Result from multiple runs could be combined• You deal with probabilities, not hard numbers• Measure wall clock, not CPU time

Byte code instrumentation

• Useful for all types of experiments• Produce some absolute numbers

Number of calls is accurate Time is predictably skewed

• Significantly skew performance of system under test• Require careful targeting• Could measure CPU time

Threading MBean

Standard JVM Threading MBean CPU usage per thread (user/sys) Memory allocation per thread Blocking / waiting statistics

Tools

Swiss Java Knife (SJK) - https://github.com/aragozin/jvm-tools

• Thread monitor• Stack Trace Recorder/Analyzer• Dead object histogram• and more

BTrace - https://kenai.com/projects/btrace • Incrementing profiler• Used via CLI or API• Scriptable in Java

Profiling strategy

Classify your bottleneck CPU• Application CPU usage• Excessive memory allocation / GC

Disk IO Network latency Network bandwidth Inter thread contention

Recipes: CPU hogs

1. You sampling to identify suspects• Frame frequency histogram + meditation• Call tree + deep meditation• Iterative classification

2. Instrument methods to get hard prove• relative number of calls between methods

3. Iterate to pin point root cause

Recipes: CPU hogs

Iterative classification (JBoss + Seam)

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Base

Other

DefaultServlet.doGet

LifeCycleImpl (render)

LifeCycleImpl (execute)

Business logic

Seam interceptor (lock contention)

Seam interceptor (inject/outject)

Resource bundle (getObject)

Resource bundle (missing)

Facelet compile

Hibernate (rest)

Hibernate (autoFlush)

JDBC

Recipes: CPU hogs

Iterative classification (JBoss + Seam)

[ROOT] org.apache.catalina.connector.CoyoteAdapter.service

[JDBC] org.jboss.jca.adapters.jdbc.

[Hibernate (autoFlush)] org.hibernate.internal.SessionImpl.autoFlushIfRequired

[Hibernate (rest)] org.hibernate.

[Facelets compile] com.sun.faces.facelets.compiler.Compiler.compile

[XML] javax.xml.

[ResourceBundle - missing resource] java.util.ResourceBundle.getObject(ResourceBundle.java:395)

[ResourceBundle - getObject] java.util.ResourceBundle.getObject

[Seam bean interceptor - inject/disinject/outject] !REQUIRE !LAST org.jboss.seam.core.BijectionInterceptor.aroundInvoke org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke !NOT FOLLOWED **.proceed !ANY org.jboss.seam.Component.inject

org.jboss.seam.Component.disinject org.jboss.seam.Component.outject

[Seam bean interceptor - lock contention] !REQUIRE !LAST org.jboss.seam.core.BijectionInterceptor.aroundInvoke org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke !NOT FOLLOWED **.proceed java.util.concurrent.locks.ReentrantLock [Seam bean interceptor - other] !REQUIRE !LAST org.jboss.seam.core.BijectionInterceptor.aroundInvoke org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke !NOT FOLLOWED **.proceed

[WorkItemController.doselect] org.cp.shark.WorkItemController.doselect

[LifecycleImpl.execute] com.sun.faces.lifecycle.LifecycleImpl.execute

[LifecycleImpl.render] com.sun.faces.lifecycle.LifecycleImpl.render

[DefaultServlet.doGet] org.apache.catalina.servlets.DefaultServlet.doGet

[other]

Recipes: Memory issues

1. Indentify thread producing garbage• SJK: ttop command

2. Classify garbage• Class histogram of dead object in heap• SJK: hh --dead

3. Investigate suspect classes• Heap dump• Debugging / Instrumentation

Recipes: contention

• Stack trace sampling Good for “bad” contention cases

• Thread MBean Contention statistics could be enabled

• BTrace Analyze lock access pattern using your

codebase knowledge

Sub millisecond profiling

Low latency profiling requires indirect measuring.You should not measure sub ms directly!

• Make a hypothesis• Arrange experiment to prove / disprove• Collect aggregated metrics• Compare, analyze and iterate

Flight record / Mission Control

Tight integration with JVM• Low overhead• Access to intimate JVM areas

Well balanced set of metrics• Unobtrusive• Indicative

Good user interfaceMay be one day I will abandon my SJK

Thank you

Alexey Ragozin alexey.ragozin@gmail.com

http://blog.ragozin.info- my articleshttp://aragozin.timepad.ru- community events in Moscow