Download - Java black box profiling
![Page 2: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/2.jpg)
The Problem
• There are application• It doesn’t meat its SLA• Users/client/stakeholders are unhappy
You are one to fix it!
![Page 3: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/3.jpg)
What to do?
• Do not panic!• Write out all moving parts in system• Elaborate acceptance criteria• Make sure you understand KPI and SLA• Is problem in Java application?
Yes - You are ready to start profiling
![Page 4: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/4.jpg)
Moving parts
ServerBrowser
Application
Database
Caching
JavaScript
HTTP
Network Network
SQL performance
CPU Memory / Swapping
Disk IO Virtualization
![Page 5: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/5.jpg)
Understanding targets
• Business transaction ≠ Page• Business transaction ≠ SQL transaction• Page ≠ HTTP request• HTTP request ≠ SQL transaction
Make sure you know how your KPI are translating into milliseconds
![Page 6: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/6.jpg)
Just be before you start profiling
Take a deep breath Make sure problem is in Java part Make sure you understand end goal
These points are trivial so it is very easy to forget them under stress.Do not make that mistake!
![Page 7: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/7.jpg)
Few more rule before start
Every system is a black box Do you think that know what is inside? You don’t!
There 3 kinds of data produced by profiling Lies Darn lies Statistics
Data incorrectly interpreted
Data incorrectly collected
![Page 8: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/8.jpg)
Profiling
![Page 9: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/9.jpg)
Reproducing performance issue
Stationary process• Uniform load reasonable simulating real one• Goal is optimizing average performance indicators
Transition process• Goal is optimize prolonged process• Load profile is changing over time systematically
Profiling single operation• Goal is to optimize particular operation
![Page 10: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/10.jpg)
Profiling in JVM
• Stack trace sampling• Byte code instrumentation• Thread MBeans• GC logs• Heap dumps / Heap histogramMore diagnostic sources• JMX• Performance counters• Flight recorder / mission control
![Page 11: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/11.jpg)
Stack trace sampling
• Work best with stationary process• Uniformly impacts system under test• Do not require “calibrating”• Result from multiple runs could be combined• You deal with probabilities, not hard numbers• Measure wall clock, not CPU time
![Page 12: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/12.jpg)
Byte code instrumentation
• Useful for all types of experiments• Produce some absolute numbers
Number of calls is accurate Time is predictably skewed
• Significantly skew performance of system under test• Require careful targeting• Could measure CPU time
![Page 13: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/13.jpg)
Threading MBean
Standard JVM Threading MBean CPU usage per thread (user/sys) Memory allocation per thread Blocking / waiting statistics
![Page 14: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/14.jpg)
Tools
Swiss Java Knife (SJK) - https://github.com/aragozin/jvm-tools
• Thread monitor• Stack Trace Recorder/Analyzer• Dead object histogram• and more
BTrace - https://kenai.com/projects/btrace • Incrementing profiler• Used via CLI or API• Scriptable in Java
![Page 15: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/15.jpg)
Profiling strategy
Classify your bottleneck CPU• Application CPU usage• Excessive memory allocation / GC
Disk IO Network latency Network bandwidth Inter thread contention
![Page 16: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/16.jpg)
Recipes: CPU hogs
1. You sampling to identify suspects• Frame frequency histogram + meditation• Call tree + deep meditation• Iterative classification
2. Instrument methods to get hard prove• relative number of calls between methods
3. Iterate to pin point root cause
![Page 17: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/17.jpg)
Recipes: CPU hogs
Iterative classification (JBoss + Seam)
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Base
Other
DefaultServlet.doGet
LifeCycleImpl (render)
LifeCycleImpl (execute)
Business logic
Seam interceptor (lock contention)
Seam interceptor (inject/outject)
Resource bundle (getObject)
Resource bundle (missing)
Facelet compile
Hibernate (rest)
Hibernate (autoFlush)
JDBC
![Page 18: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/18.jpg)
Recipes: CPU hogs
Iterative classification (JBoss + Seam)
[ROOT] org.apache.catalina.connector.CoyoteAdapter.service
[JDBC] org.jboss.jca.adapters.jdbc.
[Hibernate (autoFlush)] org.hibernate.internal.SessionImpl.autoFlushIfRequired
[Hibernate (rest)] org.hibernate.
[Facelets compile] com.sun.faces.facelets.compiler.Compiler.compile
[XML] javax.xml.
[ResourceBundle - missing resource] java.util.ResourceBundle.getObject(ResourceBundle.java:395)
[ResourceBundle - getObject] java.util.ResourceBundle.getObject
[Seam bean interceptor - inject/disinject/outject] !REQUIRE !LAST org.jboss.seam.core.BijectionInterceptor.aroundInvoke org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke !NOT FOLLOWED **.proceed !ANY org.jboss.seam.Component.inject
org.jboss.seam.Component.disinject org.jboss.seam.Component.outject
[Seam bean interceptor - lock contention] !REQUIRE !LAST org.jboss.seam.core.BijectionInterceptor.aroundInvoke org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke !NOT FOLLOWED **.proceed java.util.concurrent.locks.ReentrantLock [Seam bean interceptor - other] !REQUIRE !LAST org.jboss.seam.core.BijectionInterceptor.aroundInvoke org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke !NOT FOLLOWED **.proceed
[WorkItemController.doselect] org.cp.shark.WorkItemController.doselect
[LifecycleImpl.execute] com.sun.faces.lifecycle.LifecycleImpl.execute
[LifecycleImpl.render] com.sun.faces.lifecycle.LifecycleImpl.render
[DefaultServlet.doGet] org.apache.catalina.servlets.DefaultServlet.doGet
[other]
![Page 19: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/19.jpg)
Recipes: Memory issues
1. Indentify thread producing garbage• SJK: ttop command
2. Classify garbage• Class histogram of dead object in heap• SJK: hh --dead
3. Investigate suspect classes• Heap dump• Debugging / Instrumentation
![Page 20: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/20.jpg)
Recipes: contention
• Stack trace sampling Good for “bad” contention cases
• Thread MBean Contention statistics could be enabled
• BTrace Analyze lock access pattern using your
codebase knowledge
![Page 21: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/21.jpg)
Sub millisecond profiling
Low latency profiling requires indirect measuring.You should not measure sub ms directly!
• Make a hypothesis• Arrange experiment to prove / disprove• Collect aggregated metrics• Compare, analyze and iterate
![Page 22: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/22.jpg)
Flight record / Mission Control
Tight integration with JVM• Low overhead• Access to intimate JVM areas
Well balanced set of metrics• Unobtrusive• Indicative
Good user interfaceMay be one day I will abandon my SJK
![Page 23: Java black box profiling](https://reader036.vdocuments.us/reader036/viewer/2022062313/557fbaf0d8b42a40118b4b41/html5/thumbnails/23.jpg)
Thank you
Alexey Ragozin [email protected]
http://blog.ragozin.info- my articleshttp://aragozin.timepad.ru- community events in Moscow