production time profiling
TRANSCRIPT
Production time profiling On-Demand with Java Flight Recorder
Klara Ward Principal Software Developer Java Platform Group, Oracle
Oracle Confidential – Internal/Restricted/Highly
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Using Java Mission Control & Java Flight Recorder
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
About me
– Developer in the Oracle Java Mission Control team in Stockholm, Sweden
– Sometimes tweets @klaraward
Celebrating Java 20 Years at JFokus
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
5
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Agenda
Overview of Java Mission Control
Overview of Java Flight Recorder
Demo
Customization, Future, Links
Q&A
1
2
3
4
5
6
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
“I managed to do in one day what I've failed to do in 2+ weeks using <profiling tool> and <another profiling tool>.”
“ JMC is my main tool for getting insight into the rhythm of a JVM and the running applications. … I have used recordings to resolve critical production issues caused by latency, memory-leaks or threading. ”
Maurizio Cimadamore, Oracle (Java LangTools)
Allan Thrane Andersen, Trygg
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
RebelLabs Developer Productivity Report 2015, Java Performance Survey
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
”Java Mission Control profiling tool”
Probably:
– Data from Java Flight Recorder
– Visualized in Java Mission Control
What do they mean?
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Java Flight Recorder (JFR) & Java Mission Control (JMC) Brief overview
JMC
JDK
JVM
Low overhead
JFR Engine
JFR Events
Recording data myrecording.jfr
Control recordings start/stop/dump
java -XX:+FlightRecorder JDK/bin/jmc
or Eclipse plug-ins
JDK/bin/jcmd <pid> <cmd>
java -XX:StartFlightRecording
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Overview of Java Mission Control The graphical client
11
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Java Mission Control Overview
• A tools suite for production use (fine in development too)
– Basic monitoring
– Production time profiling and diagnostics
• Free for development and evaluation
– Tool usage is free, data creation in production
requires a commercial license
tiny.cc/javalicense
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
History of Mission Control
• JRockit Flight Recorder
• Appeal(JRockit) -> BEA Systems -> Oracle <- Sun
• Best JRockit features -> HotSpot JVM
• JFR and JMC released with 7u40, September 2013
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Java Mission Control Main Tools
Two main tools:
• JMX Console
– Online monitoring
• Flight Recorder
– Offline low overhead profiler
– Control and visualization in JMC
• JRockit Mission Control also had the Memory Leak Analyzer
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Experimental Plugins
• DTrace
– JFR style visualization of data produced by DTrace
• JOverflow
– Memory anti-pattern analysis from hprof dumps
• JMX Console plug-ins
• Java Flight Recorder plug-ins
– WLS
– JavaFX
Downloadable from within Mission Control
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
JMC installation/startup
<JDK>/bin/jmc
– Mac: (/usr/bin/) jmc
Add if needed:
–consoleLog –debug ( | more 2>&1 )
• Eclipse plug-ins
– Install from update site on OTN: http://oracle.com/missioncontrol, Eclipse Update Site
• Experimental plug-ins: Install from within the JMC app, or from
https://oracle.com/missioncontrol, Eclipse Experimental Update Site
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Overview of Java Flight Recorder Low overhead profiling
17
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Flight Recorder
• High Performance Event Recorder
• Built into the JVM
– Already available runtime information
– Measuring the real behavior, doesn’t disable JVM optimizations
• Binary recordings
– Self contained self describing chunks
• Very detailed information
• Extremely low overhead (~1-2%)
– Can keep it always on, dump when necessary
101
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Java Flight Recorder (JFR) & Java Mission Control (JMC) Brief overview
JMC
JDK
JVM
JVM
Events
JFR
Engine Recording data
myrecording.jfr
Control recordings start/stop/dump
java -XX:+FlightRecorder JDK/bin/jmc
or Eclipse plug-ins
JDK/bin/jcmd <pid> <cmd>
java -XX:StartFlightRecording
JFR Java API
JFR JMX API
Java
Events
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Data collected by JFR
• Java application behavior
– Threads/Locks
– I/O
– Exceptions
• JVM behavior (indirect Java application behavior)
– Garbage collection, allocation
– JIT Compiler
Implemented by the different subsystem teams
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Method sampling
• Sampling profiler
– Not displaying every single call to your method
• This is partly why we get the low overhead
• Detects hot methods
– Does not require threads to be at safepoints
• (Flags currently needed to give more accurate non-safepoint data)
-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints
(will be default in coming releases)
• Not sampling threads in native
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
RebelLabs Developer Productivity Report 2015, Java Performance Survey
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
RebelLabs Developer Productivity Report 2015, Java Performance Survey
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Different Kinds of Events
• Instant Event - Exception
• Duration Event – Thread.sleep
– Configurable threshold
• Requestable Event – Method profiling sample
– Polled from separate thread
– Configurable period
• Period and threshold settings impact the performance overhead
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Event settings
• Predefined settings
– ‘default’ – designed to get max information within <= 1 % overhead
– ‘profile’ – even more information, ~2 % overhead
– Enabling of event types, configuring periods and thresholds
– jre/lib/jfr/*.jfc
• Design your own from the Mission Control GUI
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Different Kinds of Recordings
• Continuous Recordings
– Have no end time
– Must be explicitly dumped
– Example use case: Enable at startup, dump the last X minutes when needed
• Time Fixed Recordings (‘profiling recordings’)
– Have a fixed time
– If started from Java Mission Control, opened automatically in the GUI
– Example use case: Performance testing under load, do a 1 minute recording
• Resulting *.jfr file also called “Recording”
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Creating recordings More than one way
27
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Preparations
• Start the JVM from which to get recordings with: -XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
• In 8u40 and later, possible to enable at runtime if needed
– Using JMC or jcmd
– “On-demand”
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Creating Recordings Using Mission Control
1. Find a JVM to do a recording on in the JVM Browser
2. Double click the Flight Recorder node under the JVM
3. Follow the wizard
NEW: No need for the JVM flags,
automatic enablement from JMC
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Creating Recordings Using Startup Flags
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
• Time fixed
– -XX:StartFlightRecording= delay=20s,duration=60s,
filename=C:\tmp\myrecording.jfr,settings=profile,name=JavaLand
• Continuous w/ dumponexit – -XX:StartFlightRecording=settings=default
-XX:FlightRecorderOptions=dumponexit=true,
dumponexitpath=C:\tmp\myrecordings
– (Needed before 8u20: -XX:FlightRecorderOptions=defaultrecording=true )
• See documentation for Java options (google “java options”)
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Creating Recordings Using JCMD
Usage: jcmd <pid> <command>
• Starting a recording: jcmd 7060 JFR.start name=MyRecording settings=profile
delay=20s duration=2m filename=c:\tmp\myrecording.jfr
• Dumping a recording: jcmd 7060 JFR.dump name=MyRecording
filename=C:\tmp\dump.jfr
• Unlocking commercial features (if JVM not started with the flag):
jcmd 7060 VM.unlock_commercial_features
Don’t forget to try jcmd <pid> help to see what else jcmd can do
Useful for controlling JFR from the command line
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Creating Recordings Using JMX Console triggers
• Start JMC
• Connect a JMX Console to your application
• Configure and enable rules on the Triggers tab
• Recording will be started or dumped when the trigger occurs
When realtime monitoring your application
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Remote production systems
• Remote access – either using -Dcom.sun.management.jmxremote…, connect with JMC
– or start recording from local commandline using jcmd, transfer JFR to workstation
• Enabling JFR
– at startup • some very small initiation overhead at startup, threads allocate small amount of
extra memory
– dynamically • 0% overhead at startup, initiation overhead happens at time of enabling, might
cause some classes to be deoptimized etc.
• If you want to avoid restart
No GUI, security restrictions
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Analyzing recordings Using the graphical client
34
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
How to think about the information shown
• Only you know what your application is supposed to be doing
– Batch job, or real time trading?
– Do you want the CPU usage to be high or low?
– If you have a theory about what is wrong, you can find out why
When analyzing Flight Recordings
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
How to think about the information shown
• CPU load
– Low load – method sampling not very interesting
– Full load – latencies not very interesting/likely
• Event thresholds
– Keeping performance up but still detecting outliers
– Is the thread running normally during 30%?
– Hint: Default threshold is 10 ms.
Common pitfalls
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
DEMO! Flight Recorder startup and analysis
37
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 39
import com.oracle.jrockit.jfr.*; public class Example { private final static String PRODUCER_URI = "http://www.example.com/demo/"; private Producer myProducer; private EventToken myToken; public Example() throws URISyntaxException, InvalidEventDefinitionException, InvalidValueException { myProducer = new Producer("Demo Producer", "A demo event producer.", PRODUCER_URI); myToken = myProducer.addEvent(MyEvent.class); } @EventDefinition(path="demo/myevent", name = "My Event", description="An event triggered by doStuff.", stacktrace=true, thread=true) private class MyEvent extends TimedEvent { @ValueDefinition(name="Message", description="The logged important stuff.") private String text; public MyEvent(EventToken eventToken) { super(eventToken); } public void setText(String text) { this.text = text; } } public void doStuff() { MyEvent event = new MyEvent(myToken); event.begin(); String importantResultInStuff = ""; // Generate the string, then set it... event.setText(importantResultInStuff); event.end(); event.commit(); } }
Adding Your Own Events (unsupported)
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 40
... private MyEvent event = new MyEvent(myToken); public void doStuffReuse() {
event.reset(); event.begin(); String importantResultInStuff = ""; // Generate the string, then set it... event.setText(importantResultInStuff); event.end(); event.commit();
} ...
• NB: If JFR is not enabled, the custom events gives no (0%) overhead
Adding Your Own Events (unsupported) Reusing the event (not thread safe)
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 41
• The JDK parser • import oracle.jrockit.jfr.parser.*; • SAX style parser
• The JMC parser • import com.jrockit.mc.flightrecorder.FlightRecording; •DOM style parser
• http://hirt.se/blog/?p=446
Parsing recordings (unsupported) Two unsupported options
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Built in GUI editor (unsupported)
• Show view -> Designer
• Customize the existing GUI or produce entirely new GUIs for events
• Export the created GUI to share it with others
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Future
• JFR - Supported API for adding your own JFR events
• JMC - Automatic analysis of Flight Recordings
–”You have used a JVM flag that is not recommended”
–”Your application is doing a lot of GC, investigate more here”
– ...
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Resources Homepage:
http://oracle.com/missioncontrol (Click Discussion to see the forum)
http://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/index.html
Twitter: @javamissionctrl
@klaraward
@hirt
Blog: http://hirt.se/blog? (http://hirt.se/blog/?p=611 for JMC tutorial)
Facebook: https://www.facebook.com/javamissionctrl
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 46
Questions?
Q: Can you repeat those JVM flags again? A: -XX:+UnlockCommercialFeatures –XX:+FlightRecorder
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
47