distributed jvm monitoring - rainfocus...“once more @javamissionctrlis saving my day!...

Copyright©2017, Oracleand/oritsaffiliates.Allrightsreserved. |

DistributedJVMMonitoringandProfilingMadeSimpleCON6836

MarcusHirtJavaProductGroup

DaveCabelusCloudNativeApplicationDevelopment

October2,2017

JavaOneOctober2017 3


SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.



JavaPerformanceTroubleshootingChallenges• Memory

– ToofrequentGC,longpausesforGC,Allocationhotspots

– Leakedmemory

• Concurrency– Threaddeadlocks– Threadgridlocks– CPUunderutilization– Threadpoolmisconfiguration(toomany/toofewthreads)


Thesearehardtosolve,especiallyinproduction!

Photocredit:https://www.flickr.com/photos/ofernandezberrios/

Copyright©2017, Oracleand/oritsaffiliates.Allrightsreserved. | JavaOneOctober2017 6

Nowaddthosechallengestonewdevelopmentandhostingtrends

Containers Microservices Serverless/Functions Cloudscale

Alongtimeago… Presentdaycontainer-native…

WebServerorLoad

Balancer

PresentationTier

BusinessLogic DataTier

OrderError

Function

CreditFraud

FunctionInventoryLocationService

BillingService

ShippingService

UserinfoService

ProductDisplayService Product

RatingService

CustomDiscountFunction

ProductPricingService

DataValidationFunction

ProductSearchServiceProductSearchService

ProductSearchService

ProductCatalogService




“Thebigchallengeisnolongerreallyperformance.Thebigchallengeisprofiling,andespeciallyprofilinginproduction.”

- TonyPrintezis,JVMengineer,Twitter(Devoxx 2015,”Lifeof aTwitterJVMEngineer”,49:49)



JavahasPowerfulBuilt-inProfilingCapabilities• JavaFlightRecorder(JFR)

– HighlyEfficientDataRecorder,builtintotheJavaRuntime

• JavaMissionControl(JMC)– Theclientapplication– VisualizationofFlightRecordings– JMXconsole– Heapdumpanalyzer– ...

9


FlightRecorderPerformance

• BuiltintotheJVM/JDK,bythepeopledevelopingtheJVM• Highperformanceflightrecordingengineandhighperformancedatacollection– Accesstodataalreadycollectedintheruntime– Threadlocalnativebuffers– InvariantTSCfortimestamping–Methodprofilingdataevenfromoutsidesafe-points– Fasterandmoreaccurateallocationprofiling(scalarizationnotundonebyprofiler)

ExtremelyLowOverhead


"JavaMissionControlisthebestprofilerbyfar.“- TJakeLuciani,PMCCassandra

“JavaMissionControlismycurrentfavourite profilerfortheOracleJVM."- Nitsan Wakart,Azul

“…Ourreal-timemessagingproductscanpublishmillionsofmessagesasecondtomanythousandsofconnections- onlyJMCcankeepupwiththislevelofload."- PhilAston,ProductArchitect,PushTechnology

"JMCnotonlysavestimetryingtoresolveperformanceissuesandbugs,itcangiveyouadetailedviewonyourapplicationyoucannotgetwithothercommercialprofilers"- PeterLawrey,CEO,ChronicleSoftware

“Oncemore@javamissionctrl issavingmyday!"- MichaelNitschinger,SDKEngineer,Couchbase

"Fortherecord:JavaMissionControlisthebestprofilerever,Iuseitdaily,andsoshouldyou"

- MarcusLagergren,LeadArchitect,Nasdaq


"I am ACS engineer since 2008, delivering local Middleware support to several customers. Since I started to work with Java/JRockit Mission Control, it became a key tool for my work, helping me to troubleshooting, identifying root causes and bottlenecks, and also for doing proactive follow up services to customers. Without it, I would be blind." - Iratxe Etxebarria, Oracle (ACS)

"In Fusion we create hundreds of thousands of Flight Recordings, and we can figure out 95% of the issues using nothing but the recordings." - Joe Albowicz, Oracle (Fusion Application Development)


NewAvailability forJVMProfiling TechnologyBoth JFRandJMCwill beinOpen Source!• Current target forJFRis18.9• Current target forJMCis18.9• Both are subject tochange without notice(due dilligence - code reviews,IPreviews,license incompatibilities,etc)


JDK9Upgrades

• New,supported,APIsfor:– controlling therecorder– custom events– consuming recordings

• Performance improvements• Newevents• Advanced APIforconsuming recordings• Automated Analysis of FlightRecordings

14

JFRandJMC


APIImprovements,Performance

15


NewAPIforCustomEvents

16

Example

import jdk.jfr.Event;import jdk.jfr.Label;

public class Hello {

@Label("Hello World")static class HelloWorldEvent extends Event {

@Label("Message")String message;

}

public static void main(String... args) {HelloWorldEvent event = new HelloWorldEvent();event.message = "Hello World event message!";event.commit();

}}


NewJDKAPIforParsingFlightRecorderFiles

17

public class ParserExample {public static void main(String[] args) throws IOException {

Duration maxNanos = Duration.of(0, ChronoUnit.NANOS);RecordedEvent maxEvent = null;for (RecordedEvent event : RecordingFile.readAllEvents(Paths.get(args[0]))) {

if (event.getEventType().getName().equals("se.hirt.jfr4junit.TestEvent")) {Duration nanos = event.getDuration();if (!nanos.minus(maxNanos).isNegative()) {

maxNanos = nanos;maxEvent = event;

}}

}System.out.printf("Longest running test was: %s for %ds\n",

maxEvent.getValue("displayName"), maxNanos.getSeconds());System.out.println("Event was:\n" + maxEvent);

}}


AlsoJMCAPIforParsingFlightRecorderFiles

18

public static void main(String[] args) throws Exception {IItemCollection events = JfrLoaderToolkit.loadEvents(new File(args[0]));IQuantity aggregate = events.apply(ItemFilters.type(JdkTypeIDs.MONITOR_ENTER))

.getAggregate(Aggregators.stddev(JfrAttributes.DURATION));

System.out.println(“Standard deviation for Java monitor enter was “ + aggregate.displayUsing(IDisplayable.AUTO));

}


RunningJMCRulesHeadlesspublic class RunRulesOnFileSimple {

public static void main(String[] args) throws Exception {File recording = DemoToolkit.getRecordingArgument(args);IItemCollection events = JfrLoaderToolkit.loadEvents(recording);

for (IRule rule : RuleRegistry.getRules()) {RunnableFuture<Result> future = rule.evaluate(events,

IPreferenceValueProvider.DEFAULT_VALUES);future.run();Result result = future.get();if (result.getScore() > 50) {

DemoToolkit.printResult(result);}

}}

}



DemoMissionControl



Summary• JFRAPIsareSUPPORTED inJDK9

– Controllingtherecorder– Creatingcustomevents– Parsingtherecordings

• JMC6JFRpartgreatlyrevised– Automatedanalysisofrecordings– Easytoaddcustomrules–WillworkonJDK7,8and9recordings– Canbeusedstandaloneforscripting(initiallyunsupported)

• Comingsoontoopensource

21


JMC6.0iscool,butwestillhavetheproblemsfromthenatureofdistributedcomputing…



Imagine…

• Low-impactJFRdatacollectionbuiltintoaK8sservice…• Profile anyjavaapplicationormicroserviceorfunction… anywhere andinanycombination…

• Centraltimemachinetorollbacktowhereandwhentheproblemoccurred…

• Thesystemtellsyouhowtosolvetheproblem

JavaOneOctober2017 23Photosourcehttps://www.flickr.com/photos/tom_hall_nz/


OracleCloudAppDev Platform– ContainerNative

24

Microservice/FunctionFramework

Devops AutomationCICD for Docker Container Cloud

API Management

Management CloudOps/Diagnostics

IT Analytics

Logs

APMDiagnostics

BuildEnterpriseScaleMicroservices &FunctionsonManagedDocker Infrastructure

Microservices

Functions

KubernetesContainer

Service

JavaOneOctober2017


DEPTH BREADTH AUTOMATION EASEOFUSE


ContinuousGatheringofProfile Data

ContinuousGatheringofProfileData



DataProcessingandAutomatedAnalysis

ServiceDashboard

Auto-AnalysisDashboard

RuleResults

RuleResults

RuleResults

RuleResults

RuleResults

RuleResults

RuleResults

RuleResults

RuleResults

RuleResults

IDEPlugins

APIsforIntegrationpublicclassMyMethods{privatestaticfinalintNUMBER_OF_THREADS=4privatestaticfinalStringPROD_URI="http://www.or

WithTracesandSpans

JVMAutotuning


DemoContainerDiagnostics



Take-Aways• JFRandMissionControlnewfeatures

– Automatedanalysis,APIsforJFR,performance

• PlansforJFRandJMCinopensource• EnablesthingslikeOracle’snewContainerDiagnosticsservice– Builtonwell-proventechnology!– Builtintocloud-nativeplatform– Alwayson,depthandbreadth,automatedanalysisandtuning



• JavaFlightRecorderinJDK9HOL3018– Wednesday13:30to15:30HiltonSanFranciscoUnionSquare(LobbyLevel)– GoldenGate4/5

• JMCHomepagehttp://oracle.com/missioncontrol

• Hirt’s bloghttp://hirt.se/blog

• Twitter@javamissionctrl

• UpdatedJMCTutorialwillbeuploadedtotheblogafterJavaOne

• Devops Corner– DeveloperLoungeinMoscone West

• DevelopersaretheNewKingmakers– CON6493- Tue10/311:30AM,Moscone West3024

• SteppingUptotheNewStack– CON6476– Mon10/211:00AMMoscone West3024(Look forvideorecording)

• http://www.wercker.com/

28

AdditionalResources

@dave_cabelus

distributed jvm monitoring - rainfocus...“once more @javamissionctrlis saving my day!...

Documents