1 @Dynatrace

- More on http://blog.dynatrace.com

- Dynatrace Free Trial: http://bit.ly/dttrial

Andreas Grabner - @grabnerandi

Java Performance:Biggest Mistakes

2 @Dynatrace

And this is where I am from

3 @Dynatrace

4 @Dynatrace

5 @Dynatrace

Can you tell my age by looking at my first computer?

6 @Dynatrace

That’s why I ended up talking about performance

7 @Dynatrace

Nobody wants this …

8 @Dynatrace

9 @Dynatrace

Nor this …

10 @Dynatrace

11 @Dynatrace

As it leads to this …

12 @Dynatrace

The “War Room”

Facebook – December 2012

13 @Dynatrace

And potentially to this …

14 @Dynatrace

15 @Dynatrace

And this …

16 @Dynatrace

17 @Dynatrace

And that’s why Business doesn’t like it either …

18 @Dynatrace

~80% of problems

caused by ~20% patterns

YES we know this

80% Dev Time in Bug Fixing

$60B Defect Costs

BUT

19 @Dynatrace

6 Situations on

WHY this happened,

HOW to avoid it

20 @Dynatrace

21 @Dynatrace

“Blindly” (Re)use Existing

Components

22 @Dynatrace

Requirement: We need a report

23 @Dynatrace

Using Hibernate results in 4k+ SQL Statements to display 3 items!

Hibernate Executes 4k+ Statements

Individual Execution VERY FAST

But Total SUM takes 6s

24 @Dynatrace

25 @Dynatrace

Not every Architect makes good decisions

26 @Dynatrace

• Symptoms

• HTML takes between 60 and 120s to render

• High GC Time

• Assumptions

• Bad GC Tuning

• Probably bad Database Performance as rendering was simple

Project: Online Room Reservation System

27 @Dynatrace

Developers built own monitoring

void roomreservationReport(int roomid)

{

long startTime = System.currentTimeMillis();

Object data = loadDataForRoom(roomid);

long dataLoadTime = System.currentTimeMillis() - startTime;

generateReport(data, roomid);

}

Result:

Avg. Data Load Time: 45s!

DB Tool says:

Avg. SQL Query: <1ms!

28 @Dynatrace

#1: Loading too much data24889! Calls to the

Database API!

High CPU and High Memory Usage to keep all

data in Memory

29 @Dynatrace

#2: On individual connections 12444!individual

connections

Classical N+1 Query Problem

Individual SQL really <1ms

30 @Dynatrace

#3: Putting all data in temp Hashtable

Lots of time spent in

Hashtable.get

Called from their Entity Objects

31 @Dynatrace

• Custom Measuring

• Was impacted by Garbage Collection

• Just measured overall time but not # SQL Executions

• Learn SQL and don’t use Hashtables as Workaround

Lesson Learned

void roomreservationReport(int roomid)

{

long startTime = System.currentTimeMillis();

Object data = loadDataForRoom(roomid);

long dataLoadTime = System.currentTimeMillis() - startTime;

generateReport(data, roomid);

}

32 @Dynatrace

33 @Dynatrace

Implementation Flaws

34 @Dynatrace

Business Impact requires Action!

35 @Dynatrace

Solution: Cache to the RESCUE!!

36 @Dynatrace

Implementation and Rollout

Implemented InMemory Cache

Worked well in Load Testing

37 @Dynatrace

Result: Out of Memory Crashes!!

Still crashes

Problem fixed!Fixed Version Deployed

38 @Dynatrace

39 @Dynatrace

Disconnected Teams

40 @Dynatrace

“Teamwork” between Dev and Ops

SEV1 Problem in Production

Need access to log files

Where are they? Can’t get them

Need to increase log level

Can’t do! Can’t change configfiles in prod!

41 @Dynatrace

Solution: Implement a Custom “On Demand” Remote Logger

42 @Dynatrace

Implementation and Rollout

Implemented Custom Logger

Worked well in Load Testing

43 @Dynatrace

What happened?

~ 1Mio Lock Exceptions in 30 mins

44 @Dynatrace

Root Cause: A special WebSphere Setting!

Log Service provides a synchronized log file across ALL JVMs

Log Service provides a synchronized log file across

ALL JVMs

45 @Dynatrace

46 @Dynatrace

“Deployment” Gone Bad!

47 @Dynatrace

Test Environment

Production Environment

8x slower3x more SQL

48 @Dynatrace

Test Environment Production Environment

Hibernate, Classloading,

XML – The Key Hotspots

Hibernate, Classloading, XML

– The Key Hotspots

I/O for Web Requests doesn’t

even show up!

That’s Normal: Having I/O for Web

Request as main contributor

49 @Dynatrace

Top Contributor Class.getInterfaces

Called from Hibernates FieldInterceptionHelper

These calls all originate form thousands of calls to

find item by code

50 @Dynatrace

Top Methods related to XML Processing

Classloading is triggered through CustomMonnkey and the Xalan

Parser

Classloading is triggered through CustomMonkey and the Xalan

Parser

51 @Dynatrace

52 @Dynatrace

#Push without a Plan

53 @Dynatrace

Mobile Landing Page of Super Bowl Ad

434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …

Total size of ~ 20MB

54 @Dynatrace

Fifa.com during Worldcup

http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/

55 @Dynatrace

Waiting for Bug Reports

56 @Dynatrace

Look behind the scenes

57 @Dynatrace

58 @Dynatrace

Performance Clinic: Bring Your Own AppWhen: Thursday, December 18th

Lunch Session: NOON – 3PMAfter Work Session: 4PM – 7PM

Where: Dynatrace Office404 Wyman Street, Waltham

Register: http://bit.ly/onlineperfclinic

59 @Dynatrace

Quick Demo

60 @Dynatrace

• Get it here: http://bit.ly/dttrial

• YouTube Tutorials: http://bit.ly/dttutorials

• Live Q&A Sessions: http://bit.ly/onlineperfclinic

• Contact me: agrabner@dynatrace.com - @grabnerandi

•Special Offer:

• SaaS VIP Program: http://bit.ly/dtaasvip

30 Days Dynatrace Free Trial

61 @Dynatrace

Andreas GrabnerDynatrace Developer Advocate

@grabnerandi

http://blog.dynatrace.com