top .net, java & web performance mistakes - meetup jan 2015

1 @Dynatrace

- More on http://blog.dynatrace.com

- Dynatrace Free Trial: http://bit.ly/dttrial

Andreas Grabner - @grabnerandi

Biggest Performance Mistakes (.NET, Java, Web …)

2 @Dynatrace

That’s why I ended up talking about performance

3 @Dynatrace

Nobody wants this …

4 @Dynatrace

Unless you work for Google or Microsoft

5 @Dynatrace

Nor this …

6 @Dynatrace

7 @Dynatrace

As it leads to this …

8 @Dynatrace

The “War Room”

Facebook – December 2012

9 @Dynatrace

And potentially to this …

10 @Dynatrace

11 @Dynatrace

And this …

12 @Dynatrace

13 @Dynatrace

And that’s why Business doesn’t like it either …

14 @Dynatrace

~80% of problems

caused by ~20% patterns

YES we know this

80% Dev Time in Bug Fixing

$60B Defect Costs

BUT

15 @Dynatrace

6 Situations on

WHY this happened,

HOW to avoid it,

METRICS to look at

16 @Dynatrace

17 @Dynatrace

Incorrect Sizing of Pools and

Queues

18 @Dynatrace

Online Banking: Slow Balance Check

1.69s (=101s!) To Check Balance!

87% spent in IIS 600! SQL Executions

19 @Dynatrace

#1 Time really spent in IIS?

Tip: Elapsed Time tells us WHEN a Method was executed!

Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET

Tip: Thread# gives us insight on Thread Queues / Switches

20 @Dynatrace

#2 What about these SQL Executions?

Finding: EVERY SQL statement is executed on

ITS OWN Connection!

Tip: Look at “GetConnection”

21 @Dynatrace

#2 SQL Executions! continued …

#1: Same SQL is executed 67! times

#2: NO PREPARATIONbecause everything

executed on new Connection

22 @Dynatrace

Lessons Learned!

ASP.NET Worker Thread Pool Sizing!

DB Connection PoolsMore Efficient SQL

23 @Dynatrace

Idle vs. Busy Threads

# SQLs / Request

# GetConnection

24 @Dynatrace

25 @Dynatrace

“Blindly” (Re)use Existing

Components

26 @Dynatrace

Requirement: We need a report

27 @Dynatrace

Using Hibernate results in 4k+ SQL Statements to display 3 items!

Hibernate Executes 4k+ Statements

Individual Execution VERY FAST

But Total SUM takes 6s

28 @Dynatrace

• Proper Sizing Specifications

• Understand the Frameworks you are using

Lessons Learned

29 @Dynatrace

# SQLs / Request

# Same SQL / Request

30 @Dynatrace

31 @Dynatrace

Wrong Tools for the Job

32 @Dynatrace

#1 Banking: Transaction History CSV Download!

Remember: High GC doesn’t necessarily mean that this request has a memory problem – BUT - …

Root Cause: Heavy use of String.Concat to build the CSV output in Memory

Recommendations:#1: Use System.Text.StringBuilder instead of String.Concat#2: Write directly back to OutputStream

Problem: Takes 207s! To download. 87% of Time spent in Garbage

Collection

33 @Dynatrace

#2 Store: Rendering Search ResultProblem: 4.4s to render

result page

Root Cause: Custom RegExLibrary with performance

issues on large strings

34 @Dynatrace

#3 Store: Occasional Performance Spikes

Problem: Occasional Spikes!Never seen in Development

35 @Dynatrace

First Analysis: His own code

36 @Dynatrace

DNS Lookup Issue during Login

37 @Dynatrace

Reverse Lookup by IP fails on these IPs

Root Cause: Issues from Data Center Hoster

38 @Dynatrace

• CSV Export Example

• Don’t build large Strings in Memory with String.Concat

• Write to output stream and use StringBuilder

• Search Result Page

• Don’t implement custom RegEx Library – existing Tools are better

• Test with large results and not just small test data

• Geo Lookup

• Don’t trust your Hosting Provider

• E.g: Use GeoIP Databases for accurate reverse lookups

Lessons Learned – Use Better Tools / Libraries

39 @Dynatrace

Memory Usage

Time Spent in APIs

40 @Dynatrace

41 @Dynatrace

Not every Architect makes good decisions

42 @Dynatrace

• Symptoms

• HTML takes between 60 and 120s to render

• High GC Time

• Assumptions

• Bad GC Tuning

• Probably bad Database Performance as rendering was simple

Project: Online Room Reservation System

43 @Dynatrace

Developers built own monitoring

void roomreservationReport(int roomid)

{

long startTime = System.currentTimeMillis();

Object data = loadDataForRoom(roomid);

long dataLoadTime = System.currentTimeMillis() - startTime;

generateReport(data, roomid);

}

Result:

Avg. Data Load Time: 45s!

DB Tool says:

Avg. SQL Query: <1ms!

44 @Dynatrace

#1: Loading too much data24889! Calls to the

Database API!

High CPU and High Memory Usage to keep all

data in Memory

45 @Dynatrace

#2: On individual connections 12444!individual

connections

Classical N+1 Query Problem

Individual SQL really <1ms

46 @Dynatrace

#3: Putting all data in temp Hashtable

Lots of time spent in

Hashtable.get

Called from their Entity Objects

47 @Dynatrace

• Custom Measuring

• Was impacted by Garbage Collection

• Just measured overall time but not # SQL Executions

• Learn SQL and don’t use Hashtables as Workaround

Lesson Learned

void roomreservationReport(int roomid)

{

long startTime = System.currentTimeMillis();

Object data = loadDataForRoom(roomid);

long dataLoadTime = System.currentTimeMillis() - startTime;

generateReport(data, roomid);

}

48 @Dynatrace

# SQL Executions

# of SAME SQLs

Conn. Acquisition Time

49 @Dynatrace

50 @Dynatrace

#Push without a Plan

51 @Dynatrace

Mobile Landing Page of Super Bowl Ad

434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …

Total size of ~ 20MB

52 @Dynatrace

Fifa.com during Worldcup

http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/

53 @Dynatrace

# Images

# Redirects

Size of Resources

54 @Dynatrace

55 @Dynatrace

#No “Agile” Deployment

56 @Dynatrace

Load Spike resulted in UnavailabilityAd

on

air

57 @Dynatrace

Alternative: “GoDaddy goes DevOps”

1h before SuperBowl KickOff

1h after Game ended

58 @Dynatrace

# of Domains

Total Size

59 @Dynatrace

Waiting for Bug Reports

60 @Dynatrace

Look behind the scenes

61 @Dynatrace

•# Images

•# Redirects

•Size of Resources

•# SQL Executions

•# of SAME SQLs

•# Items per Page

•# AJAX per Page

Look at these Metrics Before Checking In Code

•Time Spent in API

•# Calls into API

•# Functional Errors

•3rd Party calls

•# of Domains

•Total Size

•…

62 @Dynatrace

Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers

Automated Acceptance

Testing

Automated Capacity Testing

Manual testing• Key showcases• Exploratory testing Release

Unit & Integration Tests

Functional Tests

Performance TestsProductionMonitoring

Functional Tests

(R)Evolutionize Performance Optimization

63 @Dynatrace

64 @Dynatrace

Example from Web Diagnostics 282! Objects

on that page9.68MB Page Size

8.8s Page Load Time

Most objects are images delivered from your main

domain

Very long Connect time (1.8s) to your CDN

65 @Dynatrace

Example from Server-Side Diagnostics526s to render that

report

1 SQL running

210s!

Lots of time spent in logging to Log4J

Lots of time spent in rendering

66 @Dynatrace

Online Performance Clinics

Every Other week @

bit.ly/onlineperfclinic

67 @Dynatrace

Your Benefits

• Free Performance Review

• Extended Dynatrace License

“Share Your PurePath”

bit.ly/sharepurepathMy Benefits

• More blog material for next year

• Gratification that I could help you

68 @Dynatrace

Questions and/or Demo

Slides: slideshare.net/grabnerandi

Get Tools: bit.ly/dttrial

YouTube Tutorials: bit.ly/dttutorials

Contact Me: agrabner@dynatrace.com

Follow Me: @grabnerandi

Read More: blog.dynatrace.com

69 @Dynatrace

Andreas GrabnerDynatrace Developer Advocate

@grabnerandi

http://blog.dynatrace.com