1
Checking Performance Along your Build Pipeline
@grabnerandihttp://apmblog.compuware.com
http://bit.ly/dttrial
2
Today we are here for …
Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers
Automated Acceptance
Testing
Automated Capacity Testing
Manual testing• Key showcases• Exploratory testing Release
Cont. Delivery / Deploy
3
Who is doing it? How many successful deployments can they do?
300 Deployments / Year
50-60 Deployments / Day
10+ Deployments / Day
Every 11.6 seconds
4
More on Amazons Story
75% fewer outages since 2006
90% fewer outage minutes
~0.001% of deployments cause a problem
Instantaneous automatic rollback
Deploying every 11.6s
5 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
Recommended BookRecommended Book
https://itrevolution.wufoo.com/forms/phoenix-project-ebook-offer/
66 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
Status Quo: Unreliable Software Impacts BusinessStatus Quo: Unreliable Software Impacts Business
7
The “War Room” – back then
'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970
The “War Room” – back then
'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970
8
The “War Room” – NOW
Facebook – December 2012
The “War Room” – NOW
Facebook – December 2012
9 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
Status Quo: Bugfixing in Production happensStatus Quo: Bugfixing in Production happens
~80% of problems
caused by ~20% patterns
YES we know this
80% Dev Time in Bug Fixing
$60B Defect Costs
BUT
10
4 Situations on
WHY this happens,
HOW to avoid it
4 Situations on
WHY this happens,
HOW to avoid it
11
12
“Blindly” (Re)use Existing
Components
13
Requirement: We need a report
14
Using Hibernate results in 4k+ SQL Statements to display 3 items!
Hibernate Executes 4k+ Statements
Individual Execution VERY
FAST
But Total SUM takes 6s
15
Requirement: We need a fancy UI
16
Using Telerik Controls Results in 9s for Data-Binding of UI Controls
#1: Slow Stored ProcedureDepending on Request
execution time of this SP varies between 1 and 7.5s
#2: 240! Similar SQL StatementsMost of these 240! Statements are
not prepared and just differ in things like Column Names
17
Metrics: # Total SQLs# SQLs / Web Request# Same SQLs / Request
Transferred Rows
Test: With realistic Data
Dev: “Learn” Frameworks
18
19
Implementation Flaws
20
Business Impact requires Action!
21
Solution: Cache to the RESCUE!!
22
Implementation and Rollout
Implemented InMemory Cache
Worked well in Load Testing
23
Result: Out of Memory Crashes!!
Still crashes
Problem fixed!Fixed Version Deployed
24
Metrics: Heap Size, # Objects Allocated,# Objects in Cache
Cache Hit Ratio
Test: With realistic Data
25
12 000 000 $
26
#No “Agile” Deployment
27
Ad on air
Availability dropped to 0%
Load Spike resulted in Unavailability
28
Alternative: “GoDaddy goes DevOps”
Response time improved 4x
1h before SuperBowl KickOff
1h after Game ended
29
Behind the Scenes
30
Metrics: AvailabilityPage Size, # Objects
# Hosts, # Connections
DevOps: “Feature” Switches
31
32
#Push without a Plan
33
Mobile Landing Page of Super Bowl Ad
434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~ 20MB
34
m.store.com redirects to www.store.com
ALL CSS and JS files are redirected to the www domain
This is a lot of time “wasted” especially on high latency mobile
connections
35
Metrics: Load Time, # Resources (Images, …),
# HTTP 3xx, 4xx, 5xx
Dev: Build for Mobile
Test: Test on Mobile
Ops: Monitor Mobile
36
37
#1: Define Measures# of Requests / User
# of Log Messages
# of Exceptions
# Objects Allocated
# Objects In Cache
Cache Hit Ratio
# of Images
# of SQLs
# SQLs per RequestAvailability
# HTTP 3xx, 4xx
Page Size
38
#2: Automate
Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers
Automated Acceptance
Testing
Automated Capacity Testing
Manual testing• Key showcases• Exploratory testing Release
Unit & Integration Tests
Functional Tests
Performance TestsProductionMonitoring
Functional Tests
How? Performance Focus in Test AutomationAnalyzing All Unit / Performance Tests
Analyze Perf Metrics
Identify Regressions
How? Performance Focus in Test Automation
How? Performance Focus in Test Automation
Cross Impact of KPIs
Identify Regressions
4242
#3: Share Results
43
#4: Integrate
44
If we do all that
CANCELLED
46
FREE Products & More Info
dynaTrace Free Trial30 Days Free Trial on http://bit.ly/dttrial After that totally free for local usage
Our Blog: http://apmblog.compuware.com
Follow Me: @grabnerandi
