fostering a devops culture through the use of an apm platformfiles.meetup.com/2331301/triangle dev...
TRANSCRIPT
![Page 1: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/1.jpg)
1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Fostering a DevOps Culture Through the use of an APM Platform
Brien Lay
Sales Engineer, Dynatrace
![Page 2: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/2.jpg)
2 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
My Background
• CSC Employee for 15.5 years
• Held the following titles while at CSC
– Account manager – BPO (1year)
– Web Developer – Infrastructure (6 years – ASP, JS, SQL)
– Front end developer – POINT IN (3 years - Java, JS)
– SR Supervisor – Technical Implementation manager – POINT IN (2 years)
– SR System Administrator – Infrastructure (2 years)
– Operations, Technical Manager – Infrastructure (1 year)
• Came to Dynatrace one year ago before our DevOps transformation was complete.
![Page 3: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/3.jpg)
3 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
CSC Environment / History
• Build and host Property and Casualty Insurance systems
• @85 customers – @1,200 servers (Java on WebSphere, .NET, SQL Server) Win Servers
• Created SiteManager – moves all server based code and does SQL moves.
• Use SVN for version control.
• Regression testing = manual
• Load testing – QA Director, load runner
• Operations / configuration management staff – 12 employees (mostly non-skilled).
• Infrastructure staff – 18
• Mostly Waterfall (some teams do Agile, but in reality they are all waterfall).
• Prior to 2012 – no focus on performance. Insurance systems have a mostly captive audience.
• Started to loose deals to our competition.
![Page 4: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/4.jpg)
4 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 5: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/5.jpg)
5 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 6: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/6.jpg)
6 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 7: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/7.jpg)
7 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Dev Ops
![Page 8: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/8.jpg)
8 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 9: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/9.jpg)
9 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace9
Solution: DevOps + Performance Focus
![Page 10: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/10.jpg)
10 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
The Ideal End State
![Page 11: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/11.jpg)
11 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 12: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/12.jpg)
12 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 13: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/13.jpg)
13 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 14: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/14.jpg)
14 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 15: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/15.jpg)
15 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 16: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/16.jpg)
16 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
The New Technical Reality
Continuous Delivery creates rapidly
changing applications
Built in diverse languages and running
on diverse language runtimes
Deployed on multiple virtualization
platforms
Running on scaled out commodity
hardware
Your Cloud Public CloudHybrid CloudLocated in multiple clouds with multiple
owners
![Page 17: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/17.jpg)
17 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
New Reality, New Questions
+ How can realistic performance
tests be automated into my CI
chain?
+ How can I catch more
performance problems before
they hit production?
+ How can development, QA and
Ops teams have common data
they can trust?
+ How can I
troubleshoot problems
faster? + How can I reduce
unplanned work?
![Page 18: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/18.jpg)
18 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
What’s preventing faster AND higher quality releases?
• Developers can not easily recreate production problems, leading to unplanned work
• Data captured in Ops can’t be actioned by Dev
• Scalability and performance hotspots are not properly detected in test/QA
• No way to compare performance from build to build
+ How can realistic
performance tests be
automated into my CI
chain?
+ How can I catch more performance
problems before they hit
production?
+ How can development, QA and
Ops teams have common data
they can trust?
+ How can I
troubleshoot
problems faster?
+ How can I reduce
unplanned work?
![Page 19: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/19.jpg)
19 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
High performing organizations are more agile and more reliable
30x 50xmore frequent
deployments
Fewer failures
Source: Puppet Labs 2014 State Of DevOps https://puppetlabs.com/2014-devops-report
2x
the change
success rate
12xFaster mean
time to recover
(MTTR)
![Page 20: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/20.jpg)
20 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Gap-Free
Data
cross-tier method-level
analysis
line-of-code facts for
problem resolution
method arguments, bind
values and more for
individual transactions
![Page 21: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/21.jpg)
21 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Gap-Free
Data
Performance Build
& Test Automation
quantify build-over-build
progress
summary test results for
integration, browser, …
![Page 22: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/22.jpg)
22 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Gap-Free
Data
Performance Build
& Test Automation
Anti-Pattern
Analytics
overview of all executed
tests with status
detailed metrics per testtrending with automatic
baslines
![Page 23: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/23.jpg)
23 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Gap-Free
Data
Performance Build
& Test Automation
Anti-Pattern
Analytics
DevOps Toolchain
Integration
IDE
Support
IntelliJ IDEA
Build
Automation
Ant
Test
Automation
Silk Performer
Selenium and
WebDriver
Build Server
/ CI
![Page 24: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/24.jpg)
24 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Gap-Free
Data
Performance Build
& Test Automation
Anti-Pattern
Analytics
DevOps Toolchain
Integration
view trends and status
within build automation
tooling
![Page 25: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/25.jpg)
25 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Gap-Free
Data
Performance Build
& Test Automation
Anti-Pattern
Analytics
DevOps Toolchain
Integration
Production
Problem Capture
deliver facts to avoid
blamestorming
![Page 26: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/26.jpg)
26 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Measurement: Define KPIs accepted by all teams
# of SQL Executions
# of Log Lines
MBs / User
Time for Deployment
Time for Rollback
Response TimesPerf Test Code Coverage
![Page 27: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/27.jpg)
27 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
How? Performance Focus in Test Automation
12 0 120ms
3 1 68ms
Build 20 testPurchase OK
testSearch OK
Build 17 testPurchase OK
testSearch OK
Build 18 testPurchase FAILED
testSearch OK
Build 19 testPurchase OK
testSearch OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms
3 1 68ms
12 5 60ms
3 1 68ms
75 0 230ms
3 1 68ms
Test Framework Results Architectural Data
We identified a
regresesion
Problem solved
Lets look behind the
scenes
Exceptions probably
reason for failed testsProblem fixed but now we have
an architectural regression
Problem fixed but now we have
an architectural regression
Now we have the functional and
architectural confidence
![Page 28: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/28.jpg)
28 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #DynatraceCOMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
Performance Focus in Test Automation
Embed your Architectural Results in Jenkins
![Page 29: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/29.jpg)
29 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #DynatraceCOMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
Performance Focus in Test Automation
Build-levelPerformance Metrics
![Page 30: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/30.jpg)
30 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
![Page 31: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/31.jpg)
31 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Insert headshot
image herenordstrom
Dynatrace was instrumental
in reducing our release cycle
from 4 times a year to once
a month”
“
-Gopal Brugalette, Performance Architect
![Page 32: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/32.jpg)
32 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Internal CSC Successes using Dynatrace
• Incorporation into our software development lifecycle.
– Base development teams for our major products now use Dynatrace
• Exceed, POINT IN, Agency Link, Riskmaster
– Performance Lab – all releases strictly tested using Dynatrace integration.
– Deployments went from months down to days.
• Customer Relationships
– Major impact to improving systems = happy customers
– Sharing of session files / co-development with customers
• Dynatrace displaced CA at CSC as the APM Vendor of Choice.
![Page 33: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/33.jpg)
33 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Internal Success – Measured at 6 Months – Nov. 2012
THE CHALLENGE
• CSC’s P&C solution provides leading insurance companies with the most advanced insurance software on the market via a globally connected network of outsourcing centers.
• Larger customers require extensive software customization which introduces defects
• Serious defects involved “war rooms” of 6-12 people, and could take 1-2 days to resolve.
• Defects are costly, disruptive and time-consuming to resolve, especially for “live” customers.
• The Arch Insurance project alone generated 2963 defects which cost >$1.33M to resolve
• Defects impact SLAs – bug fix delays result in additional costs
THE BUSINESS BENEFITS
• dynaTrace has reduced the cost of fixing defects by about 40%
• Prior to using dT, a typical defect would cost about $449 to fix.
• Using dT, 80% of defects can be found and resolved at a cost of about $225
• Swiss RE : 3,885 defects
• Without dT
– 3,885 defects @ $449 = $1.744M
• With dT– 80% fixed@ $225 = $699K
– 20% fixed@ $449 = $349K
– Total cost = $1.048M
– Savings = $696K (40%) in 5 mos.
• Improved the quality of SLA reporting & tracking violations
![Page 34: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/34.jpg)
34 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #DynatraceCOMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
ROME WAS NOT BUILT IN A DAY
SPONSORSHIP
DON’T BE AFRAID TO FAIL
TALENTKNOWLEDGE
Tie code quality and performance to developers
Share your successes
![Page 35: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/35.jpg)
35 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
• It starts and end with the user - Not the app
•Gap-free data is essential
•The goal is Dev/Ops not Just/Ops
What we believe
![Page 36: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/36.jpg)
36 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
• DevOps / Lifecycle blog -http://apmblog.dynatrace.com/category/lifecycle-devops/
• Dynatrace recorded Webinars -http://www.dynatrace.com/en/resources/webinars.html
• Load testing tool integration -https://www.youtube.com/watch?v=26nVM_JgStw&list=PLqt2rd0eew1bmDn54E2_M2uvbhm_WxY_6&index=15
• Dynatrace helps build a DevOps culture -
Additional Resources
![Page 37: Fostering a DevOps Culture Through the use of an APM Platformfiles.meetup.com/2331301/Triangle Dev Ops.pdf · Fostering a DevOps Culture Through the use of an APM Platform Brien Lay](https://reader033.vdocuments.us/reader033/viewer/2022042621/5f586291d762d654485a16e2/html5/thumbnails/37.jpg)
37 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Dynatrace
Thank Youwww.dynatrace.com