metrics driven development and devops - agile 2014
DESCRIPTION
Karthik and I explore the use of metrics in development and DevOps in the form of three Epic Rap Battles of History - Dev vs Ops, Small vs Large Org, and Scrum vs Kanban. With special appearance by Dr. Evil who explains how to use metrics for evil. Presented at Agile 2014.TRANSCRIPT
![Page 1: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/1.jpg)
• Ernest Mueller (@ernestmueller)
• Karthik Gaekwad (@iteration1)
![Page 2: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/2.jpg)
@ernestmueller @iteration1#Agile2014
![Page 3: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/3.jpg)
• Senior Engineer @Signal
Sciences
• Previous:
• 10 years building products-
agile/cloud/devops teams
@ernestmueller @iteration1#Agile2014
![Page 4: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/4.jpg)
• Product Manager at
Copperegg
• Previous:
• 20 years in IT – dev, ops,
management@ernestmueller @iteration1#Agile2014
![Page 5: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/5.jpg)
Our Goal For You Today
• Empower you with new ideas to bring your organization together!
• Metrics. What are they?
• How to use metrics for good, as illustrated by three Epic Rap Battles of History!
– Dev vs Ops (What is this… DevOps?)
– Small vs Large Org
– Scrum vs Kanban
@ernestmueller @iteration1#Agile2014
![Page 6: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/6.jpg)
@ernestmueller @iteration1#Agile2014
![Page 7: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/7.jpg)
What Are Metrics?
• A quantifiable measure of any component or
process whose change is of interest to your
business.
– Business!
– Application!
– System!
– People!
– Process!
– Not: Meaningless numbers!
@ernestmueller @iteration1#Agile2014
![Page 8: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/8.jpg)
@ernestmueller @iteration1#Agile2014
![Page 9: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/9.jpg)
![Page 10: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/10.jpg)
@ernestmueller @iteration1#Agile2014
![Page 11: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/11.jpg)
11
![Page 12: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/12.jpg)
![Page 13: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/13.jpg)
@ernestmueller @iteration1#Agile2014
![Page 14: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/14.jpg)
@ernestmueller @iteration1#Agile2014
![Page 15: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/15.jpg)
@ernestmueller @iteration1#Agile2014
![Page 16: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/16.jpg)
![Page 17: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/17.jpg)
@ernestmueller @iteration1#Agile2014
![Page 18: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/18.jpg)
@ernestmueller @iteration1#Agile2014
![Page 19: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/19.jpg)
@ernestmueller @iteration1#Agile2014
![Page 20: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/20.jpg)
• Tasked with building new cloud business for the
organization.
• Understand how cloud technologies can impact
bottom line.
• Build products customers will want from the new
business unit.
– Read, startup inside a bigger organization
Story Time: Our context
@ernestmueller @iteration1#Agile2014
![Page 21: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/21.jpg)
![Page 22: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/22.jpg)
![Page 23: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/23.jpg)
• Used ‘Lean Startup’ ideas to power new area.
– Able to define an MVP (Minimum Viable
Product).
– Easier to define workflow for something brand
new.
– No confusion with existing processes.
• Once we started to see value, retrofitted to
other parts of the org.
Lean Startup Applied
@ernestmueller @iteration1#Agile2014
![Page 24: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/24.jpg)
Showing progress
• Initially- we had weekly
progress/status meetings with
stakeholders.
• Cross functional team with
business/marketing/engineering.
@ernestmueller @iteration1#Agile2014
![Page 25: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/25.jpg)
@ernestmueller @iteration1#Agile2014
![Page 26: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/26.jpg)
@ernestmueller @iteration1#Agile2014
![Page 27: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/27.jpg)
Metrics
• Pivot: change conversations to metrics instead.
• Agreed on metrics that we wanted to track
– Stakeholder input
• “What do you want out of this?”
• “How quickly do you want this?”
• …Okay, let’s measure this!
@ernestmueller @iteration1#Agile2014
![Page 28: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/28.jpg)
Tracked Metrics
• Tracked actionable metrics (dev and business):
– # Users signing up per week
– # Active sessions per day/week
– # of compiles sent per week
– # unique data points sent per week
![Page 29: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/29.jpg)
@ernestmueller @iteration1#Agile2014
Pro Tip: Metrics
• Link all your metrics from one
dashboard.
– Business (Ex: User logins)
– Dev (Ex: Performance
metrics)
– Ops (Ex: DB CPU Usage)
• One bookmark to rule them
all.
![Page 30: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/30.jpg)
@ernestmueller @iteration1#Agile2014
Pro Tip: Metrics
• Don’t use yet another
username/password scheme.
• You’ll lose your users really
fast!
![Page 31: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/31.jpg)
Pro Tip: Metrics
• Try to use a tool that can handle different kinds
of metrics.
• Shoutouts:
– Statsd
– Datadog
@ernestmueller @iteration1#Agile2014
![Page 32: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/32.jpg)
End Result
• Business and engineering on the same
page.
• Management looking at metrics without
having “meetings to look at metrics”.
• Became a part of the culture.
• Innovate faster because different
teams were in sync.
@ernestmueller @iteration1#Agile2014
![Page 33: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/33.jpg)
@ernestmueller @iteration1#Agile2014
![Page 34: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/34.jpg)
OperationsWhat is it?
Why Do You Care?
@ernestmueller @iteration1#Agile2014
![Page 35: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/35.jpg)
Other Kinds Of Operations
• Wikipedia quoth:
• Business operations is the harvesting of
value from assets owned by a business
• Operations management is […] overseeing,
designing, and controlling the process of
production and redesigning business
operations in the production of goods or
services.
@ernestmueller @iteration1#Agile2014
![Page 36: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/36.jpg)
Technical Operations
• “Operations: The New Secret Sauce” – Tim
O’Reilly (2006)
• Without the ability to
– Release changes
–Quickly respond to change
– Provide a service without interruption
–Operate cost effectively
Your service is borked.@ernestmueller @iteration1#Agile2014
![Page 37: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/37.jpg)
What Does Operations Do?
• Build Servers, OS, Virtualization/Cloud
• Install/Upgrade Software
• Install Applications/Release Process/Move to Prod
• Configure Network, Load Balancers, Storage, etc.
• Security testing, reporting, and hardening
• Reliability (scaling, backups)
• Performance management (apps, systems)
• Scalability (capacity planning to autoscaling)
@ernestmueller @iteration1#Agile2014
![Page 38: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/38.jpg)
What Else Does Operations Do?
• Availability – Responsible for service being up
• Incident Response
• Fulfill Requests
• Budgeting/Contracts/Cost Tracking/Reduction
• Monitor all of that
• Much more
• So besides “they run the services,” the critical
final piece of your value chain, they have access
to many of the things you want metrics from
![Page 39: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/39.jpg)
Code – Operations =
@ernestmueller @iteration1#Agile2014
![Page 40: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/40.jpg)
Story Time: Black Friday
• Every year, a huge spike in usage
• Uptime and performance critical to retailers
during the period
• Product directly contributed to conversion
• Metrics crucial to plan the period, execute
through the period, report how we did
@ernestmueller @iteration1#Agile2014
![Page 41: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/41.jpg)
@ernestmueller @iteration1#Agile2014
![Page 42: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/42.jpg)
Metrics From…
From:
• Servers
• Applications/App Logs
• App Servers/Software
• Network
• Data Stores
• Web Servers/CDNs
• Client Browsers
• Alerts
• Tickets
Using:
• Open source monitoring tools (Zabbix, nagios)
• SaaS (SumoLogic, PagerDuty)
• JIRA
• Custom (Web front end analytics w.Hadoop)
• Custom (Amazon cost analytics w.GoodData)
• Custom (Metrics Dashboard)
![Page 43: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/43.jpg)
Many Tools Are Awful
![Page 44: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/44.jpg)
![Page 45: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/45.jpg)
![Page 46: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/46.jpg)
YOU
DECIDE
![Page 47: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/47.jpg)
DEVOPS
@ernestmueller @iteration1#Agile2014
![Page 48: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/48.jpg)
![Page 49: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/49.jpg)
Traditional Dev and Ops
![Page 50: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/50.jpg)
What is DevOps?
@ernestmueller @iteration1#Agile2014
![Page 51: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/51.jpg)
What is DevOps?
@ernestmueller @iteration1#Agile2014
![Page 52: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/52.jpg)
![Page 53: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/53.jpg)
Scrumming
away….
![Page 54: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/54.jpg)
Ready to deploy…
![Page 55: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/55.jpg)
![Page 56: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/56.jpg)
Metrics Promote DevOps
• How do you get the cat inside the circle? Herding cats is hard. Some people aren’t cat people.
• Metrics can be used to promote culture, understanding, and collaboration
• Metrics help keep those different disciplines in sync by providing tangible collaboration points
• MTTD, MTTR, performance metrics, events
• Bringing all the discipline’s metrics together cover your whole value chain “code to cash”
![Page 57: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/57.jpg)
Operations – Code = IT
![Page 58: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/58.jpg)
![Page 59: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/59.jpg)
IT + DevOps = ?
• Many IT teams implement Agile today
• They can implement DevOps too
• But to do either, they have to change how they
interact with others
• Focus on customer’s needs not own needs;
cloud/SaaS providing “competitive pressure”
• Practice Theory of Constraints – embed when
possible, even if you need to add some
• Add devs and automate
![Page 60: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/60.jpg)
![Page 61: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/61.jpg)
![Page 62: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/62.jpg)
SMALL
ORG
![Page 63: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/63.jpg)
![Page 64: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/64.jpg)
![Page 65: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/65.jpg)
![Page 66: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/66.jpg)
![Page 67: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/67.jpg)
![Page 68: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/68.jpg)
![Page 69: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/69.jpg)
![Page 70: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/70.jpg)
![Page 71: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/71.jpg)
Metrics 101: Culture of communication
• Talk in terms of metrics
– Builds common ground between different
roles.
– Understand different perspectives.
– Find the best way to get everyone talking in 1
place.
![Page 72: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/72.jpg)
Metrics 201
• Push your metrics into your conversation tool
• Use tools that everyone likes:
– IRC v/s Hipchat/slack
• Integrate your metrics into a channel
– “Deployment channel” in your chat
![Page 73: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/73.jpg)
Culture of communication
• Find a way to get people talking.
• Find face to face time with stakeholders.
• Metrics are that specific item to have a
conversation around.
• Engineering teams love IRC, but business and
PM’s might not as much.
• Transitioned to Slack/Hipchat (integrations and
message history)
• Leads to visibility and builds trust74
![Page 74: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/74.jpg)
End Result
• Metrics drive conversations between everyone.
• Enhances productivity.
• Helped us streamline our process.
![Page 75: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/75.jpg)
LARGE
ORG
![Page 76: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/76.jpg)
Large Mature Org
• Hundreds of developers
• Many teams (many goals, processes)
• Distributed teams
• International teams
• Outsourcers
• Various Weird Partner Relationships
77
![Page 77: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/77.jpg)
Large Org Problems
• Silos Galore
• Communication Problems
• Annoying Compliance Requirements
• Profitability Actually Important
• Less pure greenfield work – also
responsibility for many existing
mature systems
![Page 78: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/78.jpg)
Story Time
• Story Time: SaaS product, 40 Engineers, 2/3
outsourced, mostly maintenance but extreme
scale (1/3 of staff were Ops)
• Lots of support initiated urgent customer
requests
• Dev still required for features,
integration/transition with newer services, bug
fix, scaling
• Team morale issues
![Page 79: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/79.jpg)
Metrics 101
• First, add Agile. (Previously the ‘stew method’)
• Basic Metrics – number of tickets (100+ in queue
at any time), size of backlog (500 or so bugs
and stories), rate of new inflow and completion.
• Used to fix misunderstanding from upper
management and correct resourcing
• Next step on metrics – how to balance the
support work and new work?
![Page 80: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/80.jpg)
Metrics 201
Support SLA
![Page 81: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/81.jpg)
Metrics 201
• Metrics 201
Velocity
![Page 82: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/82.jpg)
Metrics 201
• Balancing these two metrics was the key to
satisfying customers short and long term.
• But it’s not an either-or - by seeing the effects
of people, process, and technology changes on
those metrics we drove SLA from <50% to
100% and kept velocity growing (20.. 50… 200…)
• Having the metrics to focus on gave shared
purpose and eased communication with the large
distributed team
• Experiment, see the impact, pivot.
![Page 83: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/83.jpg)
Metrics 301
• Monthly “Operational
Excellence (Metrics)
Meeting”
• Teams presented their
metrics portfolio – with
some variation as
appropriate
• Drawn from system info,
app metrics, db reports,
Salesforce, surveys, etc.
• Keep it lean!!!
• Revenue and Cost
• Product Usage
• Performance
• Availability
• Client Satisfaction
• Employee Satisfaction
• Quality
• Security
![Page 84: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/84.jpg)
![Page 85: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/85.jpg)
Metrics 401 - A/B Testing
• All features had usage measured
• Feature flags would turn features on for
customer subsets to measure usage, effect on
conversion, etc. before committing
• Sometimes you had to kill it despite work spent
• Retooling could save a high profile failure
• “Yes, product guy, you have to.”
• Look for things metrics say you can kill – it’s the
only way to stay lean long term
![Page 86: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/86.jpg)
YOU
DECIDE
![Page 87: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/87.jpg)
@ernestmueller @iteration1#Agile2014
![Page 88: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/88.jpg)
![Page 89: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/89.jpg)
KANBAN
@ernestmueller @iteration1#Agile2014
![Page 90: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/90.jpg)
![Page 91: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/91.jpg)
Here’s why…
• How many meetings?
• “Short planning meeting”?
• How often do these go long?
• Wait how long before prioritizing a feature/bug?
• Role of a dedicated scrum master is a luxury.
• Derailed sprints because of changing business
priorities…
@ernestmueller @iteration1#Agile2014
![Page 92: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/92.jpg)
![Page 93: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/93.jpg)
Why Kanban?
• Limited number of WIP tasks in play.
• Easier to prioritize. There is only 1 list!
• Standups are simpler.
• Task estimates in days versus hours (1/2 day->7
day).
• Research tasks to figure out how long something
may take.
94@ernestmueller @iteration1#Agile2014
![Page 94: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/94.jpg)
Kanban benefits
• Kanban + CI == Solved our issue of “when to
release”. Didn’t have to wait for release windows
like in scrum.
• Less stressful == Only x number of tasks going
on at once. Easier to measure.
• Velocity is awesome!
95@ernestmueller @iteration1#Agile2014
![Page 95: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/95.jpg)
Kanban Metrics
• Things we track:
– Visualized board (JIRA Greenhopper)
– Cycle Time (How fast something gets done)
– WIP (Work/Tasks in progress)
– Flow diagram
–% of bugs
![Page 96: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/96.jpg)
SCRUM@ernestmueller @iteration1#Agile2014
![Page 97: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/97.jpg)
Scrum Rules
• Have used Scrum for both pure Ops and
Dev+Ops teams
![Page 98: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/98.jpg)
Kanban Drools
• Deadlines help maintain tempo – we had multiple
releases a sprint, don’t need to tie them together
• You can reliably commit to a near term ETA with
Scrum instead of just “when it’s done”
• Scrum has a better backlog (esp. in JIRA!)
• Many people “doing Kanban” are really “doing
nothing”, like some doing “Agile” are really doing
“cowboy coding.” Kanban takes more discipline
and training than Scrum.
@ernestmueller @iteration1#Agile2014
![Page 99: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/99.jpg)
Scrum and Metrics
• Velocity is easier for people to understand than
flow diagrams
@ernestmueller @iteration1#Agile2014
![Page 100: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/100.jpg)
But I Hear Kanban Is Better For Ops
• In a DevOps world, most Ops work SHOULD
NOT be interrupt driven – it’s project work just
like the devs are doing
• Dev and Ops expedite work approach each other
in magnitude over time assuming appropriate
investment in automation
• You may be thinking of “Level 1 Support” or “The
Helpdesk” – that is NOT an Ops Engineer
@ernestmueller @iteration1#Agile2014
![Page 101: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/101.jpg)
Scrum for Ops?
• Devs have to be involved in major incidents too!
• Over the length of a sprint, the interrupt level
evens out – my metrics show that velocity
doesn’t vary more than with dev teams
• You manage WIP in your scrum too
@ernestmueller @iteration1#Agile2014
![Page 102: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/102.jpg)
Complications Scrum Helps
• Distributed teams need more communication
ceremonies
• Foreign/contract workers need more
communication ceremonies
• Same process across teams is better – in most
cases other teams were using Scrum
• Simple common metrics -> better collaboration
• When starting from zero, Scrum was the
quickest path to team continuous improvement
![Page 103: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/103.jpg)
YOU
DECIDE@ernestmueller @iteration1#Agile2014
![Page 104: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/104.jpg)
![Page 105: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/105.jpg)
Using Metrics For Evil
@ernestmueller @iteration1#Agile2014
![Page 106: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/106.jpg)
Too Many Metrics
![Page 107: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/107.jpg)
Cargo Cult Metrics
![Page 108: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/108.jpg)
Demand Perfection
@ernestmueller @iteration1#Agile2014
![Page 109: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/109.jpg)
Weaponized Metrics
![Page 110: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/110.jpg)
![Page 111: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/111.jpg)
Recap
• Metrics are good - use them, be
guided by them, communicate with
them.
@ernestmueller @iteration1#Agile2014
![Page 112: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/112.jpg)
Recap
• Metrics can enhance your:
–Culture
–Productivity
–Process
![Page 113: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/113.jpg)
Recap
• Use them for good, not for evil.
![Page 114: Metrics Driven Development and DevOps - Agile 2014](https://reader033.vdocuments.us/reader033/viewer/2022061103/540d73c08d7f72747e8b49bf/html5/thumbnails/114.jpg)
@ernestmueller
@iteration1
theagileadmin.com