devops culture shift: expanding on-call responsibilties
TRANSCRIPT
DevOps Culture Shift: Expanding On-Call Responsibilities
Todd Vernon - CEO, VictorOpsKurt Bittner - Research Analyst,
Forrester ResearchNick Goodman - Director of Platform Engineering, BunchballPaul Beltrani - TechOps, Onshape
#DevOpsCulture
Moderator
Todd Vernon, CEO VictorOps
Industry veteran of SaaS business
CTO, Raindance Communications CEO, Lijit NetworksCEO, VictorOps@toddvernon
#DevOpsCulture
Yesterday (Developers Developed)
We spent a long time writing product requirements
We spent a longer time writing software
We pushed new code every 3-6 months to production
When it worked (12 hours later), we locked the data center
Reliability was obtained by NOT CHANGING THINGS
#DevOpsCulture
Today (Development is the new NOC)
Agile replaced Waterfall
Virtual replaced Physical
Puppet and Chef replaced System Admins
Continuous Delivery replaced lack of delivery
DevOps is replacing Operations
Developers are replacing the NOC #DevOpsCulture
Housekeeping (What you need to know)
• Half of today’s webinar will be a panel discussion seeded by questions from attendees
• Chat in your questions as we go. Two ways.– Chat directly into the platform– Twitter @VictorOps or #DevOpsCulture
• Today’s webinar will be recorded and the slides made available
• All attendees will receive a copy of the recent Forrester brief “Putting Devs On The Front Lines”
#DevOpsCulture
Poll (Audience Participation)
• How many currently include developers in their on-call rotations?
#DevOpsCulture
© 2015 Forrester Research, Inc. Reproduction Prohibited
Putting Devs On The Front LinesDevOps Culture Begins With Visibility and Responsibility
#DevOpsCulture
© 2015 Forrester Research, Inc. Reproduction Prohibited 10© Golden Gate Bridge, Highway and Transportation District
Making Software Is Nothing Like Building A Bridge
http
://b
etan
ews.
com
/201
5/02
/26/
agile
-ver
sus-
wat
erfa
ll-de
velo
pmen
t-th
e-ca
se-f
or-a
gile
/
http://1
08
0pa
nim
ew
allpa
pers.co
m/n
etw
ork/ne
two
rk-op
eratio
ns-ce
nter.h
tml
Dev and Ops live in different worlds, with different cultures and values
© 2015 Forrester Research, Inc. Reproduction Prohibited 11© Golden Gate Bridge, Highway and Transportation District
Making Software Is Nothing Like Building A Bridge
http
://b
etan
ews.
com
/201
5/02
/26/
agile
-ver
sus-
wat
erfa
ll-de
velo
pmen
t-th
e-ca
se-f
or-a
gile
/
http://1
08
0pa
nim
ew
allpa
pers.co
m/n
etw
ork/ne
two
rk-op
eratio
ns-ce
nter.h
tml
Dev and Ops live in different worlds, with different cultures and values
Innovation (Encourage Change)
Stability(Prevent Change)
© 2015 Forrester Research, Inc. Reproduction Prohibited 12© Golden Gate Bridge, Highway and Transportation District
Making Software Is Nothing Like Building A Bridge
http
://b
etan
ews.
com
/201
5/02
/26/
agile
-ver
sus-
wat
erfa
ll-de
velo
pmen
t-th
e-ca
se-f
or-a
gile
/
http://1
08
0pa
nim
ew
allpa
pers.co
m/n
etw
ork/ne
two
rk-op
eratio
ns-ce
nter.h
tml
Dev and Ops live in different worlds, with different cultures and values
Innovation (Encourage Change)
Stability(Prevent Change)
Customer Success Requires Both
© 2015 Forrester Research, Inc. Reproduction Prohibited 13
Measure everyone the same way.
Reward improved customer experiences.
http://e-strategyblog.com/2011/05/daily-numbers-killing-time-on-mobile/
© 2015 Forrester Research, Inc. Reproduction Prohibited 14
Walk in the other person’s shoes
http://newsroom.blogs.cnn.com/2009/11/12/shoes-for-africa-sat-nov-14-at-10-p-m-et-on-cnn/
© 2015 Forrester Research, Inc. Reproduction Prohibited 15http://www.cnn.com/2013/11/27/world/gallery/most-beautiful-libraries-in-the-world/
Expand Application Support Knowledge
© 2015 Forrester Research, Inc. Reproduction Prohibited 16http://www.huffingtonpost.com/2013/05/22/blended-family-advice_n_3322007.html
“If you write code, you’re on call.
If you want to take vacations, write good tests.”
- Mario Cruz, CIO at Choose Digital
© 2015 Forrester Research, Inc. Reproduction Prohibited 17http://aragec.com/winding+roads.html#prettyPhoto[galeri]/1/
Think End-to-End
“The more time that developers spend
working in production on operations
issues with operations staff, …
… the more that they will learn about
what it takes to design and build a real-
world system.”
© 2015 Forrester Research, Inc. Reproduction Prohibited 18https://leadershipfreak.files.wordpress.com/2015/01/winston-churchill-the-price-of-greatness.jpg
DEVOPS is a Culture of InnovationCulture of Creation
Everybody creates something Technology choices driven by values
Culture of Empowerment It's all code, so everybody can work on it Developers feel responsible for their code's behavior
Product-Centric Culture Ignore IT Our roles exist to deliver a product to the customer Focus on the things we want to be good at
DEVOPS is HARDConvincing People is Hard
“I'm not a sys admin, and I don't want to be one” “I didn't write that, so I can't fix it”
DEVOPS Technologies Create Different Problems Biggest benefit comes from wide adoption within the code Such problems require a DEVOPS team to address
Onshape• https://www.onshape.com/• Founded by same people who founded
Solidworks• Disrupted the workstation era with CAD on MS Windows• Disrupting desktop era CAD with Cloud based CAD
•Many, smart developers who are new to WebOps
• Challenge, foster a DevOps culture• Started by getting developers involved with support rotations (on-
call)
How we brought Dev to Ops• Developers participate in the on-call rotation
• Change the traditional role of on-call• Incident manager VS repair person
• Lots of support and guidance• Paired on-call, person from Dev and a person from Ops
• Knowledge transfer in both directions
• Documentation and run books
• Group / User autonomy• Teams manage their own rotation
• Users manage their own alerting
Primary On-call• Responsible for making sure the service is
available• keeps an eye on the service and supporting infrastructure• responds to alerts 24/7*
• Manages incidents• Doesn’t have to fix the issue but is responsible for coordinating
other resources and managing the incident to resolution
• Point of contact• Keeps others informed, e.g. Customer Service• Provides updates to the teams. (Has authority to delegate if
they're busy.)
Panel (Q&A)
• Why change the status quo?• How do I align two disparate groups?• How do I remove the obstacles that
separate Dev and Ops teams?• Will customers see a difference if devs are
on call? • Can IT start to think differently?
#DevOpsCulture