march 14, 2017 srecon17 americas changing old habits: rich ... · august 2016 create sre best...
TRANSCRIPT
![Page 1: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/1.jpg)
Changing Old Habits:Meetup’s Path to SRERich Hsieh, Core Engineering Manager, Meetup
SRECon17 AmericasMarch 14, 2017
![Page 2: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/2.jpg)
About Me● Core Engineering Manager● Been at Meetup since 2007● Organizer of the Dashing Whippets Running Team
![Page 3: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/3.jpg)
![Page 4: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/4.jpg)
A Quick Poll
![Page 5: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/5.jpg)
Defect EscalationMonitoring
AlertingPostmortems
SLOsSLAs
Automation
Error Budget
Playbooks
SRE Best Practices
![Page 6: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/6.jpg)
Outline
I. Getting Buy-in for an SRE Team
II. Changing Old Habits to Implement SRE Best Practices
![Page 7: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/7.jpg)
State of Meetup 2015
Product Engineers
SystemsEngineers
Ops?
Help!
SRE?
![Page 8: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/8.jpg)
Part I: Getting Buy-In
Job Description Six-Month Roadmap Exit Strategy
![Page 9: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/9.jpg)
Name of presenter Presentation name
Getting Buy-in #1: Job Description
● What exactly would I be doing?● How does this role fit within Meetup?● Are my expectations aligned with others?
![Page 10: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/10.jpg)
Getting Buy-in #2: Roadmap
July 2016 Start doing SRE-ish Things
August 2016 Create SRE Best Practices @ Meetup
September 2016 Work with a Product Team
October 2016 Making Product Better through SRE Development
November 2016 Reflection of Last 4 Months
December 2016 What is the future of SRE at Meetup?
Each month: Theme, Focus, and Expected Output
![Page 11: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/11.jpg)
Getting Buy-in #2: Roadmap
July 2016Theme: Start doing SRE-ish Things
Focus Create process to address on-call issues through Product Health
Expected Output Eliminate 1+ repeating on-call issues working w/responsible teams
Focus Start 1:1 conversations with stakeholders to raise awareness
Expected Output Create future action items based on 1:1 conversations
![Page 12: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/12.jpg)
Getting Buy-in #3: Exit Strategy
November 2016Theme: Reflection of Last 4 Months
Focus Determine if this SRE team is sustainable
Expected Output Leave or continue with SRE
● Make sure SRE best practices are instilled in other teams● “End well” -- no hard feelings
![Page 14: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/14.jpg)
Part II: Changing Old Habits
“a settled or regular tendency or practice,especially one that is hard to give up.”
-- Oxford Dictionary
“a settled or regular tendency or practice,especially one that is hard to give up.”
-- Oxford Dictionary
![Page 15: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/15.jpg)
Science!
*Click!*
![Page 16: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/16.jpg)
Science!
![Page 17: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/17.jpg)
REWARDROUTINECUE
The Habit Loop
![Page 18: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/18.jpg)
REWARDROUTINECUE
The Habit Loop (An Example)
![Page 19: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/19.jpg)
HABITS
PROCESSES
CULTURE
![Page 20: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/20.jpg)
To implement new processes, first focus on changing individual habits.
![Page 21: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/21.jpg)
Habit #1
![Page 22: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/22.jpg)
Habit #1
? !
![Page 23: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/23.jpg)
Habit #1: Old----?
----!
? !
#production-status#core-engineering
#engineering#orgx
#qa
and more...Team AlliesCommunity ExperienceProject Management
![Page 24: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/24.jpg)
Avoid using backchannels when communicating breakage.
![Page 25: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/25.jpg)
Habit #1: New!
? !
----?
----!
#production-status#core-engineering
#engineering#orgx
#qa
+ @oncall!
Yes, critical!
and more...
![Page 26: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/26.jpg)
Habit #1: New!
#engineering+ @oncall
Yes, critical!
![Page 27: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/27.jpg)
#engineer+ @oncall
HABITS
PROCESSES
CULTURE
Defect EscalationTeam AlliesCommunity ExperienceProject Management
CommunicationTransparency
![Page 28: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/28.jpg)
Habit #2
![Page 29: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/29.jpg)
Habit #2: Old
? !??“Brent knows!
He’ll fix!”
![Page 30: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/30.jpg)
Stop relying on a single engineer to fix an incident.
![Page 31: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/31.jpg)
Habit #2: New!
? !??“I got this.”
![Page 32: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/32.jpg)
It’s Time for a Playbook
??? > 1 ?
!!
![Page 33: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/33.jpg)
HABITS
PROCESSES
CULTURE
Playbooks
CollaborationKnowledge Sharing
? !??
![Page 34: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/34.jpg)
Habit #3
![Page 35: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/35.jpg)
Habit #3: Old
My fault.I see what I did
wrong.
???
![Page 36: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/36.jpg)
Don’t just take full personal responsibility.
![Page 37: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/37.jpg)
Habit #3: New!
Let’s see what went wrong in the team’s
process.It wasn’t just
my fault.
![Page 38: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/38.jpg)
First Step Toward Postmortems
Team AlliesProduct ManagersProject Management
Critical or
Major?
Facilitators in#postmortems
POSTMORTEM
* What went well* What could have gone better* Corrective and preventive measures
![Page 39: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/39.jpg)
HABITS
PROCESSES
CULTURE
Postmortems
BlamelessLearning
![Page 40: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/40.jpg)
Old Routine
REWARDROUTINECUE
The Habit Loop
![Page 41: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/41.jpg)
1. Job DescriptionAlign Expectations
2. RoadmapDemonstrate Impact
3. Exit StrategyIs it the right time?
In Review: Getting Buy-in for SRE
![Page 42: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/42.jpg)
1. Avoid Backchannels → Incident Escalation
2. Stop Relying on “Brent” → Playbooks
3. Don’t Take All Responsibility → Postmortems
In Review: Change Old Habits
? !?
?
#engineering+ @oncall
Yes, critical!
![Page 45: March 14, 2017 SRECon17 Americas Changing Old Habits: Rich ... · August 2016 Create SRE Best Practices @ Meetup September 2016 Work with a Product Team October 2016 Making Product](https://reader033.vdocuments.us/reader033/viewer/2022050410/5f86df0f29c238269633fb31/html5/thumbnails/45.jpg)
What habit are you going to change?
“a settled or regular tendency or practice,especially one that is hard to give up.”
-- Oxford DictionaryThanks!Meetup is hiring!