Starting with C:A DevOps Culture from the Ground Up

Bio - Jeff Smith

• Manager, Site Reliability Engineering at Grubhub

• Yes, we are also hiring.

• Yes, there is free food. Yes, it's totally awesome to work here.

Email: jeff@allthingsdork.comTwitter: @DarkAndNerdyBlog: http://www.allthingsdork.com

What Does DevOps Look Like?

The 4 Pillars of DevOps• Culture• Automation

• Metrics

• SharingCulture isn't the first pillar just for the sake of the acronym.

What is Culture?The values and behaviors that contribute to the unique social and psychological environment of an organization.

• Cultures are both micro and macro.

• Culture is contagious, both good and bad

• Expectations are set by management. Culture is set by employees, so it can be changed by employees.

Culture StatementsRespect, Integrity, Communication and Excellence. We treat others as we would like to be treated ourselves....We do not tolerate abusive or disrespectful treatment. Ruthlessness, callousness and arrogance don't belong here.

-- Enron

Every time we serve a customer, we should ask ourselves, “If I were the customer in this situation, how would this experience feel for me?— Wells Fargo

But How Does Culture Change?

The Change Agent

What Makes a Good Change Agent?Change agents are catalysts for change, but they are not the sole source of change.

• Strong relationships in the organization

• Knowledgable and leads by example

• Patience, patience, patience

• Empathy

Tips for Building Your CultureA small list of steps, actions you can take

Forming the Team

Teams exist outside of formal org structures because the act of teamwork is what makes a team.

Simple actions can facilitate Team Work

Pairing on Problems

Pairing on a problem maximizes information flow between Dev and Ops

• Helps each team member understand the workflow, tools, processes of the other

• Establishes a shared sense of ownership on the problem

• Subconsciously forces the pronoun of the problem to "We"

• Pairing is an opportunity to ask questions

• Pairing reveals dumb limitations

Overcommunicate ChangesChange is what drives a wedge between Dev and Ops. Change doesn't happen in a vaccum.

• Systems are too complex. Assume all changes are impactful

• It takes 10 seconds to send an email. Save in-depth technical details for follow ups

• Cast a wide net and then narrow down from there

Bonding RitualsGood teams share a level of camaraderie, but that camaraderie needs to be built.

• Create a bonding ritual amongst the team members. Happy hours, eating lunch together, LAN Parties or boardgame sessions

• Make sure the ritual is open and public

A by-product of bonding rituals is that the team begins to know each other, which is an important recipe for building trust.

Failure Is Inevitable - Fix the SystemComplex systems will fail from time-to-time. The complexity of the system typically means there's never a single failure point.

• Never discount what portions of the system might be at fault. Treat failures as a team event.

• Rally around fixing the problem

• Accept accountability for the injection of the problem, but fix the system not the individual.

I have no idea how I missed this index-dropping script as commissar.

I definitely reviewed the scripts that went into the release, because I pinged Someone about their feature toggle (script #1) and I remember convincing myself that the additional table_name (script #3) were trivial enough to not need any follow-up. I don't have any specific recollection of handling script #2.

All that to say, my bad. I really should've asked more questions about this script before the release.

Moving forward, could we talk about adjusting our process for DB scripts? I'd like to avoid the commissar-as-single-point-of-failure scenario in the future. Additionally, we talked about the need for a new baseline/horizon, and potentially a new tool to replace dbmaintain.

Again, sorry everybody.

Empathy Builds ConnectionsShowing empathy for your co-workers and the different challenges that they face in their role goes a long way. Everything is easy on the outside.

• Try to avoid hindsight bias. The facts on the ground might have been different 10 years ago when a decision was made

• Be vulnerable. (Especially Tech leaders) Show and communicate your weaknesses. "I Don't Know" is a perfectly acceptable response

Learn a Language Besides Bash

Don't Assign Responses to PeopleIt's easy to get into a mode where you think you know how people will respond and therefore don't take action. NEVER DO THIS

• Escalate issues specifically and directly

• Detail is your friend

• If a person can't help you, ask them for a recommendation of someone that can

Lunch and LearnCreate an environment of learning. Pick a topic, application, subsystem, process, anything people might not know a ton about and share your knowledge. Ask someone to either do the next one or to suggest a topic to learn more about.

Incident ManagementWhen things break, it's an awesome opportunity to build (or destroy) your team building.

• Incidents are not only an OPS or DEV problem. Have representaiton from both.

• Focus on the problem. The why's are only important if they're relevant to your troubleshooting

• Regroup with the entire team after resolution. (Post-mortems) Focus on events, not people. Discuss resolution options from all facets of the problem.

WHOA!This is a LOT

Start Small• Organize a few "team" lunches. Your org will probably even

cater them (use grubhub.com)

• Organize a Lunch and Learn

• Pair on your next problem

You will see progress. If you don't....

www.grubhub.com/careers

Thank You!