saltconf14 - justin carmony, deseret digital media - teaching devs about devops
DESCRIPTION
Let's set aside the buzzwords for a moment and have an honest discussion about DevOps. There is the idea of putting more Dev into Ops, but just as crucial (if not more crucial) is getting your Devs to think more like Ops. Most developers have little to no experience dealing with production environments, and helping them add value to DevOps efforts can be difficult. This talk will cover practical ways of mentoring Devs into more DevOps skills and responsibilities. Ultimately, the goal is to help your Devs gain the skills leading to better production health, application performance and uptime. Of course, we'll also consider how SaltStack can help.TRANSCRIPT
![Page 1: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/1.jpg)
Mentoring Devs Into DevOpsJustin Carmony Director of Development Deseret Digital Media
![Page 2: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/2.jpg)
![Page 3: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/3.jpg)
Lets Measure The Audience
• Who here is a…
• System Administrator?
• Developer?
• Manager / Management?
• “DevOp?”
![Page 4: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/4.jpg)
Confession:I’m a Developer
![Page 5: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/5.jpg)
![Page 6: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/6.jpg)
Self-Taught OpsBecause There Was No One Else To Do It
![Page 7: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/7.jpg)
About Me• Director of Development
for Deseret Digital Media
• Utah PHP UsergroupPresident
• I Make (and Break) Web Stuff (10 years)
• Salt User in Production since 0.8
(I <3 Salt)
![Page 8: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/8.jpg)
This Presentation• Lessons learned at DDM & previous jobs
• Insight into our process of increasing “DevOps”
• We’re still learning, but this what we’ve found.
• Slides will be posted online, so don’t worry about copying slide content.
• Feel free to ask on-topic questions during, and we’ll have questions at the end.
![Page 9: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/9.jpg)
About DDM
• Deseret Digital Media runs local website like KSL.com, DeseretNews.com
• Running National and International Websites like OK.com, familia.com.br, etc.
• ~10 million pageviews a day across sites.
• ~150 internal VMs, a few dozen physical machines, some AWS sprinkled around.
![Page 10: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/10.jpg)
Lets Start With a Story!
![Page 11: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/11.jpg)
You Work for an Awesome Tech Company
![Page 12: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/12.jpg)
Team Is Working Hard to Build New Things!
![Page 13: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/13.jpg)
You launch your awesome product!
![Page 14: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/14.jpg)
A Few More Features…
![Page 15: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/15.jpg)
… and next thing you know…
![Page 16: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/16.jpg)
![Page 17: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/17.jpg)
Awesome Job Team, We Rock!
![Page 18: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/18.jpg)
![Page 19: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/19.jpg)
We Need !Real-Time XYZ Feature!
ASAP!
![Page 20: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/20.jpg)
We Need !Real-Time XYZ Feature!
ASAP!
![Page 21: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/21.jpg)
We Need !Real-Time XYZ Feature!
ASAP!
&#$%!
![Page 22: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/22.jpg)
![Page 23: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/23.jpg)
![Page 24: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/24.jpg)
“Huh, it works if you ! just turn off caching…”!
- Dev @ 80th Hour This Week
![Page 25: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/25.jpg)
![Page 26: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/26.jpg)
“I’m sure this ! will work…”
![Page 27: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/27.jpg)
![Page 28: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/28.jpg)
![Page 29: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/29.jpg)
![Page 30: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/30.jpg)
“Our servers are melting!”
![Page 31: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/31.jpg)
![Page 32: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/32.jpg)
![Page 33: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/33.jpg)
![Page 34: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/34.jpg)
![Page 35: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/35.jpg)
![Page 36: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/36.jpg)
![Page 37: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/37.jpg)
“We Need a Better Solution!”
![Page 38: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/38.jpg)
So…
![Page 39: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/39.jpg)
Where Do We Start?
![Page 40: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/40.jpg)
![Page 41: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/41.jpg)
![Page 42: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/42.jpg)
![Page 43: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/43.jpg)
![Page 44: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/44.jpg)
We Have This Problem
![Page 45: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/45.jpg)
Challenges We Faced
• Giant mesh-up of technologies
• Tightly-coupled & fragile infrastructure
• Debugging production only bugs was difficult
• Bugs that were part code, part environment were a nightmare to track down.
![Page 46: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/46.jpg)
![Page 47: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/47.jpg)
![Page 48: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/48.jpg)
So One Day… We Had A Genius Idea!
![Page 49: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/49.jpg)
Lets Hire a DevOp!
![Page 50: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/50.jpg)
I’m Not JokingWe Actually Said This
![Page 51: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/51.jpg)
Two Problems with this “Idea”
![Page 52: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/52.jpg)
Problem #1 - We Didn’t Understand What We Really Wanted
![Page 53: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/53.jpg)
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit!
![Page 54: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/54.jpg)
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !Perfectly!
![Page 55: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/55.jpg)
Problem #2 - People Who Are Great At Dev & Ops Are Hard To Find
![Page 56: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/56.jpg)
Expectation:
![Page 57: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/57.jpg)
Reality:
![Page 58: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/58.jpg)
Honest Team Discussion: What is it we’re really looking for?
![Page 59: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/59.jpg)
We Discovered a Few Things
![Page 60: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/60.jpg)
What does DevOps Mean To Us?
• DevOps: Dev & Ops, a Culture of Collaboration
• Our Goal: “10 deploys a day without issues”
• Everyone shares the goal of quick development of features AND a stable system that stays up.
![Page 61: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/61.jpg)
Team StructureDevs: 30 Ops: 2
![Page 62: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/62.jpg)
Team StructureDevs: 30 Ops: 2
DevOps: 1
![Page 63: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/63.jpg)
Team StructureDevs: 30 Ops: 2
DevOps: 1
![Page 64: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/64.jpg)
Team StructureDevs: 30 Ops: 2
DevOps: 1
Hiring one person won’t just solve all our problems!
![Page 65: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/65.jpg)
Team Realizations
• Hardest problem already solved: awesome team
• No foreseeable rapid expansion, must operate at our current scale
• Each Project’s Director of Development was acting as the bridge between Dev and Ops, but would become a bottleneck.
![Page 66: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/66.jpg)
Teams Already Had Some Ad-Hoc DevOps Tools
- Real-time Logging - Capistrano Deploys
- Nagios Alerts - Server Metrics - Puppet for File Mgmt
- App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management
- Homebrewed Metrics Sys. - Homebrewed Alert System
![Page 67: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/67.jpg)
Teams Already Had Some Ad-Hoc DevOps Tools
- Real-time Logging - Capistrano Deploys
- Nagios Alerts - Server Metrics - Puppet for File Mgmt
- App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management
- Homebrewed Metrics Sys. - Homebrewed Alert System
![Page 68: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/68.jpg)
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !Perfectly!
![Page 69: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/69.jpg)
Step 1: Hire a DevOp!
Step 2: ????????????!
Step 3: Profit! Everything Works !Perfectly!
![Page 70: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/70.jpg)
We Formed A Strategy
![Page 71: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/71.jpg)
Step #1: Promote Dev to DevOp Role
![Page 72: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/72.jpg)
WAIT!Isn’t that the advice you just
said was a bad idea?!
![Page 73: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/73.jpg)
DevOp Engineer
• Well Defined Role:
• Ownership over the TOOLS to improve DevOps efforts.
• Resource for other teams to help use DevOps Tools.
• Easy to work with, aptitude for systems & ops, likes to try new things.
![Page 74: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/74.jpg)
Promoting From Within
• A seasoned dev for your team already knows:
• Your Pain Points
• Your System’s Quirks
• How the “Chaos Works”
• Knows the people & personalities on your team
![Page 75: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/75.jpg)
Step #2: Change Team Structure
![Page 76: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/76.jpg)
Team StructureDevs: 30 Ops: 2
![Page 77: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/77.jpg)
Team StructureDevs: 30 Ops: 2
![Page 78: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/78.jpg)
Team StructureGoal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops
![Page 79: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/79.jpg)
Team StructureGoal: Spread Out Expertise By Increasing
Ops Experience & Skills Among Devs
Dev Ops
![Page 80: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/80.jpg)
Team Structure
Dev Ops
![Page 81: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/81.jpg)
Team Structure
Dev Ops
![Page 82: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/82.jpg)
Increasing Ops Among Devs
• Identify Devs who liked “Ops” & wanted to Learn
• Pair Dev with Op / Director
• Learning Dev works on things, not Op /Director.
• Pair program if needed.
![Page 83: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/83.jpg)
Step #3: Increase Everyone’s Insight
![Page 84: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/84.jpg)
Step #3: Increase Everyone’s Insight
![Page 85: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/85.jpg)
![Page 86: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/86.jpg)
Metrics• Everyone has access to Network, Server, and
Application Metrics.
• Consolidate & reduce places to look. We try to pipe everything to StatsD / Graphite
• Each developer trained to add & track metrics in production.
• We’re okay with 98% uptime of stats to avoid complexity.
![Page 87: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/87.jpg)
![Page 88: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/88.jpg)
Real-Time Logging
![Page 89: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/89.jpg)
Real-Time Logging• Harder & more complicated at scale
• Still trying to solve well, we have lots of logs.
• Start with small window of data (i.e. 48 hours) and start to expand window.
• We’re trying Logstash, ElasticSearch, and Kibana right now.
• Generate Statistics off our Logs
![Page 90: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/90.jpg)
Tracking Changes
• Everything, everything, everything in git (we use GitHub)
• Everyone has access to all repos • Everyone does work through Pull Requests • Everyone has their work code reviewed *
* - Your can merge w/o a review, but must be willing to defend your choice
![Page 91: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/91.jpg)
Deploys
![Page 92: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/92.jpg)
Everyone Can Deploy
• Automated our deployment process to a single step.
• Everyone can deploy, deployments are logged
• Easy rollback is a requirement!
• Implementing feature flags to turn off single parts of our application.
![Page 93: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/93.jpg)
TestsUnit
Functional Integration
Acceptance etc
![Page 94: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/94.jpg)
Automated Tests
• If you want to trust your Devs, you need tests
• Legacy apps we wrote Integration Tests
• New Apps & Refactored Legacy Parts have Unit Tests
• Continuous Integration to make sure tests run
![Page 95: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/95.jpg)
![Page 96: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/96.jpg)
So, where’s the salt?
![Page 97: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/97.jpg)
Step #4: Devs Use The Ops Tools
![Page 98: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/98.jpg)
Devs can grok salt
![Page 99: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/99.jpg)
Safe Environment For Devs to Learn
![Page 100: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/100.jpg)
Safe Environment For Devs to Learn
salt \* cmd.run "rm -rf /tmp /*"
![Page 101: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/101.jpg)
Safe Environment For Devs to Learn
salt \* cmd.run "rm -rf /tmp /*"
Salt is awesome, but it can’t !recover from that
![Page 102: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/102.jpg)
Dev Salt MasterDevs Can Look Into Every Server
![Page 103: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/103.jpg)
Dev Salt Master• Every server has two minions:
• Admin Salt (aka root)
• Dev Salt (aka bob)
• Each connect to different master server:
• All Devs have access to Dev Salt Master
• Trusted Devs get access to Admin Salt Master
![Page 104: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/104.jpg)
Everything Salty in GitReminder:
![Page 105: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/105.jpg)
Dev Environment
• Developers own the Dev Environment
• Dev Teams manage the Salt States for their Env
• Vagrant + Salt for their Env
• Who makes changes? Developers
• DevOp helps advise & offer support
![Page 106: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/106.jpg)
Team Structure
Dev Ops
![Page 107: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/107.jpg)
Stage Environment• Stage & Production use same salt repos, different
branches
• Developers make all the changes for Application Servers
• All Changes through Pull Requests
• We’ll worry about env changes before code
• Small changes we quickly release, large or long running branches are scary & dangerous
![Page 108: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/108.jpg)
Production Environment
• Merge change to Production Branch
• salt \* state.highstate
• Reminder: Small quick changes over time, never a large change at once.
![Page 109: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/109.jpg)
Environment Caveats• Ops & DevOps Manage VM Hosts, Physical Load
Balancers, FireWalls, etc
• Ops & DevOps manage servers that deal with data:
• MySQL
• MongoDB
• etc
![Page 110: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/110.jpg)
Mentoring Devs
![Page 111: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/111.jpg)
Mentoring Devs
• Not every Dev will become an amazing DevOp
• Thats okay!
![Page 112: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/112.jpg)
Level of “DevOps” Skills
• Thinks about their impact on Ops: Everyone
• Able to debug issues with production: Most
• Able to make changes to environments: Many
• “Awesome DevOp”: Some
![Page 113: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/113.jpg)
Team’s “DevOps” Skills
0
25
50
75
100
Think Debug Change DevOp
Current Goal
![Page 114: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/114.jpg)
So Everything Is Awesome for us, right?
![Page 115: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/115.jpg)
Honesty Slide: We Have Skeletons In Our Closets
![Page 116: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/116.jpg)
Where We Are At
• All Dev Environments using Vagrant + Salt
• All New Stage & Prod Environments are Salty
• Some Legacy Stage & Production Envs are Salty
• Continuously working on getting out stuff salty.
![Page 117: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/117.jpg)
Making This Work For Your Team
![Page 118: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/118.jpg)
Honest Introspection• Determine for your
team what are your… • Strengths • Weaknesses • Problems • Goals
![Page 119: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/119.jpg)
Increase Team’s Insight • Make sure devs can see
& understand how their code performs
• Increase responsibility of team for those metrics.
• If they break it, they fix it. Do not always bail them out.
• Everyone can see everything.
![Page 120: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/120.jpg)
Increase Team’s Insight • Make sure devs can see
& understand how their code performs
• Increase responsibility of team for those metrics.
• If they break it, they fix it. Do not always bail them out.
• Everyone can see everything.
![Page 121: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/121.jpg)
Mentor Those With Desire / Aptitude
• Give Developers Safe Environment to Learn
• Let them submit code-reviewed changes for Stage & Production
• When teaching / mentoring, let the learner drive, kindly offer advice and help.
• It takes time, but worth the investment.
![Page 122: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/122.jpg)
A Few Final Thoughts
![Page 123: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/123.jpg)
Team Culture Matters
![Page 124: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/124.jpg)
Positive Influence
![Page 125: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/125.jpg)
Questions?
![Page 126: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/126.jpg)
Thank YouJustin Carmony
Email: [email protected]: @JustinCarmony
IRC: carmony #salt #uphpuWebsite: [email protected]
![Page 127: SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps](https://reader038.vdocuments.us/reader038/viewer/2022103016/554f8eacb4c9052a518b526c/html5/thumbnails/127.jpg)
p.s. we’re hiring, email / pm / tweet me