devops - nothing stays the same
TRANSCRIPT
![Page 1: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/1.jpg)
@mmaibaum
DevOps - Nothing Stays The SameMichael Maibaum
![Page 2: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/2.jpg)
@mmaibaum
Precis
Sky Betting & Gaming has become one of the largest online operators in the UK, undergoing a period of sustained high growth in customer numbers, transaction rate, staff size, and number of systems. Five years ago, the company established its first DevOps team, and since then, DevOps has become a major part of the way Sky Betting & Gaming does things. However, what that means keeps changing. Michael Maibaum describes how the DevOps function has changed repeatedly over the last few years to help the company continue to move fast and keep systems operating through organizational and technical challenges.Originally, the DevOps team was established as a group of like-minded engineers keen to smooth the delivery of software into operations and make it run better. As the business grew, the engineering teams were split and the accumulated DevOps knowledge distributed into those new groups, but the team soon found out that things didn’t fit into a distributed function and features of the platform that need ownership. As a result, platform teams were formed to produce products that other teams use. Sky Betting & Gaming’s DevOps experts now come in two categories: those that directly work in or with individual (product) engineering teams and those that deliver a platform that makes life easier for the rest of the engineering function.It is easy to see a narrow definition of DevOps as part of the function of a specific engineering team. However, in the experience of Sky Betting & Gaming, to achieve a truly effective delivery and operational culture (and indeed, DevOps) once you have hundreds of engineers requires investment in the platform as a product in and of itself.Michael outlines the history of DevOps at Sky Betting & Gaming and explains how the company has taken its DevOps philosophy into its vendors as it takes its first steps into the cloud.
![Page 3: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/3.jpg)
@mmaibaum
Introducing Sky Betting & Gaming
• One of the top 3 online gambling operators in the UK
• 3 Categories of product
• Sportsbook
• Free Sports related content
• Gaming
![Page 4: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/4.jpg)
@mmaibaum
A Diverse Technology Stack
![Page 5: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/5.jpg)
@mmaibaum
In the Beginning
![Page 6: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/6.jpg)
@mmaibaum
A Story of Change & Growth
2010 2015
£50M
£350M• Business grew slowly for the first 8
years post-acquisition
• Interactive tv was seen as the next big thing
• Major growth period starts in ~2008
![Page 7: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/7.jpg)
@mmaibaum
2008
![Page 8: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/8.jpg)
@mmaibaum
Infrastructure & Ops Only
• Tech Team
• No in-house development
• Hosting and operating third party vendors applications
• Waterfall project management and delivery
![Page 9: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/9.jpg)
@mmaibaum
Focus on the Web• Increased focus on the web, but still delivered by third party vendor software teams
• Starting to deliver real customer & revenue growth at this point, company profits start to grow.
![Page 10: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/10.jpg)
@mmaibaum
The End of the Beginning
• Business wanted to increase velocity
• More frequent change
• Cheaper to deliver new features
• More control
• Time to bring the user experience in-house
![Page 11: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/11.jpg)
@mmaibaum
The First ProblemHow to improve delivery & reliability from the in-house software teams?
Vendor
Ops
Dev
![Page 12: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/12.jpg)
Backlog Wip Done Test Live
Delivery Team 1
Infrastructure
Service Desk
1st Line On Call
The First Answer - 2011
DevOps Team
2nd Line On Call
![Page 13: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/13.jpg)
@mmaibaum
DevOps
• Build tooling focussed on developer productivity and system reliability
• First CI pipelines with Jenkins
• Load testing, capacity and scaling (large, peaky events)
![Page 14: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/14.jpg)
@mmaibaum
Typical Saturday (Bets & Logins)
![Page 15: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/15.jpg)
@mmaibaum
Centralised DevOps?
• Probably not want what you want to aspire to
• But… Can be a good way to start
• Start the cultural shift
• Solve the problem of not enough ‘DevOps’ to go around
![Page 16: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/16.jpg)
@mmaibaum
DevOps Starting to Work
• Fit in well with increasing emphasis of agile delivery (Scrum, then Kanban)
• Central team provided a concentration of capability and culture
• Demonstrable wins important for adoption
• In-House Dev going well enough that we start work on Sky Vegas ‘in-house’ front end
![Page 17: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/17.jpg)
@mmaibaum
Commits/MonthCommits/Release
![Page 18: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/18.jpg)
@mmaibaum
March 2012We go from 'does anyone think we will be using the in-house site for Grand National?' to 'does anyone think we *won't* be using the in-
house site?'
![Page 19: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/19.jpg)
@mmaibaum
Soon we had another problem…How do we manage configuration for Disaster Recovery?
![Page 20: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/20.jpg)
@mmaibaum
Config ManagementConfigureServer - Many custom perl scripts
Revision control via
something.pl.freds-test
something.pl.bak, something.pl.bak2, something.pl.old,
something.pl.not-sure-what-this-is-but-scared-to-delete-it
![Page 21: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/21.jpg)
@mmaibaum
Platform Evolution
• Another Centralised Team
• This time born out of infrastructure and the DevOps team
• Created with a specific purpose (fix config management for DR, aka Chef All the Things)
• This turned out to be hard - At least 1.5 years effort
• Lots of concurrent change, with little effective standardisation
![Page 22: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/22.jpg)
Publish
Applications
Infrastructure Code (Chef, Ruby,
ServerSpec)
Publish
Application Code (PHP, NodeJS, React,
Java)
Release
Configuration
Orchestra
tion
![Page 23: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/23.jpg)
@mmaibaum
The Beginning of the MiddleThe birth of tribes
![Page 24: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/24.jpg)
@mmaibaum
DevOps was in danger of becoming Ops
• With teams growing and changing the ways they work, a centralised devops team increasingly mis-aligned.
• DevOps engineers were spread out around different teams
![Page 25: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/25.jpg)
@mmaibaum
Tribes
• Inspired by the Spotify white paper
• Overall team getting too big
• Sub-divide into autonomous teams first at main product level (tribes, e.g. bet/gaming) then squads within those.
![Page 26: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/26.jpg)
Core Tribe
Gaming Tribe
Infra Tribe
Bet Tribe
Growing Pains - 2013
Backlog Wip Done Test Live Backlog Wip Done Test Live
Web Experience Place & Track Squad
Platform Ops
Service Desk
1st Line On Call
Backlog Wip Done Test Live Backlog Wip Done Test Live
Casino Squad Vegas Squad
Backlog Wip Done Test Live Backlog Wip Done Test Live
Platform Evo Account SquadBacklog Wip Done Test Live
Infra Squad
SLM
Security
![Page 27: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/27.jpg)
@mmaibaum
Better…
• Alignment with development
• Ownership of Ops issues in squads
• Knowledge of services each ‘DevOp’ was working with
![Page 28: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/28.jpg)
@mmaibaum
But…
• The ‘DevOps’ were still the first on-call, cross tribe
• Increasingly limited knowledge of other teams services
• Team size awkward,
• too many services for individuals to know all services,
• not big enough to populate on-call with the right Ops skills
![Page 29: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/29.jpg)
@mmaibaum
And…
![Page 30: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/30.jpg)
Publish
Applications
Infrastructure Code (Chef, Ruby,
ServerSpec)
Publish
Application Code (PHP, NodeJS, React,
Java)
Release
Configuration
Orchestra
tion
![Page 31: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/31.jpg)
Publish
Applications
Infrastructure Code (Chef, Ruby,
ServerSpec)
Publish
Application Code (PHP, NodeJS, React,
Java)
Release
Configuration
Orchestra
tion
Integration/Test/Production
![Page 32: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/32.jpg)
Dev Ops
Publish
Applications
Infrastructure Code (Chef, Ruby,
ServerSpec)
Publish
Application Code (PHP, NodeJS, React,
Java)
Release
Configuration
Orchestra
tion
Integration/Test
![Page 33: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/33.jpg)
@mmaibaum
The Middle of the Middle
aka 2014
![Page 34: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/34.jpg)
@mmaibaum
Tribes have local focus• Optimising for local concerns
• Delivery of that product
• Improvement of their technology stack
• Improving their processes
• Local service delivery teams
• Bet WebOps team (monitoring and so on)
• Bet SRE team
• Bet Delivery Engineering
![Page 35: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/35.jpg)
@mmaibaum
Squads taking ownership…
• End to end ownership
• Design, Build, Run, Change, Fix, Retire
• Full support in a team - on call
• There are specialists, but they aren’t the only people that can do things
![Page 36: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/36.jpg)
@mmaibaum
But…
Should everything be a local concern?
![Page 37: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/37.jpg)
@mmaibaum
Cross Cutting Platform Features• What happened to Platform Evolution?…. It evolved into Platform Services
• There is a wider set of ‘PaaS’ like services that would be useful across the business
• Counterbalance ‘everything local’ inefficiencies
• What
• PlatCI - Our CI as a service platform (Jenkins etc),
• Shared Kafka - Messaging Platform as a Service
• Self Contained Projects - Get rid of the Dev/Ops tooling projects/tooling splits
![Page 38: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/38.jpg)
Orchestra
tion
Application + Config
Build
JenkinsPublish
Cookbook (sbg_myapp)
• Infrastructure Code (Chef, Inspec, Custom Resources)• Application Code (PHP, NodeJS, Java)• CI Pipeline (Jenkins Pipeline, Chef)• Integration Tests (Kitchen, Chef)
![Page 39: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/39.jpg)
pscliThe ‘Glue’ - enables the consistent composition of toolsets in different environments
• Internal Tool• Written in Go• Pulls in various ‘tools’ Docker images• Executes tools in containers, e.g.
• ChefDK• Terraform• Packer• AWS Authentication• Hashicorp Vault• Code Generation
![Page 40: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/40.jpg)
pscli generate cookbook myapp
Git--volumes-from /git
/opt/chefdk
ChefDK--volumes-from
Docker Registry
Code Generator--volumes-from
/generator
{command runner} ~/workspace/myapp
![Page 41: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/41.jpg)
pscli kitchen converge
Git--volumes-from
/git
{command runner}
Kitchen Suite A
Kitchen Suite B
/opt/chefdk
ChefDK--volumes-from
Docker Registry
![Page 42: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/42.jpg)
pscli terraform apply
/opt/terraform
Terraform--volumes-from
Docker Registry
{command runner}
![Page 43: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/43.jpg)
@mmaibaum
Vendors
• Bad vendor relationships can cripple progress
• Or they can enable it
• It is in your interest to help them as much as you can
![Page 44: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/44.jpg)
@mmaibaum
Delivery Partners
• Bad vendor relationships can cripple progress
• Or they can enable it
• It is in your interest to help them as much as you can
![Page 45: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/45.jpg)
@mmaibaum
Tribes Getting Too Big
• Feeling the pain of growth again
• Bet Tribe bigger than whole tech organisation was 3 years earlier
• Break up of bet tribe into smaller, nested, tribes
• Making more specialist roles closer to each ‘product’ delivery squad (e.g. SRE as part of a squad)
![Page 46: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/46.jpg)
@mmaibaum
Two kinds of ‘DevOps’• People in every delivery team, some of these are DevOps specialists but the
whole team cares about the whole product lifecycle
• People in specialist teams working on shared platform capabilities
• Platform Services - Cross Tribe Services
• Platform Engineering (Big tribes have their own ‘shared’ services)
• Delivery Engineering (Specialists in tribes helping squads optimise reliability & delivery, especially things like release engineering, CI, etc)
![Page 47: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/47.jpg)
@mmaibaum
Path Dependency
• It really matters where you are, and where you are coming from
• At least as much as where you’d like to go to.
• There isn’t a path, because there isn’t an environment (and it changes)
![Page 48: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/48.jpg)
@mmaibaum
The End?
![Page 49: DevOps - Nothing Stays the Same](https://reader031.vdocuments.us/reader031/viewer/2022022203/58773d541a28ab342e8b5e71/html5/thumbnails/49.jpg)
@mmaibaum
There is no EndExcept of this talk
http://engineering.skybettingandgaming.com