![Page 1: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/1.jpg)
Thinking Inside the Container:A Continuous Delivery Story
Maxfield StewartEngineering Manager: Riot Games
![Page 2: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/2.jpg)
Who’s This Guy?
![Page 3: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/3.jpg)
Thinking Inside the Container
![Page 4: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/4.jpg)
![Page 5: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/5.jpg)
A Quick Note on Riot Games
![Page 6: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/6.jpg)
The Scale of LeagueMORE THAN67 MILLION
MONTHLY ACTIVE
PLAYERS
MORE THAN27 MILLION
DAILY ACTIVE PLAYERS
7.5 MILLION
PEAK CONCURRENT
PLAYERS
![Page 7: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/7.jpg)
1.25 Million Builds a Year
![Page 8: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/8.jpg)
10,000 - 14,000 Containers A Week
![Page 9: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/9.jpg)
120 Jobs An Hour
![Page 10: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/10.jpg)
A Containerized Build Farm
JENKINS SWARM
DOCKERHOSTDOCKERHOSTDOCKERHOSTDOCKERHOSTDOCKERHOST
cAdvisorDocker-GC
Container Metrics
CentOS 7.2/Docker 1.10.3
4 Core/32GB RAM/120GB LVS
DRYDOCK
BUILD HOSTS
SSH
Docker API Docker API
Docker API
![Page 11: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/11.jpg)
STORY TIME!(aka. Presenting the Problem)
![Page 12: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/12.jpg)
3500+ Unique Build Jobs
![Page 13: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/13.jpg)
650+ Builds an Hour
![Page 14: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/14.jpg)
90+ Build Slaves
![Page 15: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/15.jpg)
Expecting 100 More!
![Page 16: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/16.jpg)
Story Time....
Ticket
Engineers Build Team
![Page 17: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/17.jpg)
Story Time....Ticket Ticket
Ticket Ticket
Ticket Ticket Ticket
Ticket
Ticket Ticket
Ticket
Engineers Build Team
![Page 18: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/18.jpg)
What Did We Want?
TEAMS NEEDED TO MOVE FAST
PRODUCTS HAD TO OWN THEIR STACK
CONFIGURATION AS CODE
![Page 19: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/19.jpg)
Maybe We Want...
![Page 20: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/20.jpg)
“Along Came a Whale”
(More Story Time!)
![Page 21: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/21.jpg)
![Page 22: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/22.jpg)
Oh Look! Another Way to Deploy!
![Page 23: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/23.jpg)
Docker For Newbz
![Page 24: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/24.jpg)
![Page 25: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/25.jpg)
Jenkins Primer
![Page 26: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/26.jpg)
Jenkins Primer
JENKINS “Win32” + “Java” + ”TeamA”
Master Slaves
“Cent7” + “Java” + ”TeamB”
“Cent7” + “GoLang” + ”TeamB”
![Page 27: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/27.jpg)
Jenkins Primer
JENKINS “Win32” + “Java” + ”TeamA”
Master Slaves
“Cent7” + “Java” + ”TeamB”
“Cent7” + “GoLang” + ”TeamB”
Build Job“Win32” + “Java” + ”TeamA”
![Page 28: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/28.jpg)
Jenkins Primer
JENKINS “Win32” + “Java” + ”TeamA”
Master Slaves
“Cent7” + “Java” + ”TeamB”
“Cent7” + “GoLang” + ”TeamB”
Build Job“Win32” + “Java” + ”TeamA”
Build JobBuild JobBuild JobBuild Job
Queue
![Page 29: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/29.jpg)
Jenkins Primer
JENKINS “Win32” + “Java” + ”TeamA”
Master Slaves
“Cent7” + “Java” + ”TeamB”
“Cent7” + “GoLang” + ”TeamB”
Build Job“Win32” + “Java” + ”TeamA”
Build JobBuild JobBuild JobBuild Job
Queue
![Page 30: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/30.jpg)
What Did We Need?CONTAINER RUNNING AS A BUILD SLAVE
HOW TO SCALE ITWAY TO HAVE ENGINEERS PROVIDE DOCKERFILES
PROFIT?
![Page 31: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/31.jpg)
A Build Slave Container
![Page 32: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/32.jpg)
Add a Bit of Secret Sauce...
Base Slave ImageUniversal Riot Tools
Product/App Specific Build Stuff
![Page 33: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/33.jpg)
A Real Example
![Page 34: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/34.jpg)
Provisioning and Plugins
+ = ?
![Page 35: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/35.jpg)
Of Whales and Plugins
MESOS PLUGIN
KUBERNETES PLUGIN
DOCKER PLUGIN
![Page 36: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/36.jpg)
Of Whales and Plugins
MESOS PLUGIN
KUBERNETES PLUGIN
DOCKER PLUGIN
![Page 37: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/37.jpg)
Quick Look
![Page 38: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/38.jpg)
Provisioning and Plugins
Image Name
Label
![Page 39: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/39.jpg)
Groovy To the Rescueht
tps:
//gith
ub.c
om/m
axfie
lds2
000/
dock
erje
nkin
s_tu
toria
l/tre
e/m
aste
r/jen
kins
scrip
ts
![Page 40: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/40.jpg)
We Created A Monster
![Page 41: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/41.jpg)
We Need to Inspect Our Containers
![Page 42: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/42.jpg)
We Need to Inspect Our Containers./harbormaster --registry_user "$REGISTRY_USER" --registry_token "$REGISTRY_TOKEN" --jenkins_user $USER --jenkins_pass $PASSWORD --jenkins --dockerhost $TEST_DOCKER_HOST "$IMAGENAME" "$CLOUDNAME" "$LABELNAME"
![Page 43: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/43.jpg)
Are We Done?
![Page 44: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/44.jpg)
But It Needs to Scale!
![Page 45: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/45.jpg)
If Only...
![Page 46: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/46.jpg)
If Only...
![Page 47: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/47.jpg)
But It Needs to Scale!
JENKINS SWARM
DOCKERHOSTDOCKERHOSTDOCKERHOSTDOCKERHOSTDOCKERHOST
cAdvisorDocker-GC
Container Metrics
CentOS 7.2/Docker 1.10.3
4 Core/32GB RAM/120GB LVS
BUILD HOSTS
SSH
Docker API Docker API
![Page 48: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/48.jpg)
Putting It All Together
Engineer
REGISTRY
Image
![Page 49: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/49.jpg)
Putting It All Together
Engineer
REGISTRYHARBOR MASTER
Image Name + Jenkins Label
![Page 50: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/50.jpg)
Putting It All Together
Engineer
REGISTRYHARBOR MASTER
Runs Tests
![Page 51: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/51.jpg)
Putting It All Together
Engineer
REGISTRYHARBOR MASTER SWARM DOCKERHOST
DOCKERHOST
DOCKERHOST
Image Pulled to Swarm
![Page 52: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/52.jpg)
Putting It All Together
Engineer
REGISTRYHARBOR MASTER SWARM DOCKERHOST
DOCKERHOST
DOCKERHOST
Groovy API CalledJenkins
![Page 53: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/53.jpg)
Build Job Quick Look
![Page 54: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/54.jpg)
Dockerception
![Page 55: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/55.jpg)
Where to Build Containers?
JENKINS SWARM
DOCKERHOSTDOCKERHOSTDOCKERHOSTDOCKERHOSTDOCKERHOST
cAdvisorDocker-GC
Container Metrics
CentOS 7.2/Docker 1.10.3
4 Core/32GB RAM/120GB LVS
DRYDOCK
BUILD HOSTS
SSH
Docker API Docker API
Docker API
![Page 56: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/56.jpg)
Mai Tai’s On the Beach
Build Engineer
![Page 57: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/57.jpg)
How Do You Actually Build It?
![Page 58: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/58.jpg)
How Do You Actually Build It?
(engineering.riotgames.com)
![Page 59: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/59.jpg)
https://github.com/maxfields2000/dockerjenkins_tutorial
![Page 60: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/60.jpg)
You Make This Sound Simple.
![Page 61: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/61.jpg)
Lesson 1 - Docker Isn’t “Simple”
DOCKERFILES REQUIRE DECENT SYSTEMS ADMIN KNOWLEDGEDOCKER IMAGES THAT BUILD DOCKER IMAGES IS DOCKERCEPTIONDOCKER “VOODOO AND BLACK MAGIC” QUESTIONS
![Page 62: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/62.jpg)
Lesson 2 - Containers != VM’sCANNOT MOUNT REMOTE FILE SYSTEMS
HAVE RULES FOR DOCKERFILE VS RUN TIMEEPHEMERAL CONTAINERS DON’T MAINTAIN STATE
![Page 63: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/63.jpg)
Lesson 3 - Garbage Collection
DOCKER BUILD
DOCKER PULLDOCKER RUN
VOLUME
![Page 64: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/64.jpg)
Lesson 4 - Maintenance/FailurePULL HOSTS ON/OFFLINEUPDATE ALL IMAGESROLLING RESTARTS
![Page 65: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/65.jpg)
Lesson 5 - How Will You Upgrade?
![Page 66: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/66.jpg)
Lesson 7 - Credentials & SecurityWHAT SHOULD YOU PUT IN YOUR BASE IMAGE?
PASSWORDS IN YOUR SOURCE CODE?SSH KEYS IN YOUR REGISTRY?
PASSING EVERYTHING IN AS A ENVIRONMENT VARIABLE?
![Page 67: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/67.jpg)
![Page 68: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/68.jpg)
Over 1200 New Build Jobs Created
![Page 69: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/69.jpg)
30% of All Environments Are Containers
![Page 70: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/70.jpg)
Environment Change/Create/FixTickets Dissappeared
![Page 71: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/71.jpg)
We No Longer Have “Just” A Build Team
![Page 72: Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart](https://reader037.vdocuments.us/reader037/viewer/2022102317/586fdc141a28ab18428b6371/html5/thumbnails/72.jpg)
QUESTIONS?