![Page 1: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/1.jpg)
Scared of Huge Kubernetes Ecosystem?Adopt it incrementally!
Mazedur Rahman
![Page 2: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/2.jpg)
About Me
● Advocate building repeatable and testable infrastructure
● Love to automate everything
● Following container ecosystem since 2013
● Enjoy reading/writing papers on cloud computing
● Teach classes in spare time
![Page 3: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/3.jpg)
Kent Beck’s 3X
● eXplore○ Embarking on the greenfield project○ Risky search for a viable return○ If unexpectedly successful → leads to
● eXpand○ Now things are going nuts○ Unanticipated bottlenecks appear○ Growth becomes routine → leads to
● eXtract○ Now problem and solution spaces are clear○ Known cost pattern: $1 in → $3 out○ Playbook emerge, economies of scale matters
Explore
Expand
Extract
Success
Pay
off
![Page 4: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/4.jpg)
THE PROBLEM
![Page 5: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/5.jpg)
The Problem
● Deploy/Manage a set of stacks○ ∑(LB, Web Server, App, DB, ...)
● Needs○ Highly available
○ Scale horizontally
○ Rolling deploy, Blue/Green, …
○ Fast rollback
○ Secure runtime
○ ...
AmazonDynamoDB
![Page 6: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/6.jpg)
THE CHALLENGES
![Page 7: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/7.jpg)
The Challenges: VM is Too Heavy
Host OS
Hypervisor
GuestOS
GuestOS
GuestOS
Bin / Libs
Bin / Libs
Bin / Libs
App 1 App 3App 2
Host/Guest OS
Container Engine
Bin / Libs
Bin / Libs
Bin / Libs
App 1 App 3App 2
Infrastructure Infrastructure
VM
Co
nta
iner
![Page 8: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/8.jpg)
The Challenges
● How do we○ deploy new version of apps?
○ solve service discovery?
○ avoid downtime during rollouts?
○ auto-heal services from failures?
○ patch host OS without disruptions?
○ implement load balancing?
○ automate certificate deployment?
○ orchestrate storage?
○ ….
![Page 9: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/9.jpg)
LIFE WITHOUT ORCHESTRATOR
![Page 10: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/10.jpg)
Life without Orchestrator: Ad-hoc Style
● Poor man’s orchestrator○ manual ‘docker run …’
○ ‘docker run …’ via scripts
○ docker-compose, rancher, ...
○ custom (home-grown) tooling
○ ansible, chef, puppet, …
○ Jenkins
○ systemd, fleet, …
○ …
![Page 11: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/11.jpg)
Life without Orchestrator: Outcome
● Likely consequences○ ad-hoc everything → snowflakes
○ random failures → firefighting
○ human in the loop → slow process
○ inextensible → business blockers
![Page 12: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/12.jpg)
ENOUGH IS ENOUGH
![Page 13: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/13.jpg)
The Contenders
AWS ECS Kubernetes Marathon Nomad Swarm
![Page 14: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/14.jpg)
The Winner
AWS ECS Kubernetes Marathon Nomad Swarm
✅
![Page 15: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/15.jpg)
Kubernetes Brings...
Au
to-h
ealin
g
Service D
iscovery
Secret & Config mgmt
Auto binpacking
Horizontal Scaling
Load Balancing
Rolling d
eploy
![Page 16: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/16.jpg)
The Ecosystem
...
![Page 17: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/17.jpg)
DESIGN FOR CHANGE
![Page 18: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/18.jpg)
Design for Change: Layered Architecture (1)
● Business and Tech Change Inevitable → Design in Layers○ Keep lock-in low → Easy to move around
○ Easy to upgrade/patch
○ Swappable
Network
K8s Cluster
K8s Master K8s Worker
Services
![Page 19: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/19.jpg)
Design for Change: Layered Architecture (2)
● An ExampleServices
(Containers, FaaS, …)
Agents(Logging, Monitoring, …)
Container Engine(Docker, rkt, runc, …)
VM(OS Image, AMI, …)
Network(VPC, Route, SG, …)
Spec
ific
Agn
ost
ic Orchestration(K8s, Swarm, …)
Provision(Terraform, Pulumi, Provider tooling, …)
![Page 20: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/20.jpg)
Design for Change: Workflows, not Tools
● Tools will most likely change○ But workflow remains same
Build Test Deploy Manage
![Page 21: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/21.jpg)
DESIGN FOR STABILITY
![Page 22: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/22.jpg)
Design for Stability: API Selection
● Opt for Stable APIs Initially○ v1 > v1beta1 > v1alpha1
![Page 23: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/23.jpg)
Design for Stability: Service Selection
● Leading Candidates○ Non-user-facing API services
○ Predictable traffic/client
○ Stateless
○ ...
Online StoreFrontend
Account Cart Catalogue Order ...
![Page 24: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/24.jpg)
KILL THE SCARE
![Page 25: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/25.jpg)
● Simple but Powerful Workflows
Kill the Scare: Automate Workflow
Build Test Deploy Manage
$ ./stack build $ ./stack test $ ./stack deploy $ ./stack manage
Human or Bot
![Page 26: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/26.jpg)
● Automated tests○ Assertions for cluster/service/resource provisioning/deployment
○ Leverage automated test frameworks
● Automated health checks○ Assertions for runtime resources, cross-communication, ...
Kill the Scare: Guarantee Repeatability
![Page 27: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/27.jpg)
● Same Process Throughout○ Dev/Prod parity
○ Instrumentation
○ Troubleshooting
Kill the Scare: Boost Confidence
Traditional App Twelve-Factor App
Time between deploys Weeks Hours
Developers vs. Deployers Different people Same people
Dev vs. Prod environment Divergent As similar as possible
![Page 28: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/28.jpg)
● Safe Rollout/Rollback○ Rolling update
○ Blue/Green
○ Canary
○ A/B testing
○ ...
Kill the Scare: Stress-Free Release
github.com/ContainerSolutions/k8s-deployment-strategies
![Page 29: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/29.jpg)
ADOPT INCREMENTALLY
![Page 30: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/30.jpg)
● Defer as many tooling as possible○ Prometheus
○ Istio
○ Calico
○ ...
Adopt Incrementally: Go Minimalist Initially
Explore
Expand
Extract
Success
Pay
off
Rolling update(Off-the-shelf)
Blue/Green(via simple tooling)
Canary(Envoy, Istio, …)
App Code Rollout
![Page 31: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/31.jpg)
● Minimal templating footprint○ Avoid too many tooling initially
○ Good for getting familiar with K8s stack
○ Supported by core community
○ ...
Adopt Incrementally: Start with YAML
Explore
Expand
Extract
Success
Pay
off
YAML
Helm, Charts, ...
client-go, draft, ...
K8s Manifests
![Page 32: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/32.jpg)
● Offload Responsibilities as much as possible○ Master component management
○ Monitoring, logging, ...
○ ...
Adopt Incrementally: Start with Managed K8s if an Option
Explore
Expand
Extract
Success
Pay
off
Whatever you have
Datadog, New Relic, ...
Prometheus, ...
Monitoring
![Page 33: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/33.jpg)
● Reduce host OS management burden○ Minimalist OS pre-packaged with base tooling
■ e.g. managed workers, CoreOS, etc.
● Reduce attack surface○ Avoid bloated OS
Adopt Incrementally: Choose Right Host OS
Explore
Expand
Extract
Success
Pay
off
Supervised patching
Semi-automatic
Automatic
Upgrade/Patch Rollout
![Page 34: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/34.jpg)
● Undecided on orchestration?○ Go for Kubernetes if you must pick one
■ May feel a bit overwhelming initially
■ Investment pays off
Final Thoughts
![Page 35: Adopt it incrementally! Scared of Huge Kubernetes ...files.informatandm.com/uploads/2019/4/1320_Mazedur_Rahman.pdf · Kent Beck’s 3X eXplore Embarking on the greenfield project](https://reader036.vdocuments.us/reader036/viewer/2022081607/5ec989ceb83f5f77ec2d454e/html5/thumbnails/35.jpg)
Thanks!