the hard problems of continuous deployment
TRANSCRIPT
![Page 1: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/1.jpg)
The Hard Problems of Continuous Deployment
Timothy Fitz (.com)
![Page 2: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/2.jpg)
“Continuous deployment involves deploying early and often, so as to
avoid the pitfalls of "deployment hell". The practice aims to reduce timely rework and thus
reduce cost and development time.”
![Page 3: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/3.jpg)
“Our highest priority is to satisfythe customer through earlyand continuous delivery of
valuable software.”- Principles behind the Agile Manifesto (2001)
![Page 4: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/4.jpg)
The Hard Problems(The ones I’m going to cover)
• Data
• Mobile
• Scale
![Page 5: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/5.jpg)
Data makes CD harder
• Updating data is slow
• Moving data is slow
• Schemas live outside code deploy process
• Rollbacks are often hard or impossible
![Page 6: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/6.jpg)
Schema rev. N, N+1 compatible
• Add the column in production• Push the code that writes to that column• Optionally, run a data migration to populate
the existing rows with data• Push the code that reads from that column.
![Page 7: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/7.jpg)
Apply only cheap changes
• Only apply changes that are cheap enough to not affect live traffic
• More complex changes split into tiny steps:
– Create new table
– Write to both
– Cut over eventually
– Drop old table
![Page 8: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/8.jpg)
Apply change to standby
• Run two DB instances
• Apply change to standby
• Failover if successfully applied
• Might run 3rd db instance for availability
![Page 9: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/9.jpg)
Blue/Green Deploy
• Run two copies of entire cluster
• All databases are replicated
• Lets you test, update and rollback both code and schema in one step
![Page 10: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/10.jpg)
Schemaless NoSQL
• “Schemaless” really means schema is in your source code
– Which is great! We can CD schema changes
![Page 11: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/11.jpg)
Schemaless-style SQL
• Benefits of existing SQL databases while still being pseudoschemaless
• (thing_id, key, value) table
• (guid, blob) table
![Page 12: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/12.jpg)
Recap
• Apply only cheap changes
• Apply changes to standby
• Blue/Green Deploy
• Schemaless NoSQL
• Schemaless-style SQL
![Page 13: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/13.jpg)
Mobile
• Users must opt-in to every update
• iOS submissions take a week to be approved
• Luckily, lots of tools aimed at the space
![Page 14: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/14.jpg)
Remember the basics
• CI server / automated tests are critical
• Can’t fallback on production alarms / rollback
• Hosted options: http://cisimple.com
![Page 15: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/15.jpg)
Data-drive everything
• Build your views / content from data files
• Ping server for updates
• Hosted: http://appgrok.com/
– Lets you deploy txt, png and xib dynamically!
![Page 16: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/16.jpg)
99% HTML
• Entire app is a single UIWebView
• Glue native code to allow access to APIs
• Clutch.io is awesome (and FOSS now)
– Live reloading for local dev
– Streamlined deploys
– https://github.com/clutchio
![Page 17: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/17.jpg)
Hybrid HTML/Native
• Core app is native
• Sections can be replaced by HTML
– i.e. Facebook stream entries fallback to HTML
• Infrequently used sections are 100% HTML
![Page 18: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/18.jpg)
Recap
• Remember the basics
• Data-drive everything
• 99% HTML
• Hybrid HTML/Native
![Page 19: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/19.jpg)
Scalability
• Maintaining availability, performance and happiness
• As a function of # of people
• As a function of # of tests
![Page 20: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/20.jpg)
Availability
• The build must always be green
• Set a “green SLA”
– 99% green
– Never red for > 15m
• Measure, track and report on these numbers
![Page 21: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/21.jpg)
Performance
• Measure (intent to) commit to time of deploy
– Goal: < 5 minutes
• Measure local development test loop
– Goal: < 2s
![Page 22: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/22.jpg)
Happiness
• CI/CD System is a product
• Software Engineers are the customer
• Keep your customer happy!
![Page 23: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/23.jpg)
Testing Pyramid
![Page 24: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/24.jpg)
How do you make tests fast?
• Tests can exercise large amounts of code without being slow
• Minimize system calls (no I/O, no disk)
• Minimize test data size
• Make sure all systems are cheap to instantiate/teardown
• No external state makes tests more reliable
![Page 25: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/25.jpg)
Run Tests in Parallel
• Multiprocess
• Multimachine
• Multi-VM
• Instant multi-VM: http://circleci.com
![Page 26: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/26.jpg)
Hardware Scale
• CI Cluster will get huge
– Function of cumulative engineering man-months
– Rule of thumb: 10% of your cluster size
• You will need a CI/CD DevOps person
– CI cluster monitoring / alerting
– Configuration Management critical
![Page 27: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/27.jpg)
Scale testing infrastructure recap
• Write the right kind of tests
• Make those tests as fast as possible
• Run those tests in parallel
![Page 28: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/28.jpg)
People / Roles
• Sheriff
– Designated reverter / problem troubleshooter
– Common pattern (IMVU, Chromium, Firefox)
• CD “Product Owner”
– Held accountable for SLA / Performance
– Manage infrastructure backlog
![Page 29: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/29.jpg)
Single trunk
• Do this until it doesn’t work for you
• Gets painful in the 16 – 32 developer range
• Faster commit->deploy reduces the pain
– But effort becomes prohibitive
![Page 30: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/30.jpg)
“Try” pipeline
• Conceptually, a second tree that “doesn’t matter” but still gets tested for feedback
• Buildbot implements a patch-pushing version
• Takes a significant amount of pressure off of trunk builds
![Page 31: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/31.jpg)
CI Server takes active role
• Server automatically reverts red commits
• Server merges green commits to trunk
![Page 32: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/32.jpg)
Feature branches
• All incremental development happens on branches, branches land when feature is “ready”
• If “feature” is kept small, can be 2-3 per engineer per week on average
• Less continuous, but scales much better
– Feature branches tested before merge
![Page 33: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/33.jpg)
Merge tree
• Tree per team / feature
• Trees merged into trunk daily (if green)
• Scale up via tree of trees (of trees…)
• Again, less continuous
![Page 34: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/34.jpg)
Federation
• Each team gets their own deploy pipeline
• Requires SOA / component architecture
• Each team can set their own CD pace
• “Enterprise Ready”
![Page 35: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/35.jpg)
Recap
• Single trunk + Try pipeline / Autorevert
• Feature Branches
• Merge Tree
• Federation
![Page 36: The Hard Problems of Continuous Deployment](https://reader033.vdocuments.us/reader033/viewer/2022052508/559868dd1a28aba9738b46d5/html5/thumbnails/36.jpg)
Questions?
Timothy Fitz (.com)