continuous delivery in practice (extended)
DESCRIPTION
Extended version of a previously uploaded presentation: 10 practical field-proven tips for building a continuously delivered service, based on Kenshoo's experience with its RTB service - a critical, high throughput, highly available component, serving millions of requests per minute in under 50 milliseconds. From coding practices to test automation, from monitoring tools to feature A/B testing - the entire development chain should be focused around removing blockers and manual steps between your code and your clients, without ever settling for quality. Join to see what makes our clients and developers happy and effective.TRANSCRIPT
![Page 1: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/1.jpg)
Continuous Delivery In Practice
Lessons from Kenshoo’s RTB project
![Page 2: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/2.jpg)
Who, What, Where
Tzach Zohar:● System Architect● [email protected]
Kenshoo: ● Founded 2006● Online Marketing Technology● >500 employees● 12 World Wide locations
![Page 3: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/3.jpg)
Agenda
● Continuous Delivery: What? Why?● RTB Project● How: 10 Field Tested Tips● The Process● Appendices
![Page 4: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/4.jpg)
Continuous Delivery: Definition(s)
“Continuous Delivery (CD) is a design practice …blah blah blah… Techniques such as
automated testing, continuous integration …blah blah blah... resulting in the ability to rapidly, reliably and
repeatedly push out enhancements ...blah blah blah.”
- Wikipedia
![Page 5: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/5.jpg)
Continuous Delivery: Definition(s)
TL;DR
![Page 6: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/6.jpg)
Continuous Delivery: Definition(s)
“Continuous delivery is a set of principles and practices to reduce the cost, time, and
risk of delivering incremental changes to users.”
- Jez Humble
![Page 7: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/7.jpg)
Continuous Delivery: Definition(s)
“Continuous Delivery is a software development discipline where you build
software in such a way that the software can be released to production at any time”
- Martin Fowler
![Page 8: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/8.jpg)
Continuous Delivery: Why bother?
“Our highest priority is to satisfy the customerthrough early and continuous delivery
of valuable software”
First principle of the Agile Manifesto
![Page 9: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/9.jpg)
Continuous Delivery: Why bother?
Better suited productResponsiveness
Less wasteHigher quality
Simplicity
Recommended Further Reading on ThoughtWorks
![Page 10: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/10.jpg)
Background: RTB Project
● ~1.5 years ● ~3 developers, 1 PM, 0.5 Ops (no QA)● ~Dozens of paying clients● ~50 servers (AWS)● ~1.5M requests per minute● ~7ms average response time● ~99.9% availability
![Page 11: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/11.jpg)
Background: RTB Project
Frontend ClusterHighly available, high throughput ~20 node cluster
BackendSingle node, internal APIs
FBXFacebook RTB API
Reporting ClusterElastic Map Reduce (EMR) on-demand 16-node cluster
Cassandra ClusterHighly available, high throughput ~24 node cluster
S3Raw traffic logs
![Page 12: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/12.jpg)
Background: RTB Project
~5-10 deployments / week
![Page 13: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/13.jpg)
How?
![Page 14: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/14.jpg)
1.The Obvious
● Single branch (details later)● Full, Fast, Reliable coverage● Full deployment automation● Fast feedback● ABCD - Always Be Continuously
Deploying
![Page 15: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/15.jpg)
● Unit: complete functional coverage● Integration: with external systems - thin!● Behavioral: we use Cucumber● Staging: verify actual server upgrade
2. Four-Layer Test Suite
![Page 16: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/16.jpg)
2. Four-Layer Test Suite
![Page 17: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/17.jpg)
Staging: verify compatibility of new build with other components’ production builds
2. Four-Layer Test Suite
![Page 18: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/18.jpg)
3. Keep Builds Stable
Do not overlook a test that “sometimes fails”, trusting build status is crucial
![Page 19: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/19.jpg)
3. Keep Builds Stable
● Random data tests● Asynchronous tests● Integration tests
Be suspicious of:
![Page 20: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/20.jpg)
4. Master Is Always Shippable
On every commit? Not QuiteWe follow the “GitHub Flow”:
Local Master
Local Feature Branch
Master
Feature Branch
1. pull
3. push
2. checkout
4. Merge
![Page 22: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/22.jpg)
5. Rigorous Code Reviews
● Because “merge” means “deploy”!● Insist on proper coverage● Insist on code cleanliness● Insist on consistent design● Insist!
![Page 23: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/23.jpg)
5. Rigorous Code Reviews
https://github.com/tzachz/github-comment-counter
![Page 24: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/24.jpg)
6. Real-Time Feedback
Detect issues immediately and visually
![Page 25: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/25.jpg)
7. Keep Upgrade in Mind (1)
Use the “Parallel Change” pattern when changing cross-node APIs / Data
1.Write: oldRead: both
2.Write: new Read: both
3.Write: new Read: new
deploy deploy
![Page 26: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/26.jpg)
8. Keep Upgrade in Mind (2)
Verify backward compatibility in tests
![Page 27: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/27.jpg)
9. A/B Testing
Apply new features to a limited user-group Measure business results per-group
(Not by branching)
![Page 28: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/28.jpg)
9. A/B Testing
Splitting into groups correctly is important
![Page 29: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/29.jpg)
9. A/B Testing
It’s easy to mess up (neglecting biases, wrong grouping, wrong comparison
methods)
This excellent talk by LivePerson’s Shlomo Lahav helped us a lot
![Page 30: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/30.jpg)
10. Own It
Constantly check buildsConstantly collect feedbackConstantly check monitorsAnswer the phone at 3am
![Page 31: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/31.jpg)
10. Own It
![Page 32: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/32.jpg)
That’s It.
![Page 33: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/33.jpg)
The Process
● Greenfield? That’s easy:○ Start with deployment and build○ Deploy a Hello World application○ Every new feature is test-covered
![Page 34: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/34.jpg)
The Process (RTB)
1.Increase Unit+Integration CoverageCreate naive deployment AutomationCreate monitoringManual Staging tests
2.Automated stagingDowntime eradicatedManual (but often) deployment trigger
3.Autopilot - deploy upon commit
~ 9 Months
~ 3 Months
![Page 35: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/35.jpg)
Appendix A. Partial Tool List
Testing: JUnit, Cucumber, NoseBuild / CI: Jenkins, Gradle, JaCoCo
Code Review: GitHubProvisioning: Puppet
Deployment: Fabric, botoMonitoring: Metrics, Graphite
![Page 36: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/36.jpg)
Appendix B. Are You Ready?
Unit Coverage > 90%?
Good Staging Tests?
Informative Monitors?
Builds Are Kept Green?
No API Breaking Changes?
Rigorous Code Reviews?
Support Has Your Phone Number?
Do You Own it?
Not Ready
No Yes
credit: [email protected]
![Page 37: Continuous Delivery in Practice (extended)](https://reader033.vdocuments.us/reader033/viewer/2022051818/54b715074a7959286f8b45cd/html5/thumbnails/37.jpg)
Thanks. Questions?