continuous delivery: a journey not a destination€¦ · testing with feature toggles 26 describe...
TRANSCRIPT
![Page 1: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/1.jpg)
CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATIONAkshay Karle & Badrinath Janakiraman @snap_ci | https://snap-ci.com
1
![Page 2: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/2.jpg)
2
The essence of my philosophy to software delivery is to
build software so that it is always in a state where it
could be put into production. We call this Continuous
Delivery because we are continuously running a
deployment pipeline that tests if this software is in a
state to be delivered. - Jez Humble
http://martinfowler.com/delivery.html
![Page 3: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/3.jpg)
3
…high-performing organizations are still deploying
code 30 times more frequently, with 50 percent
fewer failures than their lower-performing
counterparts.
https://puppetlabs.com/sites/default/files/2014-state-of-devops-report.pdf
![Page 4: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/4.jpg)
4
![Page 5: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/5.jpg)
5
frequent, reliable releases of high quality
valuable software
![Page 6: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/6.jpg)
6
![Page 7: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/7.jpg)
SNAP, BACK IN 2012…
Built on 9 components (microservices, anyone?)
A message-queue for fast background job processing
Chef-server for infrastructure provisioning
Relational & NoSQL databases
Extensive Nagios monitoring
Completely automated infrastructure creation
Build and deployment pipelines for each component
So clearly - we had it nailed. Right?7
![Page 8: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/8.jpg)
8
![Page 9: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/9.jpg)
CONSEQUENCES
Likelihood of build setup passing 50%
Likelihood of changes actually triggering a build 50%
Likelihood that someone knew why 0%
Likelihood of 4 SSH sessions firing up debug 100%
Likelihood that the infrastructure was busted 50%
Frequency of production upgrades Twice a month
Consequence of upgrades Site unavailable!
Feedback to users about what was going on Minimal9
![Page 10: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/10.jpg)
10
frequent, reliablereleases of high quality
valuable software
![Page 11: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/11.jpg)
11
![Page 12: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/12.jpg)
Our journey so far..
![Page 13: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/13.jpg)
Queues per build
Messages lost, redelivery problems
Single, centralized queue
No need to manage queues
Delayed::Job
![Page 14: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/14.jpg)
one Nagios to rule them all
too many components
too many metrics and alerts
ignored alerts when things actually did go wrong
separated application monitoring
basic server monitoring
aggregated logs
we don’t need to manage anything
![Page 15: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/15.jpg)
Serverbecame cognizant of what state would go into chef
reliable and faster chef runs
nothing to manage
user build specific configuration data
dependency on the chef server for deployments
had to manage the server
Solo
![Page 16: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/16.jpg)
Single mutable environments
Too many components
READONLY MODE Long running deployments
Loosing build requests
Loosing customers
Fewer components
Blue-green deployments
![Page 17: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/17.jpg)
BLUE-GREEN DEPLOYMENTS IN SNAP
17
VZHOSTBuild Server
web server
VZHOSTBuild Server
web server
L B DATABASE
![Page 18: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/18.jpg)
COUPLED MODULES TO MONOLITH TO REAL MODULES
Highly coupled initial state
Collapsed moving parts to a monolith
Deploying frequently highlights rates of change
This was used to make new modules
Much more decoupled state today
18
![Page 19: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/19.jpg)
SO, WHAT’S THE CURRENT STATE?
Build setups almost always pass…
…but if there are failures, the team gets alerted
Deployments happen every other day
New releases happen once every 2~3 weeks
Near zero downtime
19
![Page 20: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/20.jpg)
20
![Page 21: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/21.jpg)
OUR CURRENT WORKING STYLE
Appropriate sequencing of stories
Using feature toggles
Structured evolution of data and app
Identical environments as trip-wires
21
![Page 22: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/22.jpg)
SEQUENCING OF STORIES
22
short (idealized) cycle time, more technical risk
time
time
longer (idealized) cycle time, less technical risk
![Page 23: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/23.jpg)
FEATURE TOGGLES
Hide unfinished UI elements
Control backend behaviour
Test with feature toggles
Avoid multi-component feature toggles
23
![Page 24: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/24.jpg)
FRONTEND
24
<% if feature_enabled?(:parallel_stage) %> <!-- show new html --><% else %> <!-- show old html --><% end %>
edit.html.erb
![Page 25: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/25.jpg)
BACKEND
25
if feature_enabled?(:parallel_stage) # new logicelse # old logicend
babysitter.rb
![Page 26: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/26.jpg)
TESTING WITH FEATURE TOGGLES
26
describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage) end it 'should not show the job tabs when there is only one job' do end end describe "feature disabled" do before(:each) do with_feature_enabled(:parallel_stage) end it 'should not show the job tabs but should show the logs' do end endend
stage_run_spec.rb
![Page 27: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/27.jpg)
MIGRATION OF DATA
Consider existing schema as well as data
Incremental changes
Rollbacks
Zero downtime releases
27
![Page 28: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/28.jpg)
INTRODUCE JOBS
28
jobs
id
![Page 29: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/29.jpg)
POPULATE ALL EXISTING STAGES
29
stages
id
started_at
completed_at
result
…
jobs
id
stage_id
…
![Page 30: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/30.jpg)
COPY ATTRIBUTES TO JOB
30
stages
…
started_at
completed_at
result
…
jobs
…
started_at
completed_at
result
…
![Page 31: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/31.jpg)
SWITCH THE APPLICATION TO START USING THE JOB MODEL
31
# After switchclass Stage def result results = jobs.collect { |job| job.result } return :failed if results.any?(:failed) :passed endend
# Before switchclass Stage attr_reader :resultend
# in transitionclass Stage def result if feature_enabled?(:parallel_stage) results = jobs.collect { |job| job.result } return :failed if results.any?(:failed) :passed else result end endend
![Page 32: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/32.jpg)
REMOVE UNUSED ATTRIBUTES
32
stages
id
started_at
completed_at
result
…
jobs
…
started_at
completed_at
result
…
![Page 33: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/33.jpg)
IDENTICAL ENVIRONMENTS/AUTOMATION
33
staging
buildn+1
upgradesmoketests�
ver. n
To production
![Page 34: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/34.jpg)
34
buildn+1
upgradesmoketests�
ver. n+1
To production
staging
![Page 35: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/35.jpg)
35
buildn+1
upgradesmoketests�
ver. n+1 staging
To production
![Page 36: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/36.jpg)
36
�
ver. n+1 staging
buildn+1
upgradesmoketests
buildn+2
upgradesmoketests� To production
![Page 37: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/37.jpg)
37
�
ver. n+2 staging
buildn+1
upgradesmoketests
buildn+2
upgradesmoketests� To production
![Page 38: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/38.jpg)
38
�
ver. n+2 staging
buildn+1
upgradesmoketests
buildn+2
upgradesmoketests� To production
![Page 39: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/39.jpg)
SUMMARY
Start with getting working software into hands of users
Layer on qualities of reliability, automation, frequent releases etc
Kick the can down the road
Don’t be afraid of “re-design”/”re-work”
Let CD guide your architecture and design - not the other way around
39
![Page 40: CONTINUOUS DELIVERY: A JOURNEY NOT A DESTINATION€¦ · TESTING WITH FEATURE TOGGLES 26 describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage)](https://reader033.vdocuments.us/reader033/viewer/2022042410/5f28a298b2c98130352df660/html5/thumbnails/40.jpg)
Akshay Karle, @akshay_karle
THANK YOU
Badrinath Janakiraman, @badrij
@snap_ci | https://snap-ci.com