Kanban as CodeCode level impact of a Lean mindset
LesFurets.com
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Insurance aggregator
5 Products
2,5M quotes per year
Paid by contracts initiated
Our product
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
The engine
• Complex forms
• 40-200 Fields (Expedia is around 10 fields)
• Risk pricing engine
• 50+ external webservices called in real time
• Collect answers within seconds
• 2B+ fields mapping per year
• Embedded online subscription moduleLesFuretsWebSite
Panel Partners
Data Partners
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Competitive Advantage(s)
Product Catalogue :Number of Panel members
Speed of getting new Panel membersAcquisition price compared to google, …
Quality of the offers presented to the usersQuality of the clients presented to the partners
UX of the WebSite :Speed of getting a proposal, clarity of the proposalAvoid mistakes along the form filling
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
The IT Team
• 23 Developers & 2 Ops
• 2 Functional Teams & 1 Infrastructure & Team
• 1 Single code base of 500.000 lines
• 40.000 unit tests (low level automated tests)
• 200 selenium tests (browser based automated tests)
•1-3 release per day of 1-10 changes
• 1 branch per feature : released when ready
• Soon known as “Kanban as code”
• 20-30 branches in parallel, 1-10 released per day
@beatiefurets
github.com/lesfurets
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
500.000 Lines of Code (LoC) that’s :
7 dictionaries of 500 pages (140 lines / page)
2 times the 7 Harry Poter volumes (3400 pages)
A new version daily5-10 changes by 25 devs
Intermezzo : 500.000 line of code ?
x 7x 2
Why daily releases ?What read and understood
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Manifeste agile
Principe #1
« Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. »
http://agilemanifesto.org/principles.html
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Food for thought : back to the future
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Reading survivor kit : save your life
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Minimal Viable : what you can’t ignore
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Release early, release often
http://paulhammant.com/2013/03/13/facebook-tbd-take-2/
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Learning #1 : Release early, release often
• To reduce the Time to Market :• Release small batches of features
• Release often
• Releasing often• Change the way source code is managed
• Change in the QA strategy (Dev QA their own changes)
• Test tooling is provisioned for the des
• Fast Feedbacks as a gift
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Continuous Delivery : 5 Golden Rules
1. Fast build
2. Unbreakable build
3. Easy and automated deployments
4. Runtime Monitoring and alerting
5. Root cause analysis of incidents
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Git / Git Flow / Github Flow
Powerful branching model
http://nvie.com/posts/a-successful-git-branching-model/
Master
Branch
Pull Request
Github
Code management in 2017The minimum to know, no excuse !
@beastiefurets@dbaeli
Original code:
master
@beastiefurets@dbaeli
Change 1 : Lean IT
@beastiefurets@dbaeli
Change 2 : Beastie Furets
@beastiefurets@dbaeli
Change 3 : Hello Beastie
@beastiefurets@dbaeli
A commit
@beastiefurets@dbaeli
master
Commit 1 : Lean IT
commit 1
@beastiefurets@dbaeli
A branch (copy)
@beastiefurets@dbaeli
Change 2 = commit 2
commit 2
master commit 1
@beastiefurets@dbaeli
Let’s merge the branch
@beastiefurets@dbaeli
master
Change 1 & Change 2 : combined (merged)
commit 1 merged
commit 2
@beastiefurets@dbaeli
A conflict of 2 branches
@beastiefurets@dbaeli
Commit 3 : Hello Beastie
master
commit 3
@beastiefurets@dbaeli
master
Commit 2 : Hello Lean IT
commit 2
@beastiefurets@dbaeli
???????
Change 3 : Hello Beastie
master
conflictcommit 3
commit 2
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Minimum viable vocabulary (1/2)
•Commit: • is a change within the code base
• described as a “diff” from original (or patch)
• History :• Sequence of Commits (changes)
• Branch:• A full copy of the code base at a given time
• Can be seen as an alternate history
• Merge:• Replay commits of a branch on another
branch 1
branch
commit 1
commit 2
commit 4
commit
merge
commit 3
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Minimum viable vocabulary (2/2)
• Conflict: • when 2 commits aren’t compatible
• from 2 branches
• Resolution :• give a solution to the conflict
• Rebase :• change the start point of a branch
• time travel (get into the future)
branch 1
branch 2 commit 2
commit a
conflict
commit 3
commit 2
commit a
commit 3
rebase
Release more often - Step 1Monthly releases (2012)
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Plan / estimate / code / test / release monthly
Sprints style 12 releases in a year
2012 : Monthly release
@beastiefurets@dbaeli
branch (prod)
trunk trunk
3 - 4 semaines 1 - 4 jours5 jours
Release
Code Freeze
branch
@beastiefurets@dbaeli
Storie 1
branch (prod)
trunk trunk
Release
Code Freeze
branch
3 - 4 semaines 1 - 4 jours5 jours
@beastiefurets@dbaeli
Storie 1
branch (prod)
trunk trunk
Release
Code Freeze
branch
3 - 4 semaines 1 - 4 jours5 jours
@beastiefurets@dbaeli
Storie 1
branch (prod)
trunk trunk
Storie 2
Release
Code Freeze
branch
3 - 4 semaines 1 - 4 jours5 jours
@beastiefurets@dbaeli
Storie 1
branch (prod)
trunk trunk
Storie 2
Release
Code Freeze
branch
3 - 4 semaines 1 - 4 jours5 jours
Risk of conflicts
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Agile Team of 12, SCRUM by the book
•Monthly sprints, planning, coding, testing, demo, release
• 1 Week of testing (code review, manual testing, bug fixes)
• Build Time : 15 minutes• Regression Testing : 1-2 hour
•Blockers : • Build time• Regression Testing• A Week of manual testing
2012 : Monthly releases
Release more often - Step 2Weekly release (2013)
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
2013 : Weekly release
Planifier / estimer / coder / tester Release weekly
50 releases per year
@beastiefurets@dbaeli
branch (prod)
trunk trunk
3 - 4 semaines 1 - 4 jours5 jours
Release
Code Freeze
branch
@beastiefurets@dbaeli
branch (prod)
trunk trunk
Release
Code Freeze
branch
3 - 4 semaines 1 - 4 jours5 jours
Storie 1
@beastiefurets@dbaeli
branch (prod)
trunk trunk
Release
Code Freeze
branch
3 - 4 semaines 1 - 4 jours5 jours
Storie 1
@beastiefurets@dbaeli
Storie 1
branch (prod)
trunk trunk
Storie 2
Release
Code Freeze
branch
3 - 4 semaines 1 - 4 jours5 jours
@beastiefurets@dbaeli
Storie 1
branch (prod)
trunk trunk
Storie 2
Release
Code Freeze
branch
3 - 4 weeks 1 - 4 days5 days
1 week
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
• Keep Scrum monthly planning with Weekly releases
•Changes:• Use the patch process to deliver features (cherry pick)
• Build down to 3 minutes (vs 15 minutes)
•Blockers:• Regression Testing + Recette• Code level management (release branch)• Painful weekly patch release
2013 : Getting Weekly
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Learning #2 : Patch process as release process
• When something goes wrongs on the live system
•Do you have a strong process to release a fix ?
• NO => you’re living dangerously => get one• YES => why don’t you use it for usual releases ?
•Anyway :• Get a robust and fast delivery process is not an option !
Release more often - Step 3Daily release (2014)
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
2014: Daily Releases
Ship what is ready … every day !
Marathon style
250 releases per year (already 50 in 2017 !)
@beastiefurets@dbaeli
master
trunk
1 day 1 day
Model 3: Kanban as code
Storie 1
Storie 2Storie 3
@beastiefurets@dbaeli
master
trunk
1 day 1 day
Storie 1
Model 3: Kanban as code
@beastiefurets@dbaeli
master
trunk
1 day 1 day
Storie 1
Model 3: Kanban as code
Is this branch ready ?
@beastiefurets@dbaeli
master
trunk
1 day 1 day
Storie 1
Model 3: Kanban as code
merge & test
Merge the branch and test
@beastiefurets@dbaeli
master
trunk
1 day 1 day
Storie 1
merge & test & drop
Model 3: Kanban as code
Merge the branches and test
@beastiefurets@dbaeli
master
trunk
1 day 1 day
Storie 1
Storie 2
merge & test
Model 3: Kanban as code
Merge the branches and test
@beastiefurets@dbaeli
master
trunk
1 day 1 day
Storie 1
Storie 2Storie 3
merge & test
Model 3: Kanban as code
Merge the branches and test
@beastiefurets@dbaeli
Keep the branches independentAvoid the conflict
Merge the branches and test
git-octopus by lesfurets (on GitHub)
@beastiefurets@dbaeli
master
trunk
1 day 1 day
Storie 1
Storie 2Storie 3
Model 3: Kanban as code
@beastiefurets@dbaeli
ticket1
ticket2
ticket3
ticket4
ticket5
features releaseslocal
ticket3
master
ticket3
ticket1
master
octopus-features
octopus-releases
few minutes 1 jour à 1 mois 1 - 2 jours
Validation
0 - 2 jours
release
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Learning #4 : Don’t block the others
• We keep the features separated until the last moment
• Other teams are merging by default !!!
• Continuous Merge (as test tooling):
• Detect conflicts with others (without requiring you to get their code)
• Allow you to avoid conflits
• Allow you to drop your code temporarely
•Bonus:
• Avoid unfinished code to be used by the others
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Break release branches, release from trunk
• Improvments:
• Automated release from finished development
• Per feature code review & demo
• 15 minutes regression testing (testing architecture)
• Release by 1 dev alone
• Monitoring & Root cause analysis of issues
2014: Daily Releases
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Learnings : Kanban as Code
#1 : Release early, release often #2 : Patch process as main release process #3 : Avoid merge & conflicts #4 : Don’t block the others
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Take away : Kanban as Code
The Goals:
• # A Flow of Features at code level (Branches)
• # Release what is ready when it’s ready (Start by finishing)
• # Branches are your Kanban card (Step 1 of a pull system)
Why does it work :
• # Enforce the features to be independent (Really Independent !)
• # Low risk of conflict on 500.000 Line of Code (daily)
• # No branch can block another one (or it the same)
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Beastie Furets (LesFurets IT Team)
@beastiefurets
beastie.lesfurets.com
lesfurets
@dbaeli@beastiefurets
MERCI !
69
LesFurets.com LeanKanban.fr29, 30 Novembre 2017 www.leankanban.fr
@beastiefurets@dbaeli
Avoid the conflict
@beastiefurets@dbaeli
Change 3 : Original (conflict with Change 2)
master
commit 3
@beastiefurets@dbaeli
Change 3bis : New version (avoid conflict)
master
commit 3
commit 2
@beastiefurets@dbaeli
Commit 1, 2: Easy merge
master
commit 3
commit 2
resolved
commit 1
@beastiefurets@dbaeli
Change 1, 2 & 3bis : Merged
master merged
commit 3
commit 2commit 1
@YourTwitterHandle@YourTwitterHandle@beastiefurets@dbaeli
Learning #3: Avoid merge & conflicts
• For 2 Features developed separately:
• Merge = enforce to release together
• But Merging new code is the default for many teams
• Avoid merges (or release immediately):
• Trunk Based Development
• Any alternatives ?