chrome release cycle 12-16-2010
DESCRIPTION
Slideshow explaining how Google tackles development of the Chrome Web browserTRANSCRIPT
![Page 2: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/2.jpg)
Our Philosophy
Think of any major website, even the fancy Web 2.HTML5 ones... do they have version numbers?We took the same approach to our client software as an online web service.That is... we treat releases as a means of getting features out to users and not goals in and of themselves.It's about flow.
![Page 3: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/3.jpg)
How Our Users Work
Users sign up for Chrome release channels based on the level of stability they want.Updates happen automatically in the background.New features and functionality just flow in without any effort on the user's part.
![Page 4: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/4.jpg)
How the Chrome Team Works
Developers work almost exclusively on a single central trunk (possible due to our try bots and continuous build infrastructure)Our branches stem from points on the trunkWe stabilize branches by pulling in changes from trunk (i.e. everything lands on trunk first)
![Page 5: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/5.jpg)
Where we started (v1-4) - On Paper
![Page 6: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/6.jpg)
In practice, for that final beta...we'd
Cut a branch (when we thought we had everything)Merge about ~500 patches (since we didn't have everything)Spend weeks stabilizing and re-stabilizing issues.Ended up working 1-3 months to get a release out the door, always certainly missing our 13 week plan.And at the end finally shipping something we were happy with... but that left us pretty drained (i.e. the bad flow)
![Page 7: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/7.jpg)
The chain of events
We'd hold releases, sometimes our branch points, until features were done, which meant...Lots of changes got queued up on the trunk, which caused... Engineers merging lots of changes to the branch, which led to...Instability on the branch that we had to deal with and that meant...
![Page 8: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/8.jpg)
That chain of events...
Long lived branches, which...Made supporting the branch (i.e. merging changes) more difficult as we drifted further from the trunk...All of which led to...
Unpredictable release cycles, with date targets we could never hit, and....Engineers who were always in a rush to get their features into "this" release since we couldn't make any promises about the schedule
![Page 9: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/9.jpg)
Which led me to think...
I need a long vacation.There has to be a better way.What's causing things not to flow smoothly?
...Yes...In that order.
![Page 10: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/10.jpg)
Primary GoalsShorten the release cycle and reduce the life span of a branch (make merges easier)Make releases more predictable and easier to scopeReduce the strain on the entire team
BadGood
![Page 11: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/11.jpg)
Goal to Plan - Getting down the cycle
Originally set out a plan to simplify the 12 week cycle, 6 weeks on dev, 6 weeks on beta, and then to stable.Once diagrammed, using blocks to represent the weeks, realized that with 3 channels you could have two overlapping releases running at once, which could get us a stable release ~ every 6 weeks.
![Page 12: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/12.jpg)
Block diagrams
![Page 13: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/13.jpg)
It was a start, but it wasn't the full answer
Sure, turning the wheel faster would make some things, like merges, easier.But without addressing the scope problems, the flow wouldn't be any better and would continue to interrupt releases.Things wouldn't be any more predictable and the whole cycle would just fall apart (more puddles, less of a stream).
![Page 14: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/14.jpg)
Goal to Plan - Controlling Scope
The pace of the schedule sets the boundaries for the amount of work that can be completed.It's important to have specific points in the schedule to review features and cut scope.Establish clear expectations (and engineering practice) to developers that any features not ready to ship at branch will be disabled (i.e. we only cut post branch, never add).
![Page 15: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/15.jpg)
Plan to Pitch
Reached out to the various cross functional groups on the Chrome team, who would all be impacted, before approaching our executives.
Engineering, QA, Product, Marketing, Support, Localization, Security, etc...
There were a lot of concerns to address, but the exercise forced a lot of thought on the implications of the schedule change.It took time, but it made the pitch easier to present to the leadership and the rest of the team.
![Page 16: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/16.jpg)
ConcernsWould large feature development still be possible?
"Yes, engineers would have to work behind flags, however they can work for as many releases as they need to and can remove the flag when they are done."Can the engineers keep up?
"Their pace won't need to change, since features can be disabled there should be no milestone pressure, things ship when they are ready."What would a world look like where we didn't base our marketing on releases?
"We market features, not releases."
![Page 17: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/17.jpg)
And so we implemented it
11 week overlapping schedules.
![Page 18: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/18.jpg)
Our General Rules
The branch point is the end of our development cycleFeatures only get disabled post branch pointFeatures should be engineered so that they can be disabled easily (1 patch)Only stability, security, and critical regressions can ever block a release
![Page 19: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/19.jpg)
Things we struggled withOvercoming team inertia (particularly when we cut scope)Saying no consistently to the teamFighting the urge to return to date driven management decisionsWe initially didn't solve the trunk->dev->branch time to patch problem (daily channel helped).Setting the right burn down point (i.e. branch point).Disabling features.
![Page 20: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/20.jpg)
Key Lessons
Having clear and concise rules is important.Predictable schedules cut down on communication costs and team confusion.It takes work and fore-thought to disable features.Diligent feature tracking is important.
![Page 21: Chrome Release Cycle 12-16-2010](https://reader035.vdocuments.us/reader035/viewer/2022062705/5571f92e49795991698efc30/html5/thumbnails/21.jpg)
ConclusionSpeed alone isn't always the right answer, it's about keeping things running smoothly.For us, scope was getting in the way of that goal.
We basically wanted to operate more like trains leaving Grand Central Station (regularly scheduled and always on time), and less like taxis leaving the Bronx
(ad hoc and unpredictable).