Oleg Chorny@Ciklum

Release Engineering: Google's approach

Agenda

MotivationEngineeringRelease EngineeringAutomated Release SystemConfiguration Management

Motivation: Toil ● If a machine could accomplish the task just as well as a

human, or the need for the task could be designed away, that task is toil

● If you’re performing a task for the first time ever, or even the second time, this work is not toil. Toil is work you do over and over

● Toil is interrupt-driven and reactive, rather than strategy-driven and proactive

Motivation: Why toil is bad● Your career progress will slow down or grind to a halt if you

spend too little time on projects. Too much toil leads to low morale

● Excessive toil makes a team less productive. A product’s feature velocity will slow if the team is too busy with manual work

● If you build too much toil into your team’s procedures, you motivate the team’s best engineers to start looking elsewhere for a more rewarding job

Engineering: The work of reducing toil● Requires human judgment ● Produces a permanent improvement in your service ● Guided by a strategy ● Creative and innovative ● Helps your team or organization handle a larger service, or

more services, with the same level of staffing

Release Engineering: High Velocity● Aim to roll out customer-facing features as quickly as

possible ● Frequent releases result in fewer changes between versions.

This approach makes testing and troubleshooting easier● Some teams perform hourly builds and then select the

version to actually deploy to production● Other teams have adopted a “Push on Green” release model

and deploy every build that passes all tests

Release Engineering: Self-Service Model● Individual teams can decide how often and when to release

new versions of their products ● Release engineering has developed best practices and tools

that allow product development teams to control and run their own release processes

● Releases are truly automatic, and only require engineer involvement if and when problems arise

Release Engineering: Budgeting● Managers don’t always plan and budget for release

engineering in the early stages of a project● Teams should budget for release engineering resources at

the beginning of the product development cycle● It’s cheaper to put good practices and process in place early,

rather than have to retrofit your system later

Automated Release System: Branching● All code is checked into the main branch of the source code

tree ● Most major projects don’t release directly from the main

branch● Each release has own branch ● Release branch never merges changes back into the main

branch ● Bug fixes are submitted to the main branch and then cherry

picked into the release branch for inclusion in the release

Automated Release System: Building● If two people attempt to build the same product at the same

revision number in the source code repository on different machines, we expect identical results

● The build process is self-contained and must not rely on services that are external to the build environment

● Build tools must allow us to ensure consistency and repeatability

Automated Release System: Testing● A continuous test system runs unit tests against the code in

the main branch each time a change is submitted● An independent testing environment that runs system-level

tests on packaged build artifacts ● Because of cherry picks, the release branch may contain a

version of the code that doesn’t exist anywhere on the main branch. During the release, unit tests are run once again using the release branch

Automated Release System: Packaging● Packages are named (e.g., search/frontend), versioned with

a unique hash, and signed to ensure authenticity ● Labels can be applied to package to indicate a package’s

location in the release process (e.g., dev, canary, or production)

● If a package is labeled as canary, someone subsequently installing the canary version of that package will automatically receive the newest version of the package with the label canary

Automated Release System: Deployment

● Performed by a general-purpose rollout automation framework

● Framework provides a set of Python classes that can be extended to support any deployment process

● It has a dashboard that allows for finer control on how the rollout is performed and provides a way to monitor the rollout’s progress

● A rollout is a logical unit of work that is composed of one or more individual tasks

Automated Release System: Policies● Several layers of security and access control determine who

can perform specific operations when releasing a project:○ Approving source code changes ○ Specifying the actions to be performed during the

release○ Creating a new release ○ Deploying a new release ○ Making changes to a project’s build configuration

Configuration Management● In order to simplify deployment, include configuration files

and binaries in the same package ● If you want to release binaries and configuration files

independently, put configuration files into “configuration packages”

● If projects have configuration files that need to change frequently or dynamically, read configuration files from external store

● Using the mainline for configuration can lead us to skew between the checked-in and running version of the configuration file

Summary● It is essential that Developers and Release Engineers work

together● Release Engineer needs to understand the intention of how

the code should be built and deployed. Developers shouldn’t build and “throw the results over the fence” to be handled by Release Engineer

● When equipped with the right tools, proper automation, and well-defined policies, we shouldn’t have to worry about releasing software. Releases can be as painless as simply pressing a button

Thank youOleg Chornyoleg.chorny@gmail.com