Praqmatic Software Development

CoDe:U Git Flow - a Continuous Delivery Approach

1

Being able to identify different promotion levels of even fairly small code snippets becomes an essential and important part of the agile and Continuous Delivery process. For that reason more and more teams see the underlying Version Control System’s (VCS) ability not only to branch but more importantly to merge back again as a required feature.

The modern distributed VCS, such as Mercurial and in particular Git, are being adopted and deployed everywhere - and for a natural cause they are not just a hype. They actually do the job well and easy.

A successful Git branching model Many teams have not been used to working with branches and especially not the process of merging as intensely as when they migrate to Git. Therefore, it’s natural to seek advice. The blog post by @nvie entitled A successful Git branching model - most often referred to as “the Git Flow” - has become the de facto branching strategy for many teams.

We have come to discover that in teams delivering continuously there is a requirement for even more automation and even less branch categories than outlined by @nvie.

This paper describes the branching strategies, the underlying rationale and the building blocks that automate it all to become the preferred choice, for a Continuous Delivery approach. We refer to it as the CoDe:U Git Flow.

Inspiration is free - so do not hesitate to call us!

You can reach us on phone +45 36 PRAQMA or send an email to info@praqma.com

Leif Sørensen & Lars KruseFounders of Praqma

Continuous Delivery (CoDe) is an approach that builds very much on divide and conquer. It’s bred from a lean and agile paradigm and it encourages you to minimize work in progress and continuously deliver the bits of work you are done with.

The CoDe:U Git Flow The automation can be referred to as a branchy approach to pretested integration, and the branching strategy is referred to as lazy, late, or deferred. Basically, this means that CoDe:U Git Flow branches as little and as late as possible. The Git Flow is based on three pillars:

• A set of principles• A vocabulary and a set of naming conventions• Continuous Delivery Pipelines with

automated promotions

A set of principles The 10 principles that make up the entire strategy.

1. The project integration branch is where developers continuously integrate (CI). Each developer should strive to integrate at least once every day.

2. The project integration branch in Git is master.

3. Integrations onto master must pass an automated toll-gate of minimum verification (typically build+unit test).

4. Only trivial merges than can be done without human intervention are allowed onto the integration branch.

5. Every successful integration onto master kicks off the pipeline.

6. The integration branch is always aiming for the next release. Anything that shouldn’t go into the next release should be kept out of the integration branch until it’s due.

7. Any push to the centralized repository that contains a commit on a development branch that matches the naming convention triggers an automated integration onto the corresponding integration branch.

8. Some development branches may be tied to maintenance branches rather than master.

9. All integrations onto promotion branches are automated (in our setup by Jenkins CI).

10. Any given promotion branch can only have one contributor - merges are guaranteed to be fast-forward.

A vocabulary and a set of naming conventions

Integration branch The shared work context of the developers where integrations are done frequently. master is typically the project’s integration branch. But occasionally maintenance branches may take on the role as integration branch for a dedicated context.

Promotion branchesSometimes also referred to as Big picture branches. Covers all the branches which are above and beyond the project integration branch. Any given promotion branch only has one contributor (e.g. stable merges master, release merges stable). Due to the nature of promotion branches only fast-forward merges are allowed.

Development branchesOccasionally referred to as small picture branches. These are all the branches that produce contributions to the integration branches. These are the branches where developers actually commit code, branch to and from their colleagues, rebase from the integration branches, solve their merge conflicts, share team branches, rewrite history etc.

Maintenance branchesA stream oriented branching strategy such as the one we describe here essentially implements a going forward approach - a release train. It implies that if anything is broken, you fix it with a new release. While this indeed is a healthy principle, it’s sometimes not possible to implement fully due to licensing issues, feature toggles, customizations, product variants etc. The solution when a release train strategy can not be implemented is to use a maintenance branch - a parallel integration branch, not for the project but for a particular release.

Maintenance branches should be introduced with some caution, since adapting into the main track should be the norm. Seen from the perspective of the maintenance context it is temporarily the new integration branch, but seen from the integration branch it’s just like any other development branch. Or simply put; A maintenance branch is a development branch you allow yourself to release from.

3

Hot fix branchesHot fix branches are used frequently in both @nvie’s branching strategy and also in the built-in strategy that is provided in Atlassian’s repository browser Stash - so we feel an obligation to mention it too. But in Continuous Delivery a hot fix in the meaning “a shorter or faster track to a release than the regular path” is a contradiction in terms. They simply do not exist. There is only one track to a release, and that is a successful ride through the Continuous Delivery pipeline.

We will erase the term “hot fix branch” from our vocabulary.

PipelineThe set of automated verifications that kicks off automatically upon every commit to an integration branch. The purpose of the pipeline is to promote the commit to the highest possible level of trusted quality. We believe in the doctrine that “every commit is a potential release candidate”. The pipeline is a chain of events, only the kick-off is triggered by a commit, the remainder of the events are triggered by the previous events having a successful outcome. A pipeline typically has the scope of a repository.

Pretested integration Effectively, an automated toll-gate process that merges contributions into a given target context, runs a verification job and - if the outcome is acceptable - commits the changes. This way, only tested integrations exists - the integration branch is kept pristine. Each pipeline has its own set of verification rules, adjusted to the given context. In our setup developers do not have write access to integration branches or promotion branches. They can implement their suggested changes on development branches and raise a flag indicating that they want in. The flag is seen by the pretested integration processes. Everything from this point on is automated.

When the automated integration is managed by the pretested integration plugin and branches are named by a certain pattern, things happens automatically.

These naming conventions are all configurable, so if you’d rather call your branches something else than what we recommend here, simply define your own naming convention when you configure the automation.

Analysis and metrics

Build &Unit test

Rejected

Failed

Failed

Failed

Failed

Failed

Passed

Failed

Passed

Passed

Passed

Passed

Passed

Released

Deploy

Functional test

Review & doc.

Acceptance test

“Every commit is a potential release candidate” a state-transition diagram of the process that could verify a release candidate from the point where the code is committed to the VCS and through the verifications that would finally promote it to a release candidate status.

4

Promotion branchesAny given promotion branch in our setup only has one source contributor, consequently all merges to promotion branches are always fast-forward. Fast-forward is a Git term meaning that no actual merge is required. It’s simply a matter of moving a branch head to a different commit that is already accessible from the current commit without merging anything.

Using promotion branches this way means that they behave more like floating tags, and we use them to indicate successful promotions. It’s a simple and efficient way of storing tiny bits of meta data in Git - allowing it to be easily synced between clones.

Typically, a pipeline can have many verification events, all working on the same commit. Essentially, all passed verifications represent some level of promotion, so should a promotion actually cause a merge onto another branch?

In reality, you could easily define a separate promotion branch for each individual step in your pipeline. In our setup the pipeline is executed by Jenkins CI so that would, to some extend, create redundant information of the same promotions being maintained by Jenkins CI.

We recommend the principle that every time a promotion causes a new group of people to take interest in the commit, it’s promoted to a promotion branch. In reality we very rarely see more than two promotion branches in our setups: stable and release.

stable indicates that the commit is ready to leave the project team and make sense to the rest of the organization; Testers and other development teams who want to use the delivery can keep themselves updated with the stable branch. release indicates that the commit is ready to leave the organisation. This content could be made available to end-users and customers.

Development branchesThe main goal when choosing names for development branches is to adapt into a predictable behavior that enables your colleagues to read what you are doing.

In the end, developer branches are indeed the developer’s and it is quite often that developers make branches in their own local clones that aren’t necessarily meant to be pushed anywhere - at least not for the time being. So we encourage you to allow developers to name their local branches any way they want.

But! Since the integration is automated and triggered by pushes to the remote origin, you will have to push to a remote branch name that follows a certain pattern.

The pattern we recommend is :

Where the keyword ready is the indicator that it should be integrated and the configuration you use in the Jenkins setup.

When you push a commit to origin and the commit is on a branch with the configured naming pattern (ready/** in our example) then your work will be automatically merged onto the integration branch and it will kick off the pipeline for the repository you are working in.

Your branch may be named something less informative like mybranch while you work on it in your local clone - and in this case you can push it to origin as often you want, simply for information sharing with colleagues or for backups at the end of the day.

When you are truly done and have committed you can push it to a branch that will trigger the integration. Like this:

Essentially your local branch mybranch becomes ready/case_514 on the remote - and all the magic starts.

5

Tim

emaster

Devlopment branches

Feature for future release - not integrated

yet

Major feature for next release

At any point in time “next release” literally

means “anything shippable"

Promotion branches

feat feat

Any integration into master and beyond to the promotion branches

is automated

Origin: CoDe:U by Praqma Inspired by: @nvie License: Creative Commons

Integration branch

Automatically verified and

promoted by the CI platform

Tag

1.0

Tag

1.1

Stable

Release

maint

Stable

ReleaseTag

1.0.1

Maintenance releases done

from here

All merges into promotion

branches are fast-forwards

6

Continuous Delivery Pipelines with automated promotions Continuous Delivery Pipelines have impact on the flow as well. The concept of a pipeline is that it is a set of related job executions that together verify a repository - either a component or a product - to a level where it’s a candidate to be released.

In any case a repository has a pipeline which we typically implement in Jenkins CI using the pretested integration plugin, but with a bit of adaption it can be implemented in any automation platform such as Bamboo, Travis, BuildBot, MFS, or whatever you prefer. The key concept is that the pipeline kicks off when something is changed on the respective repository’s master branch.

A pipeline is related to something that is tangible and which can be shipped, like a product or a component. Components are the building blocks, and products are the constructions that you build out of your building blocks.

The difference for you as a developer or a release manager is basically that sometimes you version control files and some times you version control dependencies. But the flow model and the branch naming strategies you use for the purpose are the same.

The pipeline will know what to do with your commits, and if the verification passes, then you’re OK. If not, then you still have more work to do.

The pretested integration plugin To support our flow, we’ve developed the pretested integration plugin for Jenkins CI. The plugin is Open Source and is installed like any other plugin from the Jenkins CI management menu.

There are already several Jenkins CI plugins that implement some kind of automated integration, including the one we made ourselves for ClearCase UCM. There are even existing solutions for doing this in Git, so why develop a new Jenkins CI plugin for this purpose?

To put it short: The pretested integration plugin for Jenkins CI introduces a generic branchy approach to pretested integrations which can be turned on by the check of a checkbox and most importantly which utilizes the existing SCM plugins.

There are many different approaches to achieve pretested integration using anything from clones, shelves, caches or branches. We argue that an approach using branches, is the right way simply because it’s the only one that is truly self-contained within the repository.

It means that the meta data required to make this work is all available within every single clone. It makes the implementation simpler and more resilient.

In a pretested integration situation you need to do several steps: Set-up a workspace and merge stuff into it before it is handed over to the build step. After the build is done a decision is required, whether to commit the changes or discard them.

Almost all current Jenkins CI plugins that support pretested integrations are in the SCM plugin category. But this type of plugin does not natively support a footprint in the post-build step, and they will all have to implement multiple extension points (typically the Notifier plugin category as well as the SCM plugin category).

Our extensive experience with plugin development on Jenkins CI has taught us, that we achieve the sleekest and most generic implementation when reusing all the features already implemented in the various SCM plugins and then implementing a small plugin based on the build-wrapper category, which natively has a footprint in both the pre- and post-build steps.

Our implementations for Git and Mercurial uses the original SCM plugins for Jenkins CI just out of the box - and we really don’t configure much.

For a detailed description on how to install the plugin and configure it to support the flow visit the pretested integration plugin’s wiki page at the Jenkins Ci community (a shortened version to ease your typing: bit.ly/pretested_integration).

If you do give it a spin, we hope that you will share your thoughts, ideas and feedback - either on the Jenkins CI wiki, or on Twitter in a mention of @praqma.

Now go deliver something!

7

Praqma A/SAllerød Stationsvej 43450 AllerødDenmarkTel: +45 36 PRAQMAEmail: info@praqma.netWeb: www.praqma.net

Continuous Deliver Users CoDe:U is a professionally driven user group. It’s an community for people who are interested and engaged in the disciplines of Continuous Delivery and DevOps and who believes in sharing experiences, worries, challenges and successes as a mean to become wiser.

The Git Flow presented in this paper was developed in collaboration with several corporations. The Jenkins CI plugin that does the automation is Open Source and the model and concept is handed to the CoDe:U community - to be maintained, developed and used.

Joint Open Source Roadmap AllianceJOSRA is an alliance of companies who use the same Open Source tools in their Continuous Delivery tool stack and who jointly maintain a strategic roadmap for these tools. Jenkins CI and a lot of plugins are included.

JOSRA is also the maintainer of the Pretested Integration Plugin.

You are encouraged to join the alliance: www.josra.org

References • Pretested Integration Plugin wiki on Jenkins: bit.ly/pretested_integration• Continuous Delivery Users: www.codeu.eu• Joint Open Source Roadmap Alliance: www.josra.org• PDF version of this paper: www.praqma.com/git-flow• @nvie’s original blog: nvie.com/posts/a-successful-git-branching-model

8