Git Release branching strategy for Continuous integration and deployment

Question

We have tried multiple branching strategies to adopt CI/CD.

1. Developers develop on the feature branch and then merging to master which get deployed to QA for testing and then master is deployed to production, so in short whatever goes out to QA/UAT goes out to production unless you put a create a PR/patch to fix the bug - Least hassle of merging, syncing etc.

2. Developers develop on the feature branch and then merge to master which gets deployed to QA/UAT for testing and based on the testing outcome, spin out a release branch for things which needs to be deployed to Production.

What are the recommendations in case of organizations with 50 odd devs and spread across ~8 scrum teams?

Remember, we always have feature flags to toggle the releases and features pushed out.

Answer 1

You need to separate the 2 concepts a bit:

• the integration part - how are the changes integrated into your master branch (or some other integration branch, that's also possible), which may be continuous or not. Technically neither approach you describe is actually continuous integration unless your feature branches have a very short lifespan - typically less than 1 day. But this part doesn't seem to be the main focus of your post.

• the delivery/deployment part - how are the integrated changes from an integration branch (master in your case) being delivered/deployed. In other words your release branch strategy. This appears to be the point of your post, I'll focus on it below.

Both approaches you describe are valid release branch strategies, I've seen both being used. Each with their own advantages and disadvantages.

1. This approach is inline with the continuous delivery/deployment strategy. The CI/CD pipeline will have the commits to the master branch (either feature branch merges or point fixes) as entry point: each such commit will trigger the pipeline verification, attempting to progress into it as far as possible. The QA/UAT is one of your pipeline steps, it that passes then it goes into production. Effectively your releases are simply selected (typically tagged) commits from the master branch, they aren't really branches.

   Advantages:

   • as you noticed - the least amount of branch activities
   • everyone is on the same page - developers, QA, etc all focus on the same branch context
   • lowest cost (but only if the project scale allows it to be effective, see also the disadvantages section)

   Disadvantages:

   • you can't have actively maintained releases - you only have one "current" release out there, each additional fix will become another release. Which isn't necessarily a bad thing, such release model can actually be desired (for example in the case of SaaS products).
   • the release cadence is not actually predictable. It may be difficult to get releases out - if a problem is detected, by the time the fix for it is merged into the master branch some other changes may be merged in and other problems detected. This gets progressively worse as the rate of branch commits goes higher (larger scale projects) and may actually be non-convergent (the project may not reach stability at all). Doing CI/CD at scale is not a trivial matter.

2. This approach is a more traditional release strategy, but technically it's not really continuous delivery/deployment.

   Advantages:

   • more predictable release cadence. The spinned-off release (aka throttle) branches allow reducing the rate of commits for a particular release. Typically a release branch would only receive fixes necessary for that particular release, fixes for future releases (or release trains) would go into the master branch instead. The reduced commit rate allows reducing the risks of new problems being introduced that would require additional commits, the branch is more stable. Typically the release branches are pulled from the master/integration branch commits that passed all (or most) of the CI/CD pipeline verification stages (for the master branch). In your particular example those branches wouldn't expect subsequent commits (they passed QA/UAT), but if say a problem is detected after deployment in production it would be possible to just commit that particular hotfix into the branch and re-deploy, without having to pick up any other commit that went into master since the branch was pulled, reducing the risk of additional regressions.
   • it supports multiple simultaneous/overlapping actively maintained releases, with potentially complex release policies in place (long-term/short term/rolling/etc, like OpenSUSE's Lifetime for example).
   • it supports multi-level releases (like major.minor.increment, for example) - aka release "trains": a major release branch could itself have multiple minor release children branches, etc.
   • each release branch could actually use the CI/CD methodology at its own level, independently of the other release branches - if it never gets upstream/bulk merges from its parent (i.e. effectively becoming its own "master")!

   Disadvantages:

   • everyone is no longer on the same page. Each release branch has its own context which may come with its own policies, processes, tools, necessary resources, etc. Each requires its own development/testing/validation. This drives development/maintenance costs up, often significantly.
   • tracking issues and propagating fixes across multiple releases may not be trivial. Merging back release branches into master (or their parent branches) appears an inviting approach, but IMHO it's a bad practice, see How to get rid of develop branch for simplified Git flow .
   • context switching becomes a major problem.

Answer 2

You didn't mention the tools you are using for CI/CD. I worked on Azure DevOps and used visual studio 2017 for these activities. If you are using it for a large number of the developer, I would suggest you use visual studio 2017 or later version. Visual studio has facilities to pull the code on the local repository. Then create a decentralized sub-branch with Master branch and perform the changes. After completing the work, you can use merge or rebase technique to merge it with the master branch by clicking on the button. This will not only store the history but also save the person identity who modifies it. Push that to the Azure DevOps and rest will be done by Azure itself. I worked on it and it is pretty amazing stuff.