How are you managing automated deployments with active development on the next release while features are still being tested in QA?

Question

• CI/CD tool: Azure Devops Services
• Branching Strategy: Git Flow
• Build Triggers: Auto build Dev branch on PR merges, Auto build on PR creation
• Release Triggers: Create release on every build
• Deploy Triggers: Deploy to DEV on PR merges, Deploy to QA once each morning, deployments to other environments are triggered manually
• Production Release/Deployment: At the end of every sprint
• Testing Strategy: Some manual, some automated. Regression takes a week (this is too long I know, but this is where we are).

When we are feature complete, we will create a release branch. That release branch will be deployed to QA. From that point on we do not want PRs into the DEV branch to be released to QA until after we have deployed to Production.

Currently we simply turn off the deploy trigger for QA and deploy manually as necessary into the QA environment. We turn the daily deploy trigger back on once the PROD release is successful.

This has always seemed a little clunky to me.

• Should we be managing/promoting within environments differently?
• Should we look into approvals?
• Is there something we should look into updating the Deploy agent pool instead?

Answer 1

Since it doesn't sound like you're using continuous deployment or continuous delivery, you may be deploying to your QA environment too frequently. I would recommend only deploying your release branches to QA for final checks before going to production.

I'm going to assume that you have sufficiently robust automated test coverage in various forms, from unit tests through end-to-end tests. A lack of automated test coverage will make everything slower, more error prone, and require even more manual testing as the system becomes more complex. It would also be helpful to have linters and static analysis in place to quickly detect style, performance, and security issues during your build pipelines.

When you deploy to your development environment on a PR merge, that is where any test setup is happening. Run your full suite of automated tests and put the changes in an environment where any manual testing can be dry run outside of a release context. This helps keep both your system and your tests stable and ready for release.

When you decide to release, deploy the release branch to your QA environment. Run your automated tests and any manual tests here. If everything is good, you're good to go to production. After triaging failures, you can make the decision to go to production with any known issues to fix later or resolve the failures and rerun testing to confirm.

During this process, you can continue to make changes for the next release in the development branch. You do need to make sure that any fixes for the release branch do get merged back into the development branch as well, if they are still relevant.

The biggest concern that I have with teams that need to take the approach is how time-consuming manual testing can be if automated testing does not keep up with system complexity. This is especially true if you're trying to fit the release process into a timeboxed iteration. But there are mitigations, such as having an independent test team handle that releases that work outside the timeboxed iteration or having the whole team involved in the testing activities prior to a production release.