Currently we are a team of 12 developers in two locations. In the past 3 quarters (3 releases) we always had an overload of bug-fixing just before the release:
- memory leaks
- functionality that worked and then it didn't for no apparent reason
- crashes in the product or binaries we integrated from other teams that blocked the testing team
Our manager suggested that we integrate blocking smoke tests on every build like another (more mature team). What blocking means is that after a pull request that gets merged into develop (the development branch), a build is made and then a smoke test (on 6 virtual machines with different OS versions) is run. If the smoke passes, the merge remains in the build, so the build passes.
- the smoke test runs in 56 minutes +- 10 minutes.
- the smoke test runs on 6 different OS versions in parallel
- the build takes ~30 minutes
- branching model : git workflow
What we thought of so far:
W: Blocking Smoke on every build:
Steps a dev would take:
- bugfix / finish feature, meanwhile doing n commits on his branch. pull request → merge in master
- manual build + automatic blocking smoke test
- two possibbilites (actually 4, but we will ignore here if the build fails on our side):
- a) happy-path: the smoke is green
- b) not so happy: there are issues in the smoke
- we look at b) :
- if there are any problems, we need to revert the merge / range of commits on the develop branch
- this can be done either manually or automatically
- if it's a manual process, we need another pull request with at least two approvals (this should be avoided and everything should be done automatically)
- after the revert, we shouldn't run the smoke again (because the state of the branch is exactly as it was before the failed smoke)
- at this moment the developer has no changes on the develop branch, only on his branch (actually they are there, but the code is reverted)
- he must then investigate why the smoke failed and fixes either his code and / or the automated tests
- makes another pull request
- pull request gets approved and is merged to develop.
- rinse and repeat the above process.
X: Blocking smoke on branches before merge to develop
Steps a dev would take:
- fix a bug / finish a feature on his branch
- branch build on bamboo
- serial blocking queue on bamboo, every dev would wait for the other builds to finish on the build machine.
- after the successful branch build, the smoke starts on 6 virtual machines in parallel. ETA 1h.
- obviously, each developer should have a clone of the 6 virtual machines to run the smoke. so we are talking about 6x machines, where x = number of developers
- this should be considered when talking about memory, storage space, processing power on the virtual center (our own cloud)
- from here we can repeat the above steps, from "happy path" to the final "pull request to fix the smoke", just that there will no merges / reverts needed as we are working on a branch.
Y: Blocking nightly smoke
Steps a dev would take:
- same as above with a smoke after each build, but with no blocking during the day.
- even if the smoke is red, no revert is done after a merge (only in special critical circumstances)
- fixes for the smoke would be incrementally be done on develop via pull requests and in the automate-testing repo if needed
- here we have a problem: if the smoke is not fixed on develop, the other builds with changes from other developers cannot be validated by that smoke
- the midnight smoke is blocking, that means the next day in the morning all devs are all hands on deck to fix it then continue as usual
Z: Blocking smoke on release
the same as the blocking smoke on every build, but this is done only after we branch of master and cherry pick fixes into it.
Small impact analysis on the above approaches:
- the virtual machines to be unavailable - we need at least two admins in our team to override smoke test in bamboo
- the more moving parts the more risks so: X is much more risky than W or Y or Z
I will try to estimate which of the above options is faster given an iteration / a release and in which order
- W vs. X: X is faster because the devs are working in paralel and there are no synchronous blocks like in W.
- X vs. Y: Y is faster because the devs won't wait for a build on each branch, there will be >2 merges / build.
- (W or X or Y) vs. Z: pretty hard to compare because, yes, Z is faster, but we need to consider the time needed to stabilise the build, because there is no blocking smoke up to that point - hence finding bugs is left towards the end.
Until now we have Y as the best approach time wise.
the more steps involved, the more moving parts, the more complex the approach
- Z < Y < W < X. X is the most complex, although it's disputable that Z < Y because there are extra steps involed in Z (stabilization of the build, etc)
X is the most resource intensive because we need 6x virtual machines for the smoke(s) ( x = number of developers )
Which of the above approaches do you use, please share any improvements / new ideas on how to best introduce the smoking phase in our build process.