Should performance tests be run using a CI and how often?

Question

There are tools that test the performance of applications, like gatling. It is possible to run such a test once in a while, but in practice is is not run periodically. We were discussing how to run it regularly and add it to every build, but the CI was getting slow. On the other hand it is important to know if a code change decreases the application performance.

Answer 1

I will assume by CI you mean one integration server. The best way to get the maximum out of your tests and environment stability is to decouple environment per test type. The best example is production which is lively "tested" by your users.

Be aware that by environment, I mean the server or servers used to deploy a specific version of your application or applications (I will refer this as the build). In such emvironment, you would tipically have similar setup as your live/production environment. Based on your team restrictions, here is what I proposed:

Restricted from having a new environment

Simply schedule a job to start load testing when the traffic is minimal. This is usually at night. Upon test completion, a report should be available and granular enough to indicate where improvements are required based on your internal NFR threshold.

No restriction on having another environment

• Create a new environment for load tests
• For each new build, if it was successfully deployed and/or tested in the CI server, deploy to the new environment
• Run the load tests and save the results. Depending on your CI tool, you might be able to configure reports and alerts according to your results quality

In both case, you will need to specify your load test suite in a way that allows you to figure out where latency is too high. I would personally suggest you the second solution. Though it requires more setup and maintenance, it is the most benificial for the following reasons:

• Tells you exactly when the build starts to be unstable. For the first solution, you might need to investigate because a lot of change is happening during the day.
• High visibility on what/who made the system slower. At this time, the teams are aware of the issue and can remedy right away. The nighly job might be overlooked on the long run.
• Can serve as a performance gate where all builds with a certain response time will stop from going further (i.e. production) until the response time is improved. This is unpractical to put in place with a nightly job. Also, you would need to wait a day to see the result of an improvement or run the test manually.

If you are in the journey to CI/CD, fast failures is prefered as delayed failures. Losing a day to fix a problem can have considerable impact on your time to market.

Answer 2

There is no right answer, in a sense you're facing a requirement conflict. You can't eat the cake and have it, too ;)

If a particular QA verification (any, really, not just performance) is important enough and you want to catch any regression right away - you have to insert it in the CI execution. But, unless you manage to somehow execute it in parallel with other existing jobs in the CI pipeline, it will extend the duration of the CI execution.

You could, if you want, perform it in a "slower" CI pipeline, executed less often than your regular CI pipeline. For example just nightly instead of after every commit. The drawback is that when a regression is detected you'll have to sift through a bigger pile of commits to identify the culprit. Which can be done either manually, though human analysis, or using automated bisections executing the affected QA verifications (or even a mix of both). But that takes time and resources. If it's a rare event it could be an acceptable compromise, if not then you have to decrease the time between the "slower" CI pipeline executions.

You have to decide what's more important: the CI execution speed or catching regressions faster. This decision can also depend on the development stage context. For example:

• if you're in the master/dev development branch where you have a lot of changes coming in the CI execution speed would likely be more important
• if you're on a release branch cut for production, where you have a significantly lower rate of commits than in the master branch, catching regressions faster would have priority (the longer CI execution would have less impact)

Finally, such decision would be easier when using a CI system capable of combining changesets because the impact of a longer CI execution over the effective changeset verification time is reduced, see From 6 months to 16 seconds? grokking DevOps speedup data from DevOps report by DORA and Puppet Labs.

Answer 3

One additional way to think about this is to consider alternate approaches, and whether they are suitable for your needs.

One of the axis I break down performance metrics on is whether they are real-user metrics or synthetic metrics. Your performance test is presumably synthetic: it runs the same set of actions every time in the same environment, so as to make it easy to compare runs against each other.

However, synthetic tests don't show the actual effect a change will have on users, which is what RUMs are all about. You can do canary rollouts, and if performance decreases in the canary past an acceptable threshold, roll it back and investigate (I saw a talk from Facebook about how they gather full debug traces on their canary servers and so can find which code was responsible for the slowness and automatically backtrace that to a set of commits and authors). This is more realistic in the user impact, but can be difficult to do without large amounts of traffic, as there is normal variation in what real users do.

Ideally, you'll be using both of these methods of tracking performance. But since we don't live in an ideal world, you may be able to retain a focus on performance at your company by shifting more emphasis on one of these versus the other, to free up resources for other needs. In your particular example, is it an ok trade-off to have slower deploys if it means reducing the CI feedback cycle? Maybe - but that depends on your company.