Skip to main content
Fix grammar and missing words
Source Link
gmolaire
  • 464
  • 4
  • 7

I will assume by CI you mean one integration server. The best way to get the maximum out of your tests and environment stability is to decouple environment per test type. The best example is production which is lively "tested" by your users.

Be aware that by environment, I mean the server or servers used to deploy a specific version of your application or applications (I will refer this as the build). In such emvironment, you would tipically have similar setup as your live/production environment. Based on your team restrictions, here is what I proposed:

Restricted from having a new environment

Simply schedule a job to start load testing when the traffic is minimal. This is usually at night. Upon the test completion, a report should be available and granular enough to indicate where improvements are required based on your internal NFR threshold.

No restriction on having another environment

  • Create a new environment for load tests
  • For each new build, if it was successfully deployed and/or tested in the CI server, deploy to the new environment
  • Run the load tests and save the results. Depending on your CI tool, you might be able to configure reports and alerts according to your results quality

In both case, you will need to specify your load test suite in a way that allows you to figure out where latency is too high. I would personally suggest you the second solution. Though it requires more setup and maintenance, it is the most benificial for the following reasons:

  • Tells you exactly when the build starts to be unstable. For the first solution, you might need to investigate because a lot of change is happening during the day.
  • High visibility on what/who made the system slower. At this time, the teams are aware of the issue and can remedy right away. The nighly job might be overlooked on the long run.
  • Can serve as a performance gate where all builds with a certain response time will stop from going further (i.e. production) until the response time is improved. This is unpractical to put in place with a nightly job. Also, you would need to wait a day to see the result of an improvement or run the test manually.

If you are in the journey to CI/CD, fast failures is prefered as delayed failures. Losing a day to fix a peoblemproblem can have considerable impact on your time to market.

I will assume by CI you mean one integration server. The best way to get the maximum out of your tests and environment stability is to decouple environment per test type. The best example is production which is lively "tested" by your users.

Be aware that by environment, I mean the server or servers used to deploy a specific version of your application or applications (I will refer this as the build). In such emvironment, you would tipically have similar setup as your live/production environment. Based on your team restrictions, here is what I proposed:

Restricted from having a new environment

Simply schedule a job to start load testing when the traffic is minimal. This is usually at night. Upon the test completion, a report should be available and granular enough to indicate where improvements are required based your internal threshold.

No restriction on having another environment

  • Create a new environment for load tests
  • For each new build, if it was successfully deployed and/or tested in the CI server, deploy to the new environment
  • Run the load tests and save the results. Depending on your CI tool, you might be able to configure reports and alerts according to your results quality

In both case, you will need to specify your load test suite in a way that allows you to figure out where latency is too high. I would personally suggest you the second solution. Though it requires more setup and maintenance, it is the most benificial for the following reasons:

  • Tells you exactly when the build starts to be unstable. For the first solution, you might need to investigate because a lot of change is happening during the day.
  • High visibility on what/who made the system slower. At this time, the teams are aware of the issue and can remedy right away. The nighly job might be overlooked on the long run.
  • Can serve as a performance gate where all builds with a certain response time will stop from going further (i.e. production) until the response time is improved. This is unpractical to put in place with a nightly job. Also, you would need to wait a day to see the result of an improvement or run the test manually.

If you are in the journey to CI/CD, fast failures is prefered as delayed failures. Losing a day to fix a peoblem can have considerable impact on your time to market.

I will assume by CI you mean one integration server. The best way to get the maximum out of your tests and environment stability is to decouple environment per test type. The best example is production which is lively "tested" by your users.

Be aware that by environment, I mean the server or servers used to deploy a specific version of your application or applications (I will refer this as the build). In such emvironment, you would tipically have similar setup as your live/production environment. Based on your team restrictions, here is what I proposed:

Restricted from having a new environment

Simply schedule a job to start load testing when the traffic is minimal. This is usually at night. Upon test completion, a report should be available and granular enough to indicate where improvements are required based on your internal NFR threshold.

No restriction on having another environment

  • Create a new environment for load tests
  • For each new build, if it was successfully deployed and/or tested in the CI server, deploy to the new environment
  • Run the load tests and save the results. Depending on your CI tool, you might be able to configure reports and alerts according to your results quality

In both case, you will need to specify your load test suite in a way that allows you to figure out where latency is too high. I would personally suggest you the second solution. Though it requires more setup and maintenance, it is the most benificial for the following reasons:

  • Tells you exactly when the build starts to be unstable. For the first solution, you might need to investigate because a lot of change is happening during the day.
  • High visibility on what/who made the system slower. At this time, the teams are aware of the issue and can remedy right away. The nighly job might be overlooked on the long run.
  • Can serve as a performance gate where all builds with a certain response time will stop from going further (i.e. production) until the response time is improved. This is unpractical to put in place with a nightly job. Also, you would need to wait a day to see the result of an improvement or run the test manually.

If you are in the journey to CI/CD, fast failures is prefered as delayed failures. Losing a day to fix a problem can have considerable impact on your time to market.

Source Link
gmolaire
  • 464
  • 4
  • 7

I will assume by CI you mean one integration server. The best way to get the maximum out of your tests and environment stability is to decouple environment per test type. The best example is production which is lively "tested" by your users.

Be aware that by environment, I mean the server or servers used to deploy a specific version of your application or applications (I will refer this as the build). In such emvironment, you would tipically have similar setup as your live/production environment. Based on your team restrictions, here is what I proposed:

Restricted from having a new environment

Simply schedule a job to start load testing when the traffic is minimal. This is usually at night. Upon the test completion, a report should be available and granular enough to indicate where improvements are required based your internal threshold.

No restriction on having another environment

  • Create a new environment for load tests
  • For each new build, if it was successfully deployed and/or tested in the CI server, deploy to the new environment
  • Run the load tests and save the results. Depending on your CI tool, you might be able to configure reports and alerts according to your results quality

In both case, you will need to specify your load test suite in a way that allows you to figure out where latency is too high. I would personally suggest you the second solution. Though it requires more setup and maintenance, it is the most benificial for the following reasons:

  • Tells you exactly when the build starts to be unstable. For the first solution, you might need to investigate because a lot of change is happening during the day.
  • High visibility on what/who made the system slower. At this time, the teams are aware of the issue and can remedy right away. The nighly job might be overlooked on the long run.
  • Can serve as a performance gate where all builds with a certain response time will stop from going further (i.e. production) until the response time is improved. This is unpractical to put in place with a nightly job. Also, you would need to wait a day to see the result of an improvement or run the test manually.

If you are in the journey to CI/CD, fast failures is prefered as delayed failures. Losing a day to fix a peoblem can have considerable impact on your time to market.