How to properly scale Jenkins?

Question

In my project we have one AWS server running Jenkins Master + 1 Jenkins slave (2 executors)... and we need more
In order to augment our build power we have three options:

1. Scale up: Make AWS instance bigger and add more executors.
2. Scale up: Make AWS instance bigger and add another jenkins slave process.
3. Scale out: Create another AWS instance with a jenkins slave and connect it to master

We want to do 2. as we are in a big organization and our current Jenkins Master has already access to every place he needs. Option 3. "New server" is complicated as it needs more bureaucratic approvals that will take weeks.

So my questions are:

• Is there any technical issues in option 2?. Maybe the executors of each jenkins slave are not aware of the other slave executors?
• In general, what is the best approach to scale Jenkins? Scaling up or scaling out?

Answer 1

There are no fundamental technical issues with running multiple jenkins slaves on the same machine. In fact Running Multiple Slaves on the Same Machine lists several good reasons for doing it:

While the correct use of executors largely obviates the need for multiple slave instances on the same machine, there are some unique use cases to consider:

• You want more configurability between the configured nodes. Say you have one node set to be used as much as possible, and the other node to be used only when needed.
• You may have multiple Jenkins master installations building different things, and so this configuration would allow you to have slaves for more than one master on the same box. That's right, with Jenkins you really can serve two masters.
• You may wish to leverage the easiness of starting/stopping/replacing virtual machines, perhaps in conjunction with Jenkins plugins such as the Libvirt Slaves Plugin.
• You wish to maximize your hardware investment and utilization, at the same time minimizing operating cost (e.g. utility expenses for running idling slaves).

In general scaling out is preferred, primarily because ability to scale up is typically limited by the types/sizes of the available physical resources.

In particular for augmenting build power I'd recommend an analysis of your actual build to determine how it uses the machine resources, which/where its bottlenecks are and what scalability limitations it raises to reveal if scaling up even helps.

For example I encountered cases in which the build time for 2 parallel builds on the same machine was longer than the combined build times of the same 2 builds executed sequentially (non-overlapping) on the same machine. In such case I wouldn't even consider scaling up as it would actually decrease the overall building capacity.

Answer 2

Use Kubernetes and Helm.

I would recommend using the Jenkins Helm chart. It installs with helm install stable/jenkins and automatically scales.

https://github.com/kubernetes/charts/tree/master/stable/jenkins

Answer 3

I think you should do neither ;)

Well kinda. I think you need more executors, maybe your builds are really resource intensive? I would run at least 4 but we run 6 to 8 depending on jobs. I like to match # of cores to exectors. So you might want to scale up your nodes, I think we run a M4 large for our 4-8 executors.

I also think you should scale-out but you should do so smartly. Jenkins has a plugin to automatically scale-out on AWS depending on what is in the build queue. Basically you tell it how many jobs and how long the wait before it stands up a slave and sends the jobs to the new slave. You can also set the max amount of slaves, the min amount etc.

Answer 4

I'd scale out instead of scale up, going for option 3. We have made a setup where we have all Jenkins agents running on an ECS (custom Docker based Jenkins) with an auto-scaling group. We have all our Jenkins masters communicating to the ECS, thus sharing the workload on the ECS, and no need to recreate the Jenkins master in a scale up exercise.