26

In my project we have one AWS server running Jenkins Master + 1 Jenkins slave (2 executors)... and we need more
In order to augment our build power we have three options:

  1. Scale up: Make AWS instance bigger and add more executors.
  2. Scale up: Make AWS instance bigger and add another jenkins slave process.
  3. Scale out: Create another AWS instance with a jenkins slave and connect it to master

We want to do 2. as we are in a big organization and our current Jenkins Master has already access to every place he needs. Option 3. "New server" is complicated as it needs more bureaucratic approvals that will take weeks.

So my questions are:

  • Is there any technical issues in option 2?. Maybe the executors of each jenkins slave are not aware of the other slave executors?
  • In general, what is the best approach to scale Jenkins? Scaling up or scaling out?
  • You'll have a pitfall, changing an instance type could be problematic if you move to a different hardware type, as your volume would have to be backed up and restored in the new instance. – Tensibai Mar 9 '17 at 10:40
  • 2
    Why not number 3? The usual way to send jobs to Jenkins is to master. And based on certain criterias master will send it seamless to appropriate slave – Romeo Ninov Mar 9 '17 at 13:32
  • FWIW, you need to also analyze your build structure to see how it uses the build machine's resources - scaling up might not help - I encountered cases in which the build time for 2 parallel builds on the same machine was longer than the combined build times of the same 2 builds executed sequentially, non-overlapping. In such case #3 would really be the only practical option available. – Dan Cornilescu Mar 9 '17 at 15:02
  • I agree that #3 is better but I don't have arguments for it or arguements against #1 and #2... – Oscar Foley Mar 9 '17 at 15:53
  • If you have the opportunity within your environment, I would look to an ephemeral solution. Seeing that you're already in AWS, you could easily bring machines up and down as needed while handling the workload. wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin – casey vega May 25 at 15:54
11

There are no fundamental technical issues with running multiple jenkins slaves on the same machine. In fact Running Multiple Slaves on the Same Machine lists several good reasons for doing it:

While the correct use of executors largely obviates the need for multiple slave instances on the same machine, there are some unique use cases to consider:

  • You want more configurability between the configured nodes. Say you have one node set to be used as much as possible, and the other node to be used only when needed.
  • You may have multiple Jenkins master installations building different things, and so this configuration would allow you to have slaves for more than one master on the same box. That's right, with Jenkins you really can serve two masters.
  • You may wish to leverage the easiness of starting/stopping/replacing virtual machines, perhaps in conjunction with Jenkins plugins such as the Libvirt Slaves Plugin.
  • You wish to maximize your hardware investment and utilization, at the same time minimizing operating cost (e.g. utility expenses for running idling slaves).

In general scaling out is preferred, primarily because ability to scale up is typically limited by the types/sizes of the available physical resources.

In particular for augmenting build power I'd recommend an analysis of your actual build to determine how it uses the machine resources, which/where its bottlenecks are and what scalability limitations it raises to reveal if scaling up even helps.

For example I encountered cases in which the build time for 2 parallel builds on the same machine was longer than the combined build times of the same 2 builds executed sequentially (non-overlapping) on the same machine. In such case I wouldn't even consider scaling up as it would actually decrease the overall building capacity.

5

Use Kubernetes and Helm.

I would recommend using the Jenkins Helm chart. It installs with helm install stable/jenkins and automatically scales.

https://github.com/kubernetes/charts/tree/master/stable/jenkins

3

I think you should do neither ;)

Well kinda. I think you need more executors, maybe your builds are really resource intensive? I would run at least 4 but we run 6 to 8 depending on jobs. I like to match # of cores to exectors. So you might want to scale up your nodes, I think we run a M4 large for our 4-8 executors.

I also think you should scale-out but you should do so smartly. Jenkins has a plugin to automatically scale-out on AWS depending on what is in the build queue. Basically you tell it how many jobs and how long the wait before it stands up a slave and sends the jobs to the new slave. You can also set the max amount of slaves, the min amount etc.

2

I'd scale out instead of scale up, going for option 3. We have made a setup where we have all Jenkins agents running on an ECS (custom Docker based Jenkins) with an auto-scaling group. We have all our Jenkins masters communicating to the ECS, thus sharing the workload on the ECS, and no need to recreate the Jenkins master in a scale up exercise.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.