Jenkins managing a pool of servers across restarts

Question

I have a pool of servers to test a provisioning process my team owns. One server is needed per branch or PR. The process requires reboots. Once the process is run, we run some tests. A Jenkins pipeline automates the provisioning process and test.

So my problem is how to manage the pool of servers so that

1. Any available pool server can be selected for a build
2. A server claimed for a build is unavailable for other builds
3. Upon successful build, the server is made available again
4. Failed builds are not made available so they retain the failed state for investigation (with manual return to the pool after)

I can't just use the executor on the slave node because we reboot the server and that would kill the job.

My current solution is:

1. Pool servers are setup as Jenkins slaves with one executor each and a common label
2. A pipeline stage is assigned to the common label, choosing one of the slaves with a free executor--this stage moves slave.jar to protect against Jenkins restarting the assigned slave (which it annoyingly will do)
3. The next pipeline stage uses node.getComputer().disconnect() to disable the slave--it seems this must run on master so must be in a different stage
4. A post { success { ... } } uses ssh to move slave.jar back into place on the server and node.getComputer().connect(true) to allow it to be picked again

This mostly works and is kinda cool, but it adds a ton of noise to the pipeline and worse, suffers from a race condition between steps 2 & 3. When jobs queue up for the pool, a new one can be assigned to the same node just before a prior jobs disconnects the slave. When this happens, things break badly.

Does anyone else use Jenkins to manage a pool of servers where you need to support reboots in the pool and how do you do it?

Any ideas for ways to manage the pool outside Jenkins, but somehow get the assigned node into Jenkins for the test run? Plugins I've missed, etc.

I've examined the few results google returns for the slave disconnect problem and the most promising one was https://groups.google.com/forum/#!topic/jenkinsci-dev/ch2lQZvZdkw -- the suggestion there failed for me because OfflineCause.UserCause class could not be resolved in my pipeline. I only know enough Java and Groovy to be dangerous!

Answer 1

I found the missing piece in the Lockable resource plugin.

lock(label: 'server-pool', quantity: 1) {
   ...
}

Now the servers to be used for testing don't need to be managed as slaves at all and that greatly simplifies things.

Answer 2

To me, but correct me if I am wrong, the Jenkins servers should not be rebooted, but they should create VMs and reboot them. When I read the question I get the impression the slaves are rebooted, i.e. I can't just use the executor on the slave node because we reboot the server and that would kill the job. To me it seems that the jenkins slave should create a VM, run the tests, reboot the system, run some additional tests and finally remove the VMs.

From a production perspective, but again correct me if I am wrong the team is not using Jenkins to provision the systems and subsequently reboot the nodes? It could be possible that they use Jenkins to start the provision step, but they will not reboot the Jenkins systems right?

I would recommend to start with the Production perspective, e.g. How does the team provision the systems at the moment? E.g. Chef? If that is the case, how to they run that? If they would run chef apply then this could be run by a Jenkins slave, but I would not reboot the jenkins slave but automate the process of creating a VM, provision it, reboot it and kill it. The latter is only applicable for testing purposes.