8

Exactly this. In our environment, we're using GCE preemptible nodes and the Jenkins GCE plugin. As a consequence of this, it's highly likely (even expected) that a node will go away during the course of a build. The problem is, the Jenkins GCE plugin just marks that node as "offline", but a parallel job "part" that was assigned an executor on that node will just keep churning away there, seemingly forever, apparently in the hope that the node will come back.

The question I have, then, is simple:

Is it possible to force Jenkins to reschedule (NOT fail) a parallel build "part" as soon as the node it's executing on goes offline? If so, how is this done?

0

There are two approaches to this, neither of them, in my opinion, are especially good as they don't strictly "NOT Fail" the build:

  1. Use a plugin such as Naginator which can be configured to restart the build until it passes. This can be quite frustrating as you need to differentiate between ephemeral/preemptable virtual machines being terminated and a truly failed build.

  2. Move to an orchestrator model that manages individual jobs and restarts them based upon a set of rules.

Perhaps a better solution would be to develop a Jenkins Plugin similar to the EC2 Fleet Plugin which is responsible for spinning up and maintaining Spot Instances (i.e. similar to GCEs Preemptable Instances) resubmitting jobs when they fail due to instance reclamation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.