Force Jenkins to reschedule parallel pipeline step as soon as node goes offline

Question

Exactly this. In our environment, we're using GCE preemptible nodes and the Jenkins GCE plugin. As a consequence of this, it's highly likely (even expected) that a node will go away during the course of a build. The problem is, the Jenkins GCE plugin just marks that node as "offline", but a parallel job "part" that was assigned an executor on that node will just keep churning away there, seemingly forever, apparently in the hope that the node will come back.

The question I have, then, is simple:

Is it possible to force Jenkins to reschedule (NOT fail) a parallel build "part" as soon as the node it's executing on goes offline? If so, how is this done?

Answer 1

There are two approaches to this, neither of them, in my opinion, are especially good as they don't strictly "NOT Fail" the build:

1. Use a plugin such as Naginator which can be configured to restart the build until it passes. This can be quite frustrating as you need to differentiate between ephemeral/preemptable virtual machines being terminated and a truly failed build.

2. Move to an orchestrator model that manages individual jobs and restarts them based upon a set of rules.

Perhaps a better solution would be to develop a Jenkins Plugin similar to the EC2 Fleet Plugin which is responsible for spinning up and maintaining Spot Instances (i.e. similar to GCEs Preemptable Instances) resubmitting jobs when they fail due to instance reclamation.