Rebooting node at end of Jenkins pipeline

Question

I have a pipeline that runs on a dedicated bare metal node for automated performance benchmarking. Using bare metal is necessary due to the nature of the project.

Near the end of the pipeline, the results are sent to users (simply seeing a pass or fail status is not sufficient).

To provide each instance of the pipeline a clean slate, the node is rebooted at the end of the pipeline. This makes the pipeline appear stuck in Jenkins. Several minutes after Jenkins is able to reconnect to the node, Jenkins realizes the pipeline that was running ended abruptly, and marks it failed:

+ sudo reboot
Cannot contact baremetal: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@35ca6b21: baremetal": Remote call on baremetal failed. The channel is closing down or has closed down
wrapper script does not seem to be touching the log file in /home/jenkins/workspace/bare-metal-benchmarks@tmp/durable-e96e9611
(JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -1
Setting status of *** to FAILURE with url *** and message: ' '
Using context: Bare Metal Benchmarks
Finished: FAILURE

The next instance of this pipeline can then start running on the rebooted bare metal node.

Is there a way to safely reboot a node at the end of a pipeline so Jenkins does not mark the pipeline status as failed?

The basic structure of the scripted pipeline is:

node('baremetal') {
    checkout scm

    try { 
        stage('Benchmarks') {
            // run benchmarks
        }

        stage('Process data') {
            // process data and send results to users
        }
    }
    finally {
        stage('Clean up') {
            sh 'sudo reboot'
        }
    }
}

Answer 1

Quick answer:

Changing sudo reboot to sudo shutdown -r +m5 or what ever time delay needed, would prevent the pipeline from failing. It forks the reboot process to the system and lets the Jenkins agent finish this pipeline cleanly.

Longer (preventing potential conflicts answer):

Depending on how the system is configured, it should prevent new logins to the server while the reboot is scheduled.

If the agents are configured to launch via ssh,

and configured to be brought offline if not in use,

Provided the agent times out, Jenkins should be unable to start a new pipeline on that agent until the reboot completes.

From the man page for shutdown (on one of my servers)

If the time argument is used, 5 minutes before the system goes down the
/run/nologin file is created to ensure that further logins shall not be
allowed.

Answer 2

Some ideas to get you started:

• Create a separate job to shut down the node and call that with a build step from your pipeline with "wait: false", e.g.: build(job: "shutdown_host", wait: false). The "shutdown_host" job would probably need to have a sleep statement or something before shutting down the node to avoid killing the node before the parent job has a chance to exit.

• Execute a short shell script that forks & detaches a new process to shut down the host. Again you would probably need a sleep statement in the forked process to avoid shutting down the node before the Jenkins job exits. Also, in this case, you would need to disable the Process Tree Killer to avoid the forked process being killed.

• Execute a step on the Jenkins master to ssh into your node to shut it down. But this would depend on the Jenkins master having ssh access to your node and administrative privileges to shut it down.

I'm sure there are some other clever and creative ideas that I'm missing as well.