1

I have a pipeline that runs on a dedicated bare metal node for automated performance benchmarking. Using bare metal is necessary due to the nature of the project.

Near the end of the pipeline, the results are sent to users (simply seeing a pass or fail status is not sufficient).

To provide each instance of the pipeline a clean slate, the node is rebooted at the end of the pipeline. This makes the pipeline appear stuck in Jenkins. Several minutes after Jenkins is able to reconnect to the node, Jenkins realizes the pipeline that was running ended abruptly, and marks it failed:

+ sudo reboot
Cannot contact baremetal: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@35ca6b21: baremetal": Remote call on baremetal failed. The channel is closing down or has closed down
wrapper script does not seem to be touching the log file in /home/jenkins/workspace/bare-metal-benchmarks@tmp/durable-e96e9611
(JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -1
Setting status of *** to FAILURE with url *** and message: ' '
Using context: Bare Metal Benchmarks
Finished: FAILURE

The next instance of this pipeline can then start running on the rebooted bare metal node.

Is there a way to safely reboot a node at the end of a pipeline so Jenkins does not mark the pipeline status as failed?

The basic structure of the scripted pipeline is:

node('baremetal') {
    checkout scm

    try { 
        stage('Benchmarks') {
            // run benchmarks
        }

        stage('Process data') {
            // process data and send results to users
        }
    }
    finally {
        stage('Clean up') {
            sh 'sudo reboot'
        }
    }
}

2 Answers 2

1

Quick answer:

Changing sudo reboot to sudo shutdown -r +m5 or what ever time delay needed, would prevent the pipeline from failing. It forks the reboot process to the system and lets the Jenkins agent finish this pipeline cleanly.

Longer (preventing potential conflicts answer):

Depending on how the system is configured, it should prevent new logins to the server while the reboot is scheduled.

If the agents are configured to launch via ssh,

launch agents via ssh

and configured to be brought offline if not in use,

take offline when idle

Provided the agent times out, Jenkins should be unable to start a new pipeline on that agent until the reboot completes.

From the man page for shutdown (on one of my servers)

If the time argument is used, 5 minutes before the system goes down the
/run/nologin file is created to ensure that further logins shall not be
allowed.
2
  • Depending on how the system is configured, it should prevent new logins to the server while the reboot is scheduled. Wow, I had no idea shutdown could prevent logins. Manual testing indicates this might work. I'll test this out in CI to confirm.
    – Sagar
    Jul 7 at 3:26
  • unfortunately the agent remains logged in if there is another pipeline already queued. So, the next pipeline starts, and is abruptly killed 5 minutes into the job. Is there a way to force each pipeline to use a new agent?
    – Sagar
    Jul 7 at 21:59
0

Some ideas to get you started:

  • Create a separate job to shut down the node and call that with a build step from your pipeline with "wait: false", e.g.: build(job: "shutdown_host", wait: false). The "shutdown_host" job would probably need to have a sleep statement or something before shutting down the node to avoid killing the node before the parent job has a chance to exit.

  • Execute a short shell script that forks & detaches a new process to shut down the host. Again you would probably need a sleep statement in the forked process to avoid shutting down the node before the Jenkins job exits. Also, in this case, you would need to disable the Process Tree Killer to avoid the forked process being killed.

  • Execute a step on the Jenkins master to ssh into your node to shut it down. But this would depend on the Jenkins master having ssh access to your node and administrative privileges to shut it down.

I'm sure there are some other clever and creative ideas that I'm missing as well.

2
  • Isn't there a race in all of these solutions? Suppose one job is executing, and a second is queued. If the second job starts while the first is sleeping in the background, then won't the second job be disrupted due to the reboot?
    – Sagar
    Jun 9 at 5:25
  • Yes, there will be. You can check in your pipeline code if there are any jobs running on a node (there should be other Stack Exchange questions that address how to do this) before rebooting, or you can use resource locks (which are kind of like semaphores) to prevent race conditions.
    – jayhendren
    Jun 9 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.