1

I have my Jenkins build configured to "Clean before Checkout" for its git source:

Job Config > Source > git > Behaviours > Clean before checkout

This results in

Resetting working tree

git reset --hard # timeout=10
git clean -fdx # timeout=10

Before the job begins.

This errors out with

stderr: warning: failed to remove web/static/node_modules/.../fonts/google-fonts: Directory not empty

If I re-run the job (simply replay), I get the same error.

If I re-run it again, git clean works fine and the job finishes with success in the end.

If I start it again it errors out 2 times again until it works at the third time. This is always the case.

What might be going on here?

Improve this question
6
  • 2
    Are by any chance builds recently executed or aborted on that node around the time these events happen? Check for any leftover processes from previous builds (or other processes) still running on that node when git clean executes, they may keep file descriptors open or create new files in the directory tree being removed, preventing removal. Even a regular shell with the cwd set somewhere within that tree can cause that.
    – Dan Cornilescu
    Commented Nov 2, 2019 at 16:54
  • 1
    Could you try to remove the workspace and start the build again?
    – 030
    Commented Nov 3, 2019 at 12:02
  • @DanCornilescu That is good input. No processes are running (checking by htop and ps aux), but I can try to restart the Container on which Jenkins is running after each job run to be absolutely certain.
    – Worp
    Commented Nov 4, 2019 at 6:54
  • 1
    @Worp Nice to hear that it works. Yes please post an answer and let me know once you created it.
    – 030
    Commented Nov 4, 2019 at 11:56
  • 1
    @Worp could you post an answer?
    – 030
    Commented Dec 20, 2019 at 11:33

2 Answers 2

Reset to default
0

We didn't find the cause, neither a real solution.

There is a workaround though:

Put a pipeline stage with rm -rf * before your scm checkout stage. This solves the problem but I will change the accepted answer to whoever posts the real reason and fix for the problem.

Sidenote:

Keep in mind that you will erase anything in your Jenkins workspace doing that. So be certain that you have everything you need in Git so it comes back with the checkout or get it back some other way.

If you rely on files in your Jenkins workspace from previous runs then this method is not for you!

(Although you might want to think about adapting your Job so files from previous runs are not needed anymore. Every run should contain the files it needs by itself and not rely on other runs.)

Improve this answer
1
  • I've had this happen when doing non-containerized Habitat package builds. In my case, the habitat studio aborted a build, but hadn't unmounted its bind mounts of, e g. /dev, /proc and friends. I discovered the hard way that a sledgehammer rm -rf was a quick way to trash the node. My solution was twofold: first, mount /var/lib/jenkins as tmpfs (so rebooting could cleanly fix the problem on a per-incident basis), and, later, switch to fully-containerized builds to ensure proper cleanup.
    – Michael Mol
    Commented Jan 3, 2020 at 17:10
0

I think this might just be a permissions issue. You can test this by manually running git clean -fdx as the user running Jenkins. If it doesn't work, try it again with sudo git clean -fdx. If it works, then it's probably an issue with permissions or ownership on one or more files/directories within the repo.

As an alternative, you can install the Workspace Cleanup plugin, which provides the cleanWs() function which is used to clean the workspace directory. There are different options, so consult the documentation to see what might work best for your environment.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.