Every jenkins build shows error: ln builds/lastSuccessfulBuild /root/.jenkins/jobs/Build/lastSuccessful failed

Question

I migrate my jenkins to new machine succesfully,but after migrate all jenkins job run succesfully. but every build give following error:

ln builds/lastSuccessfulBuild /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastSuccessful failed
java.nio.file.DirectoryNotEmptyException: /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastSuccessful
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
    at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
    at java.nio.file.Files.deleteIfExists(Files.java:1165)
    at hudson.Util.createSymlink(Util.java:1193)
    at hudson.model.Run.createSymlink(Run.java:1957)
    at hudson.model.Run.updateSymlinks(Run.java:1938)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:286)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)


ln builds/lastStableBuild /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastStable failed
java.nio.file.DirectoryNotEmptyException: /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastStable
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
    at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
    at java.nio.file.Files.deleteIfExists(Files.java:1165)
    at hudson.Util.createSymlink(Util.java:1193)
    at hudson.model.Run.createSymlink(Run.java:1957)
    at hudson.model.Run.updateSymlinks(Run.java:1939)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:286)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429) 

Answer 1

I had a similar issue with Jenkins running under Kubernetes with Azure-storage-backed PersistentVolumes.

Azure storage doesn't support symlinks by default.

As per: https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-linux-file-connection-problems#cannot-create-symbolic-links---ln-failed-to-create-symbolic-link-t-operation-not-supported

To resolve, I added the following to my PersistentVolume yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  storageClassName: azurefile
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  azureFile:
    secretName: azure-secret
    shareName: jenkins
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - mfsymlinks

The mfsymlinks allows symlinks to be created on Azure storage.

Answer 2

1. Log on on the master where you have the issue: ssh jenkins-machine
2. Shutdown the jenkins master (take care that nothing is running) sudo service jenkins stop
3. Find all erroneous directories: find /var/lib/jenkins/jobs -type d \( -name "last*Build" -o -name "lastStable" -o -name "lastSuccessful" \)
4. Review the list of erroneous directories
5. Rename them find /var/lib/jenkins/jobs -type d \( -name "last*Build" -o -name "lastStable" -o -name "lastSuccessful" \) -exec mv {} {}.err \;
6. Restart jenkins: sudo service jenkins start 1 delete these err files if jenkins server is successful.