1
| Product    | Version |
|:-----------|--------:|
| EC2 plugin |       X |
| Jenkins    | 2.190.1 |

Looking for help troubleshooting an intermittent issue;

We run an Ubuntu version of Jenkins on AWS infrastructure using on-demand spot ECS instances. I've noticed that while most agents spin down and correctly terminate their ECS counterparts, occasionally an agent will quit without killing its instance, leaving orphaned instances that it could take us weeks to notice.

This problem has been happening on and off for a year, through many different versions of Jenkins. I can't find a common thread between the orphaned instances, and would appreciate any insight on what I can troubleshoot to find the cause.

Can you clarify how you're terminating the instances?

Nodes are automatically terminated x minutes after the job they're running finishes, presumably taking the instance down with them. I'm not clear on the actual particulars; it's just a setting in Jenkins for "Idle termination time".

5
  • 1
    Can you check the jenkins log? Is it showing any errors when trying to execute the terminate/shutdown command? I'm assuming the plugin outputs to a log file somewhere.
    – Preston Martin
    Mar 27 '18 at 16:11
  • @PrestonM I can't find a specific error right now, but that's a good idea of what to check next time I spot one getting stuck, thanks!
    – Alex
    Mar 28 '18 at 13:46
  • @030 never found a solution ¯_(ツ)_/¯ it's still plaguing us maybe 1/500 jobs
    – Alex
    Dec 23 '19 at 14:11
  • Could you check the jenkins master, slave, cloudtrail and cloudwatch logs? Did you check whether there is a related issue on Jenkins' Jira?
    – 030
    Dec 23 '19 at 14:26
  • Could you indicate whether the jenkins master is often restarted if there are still VMs running that were launched by the jenkins master?
    – 030
    Dec 23 '19 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.