Looking for help troubleshooting an intermittent issue;

We run an Ubuntu version of Jenkins on AWS infrastructure using on-demand spot ECS instances. I've noticed that while most agents spin down and correctly terminate their ECS counterparts, occasionally an agent will quit without killing its instance, leaving orphaned instances that it could take us weeks to notice.

This problem has been happening on and off for a year, through many different versions of Jenkins. I can't find a common thread between the orphaned instances, and would appreciate any insight on what I can troubleshoot to find the cause.

Can you clarify how you're terminating the instances?

Nodes are automatically terminated x minutes after the job they're running finishes, presumably taking the instance down with them. I'm not clear on the actual particulars; it's just a setting in Jenkins for "Idle termination time".

  • Can you clarify how you're terminating the instances? – MrDuk Mar 27 '18 at 15:19
  • @MrDuk added to question – Alex Mar 27 '18 at 15:24
  • Can you check the jenkins log? Is it showing any errors when trying to execute the terminate/shutdown command? I'm assuming the plugin outputs to a log file somewhere. – PrestonM Mar 27 '18 at 16:11
  • @PrestonM I can't find a specific error right now, but that's a good idea of what to check next time I spot one getting stuck, thanks! – Alex Mar 28 '18 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.