Looking for help troubleshooting an intermittent issue;
We run an Ubuntu version of Jenkins on AWS infrastructure using on-demand spot ECS instances. I've noticed that while most agents spin down and correctly terminate their ECS counterparts, occasionally an agent will quit without killing its instance, leaving orphaned instances that it could take us weeks to notice.
This problem has been happening on and off for a year, through many different versions of Jenkins. I can't find a common thread between the orphaned instances, and would appreciate any insight on what I can troubleshoot to find the cause.
Can you clarify how you're terminating the instances?
Nodes are automatically terminated x minutes after the job they're running finishes, presumably taking the instance down with them. I'm not clear on the actual particulars; it's just a setting in Jenkins for "Idle termination time".