0

I've been working on an automated patching solution for some infrastructure in one of our AWS accounts. This AWS happens to host the Jenkins instance we use for CI/CD actions, among other things.

Our instance runs off AWS EC2 instances and are persistent - basically they live long enough that they would be out of compliance if we never patched them.

I have patched Jenkins slaves manually in the past, where the workflow looked something like this:

  • Log into Jenkins instance
  • Make sure there are no running jobs on the target node
  • Mark the target node as offline
  • Patch the target node
  • Log into Jenkins instance
  • Mark the node as online again

This process involves interacting with the Jenkins instance itself. Since I want this solution to be automated, I don't want to have to manually log in and facilitate this process before and after patching. Right now, I have the ability to execute arbitrary bash code from the target node locally and I've been working on a way of using the target node to mark itself offline in some way, and right now my best bet looks like using the environment variables that Jenkins sets while it is running a job to determine whether or not the node is busy.

That will solve the problem of the making sure the node isn't busy, but I have no idea how I will prevent it from accepting new jobs while automation patches the machine. I don't want to use API calls because I believe that would require me to put credentials on all of the Jenkins machines, which I am adverse to doing (Secrets management is a pain). Another possible method could be to prevent the Jenkins master from assigning the slave more tasks by shutting off the slave's SSH, but that sounds like a terrible idea.

Anyone have any ideas as to how I could mark Jenkins slaves offline and online, with my only real tool being local bash execution on the slave itself?

Improve this question
2
  • 1
    Have you thought about using ephemeral agents? Doing so typically eliminates the issue you're describing because the nodes are discarded when not in use. Now you can update the EC2 AMI whenever you want without worrying about agents and it also gives you the flexibility to roll out and back as needed.
    – casey vega
    May 20, 2019 at 18:18
  • Hey casey thanks for responding. Unfortunately, I do not have control of our Jenkins instance. We have planned on building a pipeline for Jenkins instances so we can just kill one and reboot from a fresh patched image but that’s too far down the road to not consider patching now. So to answer, yes I have considered it, but no I do not have the ability to enact that change right now.
    – drewto
    May 21, 2019 at 22:58

1 Answer 1

Reset to default
1

If you use the Self-Organizing Swarm Plug-in to connect your build agents to your master, you can spin agents up/down without having to make any changes on your master. Agents will automatically register themselves with the master. You will have to supply credentials to the agents to self-register. I would suggest using SSM Parameter Store to store them and then retrieve them in userdata.

Your agents can then be completely ephemeral, and updates performed in userdata or by updating the AMI.

See this Terraform module for a more detailed example of implementing this.

Improve this answer
2
  • Why use swarm over kubernetes? This plugin is also displaying a vulnerability warning "XXE vulnerability via UDP broadcast response in Swarm client"
    – casey vega
    May 21, 2019 at 2:41
  • The plugin is unrelated to Docker swarm. It can use normal EC2 instances without Docker. AWS does not support broadcast, I wonder if it's not vulnerable in AWS.
    – user2640621
    May 21, 2019 at 4:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.