5

Would like to ask for recommendations on ready automated solutions or templates for rolling ec2 instance upgrade on AWS, requiring a reboot with ELB/ALB. Specifically required to perform the maintenance upgrade for the kernel on hundreds of Linux machines.

For each ELB/ALB steps are:

  1. Disconnect ec2 instance from ELB/ALB wait for connection draining
  2. Upgrade kernel and reboot machine (not part of automation, e.g. running external ./upgrade.sh)
  3. Wait for instance to be online, connect to ELB/ALB, wait for health check status
  4. Continue to next instance in the load balancer

Note: not looking for migration to the solution similar to Netflix Spinnaker or immutable deployment strategy implementation, but for general and simple on-demand maintenance tool.

Improve this question
9
  • 2
    Is doing a +1 deploy possible for you? So instead of removing an instance from the ALB, you add an already upgraded instance? This would save a bunch of time from the two waiting states you listed.
    – MrDuk
    Mar 27, 2018 at 15:21
  • Thank you @MrDuk for the suggestion, this is what was mentioned as "immutable deployment" which will require currently more resources and coordination for deploying tens of different applications on hundreds of servers with different configurations, ideally this would be possible with better deployment automation, but it's not the case currently
    – rombob
    Mar 27, 2018 at 18:01
  • If each node is distinct, with custom changes to each, I don't know that a tool exists that can help you with this. However, you could script this fairly easily using the aws-cli, the most complex part being the upgrade/reboot/detecting
    – MrDuk
    Mar 27, 2018 at 18:13
  • @rombob I have to agree with MrDuk. If your instances have the capability to run different versions in parallel you should be able to use an immutable deployment strategy with no more effort than the steps you proposed.
    – Preston Martin ♦
    Mar 27, 2018 at 18:16
  • @MrDuk agreed, aws-cli or aws-sdk is what make sense to use, wondering if there ready solutions, other than custom upgrade, sequential/parallel traversing of balanced instances seems like very common task before we jump in writing our own, worth checking with community
    – rombob
    Mar 27, 2018 at 18:22

2 Answers 2

Reset to default
2

AWS have a specific service to deal with this type of use case, EC2 Systems Manager

This allows making base patch sets and then checking they are applied to target systems.

"Patch Manager is a fully-managed service (provided at no additional cost) that helps you simplify your operating system patching process, including defining the patches you want to approve for deployment, the method of patch deployment, the timing for patch roll-outs, and determining patch compliance status across your entire fleet of instances. It’s extremely configurable with some sensible defaults and helps you easily deal with patching heterogeneous clusters."

-- "Amazon EC2 Systems Manager Patch Manager now supports Linux"

You'll need to install an SSM agent on each system and add a role but other than that it should be unobtrusive

Improve this answer
3
  • looks good but this solution is not including ELB/ALB disconnect/connect orchestration while applying patches
    – rombob
    Apr 8, 2018 at 4:33
  • The AWS solution to your problem is to use an immutable solution: build AMI with the new kernel and then use ASG to cycle them out of ALB but I know you don't want this!!
    – Vorsprung
    Apr 8, 2018 at 18:05
  • 1
    If I had to solve this problem I'd be less concerned about rotating them out before the update. Apart from increased IO a kernel update will have no effect until the reboot. You could have a init.d (or upstart or systemd or whatever) script to make a controlled exit from the ALB group on a controlled rebooted
    – Vorsprung
    Apr 8, 2018 at 19:09
1

As people in the comments are saying, you can do this with aws-cli in bash or in the python boto3 library.

What about ansible? It has the ability to see your ec2 instances and it could handle the automation for maintenance tasks. If you are not using ansible I highly recommend you add that tool to your DevOps toolbox.

Here is how you might structure your playbook to do what you are wanting.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.