4

Would like to ask for recommendations on ready automated solutions or templates for rolling ec2 instance upgrade on AWS, requiring a reboot with ELB/ALB. Specifically required to perform the maintenance upgrade for the kernel on hundreds of Linux machines.

For each ELB/ALB steps are:

  1. Disconnect ec2 instance from ELB/ALB wait for connection draining
  2. Upgrade kernel and reboot machine (not part of automation, e.g. running external ./upgrade.sh)
  3. Wait for instance to be online, connect to ELB/ALB, wait for health check status
  4. Continue to next instance in the load balancer

Note: not looking for migration to the solution similar to Netflix Spinnaker or immutable deployment strategy implementation, but for general and simple on-demand maintenance tool.

  • 2
    Is doing a +1 deploy possible for you? So instead of removing an instance from the ALB, you add an already upgraded instance? This would save a bunch of time from the two waiting states you listed. – MrDuk Mar 27 '18 at 15:21
  • Thank you @MrDuk for the suggestion, this is what was mentioned as "immutable deployment" which will require currently more resources and coordination for deploying tens of different applications on hundreds of servers with different configurations, ideally this would be possible with better deployment automation, but it's not the case currently – rombob Mar 27 '18 at 18:01
  • If each node is distinct, with custom changes to each, I don't know that a tool exists that can help you with this. However, you could script this fairly easily using the aws-cli, the most complex part being the upgrade/reboot/detecting – MrDuk Mar 27 '18 at 18:13
  • @rombob I have to agree with MrDuk. If your instances have the capability to run different versions in parallel you should be able to use an immutable deployment strategy with no more effort than the steps you proposed. – PrestonM Mar 27 '18 at 18:16
  • @MrDuk agreed, aws-cli or aws-sdk is what make sense to use, wondering if there ready solutions, other than custom upgrade, sequential/parallel traversing of balanced instances seems like very common task before we jump in writing our own, worth checking with community – rombob Mar 27 '18 at 18:22
2

AWS have a specific service to deal with this type of use case, EC2 Systems Manager

This allows making base patch sets and then checking they are applied to target systems.

"Patch Manager is a fully-managed service (provided at no additional cost) that helps you simplify your operating system patching process, including defining the patches you want to approve for deployment, the method of patch deployment, the timing for patch roll-outs, and determining patch compliance status across your entire fleet of instances. It’s extremely configurable with some sensible defaults and helps you easily deal with patching heterogeneous clusters."

-- "Amazon EC2 Systems Manager Patch Manager now supports Linux"

You'll need to install an SSM agent on each system and add a role but other than that it should be unobtrusive

  • looks good but this solution is not including ELB/ALB disconnect/connect orchestration while applying patches – rombob Apr 8 '18 at 4:33
  • The AWS solution to your problem is to use an immutable solution: build AMI with the new kernel and then use ASG to cycle them out of ALB but I know you don't want this!! – Vorsprung Apr 8 '18 at 18:05
  • 1
    If I had to solve this problem I'd be less concerned about rotating them out before the update. Apart from increased IO a kernel update will have no effect until the reboot. You could have a init.d (or upstart or systemd or whatever) script to make a controlled exit from the ALB group on a controlled rebooted – Vorsprung Apr 8 '18 at 19:09
1

As people in the comments are saying, you can do this with aws-cli in bash or in the python boto3 library.

What about ansible? It has the ability to see your ec2 instances and it could handle the automation for maintenance tasks. If you are not using ansible I highly recommend you add that tool to your DevOps toolbox.

Here is how you might structure your playbook to do what you are wanting.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.