2

I recently had an issue with an EC2 instance. Briefly, the site was unavailable for 2 hours, a peak of CPU usage for no apparent reason, network traffic dropped close to 0, docker containers restarted, issues with the clock. The fault might have been not on their side, but...

Do you expect that EC2 instances might occasionally act up? After all, they seem to sort of guarantee no more than 0.5% downtime (3.6 hours per month?). Which means there might be downtime.

What do you do to avoid that? In my case I don't even know if that counts towards downtime, because technically the server was running, but network traffic was close to 0. And I'm not sure, but it seems their health/status checks didn't notice anything unusual (how do I check?)

UPD Most likely the physical server where it was running failed and the instance migrated to another server. Because there's the following line in the log:

Jan 12 21:31:29 ip-172-xx-x-xx.xx-yy-z.compute.internal chronyd[24287]: Forward time jump detected!

Improve this question

1 Answer 1

Reset to default
2

Short answer: Yes, you should expect any IaaS, PaaS, SaaS platform to have issues from time to time.

There's no bullet-proof system. Sometimes the vendor has issues with the underlying hardware that can affect your server/container. My first step if I think AWS is having a hardware/virtualization issue is stop and start the instance. This moves your instance to a different AWS server. You should rarely need to do this though.

What you can do is monitor and alert on metrics that would indicate your service/server is not responding. In your example, Network traffic (bytes in) below a certain threshold for a certain period of time could indicate an issue.

To further protect the availability of your site you can (depending on your use case) create a second (or multiple) instance and use an application or network load balancer to check host health and send traffic accordingly. (ALB/NLB).

Bottom line, you should always expect issues (whether using a cloud provider or your own hardware) and build monitors to alert when an issue may be present and take appropriate actions.

References
AWS High Availability White Paper

Improve this answer
2
  • Doesn't that mean that generally I should start with at least 2 EC2 instances?
    – x-yuri
    Commented Feb 7, 2022 at 20:28
  • 1
    @x-yuri Yes, depending on your use case. I've added use of LB's into the answer and linked an AWS white paper on HA.
    – kenlukas
    Commented Feb 8, 2022 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.