8

In order to test that our load balancers and Multi-AZ RDS instances are working the way I'd expect them to, I'd like to simulate failure of an AWS Availability Zone.

Short of locating one of the datacenters and setting fire to it (which probably wouldn't be successful anyway), how can I do this? (I'm aware of Netflix' Chaos Gorilla, but it doesn't seem to have been released to the public like Chaos Monkey has, and I'm not sure what techniques it uses to cause its chaos).

Improve this question

2 Answers 2

Reset to default
5

This (slightly old in cloud years) question on Hacker News provides a few suggestions:

  • Remove the security groups from all instances in one zone, which should technically make them all appear 'down'
  • Adjust the VPC route tables to black hole the subnets in one availability zone
  • Disable an availability zone from the load balancer (docs)

You could also probably adjust the Network ACL attached to the relevant subnets to deny all traffic.

I'm not aware of an existing, available tool that does any of this automatically, though.

Improve this answer
3
  • 1
    Also you can change the network ACL for a subnet. That can effectively sever all traffic between the AZs.
    – myron-semack
    Commented Jun 19, 2018 at 23:19
  • 1
    I did mention that ;) (although, not in the bullet points ha!)
    – Tim Malone
    Commented Jun 20, 2018 at 0:14
  • 1
    Since this Q&A was written a new service (Fault Injection Simulator) has become available, and recently it got support for simulating AZ failure: aws.amazon.com/about-aws/whats-new/2022/10/…
    – Tim Malone
    Commented Oct 31, 2022 at 11:44
4

This lab from AWS Well-Architected will show you how to implement an AWS AZ failure simulation using Bash, Java, Python, C#, or PowerShell

https://wellarchitectedlabs.com/reliability/300_labs/300_testing_for_resiliency_of_ec2_rds_and_s3/

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.