35

I had a perfect server, it was so pretty and rock solid and so I named it Petra. It was perfect in every way, everything was configured and tuned just right, it had perfect 100% service record and 753 days of uptime. I've spent a lot of time and effort making sure it run so well. No other server in the company had been this good. But last night this evil monster crashed my server for no reason.

Chaos Monkey

Of course I was notified at 2am and it took me until morning to get it up and running and everything configured and tuned up, but I'm afraid it is not going to be as good as before. It might take weeks before it is back to it's former glory. Now my uptime is gone, I don't have even measly three 9s and who knows what this will do to my reputation. Who is this Chaos Monkey and why did he do that to my server and why is he trying to ruin me?

6
  • 13
    There needs to be a badge for funniest question :) Mar 21, 2017 at 8:06
  • Single server? What's that? Why would you base your business on a unique work of art instead of a commodity that is easily scaled and replaced when it inevitably fails or goes off lease? Mar 26, 2017 at 21:18
  • Do we really consider this a good question to pre-seed the site with? Are we expecting engineers to install, configure, and run Chaos Monkey, then forget what it is, but discover that it's running on their network and ask on Stack Exchange rather than visiting the official website? There are so many steps there that are implausible. Apr 2, 2017 at 4:40
  • 2
    Obviously, it is not a real question. It was a joke to sort of portray the top of the field sysadmin of old, who's simply been outpaced by the industry. But your criticism assumes that there is a single person or even a single team taking care of the infrastructure. Quite often in large companies the existing infrastructure and teams are left in place even while they are being replaced. It is quite plausible that a new team started with new infrastructure and installation of Chaos Monkey and the old fart simply got hit at a time when it was after initial success deployed company wide. Apr 2, 2017 at 5:55
  • The obvious question is if you had such a server then why did you set up Chaos Monkey?
    – user253751
    Apr 15, 2017 at 11:54

3 Answers 3

39

TL;DR: Chaos Monkey was developed in 2010 at Netflix and released into wild in 2012 is part of the Simian Army, wildly popular among devoted followers. Built on principles of chaos engineering, the army increases resiliency to failure by injecting constant failure to the system.

Concept

Chaos Monkey was developed specifically for AWS where it will randomly kill instances within an Auto Scaling Group. It is meant to run during the business hours when engineers are alert and can quickly react to discovered failures.

Simian Army

Members of the army would sow chaos through other means:

  • Latency Monkey will introduce random delays to services.

  • Chaos Gorilla (Kong) will simulate outage of entire availability zone.

Other Monkeys are helpful and remove the weak members of the herd:

  • Conformity Monkey shuts down instances not following best practices.

  • Security Monkey looks for known security vulnerabilities in configuration and services.

  • Doctor Monkey shuts down unhealthy instances not conforming to certain metrics.

  • Janitor Monkey looks for unused resources to reclaim.

Failure is Inevitable

Failure in the System is inevitable, something will always go wrong. You might not be able to choose what, but you can try to choose when. By introducing small errors throughout the day, you ensure that your engineers are present. By killing non-conforming services quickly, you ensure that failures happen often before deployment. By making the environment more adversarial, you ensure that it will be the developers who run into issues long before any service makes its way into production. Failures will be quickly apparent in integration phase of new services with the old ones, but that is ok, because the old production services are already resilient.

Cattle not Pets

Everyone will tell you lately: Do not treat your servers as pets. There is a power in numbers and any single point of failure will bring down the system. No matter how well you can tune and optimize your server, no matter how beefy hardware you can get, how much it can handle, it will never be a match for herd of small scalable instances. Chaos Monkey encourages you to think about removing all points of failure, because sooner or later, the Monkey will come! Everyone fails and even the Amazon S3 had an unpredictable outage.

Anti-Fragile

So what is the theory and why does it work? Nassim Nicholas Taleb in his book Antifragile describes a concept where living self-aware systems, will benefit from a small levels of randomness and actually become better in face of adversity. This is similar to annealing.

He does also describe an evolutionary way, where fragility of parts in a system is transferring into antifragility of the whole. The transfer occurs on two levels:

  1. By a small random variations - developers making changes - the most fit for the environment will survive and propagate - pass tests and get deployed. Standard Development Life Cycle.

  2. By failure of parts not capable to withstand a larger level of randomness in the environment, the remaining parts that were able to withstand it compose a system that is as a whole better able to deal with changing environment than before. This is essentially Chaos Monkey.

Larger levels of randomness can be withstood using the second approach.

2
  • "Failure is Inevitable" - great mantram!
    – wogsland
    Mar 22, 2017 at 21:43
  • Upvoted because you mentioned Nassim Taleb. Super smart guy, and his ideas can be applied to basically anything.
    – maplebird
    Aug 1, 2017 at 21:11
9

Some additions to your own answer to this question ...

Additional monkeys

The article about "How chaos boosts performance" describes a few more of these monkeys, i.e.:

  • 10-18 Monkey: finds configuration and run time problems in instances that serve customers in multiple regions.
  • Chaos Kong: simulates an outage of an Amazon region.

Remark: The same article also mentions "Chaos Gorilla: simulates an outage of an Amazon availability zone", though it could well be that this has been renamed now to "Chaos Kong: simulates an outage of an Amazon region" ... Talking about Chaos! I couldn't find any confirmation/docu on that so far, at least there does not seem to be an issue for it in the issue queue. An undocumented change might have made it to production on github ... Gggggggrrrrrreat!

Setup and use your own Monkeys.

Head over to github to get in touch with the Simian Army (same link as the very first link in your own answer). Here is a quote of what you'll find there:

Simian Army consists of services (Monkeys) in the cloud for generating various kinds of failures, detecting abnormal conditions, and testing our ability to survive them. The goal is to keep our cloud safe, secure, and highly available. More details can be found at this blog.

Currently the simians include Chaos Monkey, Janitor Monkey, and Conformity Monkey.

Refer to the Quick start guide to get started setting up and using the Monkeys.

You can even configure the monkyes, so that they fit your business needs.

If you dig deep enough within those Github links (i.e within the Support link), you'll also find a link to join the SimianArmy Google Group.

2
  • Chaos Kong was renamed to Chaos Gorilla, I think or the other way around. Mar 23, 2017 at 15:54
  • @JiriKlouda you seem to confirm what I was starting to wonder about. Which is why I'd added my remark in my answer now also.
    – Pierre.Vriens
    Mar 23, 2017 at 16:45
3

One Server to rule them all, One Server to find them,
One Server to bring them all and in the outage bind them

You, Sauron, forged this One Server, in the Darkness of Mount Doom your Datacenter in a desire to rule All applications.
Hopefully the Fellowship of Devops did unite to tell you:

Gandalf - You shall PAAS

After a long fight, Frodo the Chaos Monkey has been able to melt your One Server and bring freedom to all applications, driving you to the way of reproducible Servers at the same time.

Credits:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.