TL;DR: Chaos Monkey was developed in 2010 at Netflix and released into wild in 2012 is part of the Simian Army, wildly popular among devoted followers. Built on principles of chaos engineering, the army increases resiliency to failure by injecting constant failure to the system.
Concept
Chaos Monkey was developed specifically for AWS where it will randomly kill instances within an Auto Scaling Group. It is meant to run during the business hours when engineers are alert and can quickly react to discovered failures.
Simian Army
Members of the army would sow chaos through other means:
Other Monkeys are helpful and remove the weak members of the herd:
Conformity Monkey shuts down instances not following best practices.
Security Monkey looks for known security vulnerabilities in configuration and services.
Doctor Monkey shuts down unhealthy instances not conforming to certain metrics.
Janitor Monkey looks for unused resources to reclaim.
Failure is Inevitable
Failure in the System is inevitable, something will always go wrong. You might not be able to choose what, but you can try to choose when. By introducing small errors throughout the day, you ensure that your engineers are present. By killing non-conforming services quickly, you ensure that failures happen often before deployment. By making the environment more adversarial, you ensure that it will be the developers who run into issues long before any service makes its way into production. Failures will be quickly apparent in integration phase of new services with the old ones, but that is ok, because the old production services are already resilient.
Cattle not Pets
Everyone will tell you lately: Do not treat your servers as pets. There is a power in numbers and any single point of failure will bring down the system. No matter how well you can tune and optimize your server, no matter how beefy hardware you can get, how much it can handle, it will never be a match for herd of small scalable instances. Chaos Monkey encourages you to think about removing all points of failure, because sooner or later, the Monkey will come! Everyone fails and even the Amazon S3 had an unpredictable outage.
Anti-Fragile
So what is the theory and why does it work? Nassim Nicholas Taleb in his book Antifragile describes a concept where living self-aware systems, will benefit from a small levels of randomness and actually become better in face of adversity. This is similar to annealing.
He does also describe an evolutionary way, where fragility of parts in a system is transferring into antifragility of the whole. The transfer occurs on two levels:
By a small random variations - developers making changes - the most fit for the environment will survive and propagate - pass tests and get deployed. Standard Development Life Cycle.
By failure of parts not capable to withstand a larger level of randomness in the environment, the remaining parts that were able to withstand it compose a system that is as a whole better able to deal with changing environment than before. This is essentially Chaos Monkey.
Larger levels of randomness can be withstood using the second approach.
