1

Back in the day this Q&A was created.

ChaosMonkey is a tool that is able to stop random services on a platform in a certain time period. The goal of this is to get a stable platform that is fault tolerant. If this happens during the day when most of the Engineers are working, they get proactive and the infrastructure will be hardened.

At the moment, more and more services are created in AWS, GCP and Azure. The services on these platforms are getting more and more critical. Now it is time to onboard more and more engineers. Especially most of the developers are still silo minded, i.e. some of them only want to develop, but this is shifting drop by drop.

According to Wikipedia, ChaosMonkey was invented at Netflix and additional resilience tools were created as well:

| Purpose                                        | Tool              |
|:----------------------------------------------:|:-----------------:|
| Stops random services in production randomly   | Chaos Monkey      |
| Drop an availability zone, e.g. in AWS         | Chaos Gorilla     |
| Communication delays                           | Latency Monkey    |
| Fix unhealthy services                         | Doctor Monkey     |
| Dispose unused resources                       | Janitor Monkey    |
| Detect nonconforming instance and notify owner | Conformity Monkey |
| Disable instances that are vulnerable          | Security Monkey   |
| Detect problems between 10-18                  | 10-18 Monkey      |

Based on this table one could also achieve also the same purpose by start to write some shell scripts that stop random services for example.

Based on the information that was read, the definition of resilience-testing is basically testing if an app and underlying infrastructure is error prone in production.

This is also confirmed by this blog post that refers to how resilience testing is done at AWS, but also describes IBMs view on this type of testing. According to IBM there are two parts that have to be taken into account, i.e. problem impact and service level. In an ideal world the customer will not notice any downtime at all, but if a machine dies (minutes) or a complete data center (hours) then the customer could experience downtime. The most important thing to be transparently as possible to the customer and to minimize impact.

Currently, resilience-testing is not used in the company, but the more is read about it, it becomes interesting. Especially as it is also aligned with DevOps and the company tries to make Engineers DevOps minded. It looks like that resilience-testing promotes a DevOps mindset.

Resilience testing is DevOps related

Some people could argue that testing is not part of DevOps, but according to Wikipedia, tools like ChaosMonkey are aligned with the DevOps toolchain.

4
  • You seem to be answering yourself here.. I fail to see what you're asking about out of the question title, and the rest of the question seems to self answer
    – Tensibai
    Apr 27, 2018 at 8:38
  • @Tensibai The more I think about it. I basically would like to know how other users implemented resilience testing and what the effects are since they have implemented. On one hand I think it is an opinionated question, but on the other hand it could provide new insights. Will it be fine to ask this? If not, what could be done to change this question a bit so that it is acceptable to ask?
    – 030
    Apr 27, 2018 at 8:43
  • Well, you may add this in the question itself, actually out of the title there's no question at all. I don't find it that much opinion based, but maybe it will turn into a poll list. I don't have a strong opinion for or against this question ATEOTD
    – Tensibai
    Apr 27, 2018 at 9:18
  • @Tensibai Thank you for the feedback. I have changed the title.
    – 030
    Apr 27, 2018 at 9:24

1 Answer 1

0

I think that chaos testing will promote DevOps unless people have been informed in advance and proper alert management has been enabled. If people do not get triggered then this will not work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.