How can I convince someone that 100% reliability is not the right target for anything?

Question

It's a fundamental principle of DevOps and SRE that failure is normal and setting a goal of perfect reliability is misguided. But sometimes IT execs and business leaders push back on the idea. They believe the business needs 100% reliability. What are some good way to persuade them to change their thinking?

Answer 1

I was wondering whether anyone in this forum had new arguments they have heard or tried in person to convince executives of the need to set service level objectives at a level below 100%, hence the original question.

Here are some approaches I've seen or taken in the past to tackling this problem. The Site Reliability Workbook has a few suggestions here, which include:

• 100% is not a reasonable target since every component of any system has a non-zero probability of failure.
• External components (such as an ISP) sitting between the customer and the target system aren't 100% reliable.
• 100% reliability implies you can never change the system, since all change introduces risk.
• 100% reliability implies you spend all your time reacting to reliability problems and have no time for anything else.

In addition to the above, I've also gathered the following talking points from various in-person discussions that might be useful to persuade organizational leaders:

• If an IT exec sets a target of 100% reliability, they are basically encouraging their engineers to lie to them and hide problems, which means they will discover those problems in the most expensive possible way.
• Nature has never evolved a 100% reliable system. Gene replication is unreliable. The human heart is not 100% reliable. Life continues and thrives in the face of imperfect reliability, and so can technology systems and their users.
• No group of humans has ever engineered a system that's 100% reliable in the long run.

We also have to acknowledge that for certain dimensions of reliability, such as data loss prevention, some regulated industries require levels of reliability so high and over long periods of time that they are virtually indistinguishable from 100%, like 99.999999%. For other dimensions like availability and latency, three or four nines of reliability are often more than adequate, and some of the arguments above may help persuade IT leadership that this is true.

Answer 2

If your company/manager/etc is arguing for 100% reliability, please ask them to name just one human-engineered system that has achieved that mark over any non-trivial period of time. I'm not aware of any -- and I've looked a lot. Pacemakers don't even have 100% reliability.

Remove the humans. You'd be hard pressed to find anything in nature that achieved that goal, either.

A goal of perfection is deeply counterproductive and will eventually slow your innovation asymptotically towards 0.

Here's a fuller explanation.

You will ship much greater reliability to your customers if you concentrate less on reducing errors, and focus more on detecting them and mitigating them quickly. The smaller the blast radius, the faster you can go.

The sin is not in the failing, but in the failing to notice.

Answer 3

Why do you think 100% reliability is not a good idea? Can you support your beliefs with numbers, facts and logical explanations?

Is the needed hardware too expensive? Development time too long? Maybe you can show that the probability of success of a chain of components is the multiplication of the individual parts, some of which are not under your control?

If you can support your claims then present those to your managers, if not then your managers might be right.

Answer 4

This is actually quite easy, if you know how much lost means the system being down.

That is if you take eBay or Amazon they would have considerable loss even for one second of downtime in a year. So they strive to have a 99.9999... reliability/uptime.

So this is not about 100% but how many nines does it make sense to invest for you to come close to 100%.