2

I am guiding teams setting SLOs for their services and a question came up about setting SLOs for services that are still in design. Are there standards for setting SLOs for services that don't exist yet? Or is the best practice to start the service and base the SLO on initial performance and move towards a goal after?

Improve this question

1 Answer 1

Reset to default
3

In an ideal world, the SLOs would be driven by business needs, and information about your customers.

Remember: an SLO should be the level of reliability at which most users are happy with your service.

To do this you can partition your service's behaviour into critical actions.

A critical action is a discrete piece of functionality offered by your service, and which your customers care about. It is typically defined from the point of view of the customer, and independent of the implementation (eg., "Customer successfully added an item to their shopping cart", "Customer viewed the high score table")

With some critical actions chosen, decide what the correct SLI specification should be for each action: does it succeed or fail, and needs an availability SLI; does it take some time to complete, and need a latency SLI; does it produce data to be consumed later, and needs a freshness or a correctness SLI? And so on. (Adding an item to a shopping cart should have an availability and a latency SLI; viewing a high score table should also have freshness SLIs, and potentially correctness)

With these specifications you can start to think about what the impact of unreliability would be on your users: will one failure every few days make a user unhappy? What about one failure per hour? Are failures masked by client-side retries? Will users be annoyed if clicking a button results in a 100ms delay before something happens? What about 1000ms or 10 seconds?

At this point, you have enough information to start setting SLOs and thinking about how to implement your SLI specifications: making trade-offs between fidelity, cost, and complexity. It is here that you can take advantage of the fact that the service is still being designed, and ensure it is designed with observability in mind. For example:

  • Splitting the service into microservices such that those microservices map neatly on to different SLIs and critical actions
  • Ensuring that any HTTP requests that should be counted towards different SLIs can be disambiguated at the load balancer (eg., don't have all requests be POST requests to /handler)
  • Include logs and metrics (eg., via OpenCensus) in the design from the start and consider their impact on your application's scalability and reliability.
  • If you plan to use synthetic probing, allow some way to do this (consider how much of a web page's logic is in client-side scripts, as opposed to server side, and how you would monitor this)
  • Can you include client-side instrumentation, and how would you collect the results?

The most important thing when setting an SLO for a new service is to plan to iterate: Even if you don't know what level of service is necessary to make users happy to begin with, you should have some metrics you can use to determine this (satisfaction scores, ticket levels, random surveys) and use these to tighten or relax your current SLO as necessary.

Don't hold your service to an SLO that is harder to defend than your customers require, but do ensure that your SLO is sufficient to keep them happy.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.