What are best practices for over-committing resources in a staging environment?

Question

Suppose there is a staging environment very strapped for resources as new development, data workloads and other applications grow. Ideally one would like to maintain production parity, even at the resource allocation level per each deployed item in the staging environment. For example, one of the 12 Factor App principles describes this, taking great pains to emphasize not just a shallow kind of parity with mocking or lightweight spoofed resources, but rather really maintaining parity by running the same resources in staging.

Obviously for projects that scale up a lot, you may be budget constrained and simply have no choice but to spoof some resources in staging, use "sample" data or microservice stubs with the same shape as what real dependencies will look like in production, and other sacrifices.

But one type of sacrifice that is difficult to approach is reducing the basic allocated resources, like CPU, RAM, disk access, networking resources, logging, etc., so that a deployment in the staging environment becomes quite a different entity if you are interested in testing behavior of the whole stack of components together.

There are various tools that can help you automatically reduce resource usage, such as auto-scaling in many cloud vendor environment and in kubernetes, which in effect can lead to "overcommitting" your staging environment if deployments are asking for the same kinds of resources as in production but being tuned to use less as a function of real load or other dynamic conditions.

My question is: what is the state of best practices or understanding about this practice of overcommitting resources in staging. How often does this type of requirement become a necessity from budget and what are the major principles to think about when designing a solution in that case? What about the flip side of the issue: that "the cost of doing business" for a particular deployed entity should have the cost of significant production parity in staging impounded in as a basic need?

Answer 1

Very good question; I'm interested to see what comes up here.

In terms of cost, obviously each company and team is different. Ideally, everyone should be resource and cost conscious... but most people really don't seem to penny pinch by heavily monitoring and tuning their workloads (probably as that takes a certain skill level and tool set).

My Thoughts / Habits:

• Unlike most on-prem environments, it is very easy to spawn up dynamic environments in the cloud (if you're automating properly).
• So, when cost is prohibitive, you can just spin up full-scale staging environments in order to do load testing and then shut them down.
• This will work for most use cases. Obviously some things do only break when systems are alive a certain period of time under a certain load.
• We usually keep identical but smaller environments in staging. E.g. linearly scaled down presto clusters/spark clusters, k8s micro-services with less copies, and the like.
• We try to keep the actual memory/cpu requests (or config) the same on any tool. This allows for proper load testing without config changes for anything that scales.

Additional Thought

I'm pretty fond of the DevOps Toolkit book series by Victor Farcic. If you're using kubernetes, the 2.5 book (https://www.devopstoolkitseries.com/posts/devops-25/) goes into detail on using prometheus and alert manager to detect over-sized or undersized resource requests, among other things.

If you're really into saving resources and testing well simultaneously, it may make most sense to implement and use this in staging and prod. That way you can tune your alerting and automate your response so you can keep both environments sensibly sized for themselves in a consistent and enforced manner.

I think you'd still want to ensure you could set the staging environment (or another dynamic environment) to the prod settings to load test effectively though.