We recently switched over our production environment to Kubernetes. I'd like to enforce CPU limits on the containers. I'm getting conflicting CPU metrics that do not fit together. Here's my setup:
- DataDog agents running as a
- Existing applications running without CPU Limits
- Containers in question are multi-threaded Ruby applications
- Two metrics:
c4.xlargecluster nodes (4 vCPUs or 4000m in Kubernetes terms)
kubernetes.cpu.usage.max reports ~600m for the containers in question.
docker.cpu.usage reports ~60%. It follows that a 1000m CPU limit would be more than enough capacity under normal operation.
I set the limit to 1000m. Then
docker.container.throttles goes up significantly while
docker.cpu.usage stay the same. The system all fells to it's knees during this time. This does not make sense to me.
I researched Docker stats. It seems that
docker stats (and the underlying API) normalize load according to CPU cores. So in my case,
docker.cpu.usage of 60% comes (4000m * 0.60) to 2400m in Kubernetes terms. However this does not correlate with any Kubernetes numbers. I did another experiment to test my hypothesis that Kubernetes numbers are incorrect. I set the limit to 2600m (for some extra headroom). This did not result in any throttles. However Kubernetes observed CPU usage did not change. This leaves me confused.
So my questions are:
- Does this feel like a bug in Kubernetes (or something in the stack?)
- Is my understanding correct?
My follow up question relates to how to properly determine CPU for Ruby applications. One container uses Puma. This is a multi-threaded web server with a configurable amount of threads. HTTP requests are handled by one of the threads. The second application is a thrift server using the threaded server. Each incoming TCP connection is handled by it's own thread. The thread exits when the connection closes. Ruby as GIL (Global Interpreter Lock) so only one thread can execute Ruby code at a time. That does allow for multiple threads executing IO and things like that.
I think the best approach is limit the number of threads running in each application and approximating Kubernetes CPU limits based on the number of threads. The processes are not forking so total CPU usage is harder to predict.
The question here is: how to properly predict CPU usage and limits for these applications?