Conflicting Kubernetes CPU usage & Docker Container Metrics

Question

We recently switched over our production environment to Kubernetes. I'd like to enforce CPU limits on the containers. I'm getting conflicting CPU metrics that do not fit together. Here's my setup:

• DataDog agents running as a Daemonset
• Existing applications running without CPU Limits
• Containers in question are multi-threaded Ruby applications
• Two metrics: kubernetes.cpu.usage.{avg,max} and docker.cpu.usage
• c4.xlarge cluster nodes (4 vCPUs or 4000m in Kubernetes terms)

kubernetes.cpu.usage.max reports ~600m for the containers in question. docker.cpu.usage reports ~60%. It follows that a 1000m CPU limit would be more than enough capacity under normal operation.

I set the limit to 1000m. Then docker.container.throttles goes up significantly while kubernetes.cpu.usage.max and docker.cpu.usage stay the same. The system all fells to it's knees during this time. This does not make sense to me.

I researched Docker stats. It seems that docker stats (and the underlying API) normalize load according to CPU cores. So in my case, docker.cpu.usage of 60% comes (4000m * 0.60) to 2400m in Kubernetes terms. However this does not correlate with any Kubernetes numbers. I did another experiment to test my hypothesis that Kubernetes numbers are incorrect. I set the limit to 2600m (for some extra headroom). This did not result in any throttles. However Kubernetes observed CPU usage did not change. This leaves me confused.

So my questions are:

• Does this feel like a bug in Kubernetes (or something in the stack?)
• Is my understanding correct?

My follow up question relates to how to properly determine CPU for Ruby applications. One container uses Puma. This is a multi-threaded web server with a configurable amount of threads. HTTP requests are handled by one of the threads. The second application is a thrift server using the threaded server. Each incoming TCP connection is handled by it's own thread. The thread exits when the connection closes. Ruby as GIL (Global Interpreter Lock) so only one thread can execute Ruby code at a time. That does allow for multiple threads executing IO and things like that.

I think the best approach is limit the number of threads running in each application and approximating Kubernetes CPU limits based on the number of threads. The processes are not forking so total CPU usage is harder to predict.

The question here is: how to properly predict CPU usage and limits for these applications?

Answer 1

Multiple things here:

1. Youre on AWS ec2, hence whatever you are doing on your instance for measuring CPU is calculating CPU on hypervisor level and not instance level. To very this run any load test and check iostat -ct 1 and CPU usage in cloudwatch. CPU usage in cloudwatch is always 10-20% greater than what iostat will report and that is because iostat will give CPU usage on hypervisor level.

2. As docker to see how kubernetes and docker metrics compare i suggest to run the containers with --cpuset=1 or any number to allow all containers to use only a single vCPU.

3. Also in AWS 1 CPU= 2vcpu. It is hyperthreaded to 2. You can maybe take this into consideration while calculating.

4. Finally the best metric to use to see CPU usage for a particular application is using htop and correlating with your cloudwatch metrics.

5. I have also observed sometimes docker daemon pinning itself to one of the virtual CPUs and hence when you are reducing it to 1000m, maybe the whole setup is getting slow because the reduction is happening on any one of the vpcus. You can use mpstat to get into details of this.

6. Lastly on the host level you can pin docker to a single CPU and observe more.

Hope this brings you somewhat closer. Update me if you have already found a solution.