2

I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling.

The Prometheus alert rules to identify this alert is :

alert: CPUThrottlingHigh
expr: 100
  * sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_throttled_periods_total{container_name!=""}[5m]))
  / sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_periods_total[5m]))
  > 25
for: 15m

If I look at one of the pods identified by this alert, it does not seem to have any reason to throttle :

$ kubectl top pod -n monitoring my-pod
NAME            CPU(cores)   MEMORY(bytes)   
my-pod          0m           6Mi

This pod has one container with these resources setup :

Limits:
  cpu:     100m
  memory:  128Mi
Requests:
  cpu:     25m
  memory:  64Mi

And the node that is hosting this pod is not under any heavy cpu use :

$ kubectl -n monitoring top node aks-agentpool-node-1
NAME                       CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
aks-agentpool-node-1       853m         21%       11668Mi         101%

On grafana, if I look at the chart for this pod, it never goes above 0,000022 of cpu usage

Why is it throttling ?

  • Have you validated that you need to multiple by 100? Is it a decimal format for the percentage or an integer value? – SheldonH Mar 14 at 8:06
  • I am not sure to understand. The expression 100m is the same as 0.1 cpu. – jobou Mar 14 at 11:15
  • If that's the syntax for Prometheus then disregard. Was just checking. – SheldonH Mar 14 at 13:51
3

The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better understand the problem.

In short, the problem is: When working with low CPU limits, spiky workloads can have low averages and still be being throttled.

Also, take a look at this issue (#67577) from Kubernetes project, which addresses a Kernel bug in CFS quotas that may cause unnecessary CPU throttling. The discussion is still open, and the Kubernetes project are even considering disabling CFS quotas for pods in the Guaranteed QoS (see #70585 for reference).

Consider the following options:

  • Increase (or even remove) your container CPU limits
  • Disable Kubernetes CFS quotas entirely (kubelet's flag --cpu-cfs-quota=false)
  • 2
    Thanks, it is nice to know that the issue is not on my installation. I don't want to increase or remove limits for now. And I am using AKS (a managed implementation of Kubernetes in Azure cloud) so I cannot disable CFS quotas. As per the comments in the issue, I am going to try to update the kernel version to see what happens. – jobou Apr 3 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.