I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling.
The Prometheus alert rules to identify this alert is :
alert: CPUThrottlingHigh
expr: 100
* sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_throttled_periods_total{container_name!=""}[5m]))
/ sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_periods_total[5m]))
> 25
for: 15m
If I look at one of the pods identified by this alert, it does not seem to have any reason to throttle :
$ kubectl top pod -n monitoring my-pod
NAME CPU(cores) MEMORY(bytes)
my-pod 0m 6Mi
This pod has one container with these resources setup :
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 25m
memory: 64Mi
And the node that is hosting this pod is not under any heavy cpu use :
$ kubectl -n monitoring top node aks-agentpool-node-1
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
aks-agentpool-node-1 853m 21% 11668Mi 101%
On grafana, if I look at the chart for this pod, it never goes above 0,000022
of cpu usage
Why is it throttling ?
100m
is the same as0.1
cpu.