Does Metrics Server kill pods that use too much memory?

Question

I have a pod (in my on premises Kubernetes cluster) that I setup to eventually try to exceed its memory limit. When it did that, my pod started throwing an error saying it was out of memory.

It stayed like that. With its memory maxed out. I can kill it manually, but otherwise it will stay just like that.

However, when I installed Metrics Server and repeat this test, these pods are killed fairly quickly with "out of memory" (once they get near the max memory allowed by the pod).

So to prove that it was Metrics Server causing it, I uninstalled it, and the previous functionality returned (out of memory errors).

Lastly I installed Metrics Server again and.... the out of memory errors continued....

So I am now confused. What would have been killing my containers when they ran out of memory? (I want it to happen, but I need to understand it so I can rely on it.)

Is it really Metrics Server doing the killing here? And is this configurable?

Answer 1

I felt a little bit odd by reading your question, because this would break the kubernetes design principles.

So checked the RBAC rules of the metric server provided manifests. There is no role assignment, which would allow the metrics server to delete pods:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
  - apiGroups: [""]
    resources:
      - nodes/metrics
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - pods
      - nodes
    verbs:
      ## ONLY read verbs allowed here!!!
      - get
      - list
      - watch

There is another ClusterRoleBinding for the system:auth-delegator ClusterRole, but this doesn't allow this eighter:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:auth-delegator
rules:
  - verbs:
      - create
    apiGroups:
      - authentication.k8s.io
    resources:
      - tokenreviews
  - verbs:
      - create
    apiGroups:
      - authorization.k8s.io
    resources:
      - subjectaccessreviews

This is only to integrate the API Service provided by Metrics-Server.

So there seems to be no direct way for Metrics-Server to kill pods.