0

I was able to correlate the "usable memory" going down with events in kubernetes:

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 06 Jun 2023 19:31:34 +0000   Thu, 11 May 2023 15:24:32 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available

The time of the LastTransitionTime coincides with the lowest memory and the transition to a lot of available memory.

Also, I think that during the times of low available memory I get a lot of OOM errors and processes are getting killed with exit code 137.

Is there something misconfigured in this Kubernetes cluster? I think the available memory should not ever go that low and it should start evicting pods earlier. Also I don't understand why the whole memory becomes available - this means that we are evicting ALL pods?

memory available goes lower and lower then jumps up

1 Answer 1

0

From the get-go, my educated guess would be memory leak your workloads. It seems that his happens pretty regular - every 12 hours, do you have some scheduled/cron job?

Alternative problem could be overcommitted resources.

First of all check resources.limits on workloads you are running on the EKS cluster. Make sure resources.limits.memory is set on every single container. You can play around with limit ranges to set defaults for a namespace; this way you "protect" your nodes to some extent (alternative is to use resource quotas.

I see you are using Datadog, you can leverage Kubernetes metrics to find spiking workload - kubernetes.memory.working_set (reference):

Current working set in bytes - this is what the OOM killer is watching for

The description of metric you are currently using (reference):

The system.mem.usable metric represents free memory plus cached memory plus buffers (on Linux, it reflects “MemAvailable” attribute from /proc/meminfo whenever possible).

On a "cluster" side of things you may enable "resource reservation", which may help with keeping system daemons (like kubelet) alive. Also read docs on node pressure evictions and thresholds.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.