OOM when running Bitbucket pipelines

Question

as one (sometimes failing) step of our pipeline we

• setup environment via installation of docker which runs 3 containers through own docker-compose. Unfortunately we could not figure out how to impose any memory limits on individual containers in docker-compose v3 config.

• run tests in sbt with SBT_OPTS="-Xmx1500m" and every second to third time step is failing with OOM

  definitions: services: docker: memory: 1024

Explanation on the memory usage is not clear on this case.

Could somebody please explain what's going on with the memory allocation to containers, docker host, and step env?

Interesting is that running the same tests, same containers for multiple times usually gives different results. Understandable that real load on the physical server changes over time and running times can differ - but memory?

Puzzled...

Answer 1

I suspect that Bitbucket pipelines is enforcing this memory limit, not docker-compose.

Similarly to docker-compose, the Bitbucket pipelines code is likely to use Linux CGroups to enforce the memory limit, through the Linux kernel's OOM killer.

The page you linked is about memory limits in additional containers (e.g. for a database) used by the pipeline, not the pipeline itself.

Some suggestions:

• Try editing bitbucket-pipelines.yml to include a size option for memory size - I would expect this is more useful than editing docker-compose options which may not be respected. See this doc page.

• Try running the build locally under Docker Compose to get an idea of memory usage.

It would help to include the details of your bitbucket-pipelines.yml, and the Bitbucket error message about being out of memory.

Answer 2

When you limit resources in docker-compose, the docker engine prevent containers to use more than the limits, docker do this by Linux's Cgroups. Its usage in docker-compose v3 is also straightforward: https://docs.docker.com/compose/compose-file/#resources

When your OS is in OOM state, this kills processes automatically and maybe your containers are good candidates in the state, as the problem I have faced on my Gitlab CI's build machine to build node projects!