No space left on device when running job on photon-os

Question

I'm using gitlab and gitlab-ci along with gitlab-runner.

Gitlab-runner is choking on my build jobs for a day now and I can't find a way to solve the problem.

The problem appears when the npm install command runs :

npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/array-initial-dd86c469/.jshintrc'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/color-support-612f9677/index.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/clone-94e64b37/clone.iml'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/async-each-23f1cf68/README.md'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/extend-b499a705/.travis.yml'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/function-bind-b92d92f1/implementation.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/fs.realpath-89c6342a/old.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/balanced-match-601d32f6/LICENSE.md'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/default-resolution-1ee87648/index.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/archy-6b1fc3a7/test/beep.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/array-initial-dd86c469/LICENSE-MIT'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/core-util-is-c201cb52/lib/util.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/concat-map-071c084f/example/map.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/buffer-equal-a9e86a30/example/eq.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/gulp-02a15240/package.json'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/flagged-respawn-ed938be5/lib/is-v8flags.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/extend-b499a705/CHANGELOG.md'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/function-bind-b92d92f1/index.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/default-resolution-1ee87648/node-version.js'
Unhandled rejection Error: ENOSPC: no space left on device, open '/root/.npm/_cacache/index-v5/48/51/88ad48f396d262bcea9e42159a40a93446826c57af0b1c54d2699aca3607'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/de/70'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/de/70'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/2f/98'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/5d/36'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/2d/fa'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/f8/8f'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/7d/c9'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/1e/e2'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/b8/3c'

Gitlab-runner is running in a docker container on a photon-os machine.

The cache directory of the runner itself (/builds/sites) is only 60M.

Here is the result of df -h on the host (photon-os)

Filesystem      Size  Used Avail Use% Mounted on
overlay          16G   13G  2.0G  87% /
tmpfs            64M     0   64M   0% /dev
tmpfs          1000M     0 1000M   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/root        16G   13G  2.0G  87% /etc/gitlab-runner
tmpfs          1000M  652K  999M   1% /run/docker.sock
tmpfs          1000M     0 1000M   0% /proc/acpi
tmpfs          1000M     0 1000M   0% /sys/firmware

and here, df -hi

Filesystem     Inodes IUsed IFree IUse% Mounted on
overlay          1.0M 1021K  3.8K  100% /
tmpfs            250K    15  250K    1% /dev
tmpfs            250K    16  250K    1% /sys/fs/cgroup
shm              250K     1  250K    1% /dev/shm
/dev/root        1.0M 1021K  3.8K  100% /etc/gitlab-runner
tmpfs            250K   470  250K    1% /run/docker.sock
tmpfs            250K     1  250K    1% /proc/acpi
tmpfs            250K     1  250K    1% /sys/firmware

Answer 1

Well this seems to be a docker issue. It’s keeping a whole lot of dangling volumes.

check if you have some :

docker volume ls -qf dangling=true

Get rid of them with :

docker volume rm $(docker volume ls -qf dangling=true)