0

I'm using gitlab and gitlab-ci along with gitlab-runner.

Gitlab-runner is choking on my build jobs for a day now and I can't find a way to solve the problem.

The problem appears when the npm install command runs :

npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/array-initial-dd86c469/.jshintrc'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/color-support-612f9677/index.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/clone-94e64b37/clone.iml'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/async-each-23f1cf68/README.md'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/extend-b499a705/.travis.yml'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/function-bind-b92d92f1/implementation.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/fs.realpath-89c6342a/old.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/balanced-match-601d32f6/LICENSE.md'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/default-resolution-1ee87648/index.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/archy-6b1fc3a7/test/beep.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/array-initial-dd86c469/LICENSE-MIT'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/core-util-is-c201cb52/lib/util.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/concat-map-071c084f/example/map.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/buffer-equal-a9e86a30/example/eq.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/gulp-02a15240/package.json'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/flagged-respawn-ed938be5/lib/is-v8flags.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/extend-b499a705/CHANGELOG.md'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/function-bind-b92d92f1/index.js'
npm WARN tar ENOSPC: no space left on device, open '/builds/sites/mysite/.npmcache/lib/node_modules/.staging/default-resolution-1ee87648/node-version.js'
Unhandled rejection Error: ENOSPC: no space left on device, open '/root/.npm/_cacache/index-v5/48/51/88ad48f396d262bcea9e42159a40a93446826c57af0b1c54d2699aca3607'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/de/70'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/de/70'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/2f/98'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/5d/36'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/2d/fa'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/f8/8f'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/7d/c9'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/1e/e2'
Unhandled rejection Error: ENOSPC: no space left on device, mkdir '/root/.npm/_cacache/index-v5/b8/3c'

Gitlab-runner is running in a docker container on a photon-os machine.

The cache directory of the runner itself (/builds/sites) is only 60M.

Here is the result of df -h on the host (photon-os)

Filesystem      Size  Used Avail Use% Mounted on
overlay          16G   13G  2.0G  87% /
tmpfs            64M     0   64M   0% /dev
tmpfs          1000M     0 1000M   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/root        16G   13G  2.0G  87% /etc/gitlab-runner
tmpfs          1000M  652K  999M   1% /run/docker.sock
tmpfs          1000M     0 1000M   0% /proc/acpi
tmpfs          1000M     0 1000M   0% /sys/firmware

and here, df -hi

Filesystem     Inodes IUsed IFree IUse% Mounted on
overlay          1.0M 1021K  3.8K  100% /
tmpfs            250K    15  250K    1% /dev
tmpfs            250K    16  250K    1% /sys/fs/cgroup
shm              250K     1  250K    1% /dev/shm
/dev/root        1.0M 1021K  3.8K  100% /etc/gitlab-runner
tmpfs            250K   470  250K    1% /run/docker.sock
tmpfs            250K     1  250K    1% /proc/acpi
tmpfs            250K     1  250K    1% /sys/firmware
  • 1
    Is by any chance / a btrfs filesystem? – Dan Cornilescu Apr 4 at 10:57
  • no idea, I never heard of this. How can I know ? – Armel Larcier Apr 4 at 11:09
  • check its entry in /etc/fstab or the output of the mount cmd (as root) – Dan Cornilescu Apr 4 at 11:12
  • You have too much files in your runner, there's no more inode free: /dev/root 1.0M 1021K 3.8K 100% /etc/gitlab-runner – Tensibai Apr 4 at 11:15
  • Wow I just saw that docker was keeping a whole lot of dangling volumes. Got rid of them and I’m good now! Thanks anyway for the input! – Armel Larcier Apr 4 at 11:16
0

Well this seems to be a docker issue. It’s keeping a whole lot of dangling volumes.

check if you have some :

docker volume ls -qf dangling=true

Get rid of them with :

docker volume rm $(docker volume ls -qf dangling=true)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.