How to enable the npm cache in a Jenkins build slave that runs in docker?

Question

I have a Docker image, let's call it frontend.image, that I use for a Jenkins build slave. The Jenkins Docker plugin will spin up a container from this image and build artifacts inside the container. This all works great. In this case, the frontend.image is used to build an AngularJs app. Part of building this Angular app is to install npm packages the app requires.

This process, npm install, seems to take a long time, 3 minutes it seems, npm always installs every package every time.

So I added a volume for my slave, it is a host mounted volume, the Docker plugin will use this volume every time it runs the frontend container:

The user that executes the command npm install is jenkins. npm keeps a cache which you can find with command npm config get cache which outputs /home/jenkins/.npm

That is why I have the host volume /slaves/volumes/tsl.frontend:/home/jenkins mounted to my web container slave.

I build my Angular app using a Jenkins project, builds no problem, many npm packages are installed. If ssh into my Docker host and run cmd ls /slaves/volumes/tsl.frontend I see lots of npm packages. This means my host volume mount for the slave worked.

Now I build the Jenkins project again, npm installs every single package again, even though the Docker slave build container is using the volume host mount. I can even confirm by bashing into the slave container with cmd docker exec -it <some_clever_random_container_id> bash then cmd su jenkins then cmd npm cache ls which lists many npm packages that are cached.

So, even with my host mount volume, which has permissions chmod 777 by the way so there is no permissions issues, I cannot get npm install to use the cache.

In my Jenkins build, which spins up the Docker slave container, the first cmd I run is npm cache ls and many packages are listed, doesn't this mean my host volume is working as expected and the npm cache index has integrity aka not corrupted?

I have tried the regular npm install cmd, which, when I run on my localhost machine installs all the packages the first time and almost no packages the next time. And also the npm cache "hack" npm --cache-min 9999999 install, taken from this SO answer as well as cmd npm --skip-installed --cache-min 9999999 install

A related question was posted on StackOverflow.

Answer 1

I finally solved this by using Docker image layer caching for the npm install, following this answer

This means I moved the npm install out of the Docker slave image and into the actually frontend image, here is my final Docker file that truly caches the npm install in between builds if package.config has no changes:

FROM centos:7
MAINTAINER Brian Ogden

# Not currently being used but may come in handy
ARG ENVIRONMENT
ENV NODE_VERSION 6.11.1

RUN yum -y update && \
    yum clean all && \
    yum -y install http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm \
    yum -y makecache && \
    yum -y install nginx-1.12.0 wget

# Cleanup some default NGINX configuration files we don’t need
RUN rm /etc/nginx/conf.d/default.conf

#############################################
# NodeJs Install
#############################################

#Download NodeJs package
RUN wget -q -O - https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz \
    | tar --strip-components=1 -xzf - -C /usr/local

# https://stackoverflow.com/a/35774741/1258525
# use changes to package.json to force Docker not to use the cache
# when we change our application's nodejs dependencies:
COPY ./package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir /app && cp -a /tmp/node_modules /app/

WORKDIR /app
COPY . /app

RUN npm run build-$ENVIRONMENT

RUN cd /app && cp -a dist/* /usr/share/nginx/html
COPY ./docker/conf/frontend.conf /etc/nginx/conf.d/frontend.conf
COPY ./docker/conf/nginx.conf /etc/nginx/nginx.conf


EXPOSE 80

CMD ["nginx"]

Answer 2

Another approach you can do is set up a nexus repository server where you host your npm modules and proxy the external ones. It does not leverage the cache, but because the resources are within your local network or maybe in the same swarm it shouldn't take as long.