14

I have a Docker image, let's call it frontend.image, that I use for a Jenkins build slave. The Jenkins Docker plugin will spin up a container from this image and build artifacts inside the container. This all works great. In this case, the frontend.image is used to build an AngularJs app. Part of building this Angular app is to install npm packages the app requires.

This process, npm install, seems to take a long time, 3 minutes it seems, npm always installs every package every time.

So I added a volume for my slave, it is a host mounted volume, the Docker plugin will use this volume every time it runs the frontend container:

enter image description here

The user that executes the command npm install is jenkins. npm keeps a cache which you can find with command npm config get cache which outputs /home/jenkins/.npm

That is why I have the host volume /slaves/volumes/tsl.frontend:/home/jenkins mounted to my web container slave.

I build my Angular app using a Jenkins project, builds no problem, many npm packages are installed. If ssh into my Docker host and run cmd ls /slaves/volumes/tsl.frontend I see lots of npm packages. This means my host volume mount for the slave worked. enter image description here

Now I build the Jenkins project again, npm installs every single package again, even though the Docker slave build container is using the volume host mount. I can even confirm by bashing into the slave container with cmd docker exec -it <some_clever_random_container_id> bash then cmd su jenkins then cmd npm cache ls which lists many npm packages that are cached. enter image description here

So, even with my host mount volume, which has permissions chmod 777 by the way so there is no permissions issues, I cannot get npm install to use the cache.

In my Jenkins build, which spins up the Docker slave container, the first cmd I run is npm cache ls and many packages are listed, doesn't this mean my host volume is working as expected and the npm cache index has integrity aka not corrupted?

enter image description here

I have tried the regular npm install cmd, which, when I run on my localhost machine installs all the packages the first time and almost no packages the next time. And also the npm cache "hack" npm --cache-min 9999999 install, taken from this SO answer as well as cmd npm --skip-installed --cache-min 9999999 install

A related question was posted on StackOverflow.

Improve this question
13
  • I would bet the cache index is not stored within ~/.npm according to your description
    – Tensibai
    Oct 26 '17 at 9:46
  • @Tensibai you are incorrect and I am very sure about this, the user is jenkins, because that is what you are saying in another way, because I run npm cache ls as jenkins user and that lists packages, you are saying that npm install is being executed by another user
    – Brian Ogden
    Oct 26 '17 at 16:19
  • no, I'm saying the index itself is probably stored elsewhere, in /usr/local or whatever path npm is installed in or whatelse, I've no idea. This just sounds like npm is acting as if there was nothing in the cache, so I guess it doesn't list the directory but base itself on some kind of index elsewhere.
    – Tensibai
    Oct 26 '17 at 16:26
  • @Tensibai but the cmd npm config get cache returns /home/jenkins.npm as that path don't you think that confirms the location of the cache?
    – Brian Ogden
    Oct 26 '17 at 16:27
  • The location of the cache yes, that doesn't enforce the cache index is in the same location at all. I'd add a npm cache ls and a raw ls ~/.npm/* -al in the build script itself before any other build step just to ensure the state of the container when starting the build.
    – Tensibai
    Oct 26 '17 at 16:29
7

I finally solved this by using Docker image layer caching for the npm install, following this answer

This means I moved the npm install out of the Docker slave image and into the actually frontend image, here is my final Docker file that truly caches the npm install in between builds if package.config has no changes:

FROM centos:7
MAINTAINER Brian Ogden

# Not currently being used but may come in handy
ARG ENVIRONMENT
ENV NODE_VERSION 6.11.1

RUN yum -y update && \
    yum clean all && \
    yum -y install http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm \
    yum -y makecache && \
    yum -y install nginx-1.12.0 wget

# Cleanup some default NGINX configuration files we don’t need
RUN rm /etc/nginx/conf.d/default.conf

#############################################
# NodeJs Install
#############################################

#Download NodeJs package
RUN wget -q -O - https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz \
    | tar --strip-components=1 -xzf - -C /usr/local

# https://stackoverflow.com/a/35774741/1258525
# use changes to package.json to force Docker not to use the cache
# when we change our application's nodejs dependencies:
COPY ./package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir /app && cp -a /tmp/node_modules /app/

WORKDIR /app
COPY . /app

RUN npm run build-$ENVIRONMENT

RUN cd /app && cp -a dist/* /usr/share/nginx/html
COPY ./docker/conf/frontend.conf /etc/nginx/conf.d/frontend.conf
COPY ./docker/conf/nginx.conf /etc/nginx/nginx.conf


EXPOSE 80

CMD ["nginx"]
Improve this answer
4
  • 2
    it doesn't solve your problem that was described in the questions. It's just another way to cache. Do you know the reason yet? @Brian
    – An Nguyen
    May 2 '19 at 15:59
  • @AnNguyen nope, and I spent alot of time trying to get npm cache to work. I suggest you use my solution
    – Brian Ogden
    May 2 '19 at 22:43
  • my situation is different. Every time a build is triggered a slave will be provisioned on k8s. So I cannot cache based on docker build process. I want to base on NPM cache so I can mount a persistent volume into slave every time it's provisioned
    – An Nguyen
    May 3 '19 at 3:17
  • How big is the final container?
    – Kraken
    Apr 2 at 14:48
0

Another approach you can do is set up a nexus repository server where you host your npm modules and proxy the external ones. It does not leverage the cache, but because the resources are within your local network or maybe in the same swarm it shouldn't take as long.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.