7

I am trying to improve CI/CD performance and part of that is improving the caching behaviour of Docker builds. I find myself trying to debug what exactly invalidates the cache. Are there any ways to query the Docker build process to get more information than "cached" and "not cached" ? For example, at some point, I add a large directory, which has not changed as far as I can tell, but busts the cache. I would like to find out what is causing that, exactly.

3

The docker cache needs the following:

  • Build has been run on the same docker host previously (or you explicitly added flags to trust a pulled image) and
  • The previous image needs to still be on the build host (not pruned) and
  • The same previous layer and either
  • The same command being run (including the same environment/args) or
  • The same hash on the files being copied

If you see the cache being busted on a COPY or ADD command, and not the previous step, then you need to look at the hash being generated. All the files need to be identical, same file names, cases, file permissions, file owners, and the contents need to be bit for bit identical. You can look at the resulting checksum for this hash by running a docker image history command, e.g.:

$ docker image history golang:alpine
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
6b21b4c6e7a3        3 weeks ago         /bin/sh -c #(nop) WORKDIR /go                   0B
<missing>           3 weeks ago         /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOPATH=/go               0B
<missing>           3 weeks ago         /bin/sh -c set -eux;  apk add --no-cache --v…   344MB
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.12.7    0B
<missing>           3 weeks ago         /bin/sh -c [ ! -e /etc/nsswitch.conf ] && ec…   17B
<missing>           3 weeks ago         /bin/sh -c apk add --no-cache   ca-certifica…   551kB
<missing>           3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:0eb5ea35741d23fe3…   5.58MB

You can include the --no-trunc option to get a full line. The key piece of data relevant to your question from the above image is in the ADD step where you see file:0eb5ea35741d23fe3…. If that checksum changes between two image builds, the cache will bust and you'll start a new set of layers.

| improve this answer | |
0

According to Docker documentation search for: "Leverage build cache", there is checksum calculated on the directory - please see the article for details.

What I will suggest is a multistage build.

If we have static content of directory we could create container/image with its content. Then instead of using Docker copy, we could use FROM as a build stage and copy that directory into a new layer, which in fact will reuse that layer. That is based on a fact that we don't operate on a directory (when a file could get a changed attribute like last accessed time).

Base Dockerfile

FROM alpine:latest  
COPY /myDirectory .

Then you could tag this container as storage-base O

MultiStage Dockerfile

FROM storage-base AS storage

FROM alpine:latest  
RUN a command 
WORKDIR /root/
COPY --from=storage /myDirectory .

Please check more references here: Docker Multistage Build

Edit

  1. the image/container created in step 1 shall be only recreated when directory content will change (by a trigger that knows that the data changed), so that means it will not be created on every build.
  2. then we use the image that was created once (and re-use that layer)

I just created a simple example to help express my idea - Git repo - run build-docker.sh script

I found that on reboot Docker is re-creating cache instead of reusing, but it takes just a few minutes and it is not critical.

| improve this answer | |
  • 1
    Unfortunately, this doesn't help me at all. I did look at the article in my own research. It gives no detail and no help in debugging why a directory that is bitwise identical does not get cached. Splitting the build up into multiple parts also would not help, since the first stage would get the cache busted triggering all later stages being uncached as well :( – Torque Aug 1 '19 at 14:07
  • @Torque please see my edit, as I poorly expressed my thoughts. – profesor79 Aug 1 '19 at 17:03
  • While I appreciate the effort you took, this really doesn't address the issue - writing your own programs or scripts to get around a bug in Dockers caching is not a sane way. Docker already does have this functionality, and I'm looking for ways to debug why it's not working as defined, not hack my way around it. Furthermore, like I said, we are using this in CI/CD so no old files would be there, it's always a fresh checkout. Lastly, a cache invalidation adds over an hour of compile time to our builds, not a few minutes. – Torque Aug 2 '19 at 7:14
  • @Torque thanks for the comment. Is your build system restarted? – profesor79 Aug 2 '19 at 8:05
  • Restarts are not what invalidate the cache, if that is what you are asking. – Torque Aug 2 '19 at 9:19
0

I have to deal with that same issue and the main problem is not the cache itself is the time that takes to make a build but also the problem of being sure you have the right changes on the image you are building.

I can recommend you some different options you can make separately or all together:

  • Make cronjob that cleans your docker Daemon of old images periodically, for example, once a week. With that, you are sure to have fresh images every week and have a good performance with the cache for the daily build, the only one that takes a little bit more time is the first build after the flush.

  • Make your own base image with all the content you know you don't change too often, for that you can have more than one Dockerfile on your project and launch a build process of the base image after the flush on the previous step to be sure you have the latest changes on a fresh image

  • Use multi-stage build or build patter to have a good size image at the end of the build.

| improve this answer | |
  • 1
    What you recommend is exactly what we do. The problem is in my last two sentences - this rarely changed stuff DOESN'T change, but the cache is still invalidated, which is what I am attempting to debug. – Torque Aug 2 '19 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.