Should I chunk my docker images in many small layers instead of one large?

Question

When I build docker image for existing applications I try to use as few layers as possible and clean up any unwanted files. For example building an image for moodle:

# Dockerfile for moodle instance.
# Forked from Jonathan Hardison's <[email protected]> docker version. https://github.com/jmhardison/docker-moodle
#Original Maintainer Jon Auer <[email protected]>

FROM  php:7.2-apache

# Replace for later version
ARG VERSION=37
ARG DB_TYPE="all"

VOLUME ["/var/moodledata"]
EXPOSE 80

ENV MOODLE_DB_TYPE="${DB_TYPE}"
# Let the container know that there is no tty
ENV DEBIAN_FRONTEND noninteractive \
    MOODLE_URL http://0.0.0.0 \
    MOODLE_ADMIN admin \
    MOODLE_ADMIN_PASSWORD Admin~1234 \
    MOODLE_ADMIN_EMAIL [email protected] \
    MOODLE_DB_HOST '' \
    MOODLE_DB_PASSWORD '' \
    MOODLE_DB_USER '' \
    MOODLE_DB_NAME '' \
    MOODLE_DB_PORT '3306'

COPY ./scripts/entrypoint.sh /usr/local/bin/entrypoint.sh


RUN echo "Build moodle version ${VERSION}" &&\
    chmod +x /usr/local/bin/entrypoint.sh &&\
    apt-get update && \
    if [ $DB_TYPE = 'mysqli' ] || [ $DB_TYPE = 'all' ]; then echo "Setup mysql and mariadb support" && docker-php-ext-install pdo mysqli pdo_mysql; fi &&\
    if [ $DB_TYPE = 'pgsql' ] || [ $DB_TYPE = 'all' ]; then echo "Setup postgresql support" &&\
        apt-get install -y --no-install-recommends libghc-postgresql-simple-dev &&\
        docker-php-ext-configure pgsql -with-pgsql=/usr/local/pgsql &&\
        docker-php-ext-install pdo pgsql pdo_pgsql; \ 
    fi &&\
    apt-get -f -y install --no-install-recommends rsync unzip netcat libxmlrpc-c++8-dev libxml2-dev libpng-dev libicu-dev libmcrypt-dev libzip-dev &&\
    docker-php-ext-install xmlrpc && \
    docker-php-ext-install mbstring && \
    whereis libzip &&\
    docker-php-ext-configure zip --with-libzip=/usr/lib/x86_64-linux-gnu/libzip.so &&\
    docker-php-ext-install zip && \
    docker-php-ext-install xml && \
    docker-php-ext-install intl && \
    docker-php-ext-install soap && \
    docker-php-ext-install gd && \
    docker-php-ext-install opcache && \
    echo "Installing moodle" && \
    curl https://download.moodle.org/download.php/direct/stable${VERSION}/moodle-latest-${VERSION}.zip -o /tmp/moodle-latest.zip  && \
    rm -rf /var/www/html/index.html && \
    cd /tmp &&  unzip /tmp/moodle-latest.zip && cd / \
    mkdir -p /usr/src/moodle && \
    mv /tmp/moodle /usr/src/ && \
    chown www-data:www-data -R /usr/src/moodle && \
    apt-get purge -y unzip &&\
    apt-get autopurge -y &&\
    apt-get autoremove -y &&\
    apt-get autoclean &&\
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* cache/* /var/lib/log/*

COPY ./scripts/moodle-config.php /usr/src/moodle/config.php
COPY ./scripts/detect_mariadb.php /opt/detect_mariadb.php


ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["/usr/sbin/apache2ctl", "-D", "FOREGROUND"]

Source

But when I build images for an another (production project) I noticed that once I pull newer images for existing containers only the changed layers are actually being downloaded.

So I wonder wouldn't be better to have a layer for each purpose for example a single layer for the php itself and an another layer for the application itself? So as a result during deployment only the changed parts will be downloaded instead of a HUGE single layer.

But as a downside splitting my builds on smaller layers may require to re-download and delete many required packages for building php extentions. Do you think on large applications may be a problem or does it worth to have longer builds for smaller download during deployment or I am talking nonsense?

On the other hand I understand using a single layer makes the overall image size smaller. But as a result it will need to actually download the whole layer itself. So if my image is 100MB it will need to re-download the whole 100Mb instead of few kilobytes that a changed layer will be.

Answer 1

Generally speaking, you want to:

• Combine multiple related actions/files into a single layer so you don't have tons of layers.
• Separate out unrelated actions/files which are likely to change independently.
• Order layers such that those least likely to change occur first in the file.
• Use multi stage builds to clean up if you have lots of garbage in your image/layers that does not need to be in the final image.

If you have to make a change to anything in the docker file, everything below that point has to be redone; caching is useless at that point. You want to avoid this.

So, rather than trying to combine everything into one layer or something odd like that, you should focus on understanding what is likely to change and how you can reduce the amount of layers you have to rebuild if and when that change happens.

The best explained example I've seen of this is here for Spring Boot.

Details are below... but to summarize; they determined that a huge layer/size was the application dependencies from Maven. They figured most app changes don't require new maven dependencies, so caching the maven dependencies in their own layer would generally make builds and image pulls faster by avoiding re-pulling dependencies. Then when you change the application code itself, it just pulls the later/smaller layers for the app itself which is very efficient.

---

A Better Dockerfile

A Spring Boot fat jar naturally has "layers" because of the way that the jar itself is packaged. If we unpack it first it will already be divided into external and internal dependencies.

Dockerfile

FROM openjdk:8-jdk-alpine
VOLUME /tmp
ARG DEPENDENCY=target/dependency
COPY ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY ${DEPENDENCY}/META-INF /app/META-INF
COPY ${DEPENDENCY}/BOOT-INF/classes /app
ENTRYPOINT ["java","-cp","app:app/lib/*","hello.Application"]

There are now 3 layers, with all the application resources in the later 2 layers. If the application dependencies don’t change, then the first layer (from BOOT-INF/lib) will not change, so the build will be faster, and so will the startup of the container at runtime as long as the base layers are already cached.

---

There really isn't a magic answer to this. It really just depends on your application and its dependencies and what is likely to change in your development/deployment environment over time.