There's a variety of techniques involved, with no single solution. You will likely want to do several of the following:
First, optimize your image layers for reuse. Put frequently changing steps later in the Dockerfile to increase the chances that early layers are cached from previous builds. A reused layer will show up as more disk space in a docker image ls, but if you examine the underlying filesystem, only one copy of each layer is ever stored on disk. That means 3 images of 2 GB each, but which only have 50 MB different in the last few layers the build, will only take up 2.1 GB of disk space, even though the listing makes it appear that they are using 6 GB since you are double counting each of the reused layers.
Layer reuse is why you see images with infrequently changing build dependencies install those first before copying in code. See any python example that has a pattern like:
FROM python
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
# note how the code is copied only after the pip install
# since code changes but requirements.txt doesn't
COPY . .
CMD ["gunicorn", "app:app"]
Pick a minimal base image. This is why you see people go from ubuntu to debian:slim (the slim variants are smaller, shipping with fewer tools), or even alpine. This reduces the size of your starting point, and is very helpful if you are constantly pulling new versions of the base image. However, if your base image rarely changes, layer reuse removes much of the advantage of a minimal base image.
The smallest base image you can pick is scratch, which is nothing, no shell or libraries, and is only useful with statically compiled binaries. Otherwise, pick a base image that includes the tools you need without lots of tools you don't need.
Next, any step that changes or deletes a file should be combined with previous steps that create that file. Otherwise the layered filesystem, which uses copy-on-write even on things like a file permission change, will have the original file in a previous layer and the image size will not shrink when you remove files. This is why your rm commands have no effect on resulting disk space. Instead, you can chain the commands, like:
RUN apt-get update \
&& apt-get install -y \
a-package \
wget \
&& ... \
&& apt-get purge -y wget \
&& rm -r a-build-dir \
&& apt-get purge -y a-package
Note that overuse of command chaining can slow down your builds since you need to reinstall the same toolset any time a prerequisite changes (e.g. the code being pulled with wget). See multi-stage below for a better alternative.
Any file you create that you do not need in your resulting image should be deleted, in the step that creates it. This includes package caches, logs, man pages, etc. To discover what files are being created in each layer, you can use a tool like wagoodman/dive (which I have not personally vetted and would express caution since it runs with full root access on your host), or you can build your docker images without pruning the intermediate containers and then view the diff with:
# first create and leave containers from any RUN step using options on build
docker image build --rm=false --no-cache -t image_name .
# review which layers use an unexpectedly large amount of space
docker image history image_name
# list all containers, particularly the exited ones from above
docker container ps -a
# examine any of those containers
docker container diff ${container_id}
# ... repeat the diff for other build steps
# then cleanup exited containers
docker container prune
With each of those intermediate containers, the diff will show what files are added, changed, or deleted in that step (these are indicated with an A, C, or D before each filename). What diff is showing is the container specific read/write filesystem, which is any file changed by the container from the image state using copy-on-write.
The best way to reduce image size is eliminating any unneeded components, like compilers, from your shipped image. For that, multi-stage builds let you compile in one stage, and then copy only the resulting artifacts from the build stage to a runtime image that has only the minimum needed to run the application. This avoids the need to optimize any of the build steps since they are not shipped with the resulting image.
FROM debian:9 as build
# still chain update with install to prevent stale cache issues
RUN apt-get update \
&& apt-get install -y \
a-package \
wget \
RUN ... # perform any download/compile steps
FROM debian:9-slim as release
COPY --from=build /usr/local/bin/app /usr/local/bin/app
CMD [ "/usr/local/bin/app" ]
Multi-stage is ideal with statically compiled binaries which you can run with scratch as your base image, or transitioning from a compile environment like JDK to a runtime like JRE. This is the easiest way to dramatically reduce your image size while still having fast builds. You may still perform chaining of steps in your release stage if you have steps that change or delete files created in previous steps, but for the most part, the COPY from another stage isolates the release stage from any layer bloat experienced in the earlier build stages.
Note, I do not recommend squashing images since this reduces the size of one image at the expense of eliminating layer reuse. That means future builds of the same image will require more disk and network traffic to send updates. To go back to the first example, squashing may reduce your image from 2 GB to 1 GB, but not 3 images may take up 3 GB instead of the 2.1 GB.
