Docker-in-docker has gotten better over the years and the docker dind image is getting more maturity. However there are still a number of caveats that make its use relatively difficult to get right.
The main challenge of getting docker-in-docker to work well is the storage driver of the docker installation. On your host you would have filesystems like
ext4, which docker can run
overlayfs2 on top of. However, within a docker container, docker is unable to use
overlayfs2 as the backing filesystem of another
overlayfs2. It doesn't work recursively. There are no easy setups to create a recursive docker-in-docker storage driver stack that works well.
The only storage driver that works on anything is
vfs does not have copy-on-write capabilities, meaning that every docker build layer is actually a complete copy of the previous layers. In docker builds with many layers, high amounts of duplication cause performance issues as well as they quickly exhaust disk space. So it's usually not a real solution.
A possible workaround is to use a volume mount for the directory of the inner docker daemon's storage (
/var/lib/docker). Then the inner docker daemon can use the host filesystem directly and can thus run
overlayfs2 on it.
Another workaround would be to create a loop device formatted as
ext4 which can then be used as the backing filesystem for the inner docker daemon. To avoid pre-allocating space on the disk, you can use a sparse file for the loop device. For example
dd if=/dev/zero of="$image_file" bs=1M count=0 seek="$sparse_loop_device_size_mb"
mount -n -o loop,noatime,nodiratime,noexec,noauto "$image_file" /var/lib/docker
(The size of the loop device here is the maximum size, not the initial size)
Note however that docker recommends that write-heavy containers should not use the container's writeable layer due to performance reasons. So it is anyway advisable to use a volume mount for the docker daemon directory.
Another typical issue is the cache of docker builds or of docker pulls within docker-in-docker. If you spin up docker-in-docker daemon for a build every time, it will have zero cache. Or if you spin up a docker-in-docker daemon for an integration test, you will have to pull down everything all over again. This is obvious when you think about it, but it can be a key performance issue if not taken into account. There is currently no easy way to share cache between the host docker daemon and the docker-in-docker.
However, you may be able to mount a volume for the docker dir (
/var/lib/docker), to use it across multiple runs. Just make sure you never mount it to two docker daemons at the same time as that's an easy way to get unpredictable behavior due to corruption. The docker dir was not designed to be accessed by more than one daemon concurrently.
I have seen no evidence in the community of any significant performance impact of using docker-in-docker (aside from the storage driver and cache considerations mentioned above).
Docker-in-docker requires privileged mode, which can be an additional attack vector. There is ongoing work to support docker-in-docker in rootless mode, including using a fuse variant of
overlayfs. This is less mature however.
Workaround: Mounting docker.sock
There is, of course, the possibility of mounting
/var/run/docker.sock within a container and you can use the docker CLI within that container. This is not a real docker-in-docker, as the container merely uses a client to connect to the host's docker daemon. It is however a popular, hassle-free option for certain use cases. This is far easier to achieve, if your circumstances permit.