Why is Docker-in-Docker considered bad?

Question

In August 2013 Jérôme Petazzoni created Docker in Docker, dind for short, this allowed Docker containers to be created inside of Docker Containers, this functionality proved very popular resulting in Jérôme's GitHub Repository receiving over a thousand stars and three hundred forks.

As of Docker 1.8, released two years later in August 2015, Docker in Docker is directly supported by Docker out of the box. However, the use of Docker in Docker comes with a warning, seemingly related to Jérôme's Post: Using Docker-in-Docker for your CI or testing environment? Think twice. which focuses on the reasons why Docker in Docker is not a great choice for Continuous Integration.

Why is it considered bad to use Docker in Docker? Is it simply a case for avoiding Turtles all the way down? or performance considerations?

Answer 1

Continuous Integration Concerns

In short: Docker in Docker (dind) doesn't handle concurrency well.

The reason why you shouldn't use dind for CI is because Docker was designed to have exclusive access to the directory it uses for storage (normally /var/lib/docker). Dind doesn't respect this as all child processes use this directory concurrently. Every time you rebuild (from CI for example), anything related to your app in this directory could get wiped out and forced to start from zero. How would your users like it if they entered their payment details, clicked "Purchase", and suddenly found themselves back on the login screen as if they'd never done anything? That's just not good UX. Two rebuilds occur at once? That's really going to end badly for everyone involved (including your data integrity).

Other Concerns

From the link the OP posted, security concerns arise as the system will try to apply security policies in a very "CSS-like" fashion where a lower container could have access to an outer container's resources unless explicitly prohibited. Remember when you could access web server resources by doing something like "mywebsite.com/../another_folder/private_resource.txt"? Also, sometimes filesystems just don't play well with each other when they're nested in this way.

The Fix

Thankfully, the blog post in the OP has a good solution for the issue. Unless your needs are not met by "build/run/push Docker containers from your CI system itself running on Docker", you can use -v mode (add a data volume to your container) on the Docker socket (usually /var/run/docker.sock:/var/run/docker.sock) to allow the kind of access you need to the "shared" data volume. These containers will be started alongside the parent, instead of underneath, forcing synchronous IO. Now you have the same thing (almost) as dind but without the downsides that come with Docker not being build for concurrency.

Reference (from OP): Using Docker-in-Docker for your CI or testing environment? Think twice.

Answer 2

Docker-in-docker has gotten better over the years and the docker dind image is getting more maturity. However there are still a number of caveats that make its use relatively difficult to get right.

Storage driver

The main challenge of getting docker-in-docker to work well is the storage driver of the docker installation. On your host you would have filesystems like ext4, which docker can run overlayfs2 on top of. However, within a docker container, docker is unable to use overlayfs2 as the backing filesystem of another overlayfs2. It doesn't work recursively. There are no easy setups to create a recursive docker-in-docker storage driver stack that works well.

The only storage driver that works on anything is vfs. However, vfs does not have copy-on-write capabilities, meaning that every docker build layer is actually a complete copy of the previous layers. In docker builds with many layers, high amounts of duplication cause performance issues as well as they quickly exhaust disk space. So it's usually not a real solution.

A possible workaround is to use a volume mount for the directory of the inner docker daemon's storage (/var/lib/docker). Then the inner docker daemon can use the host filesystem directly and can thus run overlayfs2 on it.

Another workaround would be to create a loop device formatted as ext4 which can then be used as the backing filesystem for the inner docker daemon. To avoid pre-allocating space on the disk, you can use a sparse file for the loop device. For example

dd if=/dev/zero of="$image_file" bs=1M count=0 seek="$sparse_loop_device_size_mb"
mkfs.ext4 "$image_file"
mount -n -o loop,noatime,nodiratime,noexec,noauto "$image_file" /var/lib/docker

(The size of the loop device here is the maximum size, not the initial size)

Note however that docker recommends that write-heavy containers should not use the container's writeable layer due to performance reasons. So it is anyway advisable to use a volume mount for the docker daemon directory.

Image cache

Another typical issue is the cache of docker builds or of docker pulls within docker-in-docker. If you spin up docker-in-docker daemon for a build every time, it will have zero cache. Or if you spin up a docker-in-docker daemon for an integration test, you will have to pull down everything all over again. This is obvious when you think about it, but it can be a key performance issue if not taken into account. There is currently no easy way to share cache between the host docker daemon and the docker-in-docker.

However, you may be able to mount a volume for the docker dir (/var/lib/docker), to use it across multiple runs. Just make sure you never mount it to two docker daemons at the same time as that's an easy way to get unpredictable behavior due to corruption. The docker dir was not designed to be accessed by more than one daemon concurrently.

Performance

I have seen no evidence in the community of any significant performance impact of using docker-in-docker (aside from the storage driver and cache considerations mentioned above).

Security

Docker-in-docker requires privileged mode, which can be an additional attack vector. There is ongoing work to support docker-in-docker in rootless mode, including using a fuse variant of overlayfs. This is less mature however.

Workaround: Mounting docker.sock

There is, of course, the possibility of mounting /var/run/docker.sock within a container and you can use the docker CLI within that container. This is not a real docker-in-docker, as the container merely uses a client to connect to the host's docker daemon. It is however a popular, hassle-free option for certain use cases. This is far easier to achieve, if your circumstances permit.