How to run experimental Docker features on CircleCI

Question

When a build is started on CircleCI one of the first things that is done is:

Setup a remote Docker engine

Specified reusable docker engine, but build has not been whitelisted.
Contact CircleCI to be whitelisted
Allocating a remote Docker Engine
...
Remote Docker engine created. Using VM 'prealloc-wrjtu1qd-1491949826270'
Created container accessible with:
  DOCKER_TLS_VERIFY=1
  DOCKER_HOST=tcp://<IP>:2376
  DOCKER_CERT_PATH=/tmp/docker-certs615987123
  DOCKER_MACHINE_NAME=51123

Subsequently when docker build --squash -t imagename . is run on CircleCI it results in:

Error response from daemon: squash is only supported with experimental mode
Exited with code 1

Discussion

It works to use experimental features locally after changing the /etc/docker/daemon.json as follows:

{
    "experimental": true
}

and restarting the docker systemctl service, but how to do that on CircleCI? It seems impossible as the build output indicates that a connection is made to a remote docker that reside on the CircleCI systems.

Answer 1

You are running into the limitations of managed services. Many managed services are great to start experimenting with ideas and can show a great value for the casual user. But they trade ease of use for flexibility and functionality, therefore as soon as we start to have serious needs and are ready to spent a significant amount of effort building up on these ideas, the first move we should do is to opt for a flexible a fully functional solution. In my judgement, you are hitting the line where you need to look for something else – and you will notice that you need to learn a different tool, because that's not free software and you cannot run it yourself.

Here we are speaking about Circle CI, but the following limitations are common to many managed services:

• Managed services put a hard contraint on software versions they allow us to use. These constraints may match what we need today, but does not need to at a later time. They may be already out of sync today. In any case this is often a show stopper, because being unable to use the same software versions in development environment and in managed environment is a useless source of bugs and difficulties of all sort.

• Managed services provide restricted access to their functionalities. This is the problem you are facing here: you want to run docker, but you do not have access to the daemon, only to a subset of its functions. These limitations are sometimes not well documented nor brought in full light by advertisement material. This just means work spent on understanding and integrating these features is worthless until we leave the managed service for another better solution.

• Managed services provide restricted access to the data we store there. So, to take the example of managed services for logging and monitoring, data access is slow, inconvenient and expansive. If we run our infrastructure, we are free to run any fancy custom analysis on our monitoring artefacts, because data is easier to reach.

Note that this is not a case against managed services. A hand of them are able to combine ease of use together with flexibility and functionality. But many of these service trade ease of use for flexibility and functionality: we can use them as an expedient to quickly build a simple functionality but they will prevent us to walk beyond that point. We need to be aware of this trade-off and consider.

Answer 2

CircleCI now allows you to use a machine executor which provisions a separate VM for you, with Ubuntu 14.04 and Docker version 17.06.0-ce installed. This allows you to turn on experimental features for the Docker daemon.

You need to use the machine key instead of the docker key, to run your job in a separate virtual machine instead of just a Docker container.

You can only choose 2 images for the machine:

• circleci/classic:latest: Ubuntu 14.04 with Docker 17.03.0-ce, or
• circleci/classic:edge: Ubuntu 14.04 with Docker 17.06.0-ce - the one with experimental features.

You'll also need to install dependencies on the machine yourself, as it is quite bare. For example, if you need PHP for your tests, you'll need to run sudo apt-get install -y php5.

Here is a sample .circleci/config.yml that builds a Docker image using experimental feature docker build --squash:

.circleci/config.yml

version: 2
jobs:
  build:
    # Run in a separate virtual machine instead of a Docker container.
    machine:
      enabled: true
      # Use Ubuntu 14.04 with bleeding edge Docker daemon 17.06.0-ce.
      image: circleci/classic:edge
    steps:
      - checkout
      - run:
          command: |
            # Restart Docker with experimental features on.
            sudo sh -c 'echo '\''DOCKER_OPTS="--experimental=true"'\'' >> /etc/default/docker'
            sudo service docker restart

            # Install dependencies for tests etc.
            sudo apt-get update
            sudo apt-get install -y php5

            # Build image with experimental feature --squash.
            docker build --squash -t myuser/myimage .

            # Login and push Docker image to registry.
            docker login -u $DOCKER_USER -p $DOCKER_PASS
            docker push myuser/myimage