What are best and comprehensive practices to consider when running docker in production?

Question

Finally, you are so much in love with Docker that you want to move your online business-critical production systems with sensitive customer data to a Docker Swarm. Some might even already have done so. The other organization can't afford it by a policy forbidding production processes running in root mode.

What could be a checklist of building blocks to consider for a Docker production environment? One does not need all of them, but all of them should be important to be assessed.

Disclaimer: I know there is a SE policy to avoid "large endless lists" but I think this checklist cannot be very big... and endless noway.

So - what are these buildings blocks?

1. If not already deployed, consider running a Linux host system with advanced security settings - hardened kernel, SELinux etc.
2. Consider using a tiny Docker base image, like alpine, busybox or even scratch e.g. start with an empty base image
3. Use USER setting other than root
4. Carefully assess to further reduce the already shrinked set of kernel capabilities granted to container
5. Consider having only one executable binary per container to launch your process, ideally statically linked
6. Those who want to break your system to get a shell access might wonder if they found out your container has all shells disabled
7. Mount read-only volumes where only possible

Question: what else?

Answer 1

The host on which the containers are running

Run the docker security bench on every node that runs docker containers https://github.com/docker/docker-bench-security

Running the following command on a node that runs docker containers:

docker run -it --net host --pid host --cap-add audit_control \
    -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
    -v /var/lib:/var/lib \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /usr/lib/systemd:/usr/lib/systemd \
    -v /etc:/etc --label docker_bench_security \
    docker/docker-bench-security

returns a list of checks:

[INFO] 1 - Host Configuration

[WARN] 1.1  - Ensure a separate partition for containers has been created

[NOTE] 4.2  - Ensure that containers use trusted base images

[PASS] 4.6  - Ensure HEALTHCHECK instructions have been added to the container image

Quote from the repository README:

The Docker Bench for Security is a script that checks for dozens of common best-practices around deploying Docker containers in production. The tests are all automated, and are inspired by the CIS Docker Community Edition Benchmark v1.1.0.

Some of the issues that are reported by the security bench could be solved by reading the official docker security article and comparing it with the bullets that are defined in the question the following things are important as well:

• protect the docker daemon socket by implementing ssl
• content trust using notary and DOCKER_CONTENT_TRUST variable

Answer 2

Docker images itself

An additional option is to use Clair.

Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including appc and docker).

In regular intervals, Clair ingests vulnerability metadata from a configured set of sources and stores it in the database.

Clients use the Clair API to index their container images; this creates a list of features present in the image and stores them in the database.

Clients use the Clair API to query the database for vulnerabilities of a particular image; correlating vulnerabilities and features is done for each request, avoiding the need to rescan images.

When updates to vulnerability metadata occur, a notification can be sent to alert systems that a change has occured.

Our goal is to enable a more transparent view of the security of container-based infrastructure. Thus, the project was named Clair after the French term which translates to clear, bright, transparent.

Answer 3

In addition to the points in this thread; the following would be my recommendation:

• Get control over Docker PID1 with dumb-init
  • ref: PID1 and Zombie reaping problem
• Do not run docker in production without a container orchestration system
  • Take your pick from Kubernetes, Mesos, Swarm etc.
• Use gosu for user control inside a docker image
• Follow the 12 factor app paradigm, if you are running stateful apps in containers, change it up.
  • If you really need to run stateful apps (mysql, zookeeper, elasticsearch) in containers, leverage orchestrator paradigms like Kubernetes Statefulsets
• Do robust secret / config management with tools like hashicorp vault / consul
• Ship the same container built by the devs to prod through a CI pipeline that takes it through staging, integration-tests thoroughly.
• Create notifications around CVEs and patches, trigger builds on patch-notify
• Have extensive logging to get insight into the running container, you do not want to give the devs SSH access to either the host or the containers
  • recommendation: fluentd
• Have both container and host metrics
  • recommendation: prometheus+node-exporter

Answer 4

Docker is still in development.

As with every other software in-dev bugs will happen, insecure features might be added, there could be architectural flaws that lead to security breaches. Do not underestimate this! Your system might be completely safe today, but with the patch from next week someone finds a bug, writes an exploit, and suddenly your system is wide open.

Unless you must, don't update to the latest version. Use the latest well-tested version instead.

Docker is no virtualization

If someone escapes from a Docker container, that attacker is on the real machine immediately. There is no second gate like virtualization that will prevent a breach.

Treat a Docker container like any other program. Run with lowest possible user rights, block all network traffic that isn't required, virtualize the entire Docker host if performance allows.

Docker is no protection

Whatever code is run inside Docker containers is run without question from Docker. Any attacker can simply install his software inside the container, and Docker would run that like any other code.

Aside from the things you mentioned in the question, consider using metrics and alerts to get notified if any Docker image is doing strange things. Is there a sudden, ongoing CPU spike? Is the program suddenly scanning network ports? Is there suspicious disk access? You should get a notification if any of that happens. There are many tools available to measure these things, you should use them.

Answer 5

If you are filling up your docker entrypoint with sed commands, consider this practice:

• Use a tool such as confd to manage your docker images configuration files and keep them updated

Confd will read data from many supported key-value stores and render configuration templates dynamically.

Answer 6

Reading the official docker security article and comparing it with the bullets that are defined in the question the following things are important as well:

• protect the docker daemon socket by implementing ssl
• content trust using notary and DOCKER_CONTENT_TRUST variable

Answer 7

One could use A2D to bake an app into a docker image while taking certain things into account, e.g. non-root, permissions, location of the app:

docker run -v $PWD:/projectName utrecht/a2d:1.0.0 \
       -projectName someProjectName -dockerfile /projectName/Dockerfile

returns:

FROM golang:1.12.4-alpine as builder
COPY . ./someProjectName/
WORKDIR someProjectName
RUN adduser -D -g '' someProjectName && \
    apk add git && \
    CGO_ENABLED=0 go build && \
    cp someProjectName /someProjectName && \
    chmod 100 /someProjectName

FROM scratch
COPY --from=builder /etc/group /etc/group
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder --chown=someProjectName:someProjectName /someProjectName /usr/local/someProjectName
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
USER someProjectName
ENTRYPOINT ["/usr/local/someProjectName"]