11

Sometimes you have to investigate a container, which is stopped, or a container which after starting up dies very quickly and stops.

docker exec -ti <id> bash only works on running containers, once it finishes, the bash prompt terminates as well.

With docker start you cannot supply a different command, and if the container dies abruptly again you won't have enough time to get into the container and do your investigations.

We can do docker commit, then docker run on the new image with a different command, but I'm wondering if there are any other alternatives.

Note: docker logs just returns whatever the apps printed to stdout/stderr. That might not be enough to figure out what the problem was.

  • After a while thinking about it: Docker main process ???? As a container aim at running only one process , either the term 'main' is to be removed, or you're doing something weird (like running an init process) , or you're taking threads as processes... I assume it's option one, but I had to say it because it bothers me – Tensibai Mar 7 '17 at 22:15
  • @Tensibai sometimes you have to run something like dumb-init, to handle the pid 1 / signalling issue in containers, if your main command cannot handle it itself. There might be also other cases where a docker container runs more than one processes – SztupY Mar 8 '17 at 9:04
  • Yes, that's what I call weird, mainly because containers have been made to isolate a process. Sometimes containers are not the solution for an app an willing to put everything inside a container is more a path to headaches than anything else. – Tensibai Mar 8 '17 at 10:23
8

General ways to track why a process in Linux failed are good. One such way is to run a process using strace which will tell you the system calls process did and usually point to the reason for a failure.

You can create a Dockerfile that looks something like this:

FROM original_image

RUN apt-get -y update && apt-get install -y strace

# build with `docker build -t debug_version`

Then run your new image using docker run debug_version strace original_cmd.

For processes that fork children (and then die) you want to run strace with the -ff option. You can also map some file using Docker data volumes and use the -o option from strace to write to it. But in general strace will leave output on stdout, which is readable using docker log.

Related Q: Linux process terminates mysteriously

  • This means I still have to docker commit my stopped container first to have an image to start from – SztupY Mar 7 '17 at 15:20
  • You said it dies on start. I assume you have an image then. For those that are stopped, yes a commit is required. – Evgeny Mar 7 '17 at 15:23
  • That's only one of the scenarios to get a stopped container – SztupY Mar 7 '17 at 15:25
  • There is also a package for strace in Alpine Linux, pkgs.alpinelinux.org/package/edge/main/x86_64/strace. Use the Alpine package manager to install it, apk install strace. – Evgeny Mar 8 '17 at 10:43
3

As far as I am aware, commit and run are the best options here to give you full access to the container as it was when it died.

Ideally, your container would spit out some more useful information when it fails, but that is another topic altogether.

Edit: to expand my answer, if the container is dying right on start, you can also use docker run to specify an alternative --entrypoint and CMD. Generally I'll set this to a loop or something that won't exit on its own. Once you're in the container, you can manually run the steps that are failing, and then inspect the outcome without having to worry about the container exiting.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.