5

I have a devops container inside of which I execute docker build and docker-compose up commands which run OK then (on same server or different server with same code) sometimes incorrectly fail to Recreate sibling containers after their images are rebuilt.

My process successfully rebuilds a pair of two underlying docker images however

docker-compose up 

sometimes fails to pickup that change and incorrectly says up-to-date on freshly rebuilt images when it should be saying Recreating

I am executing identical code across several servers each with varying OS ( ubuntu 16.04 or debian 8 or debian 9) and varying docker versions ( either 18.04.0-ce-rc1 or 18.03.0-ce ) ... am currently working to track down root cause however see good and bad behavior across each of above OS and docker versions

This issue does not seem related to https://github.com/docker/compose/issues/3574

Yes I am tagging the images and doing a push to local docker repository

docker build --tag localhost:5000/hygge/loudweb-admin .

docker push to my local docker repository is

docker push localhost:5000/hygge/loudweb-admin

it also does similar to above image loudweb-enduser

here my my docker-compose.yaml

version: '3'
services:

nodejs-admin:
  image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-admin
  container_name: loud_admin
  restart: always
  depends_on:
    - loudmongo
    - loudmail
  volumes:
    - /cryptdata4/var/log/loudlog-admin:/loudlog-admin
    - /cryptdata5/var/log/blobs:/blobs
    - /cryptdata5/var:/cryptdata5/var
    - /cryptdata5/var/tools:/tools
    - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
    - $SOURCE_REPO_DIR/tests:/tmp/tests
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle:/tmp
  environment:
    - MONGO_SERVICE_HOST=loudmongo
    - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
    - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
    - METEOR_SETTINGS=${METEOR_SETTINGS}
    - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:blah@loudmail:587/
    - GKE_NOTIF_TASK_OVERDUE=$GKE_NOTIF_TASK_OVERDUE
    - GKE_NOTIF_TASK_PUSH=$GKE_NOTIF_TASK_PUSH
    - GKE_NOTIF_PLANS_RECUR=$GKE_NOTIF_PLANS_RECUR
  links:
    - loudmongo
    - loudmail
  ports:
   - 127.0.0.1:3001:3001
  working_dir: /tmp
  command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

nodejs-enduser:
  image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-enduser
  container_name: loud_enduser
  restart: always
  depends_on:
    - nodejs-admin
    - loudmongo
    - loudmail
  volumes:
    - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
    - /cryptdata5/var/log/blobs:/blobs
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/enduser/bundle:/tmp
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-cert.pem:/private/config/apn-cert.pem
    - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-key.pem:/private/config/apn-key.pem
  environment:
    - MONGO_SERVICE_HOST=loudmongo
    - MONGO_SERVICE_PORT=$GKE_MONGO_PORT 
    - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
    - METEOR_SETTINGS=${METEOR_SETTINGS}
    - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:puMDcxNmEwMDU5MmJjZjNiNzQ3M2ZlNTJjNDYyOGI1NTggIC0Ksy@loudmail:587/
  links:
    - loudmongo
    - loudmail
  ports:
   - 127.0.0.1:3000:3000
  working_dir: /tmp
  command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

loudmongo:
  image: mongo
  container_name: loud_mongo
  restart: always
  ports:
   - 127.0.0.1:$GKE_MONGO_PORT:$GKE_MONGO_PORT
  volumes:
   - /cryptdata7/var/data/db:/data/db

loudmail:
  image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudmail
  hostname: mail
  domainname: ${GKE_DOMAIN_NAME}
  container_name: loud_mail
  restart: always
  environment:
    - DMS_DEBUG=1
    - ENABLE_MANAGESIEVE=1
    - ENABLE_FAIL2BAN=1
    - SA_TAG=2.0
    - SA_TAG2=6.31
    - SA_KILL=6.31
    - SSL_TYPE=letsencrypt
  ports:
    - 127.0.0.1:25:25
    - 127.0.0.1:110:110
    - 127.0.0.1:143:143
    - 127.0.0.1:587:587
    - 127.0.0.1:993:993
    - 127.0.0.1:995:995
    - 127.0.0.1:4190:4190


loud-devops:
  image: dind
  container_name: loud_devops
  restart: always
  ports:
    - 127.0.0.1:9000:9000
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock   
    - /usr/bin/docker:/usr/bin/docker   
    - /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7  
    - /usr/lib/x86_64-linux-gnu/libgpm.so.2:/usr/lib/x86_64-linux-gnu/libgpm.so.2
    - /home/asya/src/github.com/blahblah:/inner_home/asya/src/github.com/blahblah 
    - /home/asya/.docker:/inner_home/asya/.docker
    - /cryptdata5/var/tools/inner_home/asya:/inner_home/asya
    - $GKE_DIND_SUPERVISOR_LOG_DIR:/var/log/supervisor
    - /cryptdata5/var/tools/usr/local/go:/usr/local/go
    - /cryptdata6/var/log/tmp/asya01:/cryptdata6/var/log/tmp/asya01
    - /cryptdata6/var/log/tmp/shared:/cryptdata6/var/log/tmp/shared
    - /cryptdata5/var/tools/usr/local/bin:/usr/local/bin 
    - /cryptdata:/cryptdata
    - /cryptdata4:/cryptdata4
    - /cryptdata5:/cryptdata5
    - /cryptdata6:/cryptdata6
    - /cryptdata7:/cryptdata7
    - /etc/letsencrypt/live:/etc/letsencrypt/live
    - /usr/local/ssl:/usr/local/ssl
    - /var/lib/docker:/var/lib/docker
  command: /usr/bin/supervisord -c /etc/supervisor/supervisord.conf

The bottom most container in above ( loud-devops ) contains a webhook daemon which listens for a git push to trigger a rebuild cycle. While inside that container the system issues all my docker command including docker build create fresh images for two containers ( nodejs-admin and nodejs-enduser ) then while inside the container I issue the

docker-compose -f /docker-compose.yaml up -d 

which then sometimes incorrectly displays

loud_devops is up-to-date
loud_mongo is up-to-date
loud_mail is up-to-date
loud_admin is up-to-date
loud_enduser is up-to-date

other times it incorrectly displays

docker-compose -f /docker-compose.yml up -d
Recreating loud_devops ...
Recreating loud_mongo ...
Recreating loud_mail ...

which should NEVER happen since those 3 images are never rebuilt ... to me that scenario is a blatant docker logic bug ...

whereas other times works OK and correctly displays

loud_mongo is up-to-date
loud_mail is up-to-date
loud_devops is up-to-date
Recreating loud_admin ... done
Recreating loud_enduser ... done

On the host of each server I have docker installed then as you can see in above yaml for container loud-devops I mount as volumes various docker dirs to enable the execution of docker in docker from inside container loud-devops

A critical piece of evidence : on a server which has finished its rebuild cycle where it successfully rebuilt both images loudweb-enduser and loudweb-admin then incorrectly shows

docker-compose -f /docker-compose.yml up -d 
loud_mail is up-to-date
loud_mongo is up-to-date
loud_devops is up-to-date
loud_admin is up-to-date
loud_enduser is up-to-date

where the docker-compose up command gets executed by the system while inside container loud-devops ... IF I then manually login to that same server then manually issue the same command it works correctly :

docker-compose -f /docker-compose.yml up -d 
loud_devops is up-to-date
loud_mail is up-to-date
loud_mongo is up-to-date
Recreating loud_admin ... done
Recreating loud_enduser ... done
  • 1
    Do you tag the built images? this might happen if you build & tag an image with the tag which already exists = rewrite the previous version, then when you do docker-compose up it won't recreate because the version is the same. I would like to know if --force-rebuild and/or --always-recreate-deps solves the issue and I would also like to see the docker-compose file, I might be able to advice further afterwards – Tomáš Tibenský Apr 4 '18 at 12:45
  • @TomášTibenský to respond to your questions - the rebuild logic is outside the docker-compose.yaml so I have never used nor believe I can use --force-rebuild or --always-recreate-deps – Scott Stensland Apr 4 '18 at 17:11
  • 1
    what about running docker-compose -f /docker-compose.yml pull and then docker-compose -f /docker-compose.yml up -d ? – Tomáš Tibenský Apr 6 '18 at 11:38
0

Solution was to put a docker pull into my launch script ... as in

docker-compose -f ${GKE_COMPOSE_YAML} pull

just before issuing

docker-compose -f ${GKE_COMPOSE_YAML} up -d

which seems strange since most of the time it works OK without issuing pull ... I am using a local docker registry

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.