I have a devops container inside of which I execute docker build and docker-compose up commands which run OK then (on same server or different server with same code) sometimes incorrectly fail to Recreate sibling containers after their images are rebuilt.
My process successfully rebuilds a pair of two underlying docker images however
docker-compose up
sometimes fails to pickup that change and incorrectly says up-to-date on freshly rebuilt images when it should be saying Recreating
I am executing identical code across several servers each with varying OS ( ubuntu 16.04 or debian 8 or debian 9) and varying docker versions ( either 18.04.0-ce-rc1 or 18.03.0-ce ) ... am currently working to track down root cause however see good and bad behavior across each of above OS and docker versions
This issue does not seem related to https://github.com/docker/compose/issues/3574
Yes I am tagging the images and doing a push to local docker repository
docker build --tag localhost:5000/hygge/loudweb-admin .
docker push to my local docker repository is
docker push localhost:5000/hygge/loudweb-admin
it also does similar to above image loudweb-enduser
here my my docker-compose.yaml
version: '3'
services:
nodejs-admin:
image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-admin
container_name: loud_admin
restart: always
depends_on:
- loudmongo
- loudmail
volumes:
- /cryptdata4/var/log/loudlog-admin:/loudlog-admin
- /cryptdata5/var/log/blobs:/blobs
- /cryptdata5/var:/cryptdata5/var
- /cryptdata5/var/tools:/tools
- /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
- $SOURCE_REPO_DIR/tests:/tmp/tests
- ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle:/tmp
environment:
- MONGO_SERVICE_HOST=loudmongo
- MONGO_SERVICE_PORT=$GKE_MONGO_PORT
- MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
- METEOR_SETTINGS=${METEOR_SETTINGS}
- MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:blah@loudmail:587/
- GKE_NOTIF_TASK_OVERDUE=$GKE_NOTIF_TASK_OVERDUE
- GKE_NOTIF_TASK_PUSH=$GKE_NOTIF_TASK_PUSH
- GKE_NOTIF_PLANS_RECUR=$GKE_NOTIF_PLANS_RECUR
links:
- loudmongo
- loudmail
ports:
- 127.0.0.1:3001:3001
working_dir: /tmp
command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
nodejs-enduser:
image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-enduser
container_name: loud_enduser
restart: always
depends_on:
- nodejs-admin
- loudmongo
- loudmail
volumes:
- /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
- /cryptdata5/var/log/blobs:/blobs
- ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/enduser/bundle:/tmp
- ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-cert.pem:/private/config/apn-cert.pem
- ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-key.pem:/private/config/apn-key.pem
environment:
- MONGO_SERVICE_HOST=loudmongo
- MONGO_SERVICE_PORT=$GKE_MONGO_PORT
- MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
- METEOR_SETTINGS=${METEOR_SETTINGS}
- MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:puMDcxNmEwMDU5MmJjZjNiNzQ3M2ZlNTJjNDYyOGI1NTggIC0Ksy@loudmail:587/
links:
- loudmongo
- loudmail
ports:
- 127.0.0.1:3000:3000
working_dir: /tmp
command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
loudmongo:
image: mongo
container_name: loud_mongo
restart: always
ports:
- 127.0.0.1:$GKE_MONGO_PORT:$GKE_MONGO_PORT
volumes:
- /cryptdata7/var/data/db:/data/db
loudmail:
image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudmail
hostname: mail
domainname: ${GKE_DOMAIN_NAME}
container_name: loud_mail
restart: always
environment:
- DMS_DEBUG=1
- ENABLE_MANAGESIEVE=1
- ENABLE_FAIL2BAN=1
- SA_TAG=2.0
- SA_TAG2=6.31
- SA_KILL=6.31
- SSL_TYPE=letsencrypt
ports:
- 127.0.0.1:25:25
- 127.0.0.1:110:110
- 127.0.0.1:143:143
- 127.0.0.1:587:587
- 127.0.0.1:993:993
- 127.0.0.1:995:995
- 127.0.0.1:4190:4190
loud-devops:
image: dind
container_name: loud_devops
restart: always
ports:
- 127.0.0.1:9000:9000
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /usr/bin/docker:/usr/bin/docker
- /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7
- /usr/lib/x86_64-linux-gnu/libgpm.so.2:/usr/lib/x86_64-linux-gnu/libgpm.so.2
- /home/asya/src/github.com/blahblah:/inner_home/asya/src/github.com/blahblah
- /home/asya/.docker:/inner_home/asya/.docker
- /cryptdata5/var/tools/inner_home/asya:/inner_home/asya
- $GKE_DIND_SUPERVISOR_LOG_DIR:/var/log/supervisor
- /cryptdata5/var/tools/usr/local/go:/usr/local/go
- /cryptdata6/var/log/tmp/asya01:/cryptdata6/var/log/tmp/asya01
- /cryptdata6/var/log/tmp/shared:/cryptdata6/var/log/tmp/shared
- /cryptdata5/var/tools/usr/local/bin:/usr/local/bin
- /cryptdata:/cryptdata
- /cryptdata4:/cryptdata4
- /cryptdata5:/cryptdata5
- /cryptdata6:/cryptdata6
- /cryptdata7:/cryptdata7
- /etc/letsencrypt/live:/etc/letsencrypt/live
- /usr/local/ssl:/usr/local/ssl
- /var/lib/docker:/var/lib/docker
command: /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
The bottom most container in above ( loud-devops ) contains a webhook daemon which listens for a git push to trigger a rebuild cycle. While inside that container the system issues all my docker command including docker build create fresh images for two containers ( nodejs-admin and nodejs-enduser ) then while inside the container I issue the
docker-compose -f /docker-compose.yaml up -d
which then sometimes incorrectly displays
loud_devops is up-to-date
loud_mongo is up-to-date
loud_mail is up-to-date
loud_admin is up-to-date
loud_enduser is up-to-date
other times it incorrectly displays
docker-compose -f /docker-compose.yml up -d
Recreating loud_devops ...
Recreating loud_mongo ...
Recreating loud_mail ...
which should NEVER happen since those 3 images are never rebuilt ... to me that scenario is a blatant docker logic bug ...
whereas other times works OK and correctly displays
loud_mongo is up-to-date
loud_mail is up-to-date
loud_devops is up-to-date
Recreating loud_admin ... done
Recreating loud_enduser ... done
On the host of each server I have docker installed then as you can see in above yaml for container loud-devops I mount as volumes various docker dirs to enable the execution of docker in docker from inside container loud-devops
A critical piece of evidence : on a server which has finished its rebuild cycle where it successfully rebuilt both images loudweb-enduser and loudweb-admin then incorrectly shows
docker-compose -f /docker-compose.yml up -d
loud_mail is up-to-date
loud_mongo is up-to-date
loud_devops is up-to-date
loud_admin is up-to-date
loud_enduser is up-to-date
where the docker-compose up command gets executed by the system while inside container loud-devops ... IF I then manually login to that same server then manually issue the same command it works correctly :
docker-compose -f /docker-compose.yml up -d
loud_devops is up-to-date
loud_mail is up-to-date
loud_mongo is up-to-date
Recreating loud_admin ... done
Recreating loud_enduser ... done
