My Azure Virtual Machine Scaleset instances, used for hosting of an application, are filling up due to excessive disk space usage by Docker.

The deployment of the application is based on an Azure DevOps pipeline that uses a Dockerfile. The intention of the pipeline is to pull the latest source code for the application from Bitbucket, and use it to build the application using Docker so that it can then be deployed using kubernetes (using Azure Kubernetes Service - AKS):

enter image description here

The Dockerfile copies files for the .NET application that are under source control, and uses them to build the application inside the container. I think that what may be happening here is that the application is being copied and built into a container in the Azure container registry, rather than the container image being prepared first and then correctly pushed to the Azure container registry.

enter image description here

Does my assessment of what the pipeline is doing wrong, and what I need to do to remedy it, sound about right? And if so, how should I be doing things in the pipeline in order to correctly build and push an image based on the latest source code to the Azure container registry? I'd appreciate any links to reference information that give me an idea of the correct way to do this.

Guides I've looked at:

I've read this - https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/containers/acr-template?view=azure-devops, but it isn't helping me to understand a good way to build and push a container image that has the required .Net Core dependencies and a built version of the .NET Core app.

The following gives a process of continuous integration and deployment using VSTS and Visual Studio - https://docs.microsoft.com/en-us/azure/devops/pipelines/apps/cd/azure/aspnet-core-to-acr?view=azure-devops, but this isn't what I'm trying to do. I want to be able to fix the deployment by adapting the existing Azure Devops pipeline if it's possible, so that it pulls the latest source code for the application and correctly builds and pushes a docker container image to the Azure container registry.

Details of the server disk space usage that's occurring with the existing build and deployment process

The main use of disk space is in var/lib/docker/overlay2. A number of sub-directories in /var/lib/docker/overlay2 (each named with a large hash) are taking up space. I investigated the largest subdirectory and the main space usage is within its merged sub-directory. The merged subdirectory has a top-level file structure typical of a Linux top-level file structure (e.g. home, lib, tmp, usr etc.). The next largest subdirectory under var/lib/docker/overlay2 is similar i.e. the largest use is in its merged subdirectory.

docker system df on one of the scaleset instances shows 72% of space used by images is reclaimable

enter image description here

Inspecting dangling (i.e. unreferenced) images using docker image ls -f dangling=true only shows 295Mb use by unused images:

enter image description here

docker system prune doesn’t clear much space, but docker system prune -a clears the reclaimable space. The images removes by docker system prune -a all appear to be related to Azure/kubernetes - for example here is the beginning of the output when I check images that have been deleted:

Deleted Images:
untagged: mcr.microsoft.com/azure-policy/policy-kubernetes-webhook:prod_20200505.3
untagged: mcr.microsoft.com/azure-policy/policy-kubernetes-webhook@sha256:ae17bf0784c2bf340a304e975104ea8dbaf4a92398d25f1cc53606815f8f0464
deleted: sha256:2419efef0d87926a6b1c5afd003095a7c915f47cce7980039fe00da6c2d8a62b
deleted: sha256:9d5b880b9f9ae7d70d285dac221ec420516fbee51fabc4046611e0aa4bc67bb3
deleted: sha256:2b97796a4d7de2b8c1047f1244b0cbc3a8caf10c75435bb5cddc4800c7c40d45
untagged: mcr.microsoft.com/azure-policy/policy-kubernetes-addon-prod:prod_20210325.1
untagged: mcr.microsoft.com/azure-policy/policy-kubernetes-addon-prod@sha256:ed990be4cd33c3ead578c2e4f31907ab11f4d5fb582e974cd0529f5965b27c11
deleted: sha256:17f17127a71d7bc130274d155617433ffc0a368af61f3a2effb0b4e88ca3938c


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.