We are using Azure DevOps as our CICD platform.
We have our own self hosted Linux agents which we integrated into our CICD Azure Kubernetes Cluster .
We use these agents to build our Docker images within the cluster, meaning that we have N agents represented by N pods.
While working, we might scale up and down the amount of concurrent agents depending on different parameters.
Since we have started with this approach, we have encountered a huge issue in leveraging docker's cache.
We have tried to:
- Use kubernetes StatefulSets. But this limits us from having the cache shared between all pods (one PV/PVC per pod)
- Use Google's Kaniko - Unfortunately we have found that in many cases it is slower than just using Docker's BuildKit without cache.
- We tried to use Azure Files with SMB and NFS as we thought that a PV with ReadWriteMany will solve the problem. Unfortunately we have ended up with a fatal error.
Currently we are a bit confused as we assume that this is a common industry issue and therefore we might have got it wrong. We are looking to understand how to leverage docker's cache while using our current setup and without moving to another platform.
How should we do that? What are the best practices for this issue?