In our organization we have many small repos for shared libraries used among our applications. Particularly, we have many Python libraries and Python applications. We have a structures like:
myorg/python-lib1: produces package
myorg/python-lib2: produces package
myorg/python-app: produces a package
myorg.app, depends on
In our CI for
myorg/python-lib1 we run unit tests when PRs come in and publish wheels to PyPI on merges to the main branch.
In our CI for
myorg/python-app1 we will build test application images when PRs come, deploy them to a test kubernetes cluster, and allow developers to run integration tests against it. It has a Dockerfile like:
# Builds image reistry.myorg.com/python-app
COPY . /app
# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
RUN pip install /app
Note: We also have some projects using build-packs to achieve a similar goal.
Because we only run the integration tests for
myorg/python-app1, we can't get as much confidence when a change to
myorg/python-lib2 happens. We have to wait for the libraries to get published to PyPI and then rebuild the application images. When there's an issue in the library, this usually involves us git reverting and fixing the PyPI version. This is super cumbersome when we have a lot of images that pull in the dependency before we detect it needs to be rolled back.
I'm looking to restructure the way we build images in
myorg/python-app1 (and other apps that share this pattern) to support pulling in dependencies based off of the PR branches.
I've looked at a lot of tools for containerization but I haven't seen any strategies for building application containers given a change in a lower level dependency. I'm interested in seeing what strategies exist for this. Hopefully the strategy could be applied to other languages as well, where we have a similar strategies for applications written in languages like Java, Golang, and npm.