Re-using Python virtual environments on a build server

Question

Currently, every time we run a build through Jenkins+Ansible, we are re-creating a virtual environment and re-installing all the dependencies listed inside the requirements.txt file.

This is very slow and does not scale well. How can we improve and speed up the process? Can we re-use virtual environments?

---

Our latest idea which we have not yet implemented was to build virtual environments outside of a Jenkins workspace, name virtual environments based on a project+branch and keep an MD5 sum of the requirements for every virtual environment. Then, before building an environment the next time, we calculate the MD5 sum of the current requirements in this branch and look up existing virtual environments by this MD5 sum. If an existing environment with this sum was found, just re-use it. We are not sure if this is the best way to solve the problem.

Answer 1

It's generally a best practice to make your builds idempotent. Leaving artifacts behind only opens up opportunities for dependency management issues- and resolving those in a resilient way is exactly why you're using a build server. I suggest you take a look at WHY your Python builds are running so slow- if you discover this is because of an overabundance of packages in each project, this might be optimizable. If you discover this is because of server performance issues, those are easily fixable with more resources. If you find you are blocked by network throughput, consider using an artifact repository to cache packages locally. But building persistent virtualenvs seems like a recipe for problems that could be unpredictable when moving to another server, and extremely frustrating to troubleshoot.

Answer 2

I'm going to add more story to the answer @esoterydactyl gave, in case it goes over too many people's heads.

Instead of a monolithic build process, maybe you want to build intermediate packages that can be versioned and deployed to your workspace without all the intermediate build steps. Docker is one packaging solution which is super popular in this case. You could also use .deb or .rpm or non-Docker tarballs, or make CF "StemCells" or even zip files. Also consider Python has prebuilt "wheel" packages now.

Problems:

• all of your developers must use the same build-deps package for their sandbox or hair-pulling fun will emerge in the differences between dev sandbox and CI workspace. Thus the package must support dev laptop sandboxes and Jenkins workspace installation.
• the build-deps package must have its own CI build/test/packaging/release pipeline. It, however, can have reduced cycle frequency while nobody is committing any changes to it (or any of its components).
• This is an example of bundling. If you have, for example, a CVE to patch, everything that uses the bundle must be patched. This must be no more difficult to deploy than anything else, so don't stop work when your CI builds are fast. CI needs CD, or you could be delivering production undeployable software.
• If you're making your own prebuilt artifacts, you need infrastructure to host the artifact repository, and a versioning scheme.

The incremental solution would be to solve the minimum number of those problems which yields a significant boost in build cycle speed. This is why it makes sense to maintain a local pypi repository, and add Jenkins jobs to build and push wheel packages to it. I know that sounds like a pain, but imagine if you had to manage internal software development for all of the stuff you can just install via pip?

Answer 3

You may want to take a look at the ShiningPanda Plugin.

It allows running a build step inside a particular python virtual environment, created automatically if it doesn't exist or reused if it already exists. The virtual environment is located outside the workspace, but it can be named and made to appear as a directory with that name inside the workspace.

To address the dependencies you just have to insert a build step performing the desired pip install -r requirements.txt variant after pulling the workspace (which will probably bring in your requirements.txt file) but before any other build step that would need to use the virtual environment.

The same virtual environment can be used in multiple steps of the same build (by name) and a build can have multiple virtual environments if needed. But AFAIK you can't reuse the same virtual environment in different builds, each build will have its own copy.

The Plugin has a configuration to clear/wipe the environment at any build step using it, if/when needed. Obviously you wouldn't want this always enabled as it would void reusability.

IMHO it works best with requirements.txt files with pinned package versions (like those produced by pip-tools).

Update: I recently ran into an issue with the plugin on Windows: I had several jobs in different Jenkins folders but with the same job names and same plugin configs (notably the non-empty Name in the Advanced section). Turns out all jobs were actually sharing a single environment (and stepping over each-other), I had to switch to unique values for Name across the jobs. Apparently the 2 names are used to generate the 2 hashes from the virtual environment path, so same names -> same path:

\Jenkins\shiningpanda\jobs\<job_name_hash>\virtualenvs\<venv_name_path_hash>\

So, if you have the same job names, you actually can share the same environment if you really want to :)