Re-using Python virtual environment vs Re-building it

Question

Currently, we build our application on a build server where we create a virtual environment using virtualenv command, install all the Python dependencies into it, then "patch" it using the following bash commands to make it relocatable:

# Adopt virtualenv to the target system
echo "* Patching virtual envirinment for target system"
for f in "${VENV_PATH}/bin"/* ; do
    if [ -n "$(file "${f}" | grep "text")" ] ; then
    echo "  + ${f}"
        sed -i "s;"${VENV_PATH}";/path/to/the/target/directory/venv;" "${f}"
    fi
done
rm -rf "${VENV_PATH}/lib64"
ln -s "/path/to/the/target/directory/venv/lib" "${VENV_PATH}/lib64"

Then, we pack it into an RPM package including some other parts of the application, deploy to multiple target servers and install it via yum.

The main motivation behind making the virtual environment relocatable is to "build once, deploy multiple times" - in other words, to avoid installing the same dependencies on all the target servers.

The problem is that, this patching script looks really fragile and, from what I understand, this is quite tied to the current Python version and also brings the requirement for the build server to be the same operating system as the target servers (or, at least, to have all the python related paths match).

Is my understanding correct? Would it be better to rebuild the virtual environment on every single target? (we can accelerate pip installation commands with, for example, pip-accel or caching) What are the general ways to tackle this problem?

Answer 1

What are the general ways to tackle this problem?

The ideal way to resolve your problem would be to dockerise your application, as it would remove any worries you have about compatibility between your build and running servers. Installing dependencies using pip in a Dockerfile is trivial, and as long as you're deploying the same container image you never need to worry about inconsistency.

Edit: as luck would have it I actually found myself wanting to have a venv that worked both inside and outside Docker, so I came up with the following:

python3.6 -m venv --copies venv
sed -i '43s/.*/VIRTUAL_ENV="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}" )")" \&\& pwd)"/' venv/bin/activate
sed -i '1s/.*/#!\/usr\/bin\/env python/' venv/bin/pip*